implemented <$$> command for returning from a macro

 * <$$> is faster than jumping to the end of the macro
   and enables shorter code for returning values from macros.
 * this also replaces $$ as an immediate editing command.
   In other words, command line termination is an ordinary command
   now. The old behaviour was similar to what classic TECO did.
   Classic TECO however had no choice than to track key presses
   directly for command line termination as it did not keep track
   about the parser state as input was typed.
   This led to some glitches in the language. For instance
   "FS$$" would terminate the command line, unless the second escape
   was typed after backspace, etc. This behaviour is not worth copying
   and SciTECO did a better job than that by making sure that at least the
   second escape is only effective if it is not part of language syntax.
   This still lead to some undesirable cases like "ES...$$$" that would
   terminate the command line unexpectedly.
   To terminate the command line after something like "FS$$", you will
   now have to type "FS$$$$".
 * As it is a regular command now - just executed immediately - and
   its properties stay close to the macro return behaviour, command line
   termination may now not always be performed when $$ is typed even
   as a standalone command. E.g. "Ofoo$ !bar!$$ !foo!Obar$" will
   curiously terminate the command line now.
 * This also means that macros can finally terminate command lines
   by using the command line editing commands ({ and }) to insert
   $$ into the command line macro.
   This is also of interest for function key macros.
 * This implementation showed some serious shortcoming in SciTECO's
   current parser that yet have to be fixed.
   E.g. the macro "@^Ua{<$$>}" is currently unsafe since
   loops abuse the expression stack for storing their state and $$
   does not touch the expression stack. Calling "Ma>" would actually
   continue the loop jumping to the beginning of the command line
   since program counters referring to the macro A will be reused!
   This cannot be easily solved by checking for loop termination
   since being able to return that way from loops is a useful
   feature. This is a problem even without loops and $$, e.g. as
   in "@^Ua{1,2,3(4,5} Ma)".
   Instead, a kind of expression stack frame pointer must be
   added to macro invocation stack frames, pointing to the beginning
   of the expression stack for the current frame.
   At the end of macros or on return, the stack contents of
   corresponding to the frame can be discarded while preserving the
   immediate arguments at the time of the return or end-of-macro.
   This would stabilize SciTECO's macro semantics.
 * When a top-level macro returns in batch mode, it would
   be a good idea to use the last argument to calculate the
   process return code, so it can be set by SciTECO scripts (TODO).

7 files changed, 163 insertions, 83 deletions

diff --git a/lib/lexer.tes b/lib/lexer.tes
index 89a2fd5..82d8bd9 100644
--- a/lib/lexer.tes
+++ b/lib/lexer.tes
@@ -13,8 +13,8 @@
     Q[lexer.font],32ESSTYLESETSIZEFRACTIONAL
   ' '
   :M[color.init]
+  :Q*"=  '
   [_
-  :Q*"= Oend '
 }
 
 ! Automatically mung all the lexers and add them to "lexer.auto" !
@@ -25,12 +25,11 @@
     0X.[filename] 4R .U.p <-A-^^/"= 1; ':R;> .,Q.pX.[name]
     EMQ.[filename]
     :@EU[lexer.auto]{
-      :M[lexer.test.Q.[name]]"S :M[lexer.set.Q.[name]] Oend '
+      :M[lexer.test.Q.[name]]"S :M[lexer.set.Q.[name]] ]_ '
     }
   L>
 ]*
 
-! append "end" label to "lexer.auto" !
 :@[lexer.auto]{
-  !end! ]_
+  ]_
 }
diff --git a/lib/session.tes b/lib/session.tes
index e0bad9e..3ca5dd9 100644
--- a/lib/session.tes
+++ b/lib/session.tes
@@ -7,7 +7,7 @@ EU[session.path]Q[$SCITECOCONFIG]/.teco_session
 
 ! Save current session to the file specified by "session.path" !
 @[session.save]{
-  :Q[session.path]-1"< Oend '
+  :Q[session.path]-1"<  '
   Q*U.[curbuf]
   EJ<
     %.bEB ESGETFIRSTVISIBLELINEU.[fvline] ESGETXOFFSETU.[xoff] .U.[dot]
@@ -20,7 +20,7 @@ EU[session.path]Q[$SCITECOCONFIG]/.teco_session
 
   E%.[session]Q[session.path]
   Q.[curbuf]EB
-!end!}
+}
 
 ! Load session specified by "session.path" !
 @[session.load]{
diff --git a/sample.teco_ini b/sample.teco_ini
index cd2a6cf..fb02d34 100644
--- a/sample.teco_ini
+++ b/sample.teco_ini
@@ -27,16 +27,16 @@ EMQ[$SCITECOPATH]/session.tes
 
   !edit!
     ! Add code here to execute when a document is edited !
-    Oend
+    
 
   !close!
     ! Add code here to execute when a document is closed !
-    Oend
+    
 
   !quit!
     ! Add code here to execute when SciTECO quits !
     M[session.save]
-!end!}
+}
 0,32ED
 
 ! Tweak the default font name and size.
diff --git a/src/cmdline.cpp b/src/cmdline.cpp
index acaebfb..34a5df3 100644
--- a/src/cmdline.cpp
+++ b/src/cmdline.cpp
@@ -231,6 +231,39 @@ Cmdline::keypress(gchar key)
 	 */
 	try {
 		process_edit_cmd(key);
+	} catch (Return) {
+		/*
+		 * Return from top-level macro, results
+		 * in command line termination.
+		 */
+		interface.popup_clear();
+
+		if (quit_requested)
+			/* cought by user interface */
+			throw Quit();
+
+		undo.clear();
+		/* also empties all Scintilla undo buffers */
+		ring.set_scintilla_undo(true);
+		QRegisters::view.set_scintilla_undo(true);
+		Goto::table->clear();
+		expressions.clear();
+
+		last_cmdline = *this;
+		str = NULL;
+		len = rubout_len = 0;
+
+#ifdef HAVE_MALLOC_TRIM
+		/*
+		 * Glibc/Linux-only optimization: Undo stacks can grow very
+		 * large - sometimes large enough to make the system
+		 * swap and become unresponsive.
+		 * This will often reduce the amount of memory previously
+		 * freed that's still allocated to the program immediately
+		 * when the command-line is terminated:
+		 */
+		malloc_trim(0);
+#endif
 	} catch (Error &error) {
 		/*
 		 * NOTE: Error message already displayed in
@@ -484,49 +517,6 @@ Cmdline::process_edit_cmd(gchar key)
 		}
 		break;
 
-	case CTL_KEY_ESC: /* terminate command line */
-		interface.popup_clear();
-
-		if (States::current == &States::start &&
-		    str && str[len-1] == CTL_KEY_ESC) {
-			if (Goto::skip_label) {
-				interface.msg(InterfaceCurrent::MSG_ERROR,
-					      "Label \"%s\" not found",
-					      Goto::skip_label);
-				break;
-			}
-
-			if (quit_requested)
-				/* cought by user interface */
-				throw Quit();
-
-			undo.clear();
-			/* also empties all Scintilla undo buffers */
-			ring.set_scintilla_undo(true);
-			QRegisters::view.set_scintilla_undo(true);
-			Goto::table->clear();
-			expressions.clear();
-
-			last_cmdline = *this;
-			str = NULL;
-			len = rubout_len = 0;
-
-#ifdef HAVE_MALLOC_TRIM
-			/*
-			 * Glibc/Linux-only optimization: Undo stacks can grow very
-			 * large - sometimes large enough to make the system
-			 * swap and become unresponsive.
-			 * This will often reduce the amount of memory previously
-			 * freed that's still allocated to the program immediately
-			 * when the command-line is terminated:
-			 */
-			malloc_trim(0);
-#endif
-		} else {
-			insert(key);
-		}
-		break;
-
 #ifdef SIGTSTP
 	case CTL_KEY('Z'):
 		/*
diff --git a/src/error.h b/src/error.h
index 9264b33..a4f4660 100644
--- a/src/error.h
+++ b/src/error.h
@@ -29,12 +29,18 @@
 
 namespace SciTECO {
 
-/*
+/**
  * Thrown as exception to signify that program
  * should be terminated.
  */
 class Quit {};
 
+/**
+ * Thrown as exception to cause a macro to
+ * return or a command-line termination.
+ */
+class Return {};
+
 class Error {
 	gchar *description;
 	GSList *frames;
diff --git a/src/parser.cpp b/src/parser.cpp
index 4ae13d3..4cefb28 100644
--- a/src/parser.cpp
+++ b/src/parser.cpp
@@ -53,6 +53,7 @@ namespace States {
 	StateStart		start;
 	StateControl		control;
 	StateASCII		ascii;
+	StateEscape		escape;
 	StateFCommand		fcommand;
 	StateChangeDir		changedir;
 	StateCondCommand	condcommand;
@@ -139,7 +140,7 @@ Execute::macro(const gchar *macro, bool locals)
 	GotoTable *parent_goto_table = Goto::table;
 	GotoTable macro_goto_table(false);
 
-	QRegisterTable *parent_locals;
+	QRegisterTable *parent_locals = NULL;
 
 	State *parent_state = States::current;
 	gint parent_pc = macro_pc;
@@ -160,7 +161,20 @@ Execute::macro(const gchar *macro, bool locals)
 	}
 
 	try {
-		step(macro, strlen(macro));
+		try {
+			step(macro, strlen(macro));
+		} catch (Return) {
+			/*
+			 * Macro returned - handle like regular
+			 * end of macro, even though some checks
+			 * are unnecessary here.
+			 * macro_pc will still point to the return PC.
+			 * We are still in the "escape" state and must
+			 * reset it here, so it is not confused
+			 * with a trailing ^[ in macro.
+			 */
+			States::current = &States::start;
+		}
 
 		/*
 		 * Subsequent errors must still be
@@ -172,12 +186,22 @@ Execute::macro(const gchar *macro, bool locals)
 				throw Error("Label \"%s\" not found",
 				            Goto::skip_label);
 
-			if (States::current != &States::start)
+			if (States::current == &States::escape) {
+				/*
+				 * Due to the deferred nature of ^[,
+				 * it is valid to end in the "escape" state.
+				 * FIXME: This could be avoided by signalling
+				 * the state the end of macro but State::refresh()
+				 * cannot be used :-(.
+				 */
+				expressions.discard_args();
+			} else if (States::current != &States::start) {
 				/*
 				 * can only happen if we returned because
 				 * of macro end
 				 */
 				throw Error("Unterminated command");
+			}
 
 			/*
 			 * This handles the problem of Q-Registers
@@ -190,7 +214,7 @@ Execute::macro(const gchar *macro, bool locals)
 			if (locals)
 				QRegisters::locals->clear();
 		} catch (Error &error) {
-			error.set_coord(macro, strlen(macro));
+			error.set_coord(macro, macro_pc);
 			throw; /* forward */
 		}
 	} catch (...) {
@@ -726,7 +750,7 @@ StateStart::custom(gchar chr)
 	tecoBool rc;
 
 	/*
-	 * <CTRL/x> commands implemented in StateCtrlCmd
+	 * <CTRL/x> commands implemented in StateControl
 	 */
 	if (IS_CTL(chr))
 		return States::control.get_next_state(CTL_ECHO(chr));
@@ -1801,6 +1825,7 @@ StateControl::StateControl() : State()
 	transitions['I'] = &States::insert_indent;
 	transitions['U'] = &States::ctlucommand;
 	transitions['^'] = &States::ascii;
+	transitions['['] = &States::escape;
 }
 
 State *
@@ -1841,31 +1866,6 @@ StateControl::custom(gchar chr)
 		break;
 
 	/*
-	 * Alternatives: ^[, <CTRL/[>, <ESC>
-	 */
-	/*$
-	 * ^[ -- Discard all arguments
-	 * $
-	 *
-	 * Pops and discards all values from the stack that
-	 * might otherwise be used as arguments to following
-	 * commands.
-	 * Therefore it stops popping on stack boundaries like
-	 * they are introduced by arithmetic brackets or loops.
-	 *
-	 * Note that ^[ is usually typed using the Escape key.
-	 * CTRL+[ however is possible as well and equivalent to
-	 * Escape in every manner.
-	 * The Caret-[ notation however is processed like any
-	 * ordinary command and only works as the discard-arguments
-	 * command.
-	 */
-	case '[':
-		BEGIN_EXEC(&States::start);
-		expressions.discard_args();
-		break;
-
-	/*
 	 * Additional numeric operations
 	 */
 	/*$
@@ -1930,6 +1930,82 @@ StateASCII::custom(gchar chr)
 	return &States::start;
 }
 
+/*
+ * The Escape state is special, as it implements
+ * a kind of "lookahead" for the ^[ command (dicard all
+ * arguments).
+ * It is not executed immediately as usual in SciTECO
+ * but only if not followed by an escape character.
+ * This is necessary since $$ is the macro return
+ * and command-line termination command and it must not
+ * discard arguments.
+ * Deferred execution of ^[ is possible since it does
+ * not have any visible side-effects - its effects can
+ * only be seen when executing the following command.
+ */
+StateEscape::StateEscape()
+{
+	transitions['\0'] = this;
+}
+
+State *
+StateEscape::custom(gchar chr)
+{
+	/*$
+	 * [a1,a2,...]$$ -- Terminate command line or return from macro
+	 * [a1,a2,...]^[$
+	 *
+	 * Returns from the current macro invocation.
+	 * The numeric stack is not modified, giving the
+	 * effect of returning the arguments or stack contents
+	 * preceding the command to the macro caller.
+	 * It is generally semantically equivalent to reaching
+	 * the end of the current macro but is executed faster.
+	 *
+	 * Therefore returning from the top-level macro in batch mode
+	 * will exit the program or start up interactive mode depending
+	 * on whether program exit has been requested.
+	 * \(lqEX\fB$$\fP\(rq is thus a common idiom to exit
+	 * prematurely.
+	 *
+	 * In interactive mode, returning from the top-level macro
+	 * (i.e. typing \fB$$\fP at the command line) has the
+	 * effect of command line termination.
+	 *
+	 * Only the first \fIescape\fP of \fB$$\fP may be typed
+	 * in up-arrow mode as \fB^[$\fP \(em the second character
+	 * must be a real escape character.
+	 */
+	if (chr == CTL_KEY_ESC) {
+		BEGIN_EXEC(&States::start);
+		throw Return();
+	}
+
+	/*
+	 * Alternatives: ^[, <CTRL/[>, <ESC>
+	 */
+	/*$
+	 * ^[ -- Discard all arguments
+	 * $
+	 *
+	 * Pops and discards all values from the stack that
+	 * might otherwise be used as arguments to following
+	 * commands.
+	 * Therefore it stops popping on stack boundaries like
+	 * they are introduced by arithmetic brackets or loops.
+	 *
+	 * Note that ^[ is usually typed using the Escape key.
+	 * CTRL+[ however is possible as well and equivalent to
+	 * Escape in every manner.
+	 * The Caret-[ notation however is processed like any
+	 * ordinary command and only works as the discard-arguments
+	 * command.
+	 */
+	if (mode == MODE_NORMAL)
+		expressions.discard_args();
+	return States::start.get_next_state(chr);
+}
+
 StateECommand::StateECommand() : State()
 {
 	transitions['\0'] = this;
diff --git a/src/parser.h b/src/parser.h
index a1bfd75..fe8f08b 100644
--- a/src/parser.h
+++ b/src/parser.h
@@ -219,6 +219,14 @@ private:
 	State *custom(gchar chr);
 };
 
+class StateEscape : public State {
+public:
+	StateEscape();
+
+private:
+	State *custom(gchar chr);
+};
+
 class StateFCommand : public State {
 public:
 	StateFCommand();
@@ -310,6 +318,7 @@ namespace States {
 	extern StateStart		start;
 	extern StateControl		control;
 	extern StateASCII		ascii;
+	extern StateEscape		escape;
 	extern StateFCommand		fcommand;
 	extern StateChangeDir		changedir;
 	extern StateCondCommand		condcommand;