THE GREAT CEEIFICATION EVENT

This is a total conversion of SciTECO to plain C (GNU C11).
The chance was taken to improve a lot of internal datastructures,
fix fundamental bugs and lay the foundations of future features.
The GTK user interface is now in an useable state!
All changes have been squashed together.

The language itself has almost not changed at all, except for:

* Detection of string terminators (usually Escape) now takes
  the string building characters into account.
  A string is only terminated outside of string building characters.
  In other words, you can now for instance write
  I^EQ[Hello$world]$
  This removes one of the last bits of shellisms which is out of
  place in SciTECO where no tokenization/lexing is performed.
  Consequently, the current termination character can also be
  escaped using ^Q/^R.
  This is used by auto completions to make sure that strings
  are inserted verbatim and without unwanted sideeffects.
* All strings can now safely contain null-characters
  (see also: 8-bit cleanliness).
  The null-character itself (^@) is not (yet) a valid SciTECO
  command, though.

An incomplete list of changes:

* We got rid of the BSD headers for RB trees and lists/queues.
  The problem with them was that they used a form of metaprogramming
  only to gain a bit of type safety. It also resulted in less
  readble code. This was a C++ desease.
  The new code avoids metaprogramming only to gain type safety.
  The BSD tree.h has been replaced by rb3ptr by Jens Stimpfle
  (https://github.com/jstimpfle/rb3ptr).
  This implementation is also more memory efficient than BSD's.
  The BSD list.h and queue.h has been replaced with a custom
  src/list.h.
* Fixed crashes, performance issues and compatibility issues with
  the Gtk 3 User Interface.
  It is now more or less ready for general use.
  The GDK lock is no longer used to avoid using deprecated functions.
  On the downside, the new implementation (driving the Gtk event loop
  stepwise) is even slower than the old one.
  A few glitches remain (see TODO), but it is hoped that they will
  be resolved by the Scintilla update which will be performed soon.
* A lot of program units have been split up, so they are shorter
  and easier to maintain: core-commands.c, qreg-commands.c,
  goto-commands.c, file-utils.h.
* Parser states are simply structs of callbacks now.
  They still use a kind of polymorphy using a preprocessor trick.
  TECO_DEFINE_STATE() takes an initializer list that will be
  merged with the default list of field initializers.
  To "subclass" states, you can simply define new macros that add
  initializers to existing macros.
* Parsers no longer have a "transitions" table but the input_cb()
  may use switch-case statements.
  There are also teco_machine_main_transition_t now which can
  be used to implement simple transitions. Additionally, you
  can specify functions to execute during transitions.
  This largely avoids long switch-case-statements.
* Parsers are embeddable/reusable now, at least in parse-only mode.
  This does not currently bring any advantages but may later
  be used to write a Scintilla lexer for TECO syntax highlighting.
  Once parsers are fully embeddable, it will also be possible
  to run TECO macros in a kind of coroutine which would allow
  them to process string arguments in real time.
* undo.[ch] still uses metaprogramming extensively but via
  the C preprocessor of course. On the downside, most undo
  token generators must be initiated explicitly (theoretically
  we could have used embedded functions / trampolines to
  instantiate automatically but this has turned out to be
  dangereous).
  There is a TECO_DEFINE_UNDO_CALL() to generate closures for
  arbitrary functions now (ie. to call an arbitrary function
  at undo-time). This simplified a lot of code and is much
  shorter than manually pushing undo tokens in many cases.
* Instead of the ridiculous C++ Curiously Recurring Template
  Pattern to achieve static polymorphy for user interface
  implementations, we now simply declare all functions to
  implement in interface.h and link in the implementations.
  This is possible since we no longer hace to define
  interface subclasses (all state is static variables in
  the interface's *.c files).
* Headers are now significantly shorter than in C++ since
  we can often hide more of our "class" implementations.
* Memory counting is based on dlmalloc for most platforms now.
  Unfortunately, there is no malloc implementation that
  provides an efficient constant-time memory counter that
  is guaranteed to decrease when freeing memory.
  But since we use a defined malloc implementation now,
  malloc_usable_size() can be used safely for tracking memory use.
  malloc() replacement is very tricky on Windows, so we
  use a poll thread on Windows. This can also be enabled
  on other supported platforms using --disable-malloc-replacement.
  All in all, I'm still not pleased with the state of memory
  limiting. It is a mess.
* Error handling uses GError now. This has the advantage that
  the GError codes can be reused once we support error catching
  in the SciTECO language.
* Added a few more test suite cases.
* Haiku is no longer supported as builds are instable and
  I did not manage to debug them - quite possibly Haiku bugs
  were responsible.
* Glib v2.44 or later are now required.
  The GTK UI requires Gtk+ v3.12 or later now.
  The GtkFlowBox fallback and sciteco-wrapper workaround are
  no longer required.
* We now extensively use the GCC/Clang-specific g_auto
  feature (automatic deallocations when leaving the current
  code block).
* Updated copyright to 2021.
  SciTECO has been in continuous development, even though there
  have been no commits since 2018.
* Since these changes are so significant, the target release has
  been set to v2.0.
  It is planned that beginning with v3.0, the language will be
  kept stable.

1 files changed, 472 insertions, 373 deletions

diff --git a/src/parser.h b/src/parser.h
index 9255268..b594edc 100644
--- a/src/parser.h
+++ b/src/parser.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2012-2017 Robin Haberkorn
+ * Copyright (C) 2012-2021 Robin Haberkorn
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
@@ -14,47 +14,130 @@
  * You should have received a copy of the GNU General Public License
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
-
-#ifndef __PARSER_H
-#define __PARSER_H
-
-#include <string.h>
+#pragma once
 
 #include <glib.h>
 
+#include <Scintilla.h>
+
 #include "sciteco.h"
-#include "memory.h"
-#include "undo.h"
-#include "error.h"
-#include "expressions.h"
-
-namespace SciTECO {
-
-/* TECO uses only lower 7 bits for commands */
-#define MAX_TRANSITIONS	127
-
-class State : public Object {
-protected:
-	/* static transitions */
-	State *transitions[MAX_TRANSITIONS];
-
-	inline void
-	init(const gchar *chars, State &state)
-	{
-		while (*chars)
-			transitions[(int)*chars++] = &state;
-	}
-	inline void
-	init(const gchar *chars)
-	{
-		init(chars, *this);
-	}
+#include "string-utils.h"
+#include "goto.h"
+#include "qreg.h"
+
+/*
+ * Forward Declarations
+ */
+typedef const struct teco_state_t teco_state_t;
+typedef struct teco_machine_t teco_machine_t;
+typedef struct teco_machine_main_t teco_machine_main_t;
+
+typedef struct {
+	/** how many iterations are left */
+	teco_int_t counter;
+	/** Program counter of loop start command */
+	guint pc : sizeof(guint)*8 - 1;
+	/**
+	 * Whether the loop represents an argument
+	 * barrier or not (it "passes through"
+	 * stack arguments).
+	 *
+	 * Since the program counter is usually
+	 * a signed integer, it's ok steal one
+	 * bit for the pass_through flag.
+	 */
+	gboolean pass_through : 1;
+} teco_loop_context_t;
+
+extern GArray *teco_loop_stack;
+
+void undo__insert_val__teco_loop_stack(guint, teco_loop_context_t);
+void undo__remove_index__teco_loop_stack(guint);
+
+/**
+ * @defgroup states Parser states
+ *
+ * Parser states are defined as global constants using the TECO_DEFINE_STATE()
+ * macro, allowing individual fields and callbacks to be overwritten.
+ * Derived macros are defined to factor out common fields and settings.
+ * States therefore form a hierarchy, which is documented using
+ * \@interface and \@implements tags.
+ *
+ * @{
+ */
 
-public:
-	State();
+/*
+ * FIXME: Remove _cb from all callback names. See qreg.h.
+ * FIXME: Maybe use TECO_DECLARE_VTABLE_METHOD()?
+ */
+typedef const struct {
+	gboolean string_building : 1;
+	gboolean last : 1;
 
-	static void input(gchar chr);
-	State *get_next_state(gchar chr);
+	/**
+	 * Called repeatedly to process chunks of input and give interactive feedback.
+	 *
+	 * Can be NULL if no interactive feedback is required.
+	 */
+	gboolean (*process_cb)(teco_machine_main_t *ctx, const teco_string_t *str,
+	                       gsize new_chars, GError **error);
+
+	/**
+	 * Called at the end of the string argument to determine the next state.
+	 * Commands that don't give interactive feedback can use this callback
+	 * to perform their main processing.
+	 */
+	teco_state_t *(*done_cb)(teco_machine_main_t *ctx, const teco_string_t *str, GError **error);
+} teco_state_expectstring_t;
+
+typedef const struct {
+	teco_qreg_type_t type;
+
+	/** Called when a register specification has been successfully parsed. */
+	teco_state_t *(*got_register_cb)(teco_machine_main_t *ctx, teco_qreg_t *qreg,
+	                                 teco_qreg_table_t *table, GError **error);
+} teco_state_expectqreg_t;
+
+typedef gboolean (*teco_state_initial_cb_t)(teco_machine_t *ctx, GError **error);
+typedef teco_state_t *(*teco_state_input_cb_t)(teco_machine_t *ctx, gchar chr, GError **error);
+typedef gboolean (*teco_state_refresh_cb_t)(teco_machine_t *ctx, GError **error);
+typedef gboolean (*teco_state_end_of_macro_cb_t)(teco_machine_t *ctx, GError **error);
+typedef gboolean (*teco_state_process_edit_cmd_cb_t)(teco_machine_t *ctx, teco_machine_t *parent_ctx,
+                                                     gchar key, GError **error);
+
+typedef enum {
+	TECO_FNMACRO_MASK_START		= (1 << 0),
+	TECO_FNMACRO_MASK_STRING	= (1 << 1),
+	TECO_FNMACRO_MASK_DEFAULT	= ~((1 << 2)-1)
+} teco_fnmacro_mask_t;
+
+/**
+ * A teco_machine_t state.
+ * These are declared as constants using TECO_DEFINE_STATE() and friends.
+ *
+ * @note Unless you don't want to manually "upcast" the teco_machine_t* in
+ * callback implementations, you will have to cast your callback types when initializing
+ * the teco_state_t vtables.
+ * Casting to functions of different signature is theoretically undefined behavior,
+ * but works on all major platforms including Emscripten, as long as they differ only
+ * in pointer types.
+ */
+struct teco_state_t {
+	/**
+	 * Called the first time this state is entered.
+	 * Theoretically, you can use teco_machine_main_transition_t instead,
+	 * but this callback improves reusability.
+	 *
+	 * It can be NULL if not required.
+	 */
+	teco_state_initial_cb_t initial_cb;
+
+	/**
+	 * Get next state given an input character.
+	 *
+	 * This is a mandatory field.
+	 */
+	teco_state_input_cb_t input_cb;
 
 	/**
 	 * Provide interactive feedback.
@@ -63,32 +146,20 @@ public:
 	 * immediate interactive feedback should provide that
 	 * feedback; allowing them to optimize batch mode,
 	 * macro and many other cases.
+	 *
+	 * It can be NULL if not required.
 	 */
-	virtual void refresh(void) {}
+	teco_state_refresh_cb_t refresh_cb;
 
 	/**
 	 * Called at the end of a macro.
 	 * Most states/commands are not allowed to end unterminated
 	 * at the end of a macro.
+	 *
+	 * It can be NULL if not required.
 	 */
-	virtual void
-	end_of_macro(void)
-	{
-		throw Error("Unterminated command");
-	}
+	teco_state_end_of_macro_cb_t end_of_macro_cb;
 
-protected:
-	static bool eval_colon(void);
-
-	/** Get next state given an input character */
-	virtual State *
-	custom(gchar chr)
-	{
-		throw SyntaxError(chr);
-		return NULL;
-	}
-
-public:
 	/**
 	 * Process editing command (or key press).
 	 *
@@ -97,383 +168,411 @@ public:
 	 * editing commands (behaviour on key press).
 	 *
 	 * By implementing this method, sub-states can either
-	 * handle a key and return or chain to the
-	 * parent's process_edit_cmd() implementation.
+	 * handle a key and return, chain to the
+	 * parent's process_edit_cmd() implementation or even
+	 * to the parent state machine's handler.
 	 *
 	 * All implementations of this method are defined in
-	 * cmdline.cpp.
+	 * cmdline.c.
+	 *
+	 * This is a mandatory field.
 	 */
-	virtual void process_edit_cmd(gchar key);
-
-	enum fnmacroMask {
-		FNMACRO_MASK_START =	(1 << 0),
-		FNMACRO_MASK_STRING =	(1 << 1),
-		FNMACRO_MASK_DEFAULT =	~((1 << 2)-1)
-	};
+	teco_state_process_edit_cmd_cb_t process_edit_cmd_cb;
 
 	/**
-	 * Get the function key macro mask this
-	 * state refers to.
+	 * Whether this state is a start state (ie. not within any
+	 * escape sequence etc.).
+	 * This is separate of TECO_FNMACRO_MASK_START which is set
+	 * only in the main machine's start states.
+	 */
+	gboolean is_start : 1;
+	/**
+	 * Function key macro mask.
+	 * This is not a bitmask since it is compared with values set
+	 * from TECO, so the bitorder needs to be defined.
 	 *
-	 * Could also be modelled as a State member.
+	 * @fixme If we intend to "forward" masks from other state machines like
+	 * teco_machine_stringbuilding_t, this should probably be a callback.
 	 */
-	virtual fnmacroMask
-	get_fnmacro_mask(void) const
-	{
-		return FNMACRO_MASK_DEFAULT;
-	}
-};
+	teco_fnmacro_mask_t fnmacro_mask : 8;
 
-/**
- * Base class of states with case-insenstive input.
- *
- * This is meant for states accepting command characters
- * that can possibly be case-folded.
- */
-class StateCaseInsensitive : public State {
-protected:
-	/* in cmdline.cpp */
-	void process_edit_cmd(gchar key);
-};
-
-template <typename Type>
-class MicroStateMachine : public Object {
-protected:
-	/* label pointers */
-	typedef const void *MicroState;
-	const MicroState StateStart;
-#define MICROSTATE_START G_STMT_START {	\
-	if (this->state != StateStart)		\
-		goto *this->state;		\
-} G_STMT_END
-
-	MicroState state;
-
-	inline void
-	set(MicroState next)
-	{
-		if (next != state)
-			undo.push_var(state) = next;
-	}
-
-public:
-	MicroStateMachine() : StateStart(NULL), state(StateStart) {}
-	virtual ~MicroStateMachine() {}
-
-	virtual inline void
-	reset(void)
-	{
-		set(StateStart);
-	}
-
-	virtual bool input(gchar chr, Type &result) = 0;
+	/**
+	 * Additional state-dependent callbacks and settings.
+	 * This wastes some bytes compared to other techniques for extending teco_state_t
+	 * but this is acceptable since there is only a limited number of constant instances.
+	 * The main advantage of this approach is that we can use a single
+	 * TECO_DEFINE_STATE() for defining and deriving all defaults.
+	 */
+	union {
+		teco_state_expectstring_t	expectstring;
+		teco_state_expectqreg_t		expectqreg;
+	};
 };
 
-/* avoid circular dependency on qregisters.h */
-class QRegSpecMachine;
-
-class StringBuildingMachine : public MicroStateMachine<gchar *> {
-	enum Mode {
-		MODE_NORMAL,
-		MODE_UPPER,
-		MODE_LOWER
-	} mode;
-
-	bool toctl;
-
-public:
-	QRegSpecMachine *qregspec_machine;
+/** @} */
 
-	StringBuildingMachine() : MicroStateMachine<gchar *>(),
-				  mode(MODE_NORMAL), toctl(false),
-				  qregspec_machine(NULL) {}
-	~StringBuildingMachine();
+gboolean teco_state_end_of_macro(teco_machine_t *ctx, GError **error);
 
-	void reset(void);
+/* in cmdline.c */
+gboolean teco_state_process_edit_cmd(teco_machine_t *ctx, teco_machine_t *parent_ctx, gchar chr, GError **error);
 
-	bool input(gchar chr, gchar *&result);
-};
-
-/*
- * Super-class for states accepting string arguments
- * Opaquely cares about alternative-escape characters,
- * string building commands and accumulation into a string
+/**
+ * @interface TECO_DEFINE_STATE
+ * @implements teco_state_t
+ * @ingroup states
+ *
+ * @todo Should we eliminate required callbacks, this could be turned into a
+ * struct initializer TECO_INIT_STATE() and TECO_DECLARE_STATE() would become pointless.
+ * This would also ease declaring static states.
  */
-class StateExpectString : public State {
-	gsize insert_len;
-
-	gint nesting;
-
-	bool string_building;
-	bool last;
-
-	StringBuildingMachine machine;
-
-public:
-	StateExpectString(bool _building = true, bool _last = true)
-			 : insert_len(0), nesting(1),
-			   string_building(_building), last(_last) {}
-
-private:
-	State *custom(gchar chr);
-	void refresh(void);
-
-	virtual fnmacroMask
-	get_fnmacro_mask(void) const
-	{
-		return FNMACRO_MASK_STRING;
+#define TECO_DEFINE_STATE(NAME, ...) \
+	/** @ingroup states */ \
+	teco_state_t NAME = { \
+		.initial_cb = NULL,	/* do nothing */ \
+		.input_cb = (teco_state_input_cb_t)NAME##_input, /* always required */ \
+		.refresh_cb = NULL,	/* do nothing */ \
+		.end_of_macro_cb = teco_state_end_of_macro, \
+		.process_edit_cmd_cb = teco_state_process_edit_cmd, \
+		.is_start = FALSE, \
+		.fnmacro_mask = TECO_FNMACRO_MASK_DEFAULT, \
+		##__VA_ARGS__ \
 	}
 
-protected:
-	virtual void initial(void) {}
-	virtual void process(const gchar *str, gint new_chars) {}
-	virtual State *done(const gchar *str) = 0;
-
-	/* in cmdline.cpp */
-	void process_edit_cmd(gchar key);
-};
-
-class StateExpectFile : public StateExpectString {
-public:
-	StateExpectFile(bool _building = true, bool _last = true)
-		       : StateExpectString(_building, _last) {}
-
-private:
-	State *done(const gchar *str);
-
-protected:
-	virtual State *got_file(const gchar *filename) = 0;
+/** @ingroup states */
+#define TECO_DECLARE_STATE(NAME) \
+	extern teco_state_t NAME
 
-	/* in cmdline.cpp */
-	void process_edit_cmd(gchar key);
-};
+/* in cmdline.c */
+gboolean teco_state_caseinsensitive_process_edit_cmd(teco_machine_t *ctx, teco_machine_t *parent_ctx, gchar chr, GError **error);
 
-class StateExpectDir : public StateExpectFile {
-public:
-	StateExpectDir(bool _building = true, bool _last = true)
-		      : StateExpectFile(_building, _last) {}
+/**
+ * @interface TECO_DEFINE_STATE_CASEINSENSITIVE
+ * @implements TECO_DEFINE_STATE
+ * @ingroup states
+ *
+ * Base class of states with case-insenstive input.
+ *
+ * This is meant for states accepting command characters
+ * that can possibly be case-folded.
+ */
+#define TECO_DEFINE_STATE_CASEINSENSITIVE(NAME, ...) \
+	TECO_DEFINE_STATE(NAME, \
+		.process_edit_cmd_cb = teco_state_caseinsensitive_process_edit_cmd, \
+		##__VA_ARGS__ \
+	)
 
-protected:
-	/* in cmdline.cpp */
-	void process_edit_cmd(gchar key);
+/**
+ * Base class of state machine.
+ *
+ * @note On extending teco_machine_t:
+ * There is `-fplan9-extensions`, but Clang doesn't support it.
+ * There is `-fms-extensions`, but that would require type-unsafe
+ * casting to teco_machine_t*.
+ * It's possible to portably implement typesafe inheritance by using
+ * an anonymous union of an anonymous struct and a named struct, but it's
+ * not really worth the trouble in our flat "class" hierachy.
+ */
+struct teco_machine_t {
+	teco_state_t *current;
+	/**
+	 * Whether side effects must be reverted on rubout.
+	 * State machines created within macro calls don't have to
+	 * even in interactive mode.
+	 */
+	gboolean must_undo;
 };
 
-class StateStart : public StateCaseInsensitive {
-public:
-	StateStart();
+static inline void
+teco_machine_init(teco_machine_t *ctx, teco_state_t *initial, gboolean must_undo)
+{
+	ctx->current = initial;
+	ctx->must_undo = must_undo;
+}
 
-private:
-	void insert_integer(tecoInt v);
-	tecoInt read_integer(void);
+static inline void
+teco_machine_reset(teco_machine_t *ctx, teco_state_t *initial)
+{
+	if (ctx->current != initial)
+		teco_undo_ptr(ctx->current) = initial;
+}
 
-	tecoBool move_chars(tecoInt n);
-	tecoBool move_lines(tecoInt n);
+gboolean teco_machine_input(teco_machine_t *ctx, gchar chr, GError **error);
 
-	tecoBool delete_words(tecoInt n);
+typedef enum {
+	TECO_STRINGBUILDING_MODE_NORMAL = 0,
+	TECO_STRINGBUILDING_MODE_UPPER,
+	TECO_STRINGBUILDING_MODE_LOWER
+} teco_stringbuilding_mode_t;
 
-	State *custom(gchar chr);
+/**
+ * A stringbuilding state machine.
+ *
+ * @fixme Should contain the escape char (currently in teco_machine_expectstring_t),
+ * so that we can escape it via ^Q.
+ *
+ * @extends teco_machine_t
+ */
+typedef struct teco_machine_stringbuilding_t {
+	teco_machine_t parent;
 
-	void end_of_macro(void) {}
+	/**
+	 * A teco_stringbuilding_mode_t.
+	 * This is still a guint, so you can call teco_undo_guint().
+	 */
+	guint mode;
 
-	fnmacroMask
-	get_fnmacro_mask(void) const
-	{
-		return FNMACRO_MASK_START;
-	}
-};
+	/**
+	 * The escape/termination character.
+	 *
+	 * If this is `[` or `{`, it is assumed that `]` and `}` must
+	 * be escaped as well by teco_machine_stringbuilding_escape().
+	 */
+	gchar escape_char;
 
-class StateControl : public StateCaseInsensitive {
-public:
-	StateControl();
+	/**
+	 * Q-Register table for local registers.
+	 * This is stored here only to be passed to the Q-Reg spec machine.
+	 */
+	teco_qreg_table_t *qreg_table_locals;
 
-private:
-	State *custom(gchar chr);
-};
+	/**
+	 * A QRegister specification parser.
+	 * It is allocated since it in turn contains a string building machine.
+	 */
+	teco_machine_qregspec_t *machine_qregspec;
 
-class StateASCII : public State {
-public:
-	StateASCII();
+	/**
+	 * A string to append characters to or NULL in parse-only mode.
+	 *
+	 * @bug As a side-effect, rubbing out in parse-only mode is severely limited
+	 * (see teco_state_stringbuilding_start_process_edit_cmd()).
+	 */
+	teco_string_t *result;
+} teco_machine_stringbuilding_t;
 
-private:
-	State *custom(gchar chr);
-};
+void teco_machine_stringbuilding_init(teco_machine_stringbuilding_t *ctx, gchar escape_char,
+                                      teco_qreg_table_t *locals, gboolean must_undo);
 
-class StateEscape : public StateCaseInsensitive {
-public:
-	StateEscape();
+void teco_machine_stringbuilding_reset(teco_machine_stringbuilding_t *ctx);
 
-private:
-	State *custom(gchar chr);
+/**
+ * Parse a string building character.
+ *
+ * @param ctx The string building machine.
+ * @param chr The character to parse.
+ * @param result String to append characters to or NULL in parse-only mode.
+ * @param error GError.
+ * @return FALSE in case of error.
+ */
+static inline gboolean
+teco_machine_stringbuilding_input(teco_machine_stringbuilding_t *ctx, gchar chr,
+                                  teco_string_t *result, GError **error)
+{
+	ctx->result = result;
+	return teco_machine_input(&ctx->parent, chr, error);
+}
 
-	void end_of_macro(void);
+void teco_machine_stringbuilding_escape(teco_machine_stringbuilding_t *ctx, const gchar *str, gsize len,
+                                        teco_string_t *target);
 
-	/*
-	 * The state should behave like StateStart
-	 * when it comes to function key macro masking.
-	 */
-	fnmacroMask
-	get_fnmacro_mask(void) const
-	{
-		return FNMACRO_MASK_START;
-	}
-};
+void teco_machine_stringbuilding_clear(teco_machine_stringbuilding_t *ctx);
 
-class StateFCommand : public StateCaseInsensitive {
-public:
-	StateFCommand();
+/**
+ * Peristent state for teco_state_expectstring_input().
+ *
+ * This is part of the main machine instead of being a global variable,
+ * so that parsers can be run in parallel.
+ *
+ * Since it will also be part of a macro invocation frame, it will allow
+ * for tricks like macro-hooks while in "expectstring" states or calling
+ * macros as part of string building characters or macro string arguments.
+ */
+typedef struct {
+	teco_string_t string;
+	gsize insert_len;
+	gint nesting;
 
-private:
-	State *custom(gchar chr);
-};
+	teco_machine_stringbuilding_t machine;
+} teco_machine_expectstring_t;
 
-class UndoTokenChangeDir : public UndoToken {
-	gchar *dir;
+/**
+ * Scintilla message for collection by ES commands.
+ *
+ * @fixme This is a "forward" declaration, so that we don't introduce cyclic
+ * header dependencies.
+ * Could presumably be avoided by splitting parser.h in two.
+ */
+typedef struct {
+	unsigned int iMessage;
+	uptr_t wParam;
+	sptr_t lParam;
+} teco_machine_scintilla_t;
+
+typedef enum {
+	/** Normal parsing - ie. execute while parsing */
+	TECO_MODE_NORMAL = 0,
+	/** Parse, but don't execute until reaching not-yet-defined Goto-label */
+	TECO_MODE_PARSE_ONLY_GOTO,
+	/** Parse, but don't execute until reaching end of loop */
+	TECO_MODE_PARSE_ONLY_LOOP,
+	/** Parse, but don't execute until reaching end of conditional or its else-clause */
+	TECO_MODE_PARSE_ONLY_COND,
+	/** Parse, but don't execute until reaching the very end of conditional */
+	TECO_MODE_PARSE_ONLY_COND_FORCE
+} teco_mode_t;
+
+/** @extends teco_machine_t */
+struct teco_machine_main_t {
+	teco_machine_t parent;
+
+	gint macro_pc;
 
-public:
 	/**
-	 * Construct undo token.
-	 *
-	 * This passes ownership of the directory string
-	 * to the undo token object.
+	 * Aliases bitfield with an integer.
+	 * This allows teco_undo_guint(__flags),
+	 * while still supporting easy-to-access flags.
 	 */
-	UndoTokenChangeDir(gchar *_dir) : dir(_dir) {}
-	~UndoTokenChangeDir()
-	{
-		g_free(dir);
-	}
-
-	void run(void);
-};
+	union {
+		struct {
+			teco_mode_t mode : 8;
+
+			gboolean modifier_colon : 1;
+			gboolean modifier_at : 1;
+		};
+		guint __flags;
+	};
 
-class StateChangeDir : public StateExpectDir {
-private:
-	State *got_file(const gchar *filename);
-};
+	/** The nesting level of braces */
+	guint brace_level;
+	/** The nesting level of loops and control structures */
+	gint nest_level;
+	/**
+	 * Loop frame pointer: The number of elements on
+	 * the loop stack when a macro invocation frame is
+	 * created.
+	 * This is used to perform checks for flow control
+	 * commands to avoid jumping with invalid PCs while
+	 * not creating a new stack per macro frame.
+	 */
+	guint loop_stack_fp;
 
-class StateCondCommand : public StateCaseInsensitive {
-public:
-	StateCondCommand();
+	teco_goto_table_t goto_table;
+	teco_qreg_table_t *qreg_table_locals;
 
-private:
-	State *custom(gchar chr);
+	/*
+	 * teco_state_t-dependent state.
+	 *
+	 * Some of these cannot be used concurrently and are therefore
+	 * grouped into unions.
+	 * We could further optimize memory usage by dynamically allocating
+	 * some of these structures on demand.
+	 */
+	teco_machine_expectstring_t		expectstring;
+	union {
+		teco_string_t			goto_label;
+		teco_machine_qregspec_t		*expectqreg;
+		teco_machine_scintilla_t	scintilla;
+	};
 };
 
-class StateECommand : public StateCaseInsensitive {
-public:
-	StateECommand();
-
-private:
-	State *custom(gchar chr);
-};
+void teco_machine_main_init(teco_machine_main_t *ctx,
+                            teco_qreg_table_t *qreg_table_locals,
+                            gboolean must_undo);
 
-class StateScintilla_symbols : public StateExpectString {
-public:
-	StateScintilla_symbols() : StateExpectString(true, false) {}
+gboolean teco_machine_main_eval_colon(teco_machine_main_t *ctx);
 
-private:
-	State *done(const gchar *str);
+gboolean teco_machine_main_step(teco_machine_main_t *ctx,
+                                const gchar *macro, gint stop_pos, GError **error);
 
-protected:
-	/* in cmdline.cpp */
-	void process_edit_cmd(gchar key);
-};
+gboolean teco_execute_macro(const gchar *macro, gsize macro_len,
+                            teco_qreg_table_t *qreg_table_locals, GError **error);
+gboolean teco_execute_file(const gchar *filename, teco_qreg_table_t *qreg_table_locals, GError **error);
 
-class StateScintilla_lParam : public StateExpectString {
-private:
-	State *done(const gchar *str);
-};
+typedef const struct {
+	teco_state_t *next;
+	void (*transition_cb)(teco_machine_main_t *ctx, GError **error);
+} teco_machine_main_transition_t;
 
 /*
- * also serves as base class for replace-insertion states
+ * FIXME: There should probably be a teco_state_plain with
+ * the transitions and their length being stored in
+ * teco_state_t::transitions.
+ * This does not exclude the possibility of overwriting input_cb.
  */
-class StateInsert : public StateExpectString {
-public:
-	StateInsert(bool building = true)
-	           : StateExpectString(building) {}
-
-protected:
-	void initial(void);
-	void process(const gchar *str, gint new_chars);
-	State *done(const gchar *str);
-
-	/* in cmdline.cpp */
-	void process_edit_cmd(gchar key);
-};
-
-class StateInsertIndent : public StateInsert {
-protected:
-	void initial(void);
-};
-
-namespace States {
-	extern StateStart		start;
-	extern StateControl		control;
-	extern StateASCII		ascii;
-	extern StateEscape		escape;
-	extern StateFCommand		fcommand;
-	extern StateChangeDir		changedir;
-	extern StateCondCommand		condcommand;
-	extern StateECommand		ecommand;
-	extern StateScintilla_symbols	scintilla_symbols;
-	extern StateScintilla_lParam	scintilla_lparam;
-	extern StateInsert		insert_building;
-	extern StateInsert		insert_nobuilding;
-	extern StateInsertIndent	insert_indent;
-
-	extern State *current;
-
-	static inline bool
-	is_start(void)
-	{
-		/*
-		 * StateEscape should behave very much like StateStart.
-		 */
-		return current == &start || current == &escape;
-	}
-}
+teco_state_t *teco_machine_main_transition_input(teco_machine_main_t *ctx,
+                                                 teco_machine_main_transition_t *transitions,
+                                                 guint len, gchar chr, GError **error);
 
-extern enum Mode {
-	MODE_NORMAL = 0,
-	MODE_PARSE_ONLY_GOTO,
-	MODE_PARSE_ONLY_LOOP,
-	MODE_PARSE_ONLY_COND
-} mode;
+void teco_machine_main_clear(teco_machine_main_t *ctx);
 
-#define BEGIN_EXEC(STATE) G_STMT_START {	\
-	if (mode > MODE_NORMAL)			\
-		return STATE;			\
-} G_STMT_END
+G_DEFINE_AUTO_CLEANUP_CLEAR_FUNC(teco_machine_main_t, teco_machine_main_clear);
 
-extern gint macro_pc;
+teco_state_t *teco_state_expectstring_input(teco_machine_main_t *ctx, gchar chr, GError **error);
+gboolean teco_state_expectstring_refresh(teco_machine_main_t *ctx, GError **error);
 
-extern gchar *strings[2];
-extern gchar escape_char;
+/* in cmdline.c */
+gboolean teco_state_expectstring_process_edit_cmd(teco_machine_main_t *ctx, teco_machine_t *parent_ctx, gchar key, GError **error);
 
-struct LoopContext {
-	/** how many iterations are left */
-	tecoInt counter;
-	/** Program counter of loop start command */
-	guint pc : sizeof(guint)*8 - 1;
-	/**
-	 * Whether the loop represents an argument
-	 * barrier or not (it "passes through"
-	 * stack arguments).
-	 *
-	 * Since the program counter is usually
-	 * a signed integer, it's ok steal one
-	 * bit for the pass_through flag.
-	 */
-	bool pass_through : 1;
-};
-typedef ValueStack<LoopContext> LoopStack;
-extern LoopStack loop_stack;
+/**
+ * @interface TECO_DEFINE_STATE_EXPECTSTRING
+ * @implements TECO_DEFINE_STATE
+ * @ingroup states
+ *
+ * Super-class for states accepting string arguments
+ * Opaquely cares about alternative-escape characters,
+ * string building commands and accumulation into a string
+ *
+ * @note Generating the input_cb could be avoided if there were a default
+ * implementation.
+ */
+#define TECO_DEFINE_STATE_EXPECTSTRING(NAME, ...) \
+	static teco_state_t * \
+	NAME##_input(teco_machine_main_t *ctx, gchar chr, GError **error) \
+	{ \
+		return teco_state_expectstring_input(ctx, chr, error); \
+	} \
+	TECO_DEFINE_STATE(NAME, \
+		.refresh_cb = (teco_state_refresh_cb_t)teco_state_expectstring_refresh, \
+		.process_edit_cmd_cb = (teco_state_process_edit_cmd_cb_t) \
+		                       teco_state_expectstring_process_edit_cmd, \
+		.fnmacro_mask = TECO_FNMACRO_MASK_STRING, \
+		.expectstring.string_building = TRUE, \
+		.expectstring.last = TRUE, \
+		.expectstring.process_cb = NULL,	/* do nothing */ \
+		.expectstring.done_cb = NAME##_done,	/* always required */ \
+		##__VA_ARGS__ \
+	)
+
+gboolean teco_state_expectfile_process(teco_machine_main_t *ctx, const teco_string_t *str,
+                                       gsize new_chars, GError **error);
+
+/* in cmdline.c */
+gboolean teco_state_expectfile_process_edit_cmd(teco_machine_main_t *ctx, teco_machine_t *parent_ctx, gchar key, GError **error);
 
-namespace Execute {
-	void step(const gchar *macro, gint stop_pos);
-	void macro(const gchar *macro, bool locals = true);
-	void file(const gchar *filename, bool locals = true);
-}
+/**
+ * @interface TECO_DEFINE_STATE_EXPECTFILE
+ * @implements TECO_DEFINE_STATE_EXPECTSTRING
+ * @ingroup states
+ */
+#define TECO_DEFINE_STATE_EXPECTFILE(NAME, ...) \
+	TECO_DEFINE_STATE_EXPECTSTRING(NAME, \
+		.process_edit_cmd_cb = (teco_state_process_edit_cmd_cb_t) \
+		                       teco_state_expectfile_process_edit_cmd, \
+		.expectstring.process_cb = teco_state_expectfile_process, \
+		##__VA_ARGS__ \
+	)
 
-} /* namespace SciTECO */
+/* in cmdline.c */
+gboolean teco_state_expectdir_process_edit_cmd(teco_machine_main_t *ctx, teco_machine_t *parent_ctx, gchar key, GError **error);
 
-#endif
+/**
+ * @interface TECO_DEFINE_STATE_EXPECTDIR
+ * @implements TECO_DEFINE_STATE_EXPECTFILE
+ * @ingroup states
+ */
+#define TECO_DEFINE_STATE_EXPECTDIR(NAME, ...) \
+	TECO_DEFINE_STATE_EXPECTFILE(NAME, \
+		.process_edit_cmd_cb = (teco_state_process_edit_cmd_cb_t) \
+		                       teco_state_expectdir_process_edit_cmd, \
+		##__VA_ARGS__ \
+	)