<feed xmlns='http://www.w3.org/2005/Atom'>
<title>sciteco/src/core-commands.c, branch tail-calls</title>
<subtitle>Scintilla-based Text Editor and COrrector</subtitle>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/'/>
<entry>
<title>functions that can tail-call must be tail-called as well</title>
<updated>2026-08-23T19:51:48+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-08-23T19:51:48+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=3f3db5422397f1bf8d5af1ade256638d89b1081c'/>
<id>3f3db5422397f1bf8d5af1ade256638d89b1081c</id>
<content type='text'>
Otherwise we could "leak" stack frames, which causes crashes on platforms with small stack sizes.
This still did not result in a reproducable speedup with GCC 16.

fixup 2e097cec409182c3cb39b74489387480bf8a278f
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Otherwise we could "leak" stack frames, which causes crashes on platforms with small stack sizes.
This still did not result in a reproducable speedup with GCC 16.

fixup 2e097cec409182c3cb39b74489387480bf8a278f
</pre>
</div>
</content>
</entry>
<entry>
<title>introduced helper functions teco_qreg_table_get_string() and teco_qreg_table_get_integer()</title>
<updated>2026-08-22T22:28:23+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-08-22T22:28:23+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=32122651bdb51006edce2be3586dd09179bed52f'/>
<id>32122651bdb51006edce2be3586dd09179bed52f</id>
<content type='text'>
* Simplifies the common task of querying an integer or string from a Q-Register table.
* Undefined Q-Regs are reported as TECO_ERROR_QREGUNDEF, so you can theoretically
  handle this case. In practice however, it requires less boilerplating to
  just call teco_qreg_table_find() manually.
* It also doesn't make sense to use these functions when getting and setting
  a register at the same time as you will want to avoid repeated lookups.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* Simplifies the common task of querying an integer or string from a Q-Register table.
* Undefined Q-Regs are reported as TECO_ERROR_QREGUNDEF, so you can theoretically
  handle this case. In practice however, it requires less boilerplating to
  just call teco_qreg_table_find() manually.
* It also doesn't make sense to use these functions when getting and setting
  a register at the same time as you will want to avoid repeated lookups.
</pre>
</div>
</content>
</entry>
<entry>
<title>optimize main state machine transitions with tail calls</title>
<updated>2026-08-22T20:27:33+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-08-22T20:27:33+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=2e097cec409182c3cb39b74489387480bf8a278f'/>
<id>2e097cec409182c3cb39b74489387480bf8a278f</id>
<content type='text'>
* On newer GCC (&gt;= 15) and Clang (&gt;= 13) versions we can tail call
  at the end of input_cb() implementations to the next state's input_cb(),
  which will be optimized to jumps (often direct jumps).
  That is, the compilers always optimized tail calls, but we can
  guarantee tail calls with the __attribute__((musttail)) statement
  attribute.
* Every `return &amp;teco_state_xxx` has to be replaced with
  `TECO_RETURN(ctx, &amp;teco_state_xxx, error)`.
* This speeds up `-O2 -flto` builds by 18%
  (e.g. tested on grosciteco for sciteco(7)).
  Part of the speed up could also be because of inlining through
  TECO_RETURN().
* The other state machines (stringbuilding and q-reg spec) cannot
  currently be optimized the same way since they get their characters
  passed in from the "main" state machine.
* All loops around callbacks could be optimized the same way.
  E.g. the undo token runner could also tail call into the next runner,
  but it's probably not important to optimize undo token executions.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* On newer GCC (&gt;= 15) and Clang (&gt;= 13) versions we can tail call
  at the end of input_cb() implementations to the next state's input_cb(),
  which will be optimized to jumps (often direct jumps).
  That is, the compilers always optimized tail calls, but we can
  guarantee tail calls with the __attribute__((musttail)) statement
  attribute.
* Every `return &amp;teco_state_xxx` has to be replaced with
  `TECO_RETURN(ctx, &amp;teco_state_xxx, error)`.
* This speeds up `-O2 -flto` builds by 18%
  (e.g. tested on grosciteco for sciteco(7)).
  Part of the speed up could also be because of inlining through
  TECO_RETURN().
* The other state machines (stringbuilding and q-reg spec) cannot
  currently be optimized the same way since they get their characters
  passed in from the "main" state machine.
* All loops around callbacks could be optimized the same way.
  E.g. the undo token runner could also tail call into the next runner,
  but it's probably not important to optimize undo token executions.
</pre>
</div>
</content>
</entry>
<entry>
<title>support lookups via language servers (LSP)</title>
<updated>2026-08-10T22:38:31+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-08-02T13:24:45+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=44f5bf677282308b959411c416da2b5b08db2062'/>
<id>44f5bf677282308b959411c416da2b5b08db2062</id>
<content type='text'>
* The main interface is the `FT` command.
  `FT` was an undocumented Video TECO command for etags/ctags lookups.
  I don't want to exactly copy its interface, though.
* `FT` allows looking up symbol names,
  definitions and references.
* ctags will be supported via ctags-lsp.
  LSP support is more powerful though and works without
  regenerating TAGS files all the time.
  The LSP will also allow you to customize auto-completions
  using SciTECO itself (i.e. by writing a language server
  in SciTECO).
* For multiple results, `FT$` can be used to cycle through
  results - this should mimic repeated `S$` or `N$`.
* `:FT...$` does a fuzzy search. IMHO it's not important
  to return a status integer instead. `FT` will only
  really be used in interactive mode.
* Document synchronization is supported via hooks from
  ring.c and via Scintilla notifications.
* Currently, the LSP communication is based on blocking
  GIOChannels. This means that a misbehaving hanging
  server could "lock up" the entire editor (FIXME).
  Only on ncurses you can always kill the subprocess by
  pressing CTRL+C.
  We need helper functions in spawn.c to read and write
  with interruptions.
* The textDocument/didChange notification transmits
  not only all edits, but all files' contents as well
  during initial synchronization.
  Therefore it is optimized to write and JSON-escape
  data without copying them around in memory and without
  destroying the buffer gap.
* Use $SCITECO_LSP to configure the language server.
  You can also use `tee` to capture stdin and stdout.
  Perhaps $SCITECO_LSP should be saved in .teco_session,
  so you can change it between projects?
* $SCITECO_LSP_ROOT is used to point to the project's
  root directory. session.tes will set it up automatically.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* The main interface is the `FT` command.
  `FT` was an undocumented Video TECO command for etags/ctags lookups.
  I don't want to exactly copy its interface, though.
* `FT` allows looking up symbol names,
  definitions and references.
* ctags will be supported via ctags-lsp.
  LSP support is more powerful though and works without
  regenerating TAGS files all the time.
  The LSP will also allow you to customize auto-completions
  using SciTECO itself (i.e. by writing a language server
  in SciTECO).
* For multiple results, `FT$` can be used to cycle through
  results - this should mimic repeated `S$` or `N$`.
* `:FT...$` does a fuzzy search. IMHO it's not important
  to return a status integer instead. `FT` will only
  really be used in interactive mode.
* Document synchronization is supported via hooks from
  ring.c and via Scintilla notifications.
* Currently, the LSP communication is based on blocking
  GIOChannels. This means that a misbehaving hanging
  server could "lock up" the entire editor (FIXME).
  Only on ncurses you can always kill the subprocess by
  pressing CTRL+C.
  We need helper functions in spawn.c to read and write
  with interruptions.
* The textDocument/didChange notification transmits
  not only all edits, but all files' contents as well
  during initial synchronization.
  Therefore it is optimized to write and JSON-escape
  data without copying them around in memory and without
  destroying the buffer gap.
* Use $SCITECO_LSP to configure the language server.
  You can also use `tee` to capture stdin and stdout.
  Perhaps $SCITECO_LSP should be saved in .teco_session,
  so you can change it between projects?
* $SCITECO_LSP_ROOT is used to point to the project's
  root directory. session.tes will set it up automatically.
</pre>
</div>
</content>
</entry>
<entry>
<title>`A` without arguments is now equivalent to `0A`</title>
<updated>2026-07-26T11:47:17+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-26T11:47:17+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=cca906658aee1dcaa38787880c54fe304f9f2e94'/>
<id>cca906658aee1dcaa38787880c54fe304f9f2e94</id>
<content type='text'>
* It was equivalent to `1A` which is almost never what you want.
  I doubt that any existing macros would be broken by this.
  But neither do I replace all `0A` in the existing code base (yet).
* `A` without arguments is a completely different "append" command in TECO-11,
  but it doesn't make sense in SciTECO and I don't see what else `A` could
  be repurposed for.
  It cannot be made an insertion command since it depends on the stack state
  which we don't track in parse-only mode.
* Added test case.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* It was equivalent to `1A` which is almost never what you want.
  I doubt that any existing macros would be broken by this.
  But neither do I replace all `0A` in the existing code base (yet).
* `A` without arguments is a completely different "append" command in TECO-11,
  but it doesn't make sense in SciTECO and I don't see what else `A` could
  be repurposed for.
  It cannot be made an insertion command since it depends on the stack state
  which we don't track in parse-only mode.
* Added test case.
</pre>
</div>
</content>
</entry>
<entry>
<title>conditionals can be terminated with F" now in addition to a single-quote (')</title>
<updated>2026-07-26T10:28:34+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-26T00:36:20+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=68364bd3637b6944fa1f63a00dae5caf375afab9'/>
<id>68364bd3637b6944fa1f63a00dae5caf375afab9</id>
<content type='text'>
This is useful when writing small macros directly on the
command-line as in `sciteco --eval`. If you use double quoted shell
strings, too many characters have to be escaped.
If you use single-quotes, though, embedding the conditional end (')
is annoying -- it would have to be written as '\''.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is useful when writing small macros directly on the
command-line as in `sciteco --eval`. If you use double quoted shell
strings, too many characters have to be escaped.
If you use single-quotes, though, embedding the conditional end (')
is annoying -- it would have to be written as '\''.
</pre>
</div>
</content>
</entry>
<entry>
<title>implemented E*q to store the last command line</title>
<updated>2026-07-26T10:28:34+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-25T23:20:15+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=a07512149088ec8bb47d16ed8fbbfdefd4745626'/>
<id>a07512149088ec8bb47d16ed8fbbfdefd4745626</id>
<content type='text'>
`*q` can only be used at the very beginning of the command line.
We cannot support it everywhere as Video TECO did since we do
not follow the operators in parse-only mode.
`E*q` is a replacement, so you can store the previous command line at
any later point.

This also adds a test case.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
`*q` can only be used at the very beginning of the command line.
We cannot support it everywhere as Video TECO did since we do
not follow the operators in parse-only mode.
`E*q` is a replacement, so you can store the previous command line at
any later point.

This also adds a test case.
</pre>
</div>
</content>
</entry>
<entry>
<title>get rid of teco_qreg_vtable_t::undo_set_integer()</title>
<updated>2026-07-25T00:37:17+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-25T00:37:17+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=a53fa85dabdfc2c47ec0c9e6626ddda2379a8e38'/>
<id>a53fa85dabdfc2c47ec0c9e6626ddda2379a8e38</id>
<content type='text'>
It was never required as a separate callback/method,
but was kept merely for consistency.
Since we now call teco_current_doc_set_dot() in
teco_qreg_dot_set_integer(), we'd have to split it up into
an "undo" method as well. I decided to get rid of the
superfluous Q-Reg method instead.

It's quite likely we could get rid of the remaining
undo_set_string(), undo_exchange_string() and undo_edit()
callbacks as well (TODO).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
It was never required as a separate callback/method,
but was kept merely for consistency.
Since we now call teco_current_doc_set_dot() in
teco_qreg_dot_set_integer(), we'd have to split it up into
an "undo" method as well. I decided to get rid of the
superfluous Q-Reg method instead.

It's quite likely we could get rid of the remaining
undo_set_string(), undo_exchange_string() and undo_edit()
callbacks as well (TODO).
</pre>
</div>
</content>
</entry>
<entry>
<title>revised and improved the Unicode glyph-to-byte conversion heuristics</title>
<updated>2026-07-24T23:55:28+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-24T23:55:28+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=685507922b0b75da5935076395a5b1ec1ef58356'/>
<id>685507922b0b75da5935076395a5b1ec1ef58356</id>
<content type='text'>
Previously almost all glyph-to-byte offset conversions consulted
Scintilla's line index and counted characters on the resulting line.
For instance a simple expression like `.+1J` would scan the same line
twice completely, which would be very slow on pathologically long lines.
Even insertions did that due to having to update the ^Y ranges.
If you repeat such an operation over all characters as in `&lt;.+1:J;&gt;`
you would have complexity O(n^2) for n = line length.
Only commands with an explicit relative nature like `C` and `A` would
use teco_view_glyph2bytes_relative() which scans beginning at dot
as long as the relative movement is less than 1024 glyphs.

Wit the new heuristics almost all glyph-to-byte and byte-to-glyph
conversions can make use of that optimization.
This requires that dot must at all times be known in glyphs as well -
the byte position is managed by Scintilla (SCI_GETCURRENTPOS).
We therefore introduced teco_current_doc_set_dot() and
teco_current_doc_get_dot() to update dot in the current buffer or
Q-Register -- it cannot be stored along with the view since
Q-Registers share a single view.
A number of auxiliary functions have been introduced for
converting relative to a known (glyphs,bytes) offset pair
and for converting absolute and relative positions with regard
to the current doc and SCI_GETCURRENTPOS position.
Of course this is error-prone since the glyph and dot positions
are interdependant - they must always be kept in sync.

With these new optimizations even pathologically long lines can
(usually) be managed even in UTF-8 documents.
It does not address slow-downs in Scintilla's line layout, yet.
grosciteco.tes for instance runs twice as fast now.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Previously almost all glyph-to-byte offset conversions consulted
Scintilla's line index and counted characters on the resulting line.
For instance a simple expression like `.+1J` would scan the same line
twice completely, which would be very slow on pathologically long lines.
Even insertions did that due to having to update the ^Y ranges.
If you repeat such an operation over all characters as in `&lt;.+1:J;&gt;`
you would have complexity O(n^2) for n = line length.
Only commands with an explicit relative nature like `C` and `A` would
use teco_view_glyph2bytes_relative() which scans beginning at dot
as long as the relative movement is less than 1024 glyphs.

Wit the new heuristics almost all glyph-to-byte and byte-to-glyph
conversions can make use of that optimization.
This requires that dot must at all times be known in glyphs as well -
the byte position is managed by Scintilla (SCI_GETCURRENTPOS).
We therefore introduced teco_current_doc_set_dot() and
teco_current_doc_get_dot() to update dot in the current buffer or
Q-Register -- it cannot be stored along with the view since
Q-Registers share a single view.
A number of auxiliary functions have been introduced for
converting relative to a known (glyphs,bytes) offset pair
and for converting absolute and relative positions with regard
to the current doc and SCI_GETCURRENTPOS position.
Of course this is error-prone since the glyph and dot positions
are interdependant - they must always be kept in sync.

With these new optimizations even pathologically long lines can
(usually) be managed even in UTF-8 documents.
It does not address slow-downs in Scintilla's line layout, yet.
grosciteco.tes for instance runs twice as fast now.
</pre>
</div>
</content>
</entry>
<entry>
<title>terex is the new regular expression engine now and replaces PCRE (GRegex)</title>
<updated>2026-06-27T22:39:51+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-06-27T22:39:51+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=4fe5bc6f3867096965270c90f2e1e5df77b8825f'/>
<id>4fe5bc6f3867096965270c90f2e1e5df77b8825f</id>
<content type='text'>
* terex is based on Henry Spencer's regular expression engine for Tcl.
  It is a hybrid NFA/DFA design which has better worst-time runtimes than
  the backtracking PCRE. Memory usage is also limited and can no longer
  increase catastrophically.
* It should no longer be possible to crash SciTECO with pathological
  searches.
* Since it reliably supports partial matches (REG_EXPECT) we can
  now enable the new backwards-search algorithm by default.
  This used to be broken because of a glib bug, which I already
  fixed. It would however take a long time until this ends up
  on the majority of glib installations.
* Regexp executions can still be quite slow if you are looking
  for a pattern at the end of a huge file, which can hang the editor,
  but this can now at least theoretically be solved by adding
  hooks into terex to poll for interruptions.
* We can now also get rid of a TECO-pattern to regexp translation
  step by directly generating terex tokens (TODO).
* Performance-wise terex appears to be slower than PCRE for simple
  forward searches even when linking everything with optimzations (FIXME).
* Having a stand-alone regular expression engine is also a huge
  step in getting rid of glib.

See also: https://git.fmsbw.de/terex/about/
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* terex is based on Henry Spencer's regular expression engine for Tcl.
  It is a hybrid NFA/DFA design which has better worst-time runtimes than
  the backtracking PCRE. Memory usage is also limited and can no longer
  increase catastrophically.
* It should no longer be possible to crash SciTECO with pathological
  searches.
* Since it reliably supports partial matches (REG_EXPECT) we can
  now enable the new backwards-search algorithm by default.
  This used to be broken because of a glib bug, which I already
  fixed. It would however take a long time until this ends up
  on the majority of glib installations.
* Regexp executions can still be quite slow if you are looking
  for a pattern at the end of a huge file, which can hang the editor,
  but this can now at least theoretically be solved by adding
  hooks into terex to poll for interruptions.
* We can now also get rid of a TECO-pattern to regexp translation
  step by directly generating terex tokens (TODO).
* Performance-wise terex appears to be slower than PCRE for simple
  forward searches even when linking everything with optimzations (FIXME).
* Having a stand-alone regular expression engine is also a huge
  step in getting rid of glib.

See also: https://git.fmsbw.de/terex/about/
</pre>
</div>
</content>
</entry>
</feed>
