<feed xmlns='http://www.w3.org/2005/Atom'>
<title>sciteco/src/core-commands.c, branch lsp</title>
<subtitle>Scintilla-based Text Editor and COrrector</subtitle>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/'/>
<entry>
<title>support lookups via language servers (LSP)</title>
<updated>2026-08-10T22:38:31+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-08-02T13:24:45+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=44f5bf677282308b959411c416da2b5b08db2062'/>
<id>44f5bf677282308b959411c416da2b5b08db2062</id>
<content type='text'>
* The main interface is the `FT` command.
  `FT` was an undocumented Video TECO command for etags/ctags lookups.
  I don't want to exactly copy its interface, though.
* `FT` allows looking up symbol names,
  definitions and references.
* ctags will be supported via ctags-lsp.
  LSP support is more powerful though and works without
  regenerating TAGS files all the time.
  The LSP will also allow you to customize auto-completions
  using SciTECO itself (i.e. by writing a language server
  in SciTECO).
* For multiple results, `FT$` can be used to cycle through
  results - this should mimic repeated `S$` or `N$`.
* `:FT...$` does a fuzzy search. IMHO it's not important
  to return a status integer instead. `FT` will only
  really be used in interactive mode.
* Document synchronization is supported via hooks from
  ring.c and via Scintilla notifications.
* Currently, the LSP communication is based on blocking
  GIOChannels. This means that a misbehaving hanging
  server could "lock up" the entire editor (FIXME).
  Only on ncurses you can always kill the subprocess by
  pressing CTRL+C.
  We need helper functions in spawn.c to read and write
  with interruptions.
* The textDocument/didChange notification transmits
  not only all edits, but all files' contents as well
  during initial synchronization.
  Therefore it is optimized to write and JSON-escape
  data without copying them around in memory and without
  destroying the buffer gap.
* Use $SCITECO_LSP to configure the language server.
  You can also use `tee` to capture stdin and stdout.
  Perhaps $SCITECO_LSP should be saved in .teco_session,
  so you can change it between projects?
* $SCITECO_LSP_ROOT is used to point to the project's
  root directory. session.tes will set it up automatically.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* The main interface is the `FT` command.
  `FT` was an undocumented Video TECO command for etags/ctags lookups.
  I don't want to exactly copy its interface, though.
* `FT` allows looking up symbol names,
  definitions and references.
* ctags will be supported via ctags-lsp.
  LSP support is more powerful though and works without
  regenerating TAGS files all the time.
  The LSP will also allow you to customize auto-completions
  using SciTECO itself (i.e. by writing a language server
  in SciTECO).
* For multiple results, `FT$` can be used to cycle through
  results - this should mimic repeated `S$` or `N$`.
* `:FT...$` does a fuzzy search. IMHO it's not important
  to return a status integer instead. `FT` will only
  really be used in interactive mode.
* Document synchronization is supported via hooks from
  ring.c and via Scintilla notifications.
* Currently, the LSP communication is based on blocking
  GIOChannels. This means that a misbehaving hanging
  server could "lock up" the entire editor (FIXME).
  Only on ncurses you can always kill the subprocess by
  pressing CTRL+C.
  We need helper functions in spawn.c to read and write
  with interruptions.
* The textDocument/didChange notification transmits
  not only all edits, but all files' contents as well
  during initial synchronization.
  Therefore it is optimized to write and JSON-escape
  data without copying them around in memory and without
  destroying the buffer gap.
* Use $SCITECO_LSP to configure the language server.
  You can also use `tee` to capture stdin and stdout.
  Perhaps $SCITECO_LSP should be saved in .teco_session,
  so you can change it between projects?
* $SCITECO_LSP_ROOT is used to point to the project's
  root directory. session.tes will set it up automatically.
</pre>
</div>
</content>
</entry>
<entry>
<title>`A` without arguments is now equivalent to `0A`</title>
<updated>2026-07-26T11:47:17+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-26T11:47:17+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=cca906658aee1dcaa38787880c54fe304f9f2e94'/>
<id>cca906658aee1dcaa38787880c54fe304f9f2e94</id>
<content type='text'>
* It was equivalent to `1A` which is almost never what you want.
  I doubt that any existing macros would be broken by this.
  But neither do I replace all `0A` in the existing code base (yet).
* `A` without arguments is a completely different "append" command in TECO-11,
  but it doesn't make sense in SciTECO and I don't see what else `A` could
  be repurposed for.
  It cannot be made an insertion command since it depends on the stack state
  which we don't track in parse-only mode.
* Added test case.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* It was equivalent to `1A` which is almost never what you want.
  I doubt that any existing macros would be broken by this.
  But neither do I replace all `0A` in the existing code base (yet).
* `A` without arguments is a completely different "append" command in TECO-11,
  but it doesn't make sense in SciTECO and I don't see what else `A` could
  be repurposed for.
  It cannot be made an insertion command since it depends on the stack state
  which we don't track in parse-only mode.
* Added test case.
</pre>
</div>
</content>
</entry>
<entry>
<title>conditionals can be terminated with F" now in addition to a single-quote (')</title>
<updated>2026-07-26T10:28:34+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-26T00:36:20+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=68364bd3637b6944fa1f63a00dae5caf375afab9'/>
<id>68364bd3637b6944fa1f63a00dae5caf375afab9</id>
<content type='text'>
This is useful when writing small macros directly on the
command-line as in `sciteco --eval`. If you use double quoted shell
strings, too many characters have to be escaped.
If you use single-quotes, though, embedding the conditional end (')
is annoying -- it would have to be written as '\''.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is useful when writing small macros directly on the
command-line as in `sciteco --eval`. If you use double quoted shell
strings, too many characters have to be escaped.
If you use single-quotes, though, embedding the conditional end (')
is annoying -- it would have to be written as '\''.
</pre>
</div>
</content>
</entry>
<entry>
<title>implemented E*q to store the last command line</title>
<updated>2026-07-26T10:28:34+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-25T23:20:15+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=a07512149088ec8bb47d16ed8fbbfdefd4745626'/>
<id>a07512149088ec8bb47d16ed8fbbfdefd4745626</id>
<content type='text'>
`*q` can only be used at the very beginning of the command line.
We cannot support it everywhere as Video TECO did since we do
not follow the operators in parse-only mode.
`E*q` is a replacement, so you can store the previous command line at
any later point.

This also adds a test case.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
`*q` can only be used at the very beginning of the command line.
We cannot support it everywhere as Video TECO did since we do
not follow the operators in parse-only mode.
`E*q` is a replacement, so you can store the previous command line at
any later point.

This also adds a test case.
</pre>
</div>
</content>
</entry>
<entry>
<title>get rid of teco_qreg_vtable_t::undo_set_integer()</title>
<updated>2026-07-25T00:37:17+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-25T00:37:17+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=a53fa85dabdfc2c47ec0c9e6626ddda2379a8e38'/>
<id>a53fa85dabdfc2c47ec0c9e6626ddda2379a8e38</id>
<content type='text'>
It was never required as a separate callback/method,
but was kept merely for consistency.
Since we now call teco_current_doc_set_dot() in
teco_qreg_dot_set_integer(), we'd have to split it up into
an "undo" method as well. I decided to get rid of the
superfluous Q-Reg method instead.

It's quite likely we could get rid of the remaining
undo_set_string(), undo_exchange_string() and undo_edit()
callbacks as well (TODO).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
It was never required as a separate callback/method,
but was kept merely for consistency.
Since we now call teco_current_doc_set_dot() in
teco_qreg_dot_set_integer(), we'd have to split it up into
an "undo" method as well. I decided to get rid of the
superfluous Q-Reg method instead.

It's quite likely we could get rid of the remaining
undo_set_string(), undo_exchange_string() and undo_edit()
callbacks as well (TODO).
</pre>
</div>
</content>
</entry>
<entry>
<title>revised and improved the Unicode glyph-to-byte conversion heuristics</title>
<updated>2026-07-24T23:55:28+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-07-24T23:55:28+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=685507922b0b75da5935076395a5b1ec1ef58356'/>
<id>685507922b0b75da5935076395a5b1ec1ef58356</id>
<content type='text'>
Previously almost all glyph-to-byte offset conversions consulted
Scintilla's line index and counted characters on the resulting line.
For instance a simple expression like `.+1J` would scan the same line
twice completely, which would be very slow on pathologically long lines.
Even insertions did that due to having to update the ^Y ranges.
If you repeat such an operation over all characters as in `&lt;.+1:J;&gt;`
you would have complexity O(n^2) for n = line length.
Only commands with an explicit relative nature like `C` and `A` would
use teco_view_glyph2bytes_relative() which scans beginning at dot
as long as the relative movement is less than 1024 glyphs.

Wit the new heuristics almost all glyph-to-byte and byte-to-glyph
conversions can make use of that optimization.
This requires that dot must at all times be known in glyphs as well -
the byte position is managed by Scintilla (SCI_GETCURRENTPOS).
We therefore introduced teco_current_doc_set_dot() and
teco_current_doc_get_dot() to update dot in the current buffer or
Q-Register -- it cannot be stored along with the view since
Q-Registers share a single view.
A number of auxiliary functions have been introduced for
converting relative to a known (glyphs,bytes) offset pair
and for converting absolute and relative positions with regard
to the current doc and SCI_GETCURRENTPOS position.
Of course this is error-prone since the glyph and dot positions
are interdependant - they must always be kept in sync.

With these new optimizations even pathologically long lines can
(usually) be managed even in UTF-8 documents.
It does not address slow-downs in Scintilla's line layout, yet.
grosciteco.tes for instance runs twice as fast now.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Previously almost all glyph-to-byte offset conversions consulted
Scintilla's line index and counted characters on the resulting line.
For instance a simple expression like `.+1J` would scan the same line
twice completely, which would be very slow on pathologically long lines.
Even insertions did that due to having to update the ^Y ranges.
If you repeat such an operation over all characters as in `&lt;.+1:J;&gt;`
you would have complexity O(n^2) for n = line length.
Only commands with an explicit relative nature like `C` and `A` would
use teco_view_glyph2bytes_relative() which scans beginning at dot
as long as the relative movement is less than 1024 glyphs.

Wit the new heuristics almost all glyph-to-byte and byte-to-glyph
conversions can make use of that optimization.
This requires that dot must at all times be known in glyphs as well -
the byte position is managed by Scintilla (SCI_GETCURRENTPOS).
We therefore introduced teco_current_doc_set_dot() and
teco_current_doc_get_dot() to update dot in the current buffer or
Q-Register -- it cannot be stored along with the view since
Q-Registers share a single view.
A number of auxiliary functions have been introduced for
converting relative to a known (glyphs,bytes) offset pair
and for converting absolute and relative positions with regard
to the current doc and SCI_GETCURRENTPOS position.
Of course this is error-prone since the glyph and dot positions
are interdependant - they must always be kept in sync.

With these new optimizations even pathologically long lines can
(usually) be managed even in UTF-8 documents.
It does not address slow-downs in Scintilla's line layout, yet.
grosciteco.tes for instance runs twice as fast now.
</pre>
</div>
</content>
</entry>
<entry>
<title>terex is the new regular expression engine now and replaces PCRE (GRegex)</title>
<updated>2026-06-27T22:39:51+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-06-27T22:39:51+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=4fe5bc6f3867096965270c90f2e1e5df77b8825f'/>
<id>4fe5bc6f3867096965270c90f2e1e5df77b8825f</id>
<content type='text'>
* terex is based on Henry Spencer's regular expression engine for Tcl.
  It is a hybrid NFA/DFA design which has better worst-time runtimes than
  the backtracking PCRE. Memory usage is also limited and can no longer
  increase catastrophically.
* It should no longer be possible to crash SciTECO with pathological
  searches.
* Since it reliably supports partial matches (REG_EXPECT) we can
  now enable the new backwards-search algorithm by default.
  This used to be broken because of a glib bug, which I already
  fixed. It would however take a long time until this ends up
  on the majority of glib installations.
* Regexp executions can still be quite slow if you are looking
  for a pattern at the end of a huge file, which can hang the editor,
  but this can now at least theoretically be solved by adding
  hooks into terex to poll for interruptions.
* We can now also get rid of a TECO-pattern to regexp translation
  step by directly generating terex tokens (TODO).
* Performance-wise terex appears to be slower than PCRE for simple
  forward searches even when linking everything with optimzations (FIXME).
* Having a stand-alone regular expression engine is also a huge
  step in getting rid of glib.

See also: https://git.fmsbw.de/terex/about/
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* terex is based on Henry Spencer's regular expression engine for Tcl.
  It is a hybrid NFA/DFA design which has better worst-time runtimes than
  the backtracking PCRE. Memory usage is also limited and can no longer
  increase catastrophically.
* It should no longer be possible to crash SciTECO with pathological
  searches.
* Since it reliably supports partial matches (REG_EXPECT) we can
  now enable the new backwards-search algorithm by default.
  This used to be broken because of a glib bug, which I already
  fixed. It would however take a long time until this ends up
  on the majority of glib installations.
* Regexp executions can still be quite slow if you are looking
  for a pattern at the end of a huge file, which can hang the editor,
  but this can now at least theoretically be solved by adding
  hooks into terex to poll for interruptions.
* We can now also get rid of a TECO-pattern to regexp translation
  step by directly generating terex tokens (TODO).
* Performance-wise terex appears to be slower than PCRE for simple
  forward searches even when linking everything with optimzations (FIXME).
* Having a stand-alone regular expression engine is also a huge
  step in getting rid of glib.

See also: https://git.fmsbw.de/terex/about/
</pre>
</div>
</content>
</entry>
<entry>
<title>implemented but disabled block-wise backwards search algorithm</title>
<updated>2026-05-31T19:19:24+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-05-31T19:19:24+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=aa7b0bb1445feeefafdcf47fd639b10500b45c03'/>
<id>aa7b0bb1445feeefafdcf47fd639b10500b45c03</id>
<content type='text'>
* The block-wise search algorithm allows for efficient backwards searches
  on large files.
* On the downside the results are not entirely symmetric to forward searches.
  It therefore makes sense to still support the old correct but possibly slow
  algorithm.
  Since the old algorithm is just a special case of the new one (with a single
  block stretching the entire search range), you can configure the block size
  using `8EJ`.
* Unfortunately, the new block-wise algorithm won't work due to a bug in GRegex
  (only in the glib wrapper code).
  It is therefore disabled for the time being by default and will probably
  only be enabled once we switch to a new regexp engine.
  See https://gitlab.gnome.org/GNOME/glib/-/merge_requests/5199
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* The block-wise search algorithm allows for efficient backwards searches
  on large files.
* On the downside the results are not entirely symmetric to forward searches.
  It therefore makes sense to still support the old correct but possibly slow
  algorithm.
  Since the old algorithm is just a special case of the new one (with a single
  block stretching the entire search range), you can configure the block size
  using `8EJ`.
* Unfortunately, the new block-wise algorithm won't work due to a bug in GRegex
  (only in the glib wrapper code).
  It is therefore disabled for the time being by default and will probably
  only be enabled once we switch to a new regexp engine.
  See https://gitlab.gnome.org/GNOME/glib/-/merge_requests/5199
</pre>
</div>
</content>
</entry>
<entry>
<title>support "default colors"</title>
<updated>2026-05-09T22:15:52+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-05-09T22:15:52+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=58a0f8def5742248bf235c7c9cd9d3efcb334751'/>
<id>58a0f8def5742248bf235c7c9cd9d3efcb334751</id>
<content type='text'>
* The terminal's default foreground and background colors
  are now used by default (`sciteco --no-profile`), so SciTECO
  integrates naturally into all terminal color schemes, even
  dark-on-bright ones.
* The default Scintilla colors use only 0x000000 (COLOR_BLACK) and 0xC0C0C0 (COLOR_WHITE)
  now.
* You can use `7EJ` to configure the default colors in color
  schemes or your profile.
  All existing color schemes had to disable default colors
  (`-1,-1,7EJ`) since they wouldn't look well otherwise.
* You may add `-1,7EJ` to ~/.teco_ini when using a terminal emulator
  with a washed-out palettized COLOR_BLACK.
  We cannot detect the terminal's default colors automatically.
* Scinterm updated to v6.0.
  We require a not-yet-upstreamed patch:
  https://github.com/orbitalquark/scinterm/pull/40
* In fact, we might decide not to support default colors at all in Scinterm,
  so this feature should be considered experimental.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* The terminal's default foreground and background colors
  are now used by default (`sciteco --no-profile`), so SciTECO
  integrates naturally into all terminal color schemes, even
  dark-on-bright ones.
* The default Scintilla colors use only 0x000000 (COLOR_BLACK) and 0xC0C0C0 (COLOR_WHITE)
  now.
* You can use `7EJ` to configure the default colors in color
  schemes or your profile.
  All existing color schemes had to disable default colors
  (`-1,-1,7EJ`) since they wouldn't look well otherwise.
* You may add `-1,7EJ` to ~/.teco_ini when using a terminal emulator
  with a washed-out palettized COLOR_BLACK.
  We cannot detect the terminal's default colors automatically.
* Scinterm updated to v6.0.
  We require a not-yet-upstreamed patch:
  https://github.com/orbitalquark/scinterm/pull/40
* In fact, we might decide not to support default colors at all in Scinterm,
  so this feature should be considered experimental.
</pre>
</div>
</content>
</entry>
<entry>
<title>Curses: support arbitrary RGB colors</title>
<updated>2026-04-26T00:00:18+00:00</updated>
<author>
<name>Robin Haberkorn</name>
<email>rhaberkorn@fmsbw.de</email>
</author>
<published>2026-04-26T00:00:18+00:00</published>
<link rel='alternate' type='text/html' href='https://git.fmsbw.de/sciteco/commit/?id=282fa65c0d547c89eac6cd4c7e91496a280bece7'/>
<id>282fa65c0d547c89eac6cd4c7e91496a280bece7</id>
<content type='text'>
* Instead of supporting only 16 predefined RGB placeholder
  values in Scintilla messages and styles, you can now use
  arbitrary RGB values and colors are allocated via the terminal
  on the fly.
  You no longer need to call 3EJ to change the default color
  palette.
* The placeholder RGB values are still available.
  Since you will usually want exact RGB values when using
  anything outside of the range of 16 default colors
  and the RGB placeholders will not always exactly correspond
  to their RGB value, you can now call `0,3EJ` to ignore
  the default palette and allocate all colors dynamically.
* Allows for more than 16 colors on the screen simultaneously.
  Also simplifies the solarized.tes color scheme.
  Since both Scinterm and SciTECO try not to touch the 16
  default colors, you also no longer have to deal with
  restoring the palette after program termination
  (which was never reliable anyway).
* Color schemes with non-default colors (solarized.tes)
  may now be broken on TERM=linux-16color (Linux VT)
  since Scinterm will get only 8 colors, but solarized.tes
  needs 16.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* Instead of supporting only 16 predefined RGB placeholder
  values in Scintilla messages and styles, you can now use
  arbitrary RGB values and colors are allocated via the terminal
  on the fly.
  You no longer need to call 3EJ to change the default color
  palette.
* The placeholder RGB values are still available.
  Since you will usually want exact RGB values when using
  anything outside of the range of 16 default colors
  and the RGB placeholders will not always exactly correspond
  to their RGB value, you can now call `0,3EJ` to ignore
  the default palette and allocate all colors dynamically.
* Allows for more than 16 colors on the screen simultaneously.
  Also simplifies the solarized.tes color scheme.
  Since both Scinterm and SciTECO try not to touch the 16
  default colors, you also no longer have to deal with
  restoring the palette after program termination
  (which was never reliable anyway).
* Color schemes with non-default colors (solarized.tes)
  may now be broken on TERM=linux-16color (Linux VT)
  since Scinterm will get only 8 colors, but solarized.tes
  needs 16.
</pre>
</div>
</content>
</entry>
</feed>
