Tasks: * Have a look at TECO-86. * VEDIT and PMATE for MS-DOS * Update to Scinterm 5.5. Perhaps we can make use of the arbitrary RGB color feauture? * Macro to get the current word at dot (or by numeric argument) similar to double right click. Known Bugs: * Upgrade to Scintilla 5.5.7 requires charconv header which bumps the minimum GCC version to 8.1 (officially 9). This breaks OpenSUSE 15.5-15.6 builds. * Build problems on Fedora 41: See mail from Blake McBride. * @^Um{-$$} Mm= should probably return -1. * {@I/$$23=/} doesn't insert anything after $$. This would be necessary for an interactive screen editing script, that leaves you in always. This would require some refactoring. * Gtk: The control characters in tutorial.woman are still styled with the variable-width font since its rendered in STYLE_CONTROLCHAR (36), which is reset in woman.tes. Perhaps it should always be in lexer.font. * The current horizontal position (set by 4EJ via SCI_GETCOLUMN) is often wrong, i.e. pressing the up-cursor key can get you into the wrong column. Character representations obviously take always one column. This would have to be fixed in Scintilla's SCI_GETCOLUMN. * Excessively long lines slow down SciTECO too much, especially when enabling SCI_SETWRAPMODE(SC_WRAP_CHAR). In some cases, the internal redrawing blocks SciTECO forever. * @ES/SCI_CLEARALLREPRESENTATIONS// does nothing. Might be a Scintilla bug. * PDCurses/Wincon does not report button released events. We try to work around this with click detection, but it still behaves a bit oddly. See https://github.com/Bill-Gray/PDCursesMod/issues/330 Waiting for PDCurses in MSYS to be updated. * PDCurses/WinGUI: There is still some flickering, but it got better since key macros update the command line only once. Could already be fixed upstream, see: https://github.com/Bill-Gray/PDCursesMod/issues/322 * Win32: Interrupting will sometimes hang. Affects both PDCurses/WinGUI and Gtk. This no longer happens with ECbash -c 'while true; do true; done'$. However ECping -t 8.8.8.8$ still cannot be interrupted. * Win32 builds cannot work with the /mingw64 path as used in MSYS/MinGW. However the UNIX path translation appears to be a Cygwin feature. If SciTECO would do that, it might break other things (e.g. you might want to refer directory C:\mingw64 instead). * Win32/Wincon: cat ... | sciteco -i "Redirection is not supported." PR: https://github.com/Bill-Gray/PDCursesMod/pull/344 Waiting for an MSYS package releae. * PDCurses/Win32: Both Wincon and WinGUI crash when you press *any* non-ANSI character. This is fixed since PDCurses v4.5.2: https://github.com/Bill-Gray/PDCursesMod/issues/335 We're waiting for an MSYS package upgrade. It could also be worked around by using wget_wch() instead of wgetch(). * PDCurses/Win32: Crashes sometimes without any error message. * NetBSD Curses: scrolling apparently uses hardware idl capabilities resulting in graphical glitches on slow terminals. idlok(FALSE) is apparently ignored. * NetBSD: Very slow, even the redrawing. This does not happen with ncurses on NetBSD. * dlmalloc's malloc_trim() does not seem to free any resident memory after hitting the OOM limit, eg. after <%a>. Apparently an effect of HAVE_MORECORE (sbrk()) - some allocation is always left at the end. * S^ES^N<$ does not find the first line that does not begin with "<". This is because \s+ backtracks and can match shorter sequences. Perhaps ^ES should always be translated to \s++ (possessive quantifier)? * Colors are still wrong in Linux console even if TERM=linux-16color when using Solarized. Affects e.g. the message line which uses the reverse of STYLE_DEFAULT. Perhaps we must call init_color() before initializing color pairs (currently done by Scinterm). * Saving another user's file will only preserve the user when run as root. Generally, it is hard to ensure that a) save point files can be created and b) the file mode and ownership of re-created files can be preserved. We should fall back silently to an (inefficient) memory copy or temporary file strategy if this is detected. * Crashes on large files: S^EM^X$ (regexp: (.)+) Happens because the Glib regex engine is based on a recursive (backtracking) Perl regex library. I can provoke the problem only on Ubuntu 20.04. This is apparently impossible to fix as long as we do not have control over the regex engine build. We need something based on a non-backtracking Thompson's NFA with Unicode (UTF-8), see https://swtch.com/~rsc/regexp/ Basically only RE2 would check all the boxes. RE2 doesn't have a native C API, so we would also have to import the https://github.com/marcomaggi/cre2/ wrapper. re2 should be an optional dependency, so we can still build against the glib APIs. Optionally, I could build a PCRE-compatible wrapper for Rust's regex crate. * It is still possible to crash SciTECO using recursive functions, since they map to the C program's call stack. It is perhaps best to use another stack of macro strings and implement our own function calling. * SciTECO crashes can leave orphaned savepoint files lying around. Unfortunately, both the Windows and Linux ways of deleting files on close cannot be used here since that would disallow cheap savepoint restoration. * On UNIX, it's possible to open() and unlink() a file, avoiding any savepoint links. On abnormal program termination, the inodes are reclaimed automatically. Unfortunately, neither Linux, nor FreeBSD allow the re-linking based on file descriptors. On Linux this fails because the file's link count will be 0; On BSD linkat(AT_EMPTY_PATH) requires root privileges and does not work with unlinked files anyway. * Windows NT has hard links as well, but they don't work with file handles either. However, it could be possible to call CreateFile(FILE_FLAG_DELETE_ON_CLOSE) on the savepoint file, ensuring cleanup even on abnormal program termination. However, this flag cannot be cleared once we restore a save point, so we'd have to copy its contents just like in the UNIX case. There is also MoveFileEx(file, NULL, MOVEFILE_DELAY_UNTIL_REBOOT). * Windows has file system forks, but they can be orphaned just like ordinary files but are harder to locate and clean up manually. * The XTerm OSC-52 clipboard feature appears to garble Unicode characters. This is apparently an XTerm bug, probably due to 8-bit-uncleanliness. It was verified by `printf "\e]52;c;?\a"` on the command line. Few terminal emulators support OSC-52 sufficiently well. If there was an external tool to use with $SCITECO_CLIPBOARD_SET, we might get rid of the internal implementation altogether. * Glib (error) messages are not integrated with SciTECO's logging system. * Gtk on Unix: On ^Z, we do not suspend properly. The window is still shown. This would be a useful feature especially with --xembed on st. Perhaps we should try to catch SIGTSTP? This does not work with g_unix_signal_add(), though, so any workaround would be tricky. We could create a pipe via g_unix_open_pipe() which we write to using write() in a normal signal handler. We can then add a watcher using g_unix_fd_add() which will hide the main window. Unfortunately, it is also not trivial to get notified when the window has really been hidden/unrealized. Even if everything worked, it might well be annoying if you accidentally suspend your instance while not being connected to a terminal. Although this could be checked at runtime. Suspension from the command-line has therefore been disabled on Gtk for the time being. * Many Scintilla commands can easily crash the editor. A lot of the dangerous cases could be catched by parsing Scintilla.iface instead of the C header. This would also allow automatically mapping a part of the messages (property getters and setters) into the global Q-Reg space using special registers. * The lexer.sciteco.macrodef lexer property cannot currently be set/disabled. This doesn't even work with SCI_SETPROPERTY, probably since we do lexing "in the container". * Mac OS: The colors are screwed up with the terminal.tes color scheme (and with --no-profile) under Mac OS terminal emulators. This does not happen under Linux with Darling. See https://github.com/rhaberkorn/sciteco/issues/12 * File name autocompletion should take glob patterns into account. The simple reason is that if a filename really contains glob characters and you are trying to open it with EB, you might end up not being able to autocomplete it if a previous autocompletion inserted escaped glob patterns. Unfortunately, this would be very tricky to do right. * The git.blame macro is broken, at least on Git v2.45.2 and v2.25.1. Compare cat fallback.teco_ini | git blame --incremental --contents - -- fallback.teco_ini | grep -E '^[a-f0-9]{40}' (which is wrong and does not even contain all commits) with git blame --incremental --contents fallback.teco_ini -- fallback.teco_ini | grep -E '^[a-f0-9]{40}' which is correct. Without --incremental even the formatting is broken. This could well be a Git bug. * Margins, identions and the like are not configured on the unnamed buffer by fallback.teco_ini. And this is probably correct. However when saving a new unnamed file for the first time, nothing will change either and it's tricky to apply the correct settings. You have to EF and EB the file after the initial save to get everything set up correctly. Or manually run 0M#ED, but this cannot be rubbed out. We need a "save" ED hook to get this right. For instance, you could check whether the margin is set up as a sign of whether lexing has already been applied. On the other hand, this Save also can't be properly rubbed out and will not restore the original margins and styling unless we add native commands for ALL of the Scintilla messages involved. Automatically running EF EB...$ in the "save" hook could also have unwanted side effects. * Solaris/OmniOS: There are groff build errors. * session.vcs does not properly work in MSYS2 environments. * The Windows GTK version no longer works under Wine: "Failed to translate keypress (keycode: 88) for group 0 (00000409) because we could not load the layout." Also, all Windows builds have problems executing ECdir$ (under Wine!). See also https://github.com/fontforge/fontforge/issues/5031#issuecomment-1143098230 * At least the GTK version with --xembed is prone to unexpected crashes. Interestingly, while this does leave orphaned savepoint files around, it does not produce a core dump. * repl.tes terminates when encountering the first error. This requires error catching to be fixed. Also, the command-line redrawing is still broken in GNOME Terminal. * When config.status is run as a consequence of touching configure.ac, config.h will be wrong, forcing us to rerun ./configure. Features: * Should we support *.sgml files with the HTML lexer? Old Docbook documents are sometimes SGML based. * Folding support: Perhaps there should also be builtin commands [:]F+ and [:]F- * Gtk: special key macros for drag-and-drop interactions? This could be used to send the tabs of one SciTECO process into another instance and close the sender instance. * The opposite is also useful and could be done now already: A macro that takes the current buffer (or a buffer range?) and opens a new SciTECO instance with it. If successful, the given buffer is removed from the ring. This would work with the Curses version as well. Could we also send tabs into an existing SciTECO instance? Would probably require some kind of server... * opener.tes should try to center the opened line (SCI_SETFIRSTVISIBLELINE). * Rubout of SCI_GOTOPOS could also restore the vertical scrolling position (SCI_SETFIRSTVISIBLELINE). So e.g. rubbing out ZJ restores the exact view. On the other hand, this would disallow scrolling via repeated Ls, followed by their rubout. * Auto-indention could be implemented via context-sensitive immediate editing commands similar to tab-expansion. Avoids having to make LF a magic character in insertion commands. Perhaps more flexible would be adding key macro states for insert commands with default escape chars. Then this could be part of fnkeys.tes. * :$ and :$$ to pop/return only single values * Special macro assignment command. It could use the SciTECO parser for finding the end of the macro definition which is more reliable than @^Uq{}. Also this opens up new possibilities for optimizations. Macros could be special QRegs that are not backed by a Scintilla document but a normal string. This would immensely speed up macro calls. Perhaps more generically, we should add a number of alternative balanced string terminators (<> [] () {}) and assign one to parse SciTECO code. "' is not really an option here, since we want to be able to write @I"..." etc. Since ] and } can occur as stand-alone commands, we would have to use <> and/or () for SciTECO code parsing. The advantage would be that we save introducing a special assignment command and can use the same escape with @I<> for editing macro files while still getting immediate interactive syntax feedback. Plus: Once we have a parser-based terminator, there would be no more real need for command variants with disabled string building (as string building will naturally always be disabled in parser-terminator-mode). Q-Reg specs should support alternative balanced escapes as well for symmetry. * Numbers could be separate states instead of stack operating commands. The current behaviour has few benefits. If a number is a regular command that stops parsing at the first invalid character in the current radix, we could write hexadcimal constants like 16^R0BEEF^D (still clumsy...). On the other hand, the radix is runtime state and parsing must not depend on runtime state in SciTECO to ensure parseability of the language. We could use the @ modifier though, so a hexadecimal constant must always begin with @0-@9. @1e\ would be enough to insert the escape character. If the number starts with a non-decimal digit, you will have to prefix it with 0, as in @0FF. If it is followed by a command A-F, you can add an additional space or even brace the expression as in `@0FF A` or `(@0FF)A`. The same could be done with a caret: `^0FF`. Alternatively, ^P would still be available as a single-key-press hex-constant prefix. * Furthermore, this opens the possibility of floating point numbers. The "." command does not take arguments, so it could be part of the number syntax. This disallows constructs like "23." to push 23 and Dot which have to be replaced by "23,.". * In the most simple case, all TECO numbers could be floats/doubles with division/modulo having integer semantics. A separate floating point division operator could be introduced (e.g. ^/ with modulo being remapped to ^%). * SciTECO could also be "dynamically" typed by using integer and floating point types internally. The operator decides how to interpret the arguments and the return type. This is good since we'd need less operators and it's good to keep true integers around. * Having a separate number parser state will slightly simplify number syntax highlighting (see teco_lexer_getstyle()). * The first application of floats could be for U[lexer.font]. * ^H as shortcut for 16^R. ^H is backspace, but it won't be necessary to type very often. * Key macro masking flag for the beginning of the command line. May be useful e.g. for solarized's F5 key (i.e. function key macros that need to terminate the command line as they cannot be rubbed out properly). * Key macros could support special escape sequences that allow us to modify the parser state reliably. E.g. one construct could expand to the current string argument's termination character (which may not be Escape). In combination with a special key macro state effective only in the start state of the string building state machine, perhaps only in insertion commands, this could be used to make the cursor movement keys work in insertion commands by automatically terminating the command. Even more simple, the function key flag could be effective only when the termination character is $. * Support more function keys. We can define more function keys via define_key(3NCURSES). Unfortunately they are not really standardized - st and urxvt for instance have different escape sequences for Ctrl+Up or Alt+Up. It seems they can be looked up with tigetstr() and then passed to define_key(). Alternatively call use_extended_names(TRUE) and look up the key codes with key_defined(). At the very least PDCurses and Gtk could support much more keys and Alt and Ctrl modifiers. See also https://stackoverflow.com/questions/31379824/how-to-get-control-characters-for-ctrlleft-from-terminfo-in-zsh https://gist.github.com/rkumar/1237091 * Now that we have redo/reinsertion: When ^G modifier is active, normal inserts could insert between effective and rubbed out command line - without resetting it. This would add another alternative to { and } for fixing up a command line. * Instead of discarding a rubbed out command line once the user presses a non-matching key, a redo-tree could be built instead. When you rub out to a character where the tree branches, the next character typed always determines whether and which existing redo branch will be activated (ie become the new rubbed out command line). * some missing useful VideoTECO/TECO-11 commands and unnecessary incompatibilities: * EI$ would "resume" command input from the terminal in TECO-11. But how is that different from ^C/$$ in SciTECO? * In TECOC, searches the TEC_LIBRARY and ignores the file extension. This is generally very useful to have, how to force a file name verbatim? You can always add a leading `./` of course. * <^C> actually returns all the way back to the command-line in TECO-11, ie. it aborts the current command string. This cannot be fully reproduced in SciTECO, but we could return from all the stack frames up to the toplevel macro. * doesn't have string building enabled in classic TECO. Changing this would perhaps be a change too radical. Also, we would then need a string-building variant like <:I>. * FB for bounded search and FC for bounded search-replace. One advantage in comparison to ::S (which also supports arguments in SciTECO), would be the ability to bound comparisons by line with n:FB. * Searches can extend beyond the given bounds in DEC TECO as long as they start within the range. That's why ::S is equivalent to .,.:FB in DEC TECO. In SciTECO, matches must currently be entirely within the given bounds, just like in Video TECO. The DEC behavior could be achieved by always searching till the end of the buffer, but excluding all matches beyond the target range. * MSUTECO for MS-DOS had ^P for giving the number of characters until the next matching ( ) { } [ ] < > " '. However there is no single way to get brace skipping right, so this might be prime candidate for a macro. * Video TECO's updates the display when reading from long running processes. This might be useful as well, at least when detecting interactive invocation. But perhaps there should rather be a global configurable delay/refresh that could be considered by all loops etc. * Video TECO: EQq...$ and ER...$ accepts glob patterns and reads all files into the register. But we probably won't want to have that. * :EB -> Bool on Video TECO. * Video TECO: EV as an alias for EB, but create read-only buffer. * EN behaves differently on TECO-11. We could only emulate it by immediately traversing the entire directory. EN however is also heavily overloaded on SciTECO and we probably don't want to loose these features. * ET for controlling stdio. Currently you must execute stty. This would require binding the UNIX-specific tcsetattr(), as Curses APIs won't be available in Gtk. Does it also make sense to perform echoing of control codes in ^A as TECO-11 does by default? * Add flag to allow redirecting output to stderr. * ^A currently doesn't flush the output. Perhaps flushing should be controlled by the language. Perhaps via the ET flags. * Mini game where you can drive a tank around your source code. * `@]q` to pop keeping the numeric part of q intact. Perhaps `@@]q` to overwrite __only__ the numeric part, but keeping the string intact. * Return a random value between 1 and n Could automatically work with floats. * n:"x to leave on the stack (i.e. only peek). This simplifies expressions like Qa"N Qa ... * Perhaps there should be a pattern match construct for word characters according to SCI_SETWORDCHARS. Perhaps just reuse ^EC? It might be problematic to depend on a global variable, though. * The same would be useful for conditionals. Perhaps redefine <"C> to match for all word chars. * Perhaps there should be a command for converting absolute line numbers to positions (i.e. ESPOSITIONFROMLINE$$:^E). This should also include the opposite as currently supported by :^Q. An advantage would also be that :^Q could be made to return -1 for invalid line ranges as well. * _pattern$ as equivalence to Spattern$^SC. Well almost, since ^SC must not be executed in the failure case. * String building construct for subpattern references: ^Yx Perhaps support ^Y(x) as well. Since this is mainly useful for search-replace commands, we'd have to store the actual data in teco_range_t as the matched pattern is removed from the buffer before inserting the replacement string. * Visual selections via `...'. Allows them to be used recursively (eg. as a tool in macros). Returns the buffer range. * Perhaps there should be a built-in command for joining lines as has been requested by users. ^J (caret+J) would still be free. * Buffer ids should be "circular", i.e. interpreted modulo the number of buffers in the ring. This allows "%*" to wrap at the end of the buffer list. * instead of 0EB to show the list of buffers, there should perhaps be a special TAB-completion (^G mode?) that completes only buffers in the ring. It should also display the numeric buffer ids. * Gtk: Unicode IME support for asiatic languages. Have a look how it is done in Scintilla. * Progress indication in commandline cursor: Perhaps blinking or invisible? The problem is, this won't work so easily once we use a Scintilla minibuffer everywhere. Gtk already sets the "wait" cursor when busy. * Command to free Q-Register (remove from table). e.g. FQ (free Q). :FQ could free by QRegister prefix name for the common use case of Q-Register subtables and lists. * EF is currently disallowed when editing a Q-Reg unless a numeric argument is specified. On Video TECO it appears to free the current Q-Reg, which probably makes more sense than the current semantics. Should be changed along with implementing FQq. * multiline commandline * Perhaps use Scintilla view as mini buffer. This means patching Scintilla, so it does not break lines on new line characters and we can use character representations (extend SCI_SETLINEENDTYPESALLOWED?). Also, we cannot currently force ^I to be rendered with representations. cmdline.c can then directly operate on the Scintilla document. * A Scintilla view will allow syntax highlighting * These Scintilla enhancements will also improve hex mode (M#hx). * command line could highlight dead branches (e.g. gray them out) * Perhaps add a ^E register analogous to ":", but working with byte offsets. This would mainly be useful in ^E\^E. * EL command could also be used to convert all EOLs in the current buffer. * exclusive access to all opened files/buffers (locking): SciTECO will never be able to notice when a file has been changed externally. Also reversing a file write will overwrite any changes another process could have done on the file. Therefore open buffers should be locked using the flock(), fcntl() or lockf() interfaces. On Windows we can even enforce mandatory locks. A generic fallback could use lock files -- this would guard against concurrent SciTECO instances at least. * Multi-window support is probably never going to realize. Perhaps we could add a Gtk-frontend option like -X for opening all filenames with the same options in separate processes. For the Curses version, you will need a shell wrapper, or we could add an environment variable like $SCITECO_LAUNCH that can be set to a command for launching a new terminal. For multi-window SciTECO to work properly, file locking is probably a must as it is otherwise too easy to confuse SciTECO if multiple instances open the same file. * To faciliate data exchange between multiple SciTECO instances, there may be a TCP/UNIX server mode that allows read-only access to the Q-Register space of any running instance. I am not sure how to elegantly address instances, though. Especially without some kind of central name registry. * Allow setting arbitrary X11 clipboards, perhaps by hooking into the Q-Reg lookup and creating clipboard registers dynamically. * Touch restored save point files - should perhaps be configurable. This is important when working with Makefiles, as make looks at the modification times of files. * There should really be a backup mechanism. It would be relatively easy to implement portably, by using timeout() on Curses. The Gtk version can simply use a glib timer. Backup files should NOT be hidden and the timeout should be configurable (EJ?). * Error handling in SciTECO macros: Allow throwing errors with e.g. [n]^F^F where n is an error code, defaulting to 0 and description is the error string - there could be code-specific defaults. All internal error classes use defined codes. Errors could be catched using a structured try-catch-like construct or by defining an error handling label. Macros may retrieve the code and string of the last error. We will have to go through all the command implementations as well since must no longer rely on undo token execution after throwing errors. * Error codes should probably be logged to the console as well. Perhaps adopt a style similar to DEC TECO: Exxxx * Backtracking execution semantics, bringing the power of SNOBOL (and more!). This can be a variant of a structured error handling construct. This will also require managing our own function call stack. See https://github.com/rhaberkorn/sciteco/issues/26#issuecomment-2449983076 * Once we have our own function call stack, it will be possible, although not trivial, to add support for user-definable macros that accept string arguments, eg. EMq$ or @Mq/param/ This will have to switch back and forth between the macro and the invoking frame supplying the macro (similar to a coroutine). In the most simple case, a special command returns the next character produced by the callers string building machine including rubout and the user will have to implement rubout-support manually. For a lot of common cases, we could also allow a string building construct that symbolizes the string parameter supplied by the caller. This could activate interactive processing in the macro's current command, allowing you to easily "wrap" interactive commands in macros. The same construct would also be useful with non-interactive commands as a way to store the supplied parameter using EU for instance. * If we will never allow @Mq/foo/ for passing arguments to macros, we might at least make the state of the @ modifier (and the number of : modifiers) available to the macro, e.g. via a special EJ flag. This will also make it possible to detect whether the macro is a local call (whether is has its own local Q-Reg table). * Emscripten nodejs port. This may be a viable way to run SciTECO "cross"-platform, at least for evaluation... on UNIX-like systems in absence of prebuilt binaries. I already got netbsd-curses to build against Emscripten/nodejs using some UNIX shell wrapper calls, so practically all of SciTECO can be run against nodejs as a runtime. I'm not aware of any (working) alternatives, like cross-compiling for the JVM. See also https://gist.github.com/VitoVan/92ba4f2b68fec31cda803119686295e5 * Windows supports virtual terminals now. See: https://docs.microsoft.com/en-us/windows/console/classic-vs-vt Perhaps we should transition from PDCurses to something using terminal escape sequences like ncurses for Windows. * Improve the message line so it can log multiple messages. Especially important on GUI platforms and Win32 so we can get rid of the attached console window. * A dirtify-hook would be useful and could be used for spell checking. Naturally it could only be exected at the end of executing interactive commands) and it should be triggered only at the end of command lines. * CSS lexer config * Add a dedicated JSON lexer. JSON files are currently handled by the Javascript lexer (js.tes). * <'> and <|> should result in an error outside of If-statements. This is not strictly necessary and complicates the parser. <'> is currently a no-op outside of dead If-branches. | ... ' is basically equivalent to 0< ... >. Furthermore, it may not be trivial to detect such dangling Else/End-If statements as the parser state does not tell us enough. There is nothing that makes code after a successful If-condition special. Even if we always increased the nest_level, that variable does not discern Ifs and Whiles. * Possible Nightly Build improvements (and therefore releases): * Mac OS Arm64 builds either separately or via universal binary. See https://codetinkering.com/switch-homebrew-arm-x86/ Target flag: `-target arm64-apple-macos11` * A pkgsrc port could be based on the FreeBSD port and would benefit NetBSD, Mac OS, but can also be used on practically all other UNIX-like platforms. * Get into mentors.debian.net. First step to being adopted into the Debian repositories. * Get meta-rhaberkorn into https://layers.openembedded.org * AppImage via OBS. Apparently the AppImage repo is based on openSUSE 15.6. Have to fix openSUSE 15.6 builds. * Bash completions. * FreeBSD: rctl(8) theoretically allows setting up per-process actions when exceeding the memory limit. This however requires special system settings. * Auto-completions customization via external programs. This among other things could be used to integrate LSPs-driven autocompletions. * Wherever we take buffer positions (nJ; n,mD; nQ...), negative numbers could refer to the end of the buffer or Q-Register string. * Wherever we take a buffer range (e.g. n,mD), we could relax the requirement that n < m and automatically sort the indexes. In this case, right-click+drag would no longer have to sort the buffer pointers. * Support extended operators like in TECO-64: https://github.com/fpjohnston/TECO-64/blob/master/doc/oper.md However, instead of introducing a separate parser state, better use operators like ~=, ~< etc. * Alternatively, `~` might be a good choice as a shorter alternative to XOR, allowing for the idiom `Qa~Qb"=` when checking for equality that is faster than using `-` and works for floats as well. * ^& could be a NAND operator. * ^< left shift, ^> right shift * It should be possible to disable auto-completions of one-character register names, so that we can map the idention macro to M. * Add a configure-switch for LTO (--enable-lto). * There should be a string building construct for escaping search patterns. Since string building is performed immediately before search pattern translation, you cannot currently search for a Q-Register verbatim. * Tweak the Makefile lexer for Automake support. In the simplest case, just add the *.am file extension. * Add an fnkeys.tes alternative where moving cursor keys leaves you in the insert (I) command. That will behave very similar to classical editors. * Lexing via SciTECO macros? They would have to be in their own parser instance since Scintilla could ask us to restyle at any time and within string arguments, which would confuse the parser as it is. Also, parsers are not fully embeddable right now. At the same time, it would need access to the view/document it's supposed to style. Tricky, but not impossible. * A generic Scintillua lexer could be added if there was a way to retrieve SCI_PRIVATELEXERCALL(1, str). Perhaps there should be an @EPq/SCI.../ call. See "Lexer Detection" in https://orbitalquark.github.io/scintillua/manual.html * Lexilla: Could also add an APL lexer to Lexilla. APL has very simple lexical rules. * ^^ in string building expanding to a single caret is not consistent. Perhaps we should allow only ^Q^ as a way to insert a single caret? * Support for non-ANSI single byte encodings is still incomplete. You can set them with EE and they will be correctly displayed (on Gtk at least), but there is no way to insert text in these codepages, at least outside of the latin range of course. There are two ways this could be implemented: * Either all sorts of commands automatically iconv from/to the configured encoding. This would be very difficult and inefficient. * Or we iconv once to UTF-8 when loading the file and iconv back when saving. This is probably easier but means, you have to somehow specify the codepage on EB as you cannot change it later on. We could say that nEB...$ specifies the code page if the string argument is nonempty. On the other hand, iconv uses symbolic identifiers. Perhaps there should be FBfilename$codepage$ and EEcodepage$ commands or an "EE" Q-Register. Unfortunately, glib or POSIX iconv() doesn't return a list of supported codepages, that could be used for auto-completions. * Perhaps the Unicode "icons" should be configurable via TECO. In the easiest case there could simply be 2 Q-Reg namespaces: ^F... for filenames and ^E... for extensions. The numeric part could be used to store the codepoints. * It would be good if we could somehow run the testsuite with --valgrind during CI (Ubuntu). This did not work out of the box. * session.tes: Perhaps persist the search-replace registers "-" and "_". Unfortunately, there is currently no way to reliably escape their contents in the generated .teco_session. Ideally we would also persist any manually configured lexer, but we cannot easily store the current buffer's lexer. * session.tes: Could mung a .teco_ini in the VCS/session directory as well. It should also do so if opening files from the command line. Unfortunately, this requires a reliable way to check for the equality of two directories, so that we do not accidentally mung ~/.teco_ini. * session.tes: determine session.path by recursively descending from the current directory to the root, but only if session.vcs fails. This means you could create sessions in arbitrary directories but setting session.path once. * grosciteco: Does not currently support diacritics. This is because Groff decomposes characters in intermediate output. Either we have to consult devutf8/R or use wrap g_unichar_compose(). This will be important once we want to localize documentation. * Search and replace registers ("-" and "_") should probably be local to the current macro frame, so you don't have to save and restore them all the time in portable macros. This however means they should be in local Q-Registers which is a significant deviation from DEC TECO. Perhaps a few select local Q-Registers (-, _, ^X, ^R) should be accessible with the global reg syntax as well, so G- is equivalent to G.-? * `-` currently inverts the number sign in teco_state_start_minus(). So --1 == 1 and --C == C. There is probably no need for this "feature" and an error should be thrown instead on the second `-`. * The command line replacement register (Escape) should be automatically syntax highlighted. I am not sure however how that could be done without a special ED hook. * SciTECO syntax highlighting improvements: * The { } escapes should be styled as SCE_SCITECO_OPERATOR. * There should be two SCE_SCITECO_STRING[12] styles and second string arguments could be styled SCE_SCITECO_STRING2. * Alternatively, string building constructs could be styled with the alternate string style. * Erroneous constructs could be highlighted up to the previous start state. E.g. when redefining labels, the entire label would be highlighted. * NLS (Native Language Support). I could at least add German and Russian. There aren't many localizable strings in SciTECO. Should be optional. * Customizable layout: It should be possible to configure special format strings that are rendered into UI components, so people can customize the look of SciTECO. E.g. the current info line could be reproduced with a default like: ^U[^L0]SciTECO ^XI <^XT> ^XF$ As a first step, customizing the window title might be the most useful, as often, you don't need the "SciTECO - " prefix. * Until we have customizeable layouts, it may make sense to hardcode the current line+column in to the message line. * With Unicode icon support, we might want to replace a few more special symbols. For instance, the "dirty" file symbol could be  or 󰓨 or 󱠦 . * There should perhaps be an immediate editing command to repeat the last complete command. * Read-only buffers. * Lexers should uniformly avoid the color.variable style, except in languages that make a difference between names and variables, i.e. typically shell-like languages with "$var" constructs. * Install autocompletion scripts. They will have to be preprocessed, though. See also https://github.com/rhaberkorn/sciteco/wiki/Shell-auto-completions * There should perhaps we a --revision siteconfig option, to pass in a Git revision, that's printed by `sciteco --help`. But it would have to work with tarballs as well, so it has to be written into a separate header, that can be distributed. * Perhaps undefined operations like -(X) should check X for MININT. * Use Cirrus CI instead of Github Actions Allows us to do CI on FreeBSD and potentially other more obscure systems. Also, this will allow us to migrate to another hoster like Savannah or Sourcehut later on. * m,nQq to get a list of codepoints from char m to n. This would ease subscripting, e.g. to remove the last character: 0,:Qq-1Qq^Uq$ There should also be m,nA for consistency. * Sort order in Q-register autocompletions should be numeric, so ^A11 comes after ^A1. * FFI interface: There could be a command nFXlib$func$str$ to call C function func() from lib with a string argument and numeric arguments. Would allow extending SciTECO with external libraries as well. * Solaris libxcurses (/usr/include/xpg4/curses.h) is X/Open compatible and could be supported. Optimizations: * Use SC_DOCUMENTOPTION_STYLES_NONE in batch mode. However, we do not actually know that we do not continue in interactive mode. Also, some scripts like grosciteco.tes make use of styling in batch mode. So this would have to be requested explicitly per ED flag or command line option. * There should be a common error code for null-byte tests instead of TECO_ERROR_FAILED. * teco_string_append() could be optimized by ORing a padding into the realloc() size (e.g. 0xFF). However, this has not proven effective on Linux/glibc probably because it will already allocate in blocks of roughly the same size. Should be tested on Windows, though. * commonly used (special) Q-Registers could be cached, saving the q-reg table lookup. This is already done with ^R (radix) in teco_qreg_table_t. * Q-Reg arrays ([array.^E\a]) could be optimized by nesting RB trees. Each RB tree could contain an array, so the numeric lookup would have O(1). Unfortunately, worst case complexity is still O(log(n)) since you must still look up "array". * refactor search commands (create proper base class) * undo__teco_interface_ssm() could always include the check for teco_current_doc_must_undo(). * Avoid Scintilla Undo actions entirely. This will make undo token creation of deletions harder, since we must first query the text to be reinserted again, but will probably save lots of memory and one huge source of bugs. * Newer GCC and C23 features: * Perhaps teco_bool_t usage could be simplified using __attribute__((hardbool)). * Use `#elifdef` instead of `#elif defined`. * Use `[[gnu::foo]]` instead of `__attribute__((foo))`. * The TECO_FOR_EACH() hack could be simplified at least marginally using __VA_OPT__(). * A few macros like TECO_CTL_KEY() could be turned into constexpr functions. * Compound literals could be abused for default values in the Scintilla SSM functions. All the wrapper functions would have to be turned into macros, though. However, we cannot use macros in TECO_DEFINE_UNDO_CALL(). Also, we cannot nest such macros for some strange reason. * If GCC would give guarantees on the memory layout of bitfields with `struct __attribute__((packed))`, we may use bitfields instead of flag enums e.g. for teco_ed, where the position of bits must be known. * We could save around 500 bytes in every teco_machine_main_transition_t table (2kb in total) if we would wrap entries in a macro and decrement 32 (' '). * Parsing might be optimized with libc's Unicode handling functions and/or custom optimized versions. * The new Scintilla IDocumentEditable interface could be used to speed up Q-Register string accesses. That is, when it actually supports anything useful. This is a provisional feature and supported only via C++, so we would need a small wrapper to call it from C world. * Perhaps replace glib with gnulib. Could improve portability and reduce binary size on platforms where I currently have to ship the glib shared library. Also, we could control memory allocations more tightly. We could also make use of the "fnmatch" and "canonicalize" modules. On the other hand, this imports tons of sh*t into the repository and chains us to Autotools. * Does it make sense to import glib-2.0.m4? * MinGW might now support weak symbols which would be useful in interface.c to provide some default functions and avoid a little bit of preprocessor madness in the implementations. However, weak symbols might still be unsupported on other potential targets. * According to ChatGPT (sic) the glibc and jemalloc malloc_usable_size() do not change during the lifetime of an object, although this is an implementation detail. Perhaps we should support these system allocators by default if malloc_usable_size() and the original functions as __malloc() are defined. But how to even test for glibc's ptmalloc? Linux could use musl as well for instance. * Evaluate TLSF allocator. * Resolve Coverity Scan issues. If this turns out to be useful, perhaps we can automatically upload builds via CI? https://scan.coverity.com/projects/rhaberkorn-sciteco * <1;> and similar commands could be sped up if we cached the loop end PC in teco_loop_context_t. * Instead of introducing a streaming byte code compiler with all of its consequences and complications, we could translate macros into special strings where all whitespace characters, comments and labels are stripped. Every parser input byte has a second byte/word which stores the length of the symbol (including stripped bytes) in the source macro. This information is used to retrieve source macro locations in case of errors. Also, this "hint" word could cache PCs of <|> and <'>. On the downside, this scheme does not allow all kinds of optimizations. * The manual cleanup in main() could be avoided by using destructor priorities, so we can call the interface cleanup last. Unfortunately, destructor-priorities are not supported on all targets. atexit() callbacks are apparently called before the destructors. * Are there any numerical warnings we could activate? There have been quite some bugs due to numeric issues. -Wsign-compare -Wconversion Even -Werror? * Add build system support for some static code analyzer. scan-build, clang-tidy or CodeChecker? Or run scan-build in CI with a selection of fatal warnings. * The unix.Malloc check unfortunately produces countless bogus warnings due to g_auto. * Refactor control flow commands into flow-commands.c. * fnkeys.tes: Don't insert (0C) or (0R). * tecat.tes is too slow even though it doesn't even use Q-reg strings. An ideal test case to study. Documentation: * Doxygen docs could be deployed on Github pages automatically. * The ? command could be extended to support looking up help terms at dot in the current document (e.g. if called ?$). Furthermore, womanpages could contain "hypertext" links to help topics using special Troff markup and grosciteco support. * The command reference should include an overview. * Command reference is poorly formatted. tedoc.tes should be reviewed. * Write some tutorials for the Wiki, e.g. about object-oriented SciTECO ideoms etc. ;-) * What to do with `--xembed`: tabbed, st when used as the git editor, etc. * People are demanding a Youtube tutorial. * The HTML manuals lack monospaced fonts. This is partly because an.tmac removes the Courier family in nroff mode, but it still doesn't work if you undo this. * Keyboard view with the SciTECO commands on each key. Could perhaps be generated with gpic. * Wiki page about creating and maintaining lexer configurations. Also mention how to use the "lexer.test..." macros in the "edit" hook. * The man pages could include verbatim control characters when processed with grosciteco, so they will be rendered with the usual character representations when displayed within SciTECO. In fact, we could use post-processor-specific hacks to achieve reverse text on HTML and PDF output as well. However, this would require tedoc-support as well. * The cheat sheet could be built by default now and installed along with other documents. Then however we'd have to tweak it, so the CI-generated file always looks good. * We probably do not declare FOSS licenses properly. Except for the Debian package. Should all the licenses be mentioned in COPYING as well? We definitely have to ship licensing information in the binary distributions, so adding everything to COPYING would be easiest. On the other side, in binary distros we have to include licenses for all the shipped DLLs as well. * sciteco(7) should document the difference between state controlled by the SciTECO language and other state (scroll position, window size, folding) and what that all means for writing robust macros.