| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
^R now (refs #17)
* This way the search mode and radix are local to the current macro frame,
unless the macro was invoked with :Mq.
If colon-modified, you can reproduce the same effect by calling
[.^X 0^X ... ].^X
* The radix register is cached in the Q-Reg table as an optimization.
This could be done with the other "special" registers as well, but at the
cost of larger stack frames.
* In order to allow constructs like [.^X typed with upcarets,
the Q-Register specification syntax has been extended:
^c is the corresponding control code instead of the register "^".
|
|
The following rules apply:
* All SciTECO macros __must__ be in valid UTF-8, regardless of the
the register's configured encoding.
This is checked against before execution, so we can use glib's non-validating
UTF-8 API afterwards.
* Things will inevitably get slower as we have to validate all macros first
and convert to gunichar for each and every character passed into the parser.
As an optimization, it may make sense to have our own inlineable version of
g_utf8_get_char() (TODO).
Also, Unicode glyphs in syntactically significant positions may be case-folded -
just like ASCII chars were. This is is of course slower than case folding
ASCII. The impact of this should be measured and perhaps we should restrict
case folding to a-z via teco_ascii_toupper().
* The language itself does not use any non-ANSI characters, so you don't have to
use UTF-8 characters.
* Wherever the parser expects a single character, it will now accept an arbitrary
Unicode/UTF-8 glyph as well.
In other words, you can call macros like M§ instead of having to write M[§].
You can also get the codepoint of any Unicode character with ^^x.
Pressing an Unicode character in the start state or in Ex and Fx will now
give a sane error message.
* When pressing a key which produces a multi-byte UTF-8 sequence, the character
gets translated back and forth multiple times:
1. It's converted to an UTF-8 string, either buffered or by IME methods (Gtk).
On Curses we could directly get a wide char using wget_wch(), but it's
not currently used, so we don't depend on widechar curses.
2. Parsed into gunichar for passing into the edit command callbacks.
This also validates the codepoint - everything later on can assume valid
codepoints and valid UTF-8 strings.
3. Once the edit command handling decides to insert the key into the command line,
it is serialized back into an UTF-8 string as the command line macro has
to be in UTF-8 (like all other macros).
4. The parser reads back gunichars without validation for passing into
the parser callbacks.
* Flickering in the Curses UI and Pango warnings in Gtk, due to incompletely
inserted and displayed UTF-8 sequences, are now fixed.
|
|
|
|
|
|
|
|
This is a total conversion of SciTECO to plain C (GNU C11).
The chance was taken to improve a lot of internal datastructures,
fix fundamental bugs and lay the foundations of future features.
The GTK user interface is now in an useable state!
All changes have been squashed together.
The language itself has almost not changed at all, except for:
* Detection of string terminators (usually Escape) now takes
the string building characters into account.
A string is only terminated outside of string building characters.
In other words, you can now for instance write
I^EQ[Hello$world]$
This removes one of the last bits of shellisms which is out of
place in SciTECO where no tokenization/lexing is performed.
Consequently, the current termination character can also be
escaped using ^Q/^R.
This is used by auto completions to make sure that strings
are inserted verbatim and without unwanted sideeffects.
* All strings can now safely contain null-characters
(see also: 8-bit cleanliness).
The null-character itself (^@) is not (yet) a valid SciTECO
command, though.
An incomplete list of changes:
* We got rid of the BSD headers for RB trees and lists/queues.
The problem with them was that they used a form of metaprogramming
only to gain a bit of type safety. It also resulted in less
readble code. This was a C++ desease.
The new code avoids metaprogramming only to gain type safety.
The BSD tree.h has been replaced by rb3ptr by Jens Stimpfle
(https://github.com/jstimpfle/rb3ptr).
This implementation is also more memory efficient than BSD's.
The BSD list.h and queue.h has been replaced with a custom
src/list.h.
* Fixed crashes, performance issues and compatibility issues with
the Gtk 3 User Interface.
It is now more or less ready for general use.
The GDK lock is no longer used to avoid using deprecated functions.
On the downside, the new implementation (driving the Gtk event loop
stepwise) is even slower than the old one.
A few glitches remain (see TODO), but it is hoped that they will
be resolved by the Scintilla update which will be performed soon.
* A lot of program units have been split up, so they are shorter
and easier to maintain: core-commands.c, qreg-commands.c,
goto-commands.c, file-utils.h.
* Parser states are simply structs of callbacks now.
They still use a kind of polymorphy using a preprocessor trick.
TECO_DEFINE_STATE() takes an initializer list that will be
merged with the default list of field initializers.
To "subclass" states, you can simply define new macros that add
initializers to existing macros.
* Parsers no longer have a "transitions" table but the input_cb()
may use switch-case statements.
There are also teco_machine_main_transition_t now which can
be used to implement simple transitions. Additionally, you
can specify functions to execute during transitions.
This largely avoids long switch-case-statements.
* Parsers are embeddable/reusable now, at least in parse-only mode.
This does not currently bring any advantages but may later
be used to write a Scintilla lexer for TECO syntax highlighting.
Once parsers are fully embeddable, it will also be possible
to run TECO macros in a kind of coroutine which would allow
them to process string arguments in real time.
* undo.[ch] still uses metaprogramming extensively but via
the C preprocessor of course. On the downside, most undo
token generators must be initiated explicitly (theoretically
we could have used embedded functions / trampolines to
instantiate automatically but this has turned out to be
dangereous).
There is a TECO_DEFINE_UNDO_CALL() to generate closures for
arbitrary functions now (ie. to call an arbitrary function
at undo-time). This simplified a lot of code and is much
shorter than manually pushing undo tokens in many cases.
* Instead of the ridiculous C++ Curiously Recurring Template
Pattern to achieve static polymorphy for user interface
implementations, we now simply declare all functions to
implement in interface.h and link in the implementations.
This is possible since we no longer hace to define
interface subclasses (all state is static variables in
the interface's *.c files).
* Headers are now significantly shorter than in C++ since
we can often hide more of our "class" implementations.
* Memory counting is based on dlmalloc for most platforms now.
Unfortunately, there is no malloc implementation that
provides an efficient constant-time memory counter that
is guaranteed to decrease when freeing memory.
But since we use a defined malloc implementation now,
malloc_usable_size() can be used safely for tracking memory use.
malloc() replacement is very tricky on Windows, so we
use a poll thread on Windows. This can also be enabled
on other supported platforms using --disable-malloc-replacement.
All in all, I'm still not pleased with the state of memory
limiting. It is a mess.
* Error handling uses GError now. This has the advantage that
the GError codes can be reused once we support error catching
in the SciTECO language.
* Added a few more test suite cases.
* Haiku is no longer supported as builds are instable and
I did not manage to debug them - quite possibly Haiku bugs
were responsible.
* Glib v2.44 or later are now required.
The GTK UI requires Gtk+ v3.12 or later now.
The GtkFlowBox fallback and sciteco-wrapper workaround are
no longer required.
* We now extensively use the GCC/Clang-specific g_auto
feature (automatic deallocations when leaving the current
code block).
* Updated copyright to 2021.
SciTECO has been in continuous development, even though there
have been no commits since 2018.
* Since these changes are so significant, the target release has
been set to v2.0.
It is planned that beginning with v3.0, the language will be
kept stable.
|
|
|
|
* we were basing the glib allocators on throwing std::bad_alloc just like
the C++ operators. However, this always was unsafe since we were throwing
exceptions across plain-C frames (Glib).
Also, the memory vtable has been deprecated in Glib, resulting in
ugly warnings.
* Instead, we now let the C++ new/delete operators work like Glib
by basing them on g_malloc/g_slice.
This means they will assert and the application will terminate
abnormally in case of OOM. OOMs cannot be handled properly anyway, so it is
more important to have a good memory limiting mechanism.
* Memory limiting has been completely revised.
Instead of approximating undo stack sizes using virtual methods
(which is unprecise and comes with a performance penalty),
we now use a common base class SciTECO::Object to count the memory
required by all objects allocated within SciTECO.
This is less precise than using global replacement new/deletes
which would allow us to control allocations in all C++ code including
Scintilla, but they are only supported as of C++14 (GCC 5) and adding compile-time
checks would be cumbersome.
In any case, we're missing Glib allocations (esp. strings).
* As a platform-specific extension, on Linux/glibc we use mallinfo()
to count the exact memory usage of the process.
On Windows, we use GetProcessMemoryInfo() -- the latter implementation
is currently UNTESTED.
* We use g_malloc() for new/delete operators when there is
malloc_trim() since g_slice does not free heap chunks properly
(probably does its own mmap()ing), rendering malloc_trim() ineffective.
We've also benchmarked g_slice on Linux/glib (malloc_trim() shouldn't
be available elsewhere) and found that it brings no significant
performance benefit.
On all other platforms, we use g_slice since it is assumed
that it at least does not hurt.
The new g_slice based allocators should be tested on MSVCRT
since I assume that they bring a significant performance benefit
on Windows.
* Memory limiting does now work in batch mode as well and is still
enabled by default.
* The old UndoTokenWithSize CRTP hack could be removed.
UndoStack operations should be a bit faster now.
But on the other hand, there will be an overhead due to repeated
memory limit checking on every processed character.
|
|
optimizationa and additional checks
* undo tokens emitted by the expression stack no longer waste
memory by pointing to the stack implementation.
This uses some ugly C++ constant template arguments but
saves 4 or 8 byte per undo token depending on the architecture.
* Round braces are counted now and the return command $$ will
use this information to discard all non-relevant brace levels.
* It is an error to close a brace when none have been opened.
* The bracing rules are still very liberal, allowing you to
close braces in macros belonging to a higher call frame
or leave them open at the end of a macro.
While this is technically possible, it is perhaps a good
idea to stricten these rules in some future release.
* Loops no longer (ab)use the expression stack to store
program counters and loop counters.
This removes flow control from the responsibility of the
expression stack which is much safer now since we can control
where we jump to.
This also eased implemented proper semantics for $$.
* It is an error to leave loops open at the end of a macro
or trying to close a loop opened in the caller of the macro.
Similarily it is only possible to close a loop from the
current invocation frame.
This means it is now impossible to accidentally jump to invalid
PCs.
* Even though loop context stacks could be attached directly to
the macro invocation frame, this would be inefficient.
Instead there's a loop frame pointer now that is part of the
invocation frame. All frames will reuse the same stack structure.
* Loops are automatically discarded when returning using $$.
* Special aggregating forms of the loop start (":<") and loop
end (":>") commands are possible now and have been implemented.
This improves SciTECO's capability as a stack-oriented language.
It is no longer necessary to write recursive macros to generate
stack values of arbitrary length dynamically or to process them.
* All expression and loop stacks are still fixed-size.
It may be a good idea to implement dynamic resizing (TODO).
* Added some G_UNLIKELYs to Execute::macro(). Should improve
the branch prediction of modern CPUs.
* Local Q-Register tables are allocated on the stack now instead
of on the heap (the bulk of a table is stored on the heap anyway).
Should improve performance of macro invocations.
* Document that "F<" will jump to the beginning of the macro
if there is no loop.
This is not in standard TECO, but I consider it a useful feature.
|
|
batch mode
* by using variadic templates, UndoStack::push() is now responsible
for allocating undo tokens. This is avoided in batch mode.
* The old UndoStack::push(UndoToken *) method has been made private
to avoid confusion around UndoStack's API.
The old UndoStack::push() no longer needs to handle !undo.enabled,
but at least asserts on it.
* C++11 support is now required, so variadic templates can be used.
This could have also been done using manual undo.enabled checks;
or using multiple versions of the template with different numbers
of template arguments.
The latter could be done if we one day have to support a non-C++11
compiler.
However since we're depending on GCC 4.4, variadic template use should
be OK.
Clang supports it since v2.9.
* Sometimes, undo token pushing passed ownership of some memory
to the undo token. The old behaviour was relied on to reclaim the
memory even in batch mode -- the undo token was always deleted.
To avoid leaks or repeated manual undo.enabled checking,
another method UndoStack::push_own() had to be
introduced that makes sure that an undo token is always created.
In batch mode (!undo.enabled), this will however create the object
on the stack which is much cheaper than using `new`.
* Having to know which kind of undo token is to be pushed (taking ownership
or not) is inconvenient. It may be better to add static methods to
the UndoToken classes that can take care of reclaiming memory.
* Benchmarking certain SciTECO scripts have shown 50% (!!!) speed increases
at the highest possible optimization level (-O3 -mtune=native -march=native).
|
|
|
|
* use small values for low precedence
|
|
* SciTECO now has the same operator precedence table as C.
* It is numerically important whether different operators
have the same precedence. E.g. "5*2/4" used to be evaluated
by SciTECO as "5*(2/4)" since division had a higher precedence
than multiplication. Within in real (!) numbers this would
be the expected evaluation order.
Users of other programming languages however would expect
the expression to be evaluated as "(5*2)/4" which makes
a numerical difference when working with integers.
* Operator precedence has been implemented by encoding it
into the enumeration values used to represent different
operators.
Calculating the precedence of a given operator can then
be done very efficiently and elegantly (in our case using
a plain right shift operation).
* documentation updated. We use a precedence table now.
|
|
* necessary since in SciTECO every operator has a different
precedence. E.g. successive additions/subtractions cannot
be evaluated from left to right (by their associativity).
Perhaps this should be changed.
* subtraction must have a higher precedence than addition,
since (a+b)-c == a+(b-c)
* division must have a higher precedence than multiplication
since (a*b)/c == a*(b/c).
This is not quite true for integer arithmetics.
* this fixes expressions like 5-1+1 which were counterintuitively
evaluated like 5-(1+1)
|
|
expression stack
this was probably a regression from d94b18819ad4ee3237c46ad43a962d0121f0c3fe
and should not be in v0.5.
The return value of Expressions::find_op() must always be checked since
it might not find the operator, returning 0 (it used to be 0).
A zero index pointed to uninitialized memory - in the worst case it
pointed to invalid memory resulting in segfaults.
Too large indices were also not handled.
This was probably responsible for recent PPA build issues.
Valgrind/memcheck reports this error but I misread it as a bogus warning.
I took the opportunity to clean up the ValueStack implementation and
made it more robust by adding a few assertions.
ValueStacks now grow from large to small addresses (like stack data
structures usually do).
This means, there is no need to work with negative indices into the
stack pointer.
To reduce the potential for invalid stack accesses, stack indices are
now unsigned and have origin 0. Previously, all indices < 1 were
faulty but weren't checked.
Also, I added some minor optimizations.
|
|
|
|
* acts as a safe-guard against uninterrupted infinite loops
or other operations that are costly to undo in interactive mode.
If we're out of memory, it is usually too late to react properly.
This implementation tries to avoid OOMs due to SciTECO behaviour.
We cannot fully exclude the chance of an OOM error.
* The undo stack size is only approximated using the
UndoToken::get_size() method.
Other ways to measure the exact amount of allocated heap
(including size fields in every heap object or using sbrk(0) and
similar) are either costly in terms of memory or platform-specific.
This implementation does not need any additional memory per heap
object or undo token but exploits the fact that undo tokens
are virtual already. The size of an undo token is determined
at compile time.
* Default memory limit of 500mb should be OK for most people.
* The current limit can be queried with "2EJ" and set with <x>,2EJ.
This also works interactively (a bit tricky!)
* Limiting can be disabled. In this case, undo token processing
is a bit faster.
* closes #3
|
|
|
|
also changed precedence of + operator (higher than minus).
the effects of this should be minimal
|
|
* the GError expection has been renamed to GlibError, to avoid
nameclashes when working from the SciTECO namespace
|
|
normally, since SciTECO is not a library, this is not strictly
necessary since every library should use proper name prefixes
or namespaces for all global declarations to avoid name clashes.
However
* you cannot always rely on that
* Scintilla does violate the practice of using prefixes or namespaces.
The public APIs are OK, but it does define global functions/methods,
e.g. for "Document" that clashed with SciTECO's "TECODocument" class at
link-time.
Scintilla can put its definitions in a namespace, but this feature
cannot be easily enabled without patching Scintilla.
* a "SciTECO" namespace will be necessary if "SciTECO" is ever to be
turned into a library. Even if this library will have only a C-linkage
API, it must ensure it doesn't clutter the global namespace.
So the old "TECODocument" class was renamed back to "Document"
(SciTECO::Document).
|
|
|
|
* ValueStack destruction might have resulted in Segfaults at shutdown
|
|
* new Expressions::format()
* may be used format numbers as part of arrays (Q-Register names)
|
|
* storage size should always be 64 (gint64) to aid macro portability
* however, for performance reasons users compiling from source might
explicitly compile with 32 bit integers
|
|
|
|
|
|
|
