| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
* there is no /bin/true
* We cannot crash SciTECO using unlimited recursion, at least not
before the memory limit is reached.
Therefore, this test case is expected to succeed on Mac OS.
|
|
don't build an empty libdlmalloc
* on some platforms (eg. Darwin/mac OS) we cannot apparently build empty
convenience libraries
* instead, we use conditional subdirectories and a conditional library dependency
|
|
* I could not get malloc replacement via dlmalloc to work.
This does not work like on Linux by overwriting weak malloc() functions.
It should theoretically be possible to overwrite the default malloc zone
but I could not properly debug this since I can only build for Mac OS
via CI.
* memory polling seems to work though - test suite runs through
and it includes memory limiting test cases.
|
|
would be exceeded
* Checking whether the allocation succeeded may not prevent exceeding the memory
limit excessively.
* Even if the memory limit is not exceeded, the allocation can fail theoretically
and the program would terminate abnormally.
This however is true for all allocations in SciTECO (via glib).
* teco_memory_check() therefore now supports checking whether an allocation would
exceed the memory limit which will be useful before very large or variable allocations
in addition to the regular checking in teco_machine_main_step().
* As a sideeffect, this fixes the "Searching with large counts" test case on Mac OS
where too large allocations were not detected as expected (apparently Mac OS
happily gives out ridiculously large chunks of memory).
Now, all platforms are guaranteed to have the same behaviour.
|
|
Github logs when it fails
|
|
NOTE: Aliases and weak symbols must not be used for portability reasons!
|
|
after all
* the tools installed by default seem to lack aclocal...
|
|
meaningful artifact names
* Try to use as much of the "native" (Xcode?) tools on macOS as possible.
We can still fall back to Homebrew if we have to.
|
|
file systems
* There is a "Scintilla.h" as well.
* should fix macOS and builds on native Windows hosts
* It wasn't practical to refer to the Scintilla includes using paths since
the Scintilla location is configurable (--with-scintilla).
So we'd have to write something like #include <include/Scintilla.h>.
For Scinterm we cannot avoid collisions neither as its path is also
configurable (--with-scinterm).
Effectively, we must prevent name clashes across SciTECO and all
of Scintilla and Scinterm.
|
|
|
|
|
|
builds) and include Gtk+ versions
* The CI tests are unchanged. The workflow file has been renamed to ci.yml, though.
* Nightly builds are described by nightly.yml and are built at 4:13.
* Nightly Ubuntu package builds now include the Gtk+ 3 packages.
|
|
* Added a Freedesktop file - only as part of the debian package yet.
`make install` won't install a Desktop file since it would have to be
generated.
* Just like when installing manually, you can have a Curses and Gtk installation
side by side using the same .teco_ini.
* Common data between the Curses UI (sciteco-curses) and Gtk UI are in a new
architecture-independant package sciteco-common.
* The Gtk+ binaries are prefixed with `g` (gsciteco, gtedoc.tes, ggrosciteco.tes).
* Debian source and binary packages can be built using `./distribute.mk debian`
as usual.
It should also be possible to push everything to the PPA for the next release,
although that is not yet tested.
|
|
nightly builds in README
|
|
assignment to fix "Xvfb failed to start" (hopefully)
|
|
* We need some kind of XServer to run sciteco during the build process
and for the test suite, so we have to run everything via xvfb-run.
|
|
This will hopefully fix downloading libgtk-3-dev.
|
|
and compiler
|
|
`-fno-optimize-strlen` is not supported on Clang and there is no way
to ignore unknown arguments.
|
|
Clang and package for both versions of Ubuntu
* Testing is done only in the "build-and-test" job.
* Packages are built by the debian-packages job since
there is no need building them with Clang.
|
|
|
|
|
|
* required a new changelog entry since the target release is now v2.0.0
* Glib dependency bumped
|
|
|
|
* There are currently build errors with the message
"fatal: ambiguous argument 'graph_legend.dox': unknown revision or path not in the working tree."
But it does not seem to affect the overall result and "make devdoc" does not fail.
|
|
Will be extended for Continuous Deployment and using all sorts of build environments.
|
|
This is a total conversion of SciTECO to plain C (GNU C11).
The chance was taken to improve a lot of internal datastructures,
fix fundamental bugs and lay the foundations of future features.
The GTK user interface is now in an useable state!
All changes have been squashed together.
The language itself has almost not changed at all, except for:
* Detection of string terminators (usually Escape) now takes
the string building characters into account.
A string is only terminated outside of string building characters.
In other words, you can now for instance write
I^EQ[Hello$world]$
This removes one of the last bits of shellisms which is out of
place in SciTECO where no tokenization/lexing is performed.
Consequently, the current termination character can also be
escaped using ^Q/^R.
This is used by auto completions to make sure that strings
are inserted verbatim and without unwanted sideeffects.
* All strings can now safely contain null-characters
(see also: 8-bit cleanliness).
The null-character itself (^@) is not (yet) a valid SciTECO
command, though.
An incomplete list of changes:
* We got rid of the BSD headers for RB trees and lists/queues.
The problem with them was that they used a form of metaprogramming
only to gain a bit of type safety. It also resulted in less
readble code. This was a C++ desease.
The new code avoids metaprogramming only to gain type safety.
The BSD tree.h has been replaced by rb3ptr by Jens Stimpfle
(https://github.com/jstimpfle/rb3ptr).
This implementation is also more memory efficient than BSD's.
The BSD list.h and queue.h has been replaced with a custom
src/list.h.
* Fixed crashes, performance issues and compatibility issues with
the Gtk 3 User Interface.
It is now more or less ready for general use.
The GDK lock is no longer used to avoid using deprecated functions.
On the downside, the new implementation (driving the Gtk event loop
stepwise) is even slower than the old one.
A few glitches remain (see TODO), but it is hoped that they will
be resolved by the Scintilla update which will be performed soon.
* A lot of program units have been split up, so they are shorter
and easier to maintain: core-commands.c, qreg-commands.c,
goto-commands.c, file-utils.h.
* Parser states are simply structs of callbacks now.
They still use a kind of polymorphy using a preprocessor trick.
TECO_DEFINE_STATE() takes an initializer list that will be
merged with the default list of field initializers.
To "subclass" states, you can simply define new macros that add
initializers to existing macros.
* Parsers no longer have a "transitions" table but the input_cb()
may use switch-case statements.
There are also teco_machine_main_transition_t now which can
be used to implement simple transitions. Additionally, you
can specify functions to execute during transitions.
This largely avoids long switch-case-statements.
* Parsers are embeddable/reusable now, at least in parse-only mode.
This does not currently bring any advantages but may later
be used to write a Scintilla lexer for TECO syntax highlighting.
Once parsers are fully embeddable, it will also be possible
to run TECO macros in a kind of coroutine which would allow
them to process string arguments in real time.
* undo.[ch] still uses metaprogramming extensively but via
the C preprocessor of course. On the downside, most undo
token generators must be initiated explicitly (theoretically
we could have used embedded functions / trampolines to
instantiate automatically but this has turned out to be
dangereous).
There is a TECO_DEFINE_UNDO_CALL() to generate closures for
arbitrary functions now (ie. to call an arbitrary function
at undo-time). This simplified a lot of code and is much
shorter than manually pushing undo tokens in many cases.
* Instead of the ridiculous C++ Curiously Recurring Template
Pattern to achieve static polymorphy for user interface
implementations, we now simply declare all functions to
implement in interface.h and link in the implementations.
This is possible since we no longer hace to define
interface subclasses (all state is static variables in
the interface's *.c files).
* Headers are now significantly shorter than in C++ since
we can often hide more of our "class" implementations.
* Memory counting is based on dlmalloc for most platforms now.
Unfortunately, there is no malloc implementation that
provides an efficient constant-time memory counter that
is guaranteed to decrease when freeing memory.
But since we use a defined malloc implementation now,
malloc_usable_size() can be used safely for tracking memory use.
malloc() replacement is very tricky on Windows, so we
use a poll thread on Windows. This can also be enabled
on other supported platforms using --disable-malloc-replacement.
All in all, I'm still not pleased with the state of memory
limiting. It is a mess.
* Error handling uses GError now. This has the advantage that
the GError codes can be reused once we support error catching
in the SciTECO language.
* Added a few more test suite cases.
* Haiku is no longer supported as builds are instable and
I did not manage to debug them - quite possibly Haiku bugs
were responsible.
* Glib v2.44 or later are now required.
The GTK UI requires Gtk+ v3.12 or later now.
The GtkFlowBox fallback and sciteco-wrapper workaround are
no longer required.
* We now extensively use the GCC/Clang-specific g_auto
feature (automatic deallocations when leaving the current
code block).
* Updated copyright to 2021.
SciTECO has been in continuous development, even though there
have been no commits since 2018.
* Since these changes are so significant, the target release has
been set to v2.0.
It is planned that beginning with v3.0, the language will be
kept stable.
|
|
detect EMCurses
* Emscripten can be used (theoretically) to build a host-only platform-independant version
of SciTECO (running under node.js instead of the browser).
* I ported netbsd-curses with Emscripten for that purpose. Therefore, adaptions for running
in the browser are restricted to EMcurses now.
|
|
* a proper Arduino lexer supporting the special Arduino keywords/classes
could in principle be written, but for the time being they're treated
just like regular C++ sources
|
|
* Array allocations were not properly accounted since the compiler
would call the replacement new() which assumes that it would
always be called along with the replacement sized-deletion.
This is not true for array new[] allocations resulting in
a constant increase of memory_usage and unrecoverable situations.
This problem however could be fixed in principle by avoiding
memory counting for arrays or falling back to malloc_usable_size.
* The bigger problem was that some STLs (new_allocator) are broken, calling the
non-sized delete for regular new() calls which could in principle
be matched by sized-delete.
This is also the reason why I had to provide a non-sized
delete replacement, which in reality intoduced memory leaks.
* Since adding checks for the broken compiler versions or a configure-time
check that tries to detect these broken systems seems tedious,
I simply removed that optimization.
* This means we always have to rely on malloc_usable_size() now
for non-SciTECO-object memory measurement.
* Perhaps in the future, there should be an option for allowing
portable measurement at the cost of memory usage, by prefixing
each memory chunk with the chunk size.
Maintainers could then decide to optimize their build for "speed"
at the cost of memory overhead.
* Another solution to this non-ending odyssey might be to introduce
our own allocator, replacing malloc(), and allowing our own
precise measurements.
|
|
* it turned out to be possible to provoke memory_usage
overflows or underruns, resulting in unrecoverable states
* a possible reason can be that at least with G++ 5.4.0,
the compiler would sometimes call the (default) non-sized
delete followed by our custom sized delete/deallocator.
* This was true even after compiling Scintilla with -fsized-deallocation.
* therefore we provide an empty non-sized delete now.
* memory_usage counting can now be debugged by uncommenting
DEBUG_MAGIC in memory.cpp. This uses a magic value to detect
instrumented allocations being mixed with non-instrumented
allocations.
* simplified the global sized-deallocation functions
(they are identical to the Object-class allocators).
|
|
Automakefiles could be simplified by updating CXXFLAGS
in configure.ac instead.
|
|
* avoid warnings
* make sure Doxygen finds RBEntryOwnString
* it would be nice to strip the top level `SciTECO` namespace
but this is not supported without some macro magic that
ommit the namespace declaration when processing with
Doxygen.
|
|
* automatic conversion with `doxygen -u`
|
|
* when enabled, it will automatically upper-case all
one or two letter commands (which are case insensitive).
* also affects the up-carret control commands, so they when inserted
they look more like real control commands.
* specifically does not affect case-insensitive Q-Register specifications
* the result are command lines that are better readable and conform
to the coding style used in SciTECO's standard library.
This eases reusing command lines as well.
* Consequently, string-building and pattern match characters should
be case-folded as well, but they aren't currently since
State::process_edit_cmd() does not have sufficient insight
into the MicroStateMachines. Also, it could not be delegated
to the MicroStateMachines.
Perhaps they should be abandoned in favour of embeddedable
regular state machines; or regular state machines with a stack
of return states?
|
|
* this resulted in assertions (crashes!) for harmless typos like "+23="
* a test case has been added
|
|
|
|
* test case: HECcat$ on a large buffer (>= 64kb)
truncates the buffer or repeats its beginning
* it turns out that the incremental writing to the process' stdin
was broken. We were always writing data from the beginning of the buffer
which fails if the stdin watcher must be activated more than once.
* Also, EOLWriter::convert() can validly return 0, even if bytes have
been written on the data sink, so this value cannot be used to
check whether the process has closed its stdin.
We now make sure that the entire buffer range is written to stdin.
* Piping large buffers no longer removes the buffer gap.
This makes little difference when filtering via EC since
it will change the buffer gap anyway.
Can make a huge difference when not touching the buffer, though
(e.g. HEGAcat$).
* I did not add a test suite case since that requires
a very large test file and it cannot be easily generated automatically.
|
|
* StateQueryQReg is now derived from StateExpectQReg
whose semantics have been changed slightly.
* The alternative would have been another common base class for both
StateQueryQReg and StateExpectQReg.
|
|
support
* Since netbsd-curses can act as a drop-in replacement to ncurses,
SciTECO builds with --with-interface=ncurses as well.
However, it is unintuitive for users to build with ncurses support
when actually linking against netbsd-curses; so another option has been added.
* The UNIX/TTY specific code (which works with both ncurses and netbsd-curses)
was selected when NCURSES was detected at build-time.
This does not work for netbsd-curses, so we define a new symbol
NETBSD_CURSES. At build-time, a CURSES_TTY macro may now be defined.
* This effectively fixes the stdio in interactive mode, window titles
and the XTerm clipboard support for netbsd-curses.
Some minor features like the reduced ESCDELAY are still broken.
|
|
* this is actually another independant Curses implementation for
Unix platforms I wasn't aware of.
I tested against this portable fork of it:
https://github.com/sabotage-linux/netbsd-curses
* Only a mimimal change to Scinterm was necessary to support it.
* netbsd-curses might be useful for NetBSD support (which is
otherwise untested) and when building small statically
linked self-contained binaries since netbsd-curses is much
smaller than ncurses.
|
|
|
|
been shown to be unacceptably broken, so the fallback implementation has been improved
* mallinfo() is not only broken on 64-bit systems but slows things
down linearilly to the memory size of the process.
E.g. after 500000<%A>, SciTECO will act sluggish! Shutting down
afterwards can take minutes...
mallinfo() was thus finally discarded as a memory measurement
technique.
* Evaluating /proc/self/statm? has also been evaluated and discarded
because doing this frequently is even slower.
* Instead, the fallback implementation has been drastically improved:
* If possible use C++14 global sized deallocators, allowing memory measurements
across the entire C++ code base with minimal runtime overhead.
Since we only depend on C++11, a lengthy Autoconf check had to be introduced.
* Use malloc_usable_size() with global non-sized deallocators to
measure the approx. memory usage of the entire process (at least
the ones done via C++).
The cheaper C++11 sized deallocators implemented via SciTECO::Object still
have precedence, so this affects Scintilla code only.
* With both improvements the test case
sciteco -e '<@EU[X^E\a]"^E\a"%a>'
is handled sufficiently well now on glibc and performance is much better
now.
* The jemalloc-specific technique has been removed since it no longer
brings any benefits compared to the improved fallback technique.
Even the case of using malloc_usable_size() in strict C++ mode is
up to 3 times faster.
* The new fallback implementation might actually be good enough for
Windows as well if some MSVCRT-specific support is added, like
using _msize() instead of malloc_usable_size().
This must be tested and benchmarked, so we keep the Windows-specific
implementation for the time being.
|
|
* in a flat list of undo tokens, we need to store
the program counter (ie. command line position)
that the undo token corresponds to.
Since in general there is more than one undo token per
input character, this stored PCs redundantly.
* For input characters with no undo tokens
(only applies to NOPs like space in the command line
macro), this needs one more pointer than before.
* In case of 1 undo token per input character,
the new implementation uses approx. the same memory.
* In the most common case of more than one undo token
per input character, this saves at least 4 bytes per
undo token.
* In large macros and long loops the effect is especially
pronounced. E.g. 500000<%A> will use 8MB less memory
with the new implementation.
|
|
editing key
* StateEscape should return the same fnmacro mask as StateStart
* When rubbing out a command, we should stop at StateEscape as well.
Therefore we reintroduced States::is_start().
RTTI is still not used.
|
|
as State::process_edit_cmd() virtual methods
* Cmdline::process_edit_cmd() was much too long and deeply nested.
It used RTTI excessively to implement the state-specific behaviour.
It became apparent that the behaviour is largely state-specific and could be
modelled much more elegantly as virtual methods of State.
* Basically, a state can now implement a method to customize its
commandline behaviour.
In the case that the state does not define custom behaviour for
the key pressed, it can "chain" to the parent class' process_edit_cmd().
This can be optimized to tail calls by the compiler.
* The State::process_edit_cmd() implementations are still isolated in
cmdline.cpp. This is not strictly necessary but allows us keep the
already large compilations units like parser.cpp small.
Also, the edit command processing has little to do with the rest of
a state's functionality and is only used in interactive mode.
* As a result, we have many small functions now which are much easier to
maintain.
This makes adding new and more complex context sensitive editing behaviour
easier.
* State-specific function key masking has been refactored by introducing
State::get_fnmacro_mask().
* This allowed us to remove the States::is_*() functions which have
always been a crutch to support context-sensitive key handling.
* RTTI is almost completely eradicated, except for exception handling
and StdError(). Both remaining cases can probably be avoided in the
future, allowing us to compile smaller binaries.
|
|
and added a FreeBSD/jemalloc-specific implementation
* largely reverts 39cfc573, but leaves in minor and documentation
changes.
* further experimentation of memory limiting using malloc() wrapping
has shown additional problems, like dlsym() calling malloc-functions,
further reducing the implementation to glibc-specific means.
This means there had been no implementation for FreeBSD and checks
would have to rely on undocumented internal implementation details
of different libcs, which is not a good thing.
* Other problems have been identified, like having to wrap calloc(),
guarding against underruns and multi-thread safety had been identified
but could be worked around.
* A technique by calculating the memory usage as sbrk(0) - &end
has been shown to be effective enough, at least on glibc.
However even on glibc it has shortcomings since malloc() will
somtimes use mmap() for allocations and the technique
relies on implementation details of the libc.
Furthermore another malloc_trim(0) had to be added to the error
recovery in interactive mode, since glibc does not adjust the program break
automatically (to avoid syscalls I presume).
* On FreeBSD/jemalloc, the sbrk(0) method totally fails because jemalloc
exclusively allocates via mmap() -> that solution was discarded as well.
* Since all evaluated techniques turn out to be highly platform
specific, I reverted to the simple and stable platform-specific
mallinfo() API on Linux.
* On FreeBSD/jemalloc, it's possible to use mallctl("stats.allocated")
for the same purpose - so it works there, too now.
It's slower than the other techniques, though.
* A lengthy discussion has been added to memory.cpp, so that we
do not repeat the previous mistakes.
|
|
* shouldn't make much of a difference, since we're in deep trouble
when they return NULL, but the wrappers should be transparent
instead of crashing in malloc_usable_size().
|
|
|
|
portable and faster hack
* Works by "hooking" into malloc() and friends and counting the
usable heap object sizes with malloc_usable_size().
Thus, it has no memory-overhead.
* Will work at least on Linux and (Free)BSD.
Other UNIXoid systems may work as well - this is tested by ./configure.
* Usually faster than even the fallback implementation since the
memory limit is hit earlier.
* A similar approach could be tried on Windows (TODO).
* A proper memory-limiting counting all malloc()s in the system can make
a huge difference as this test case shows:
sciteco -e '<@EU[X^E\a]"^E\a"%a>'
It will allocate gigabytes before hitting the 500MB memory limit...
* Fixed the UNIX-function checks on BSDs.
|
