improved interruption (SIGINT, CTRL+C) support

* Just like the original LuaJIT interpreter, this will use a hook to automatically
  raise an error from Lua code.
* Unfortunately, as long as Jit compilation is enabled, this cannot reliably work.
  Therefore we still set an `interrupted` flag that must be
  polled from tight loops.
* Instead of polling via applause_push_sample() which gave interruption-support only
  to Stream:play(), we now have a separate checkint() function.
* checkint() should be manually added to all tight loops.
* Stream:foreach() and everthing based on it is now also supporting interruptions
  (via checkint()).
* This internally works via applause_is_interrupted().
  A C function was exposed only because LuaJIT does not support volatile-qualifiers and
  would optimize away reads to the interrupted-flag.
  As a side effect, we can also reset the hook.
* Flags set in signal handlers should be `volatile`.
* Added likely() and unlikely() macros to C code.
* Updated sample.ipynb Jupyter notebook: Everything important is now supported, albeit
  requiring custom ILua patches.

6 files changed, 192 insertions, 94 deletions

diff --git a/README.md b/README.md
index e6a8e5f..8642d18 100644
--- a/README.md
+++ b/README.md
@@ -125,13 +125,21 @@ APPLAUSE_OPTS="-o 2" jupyter notebook --MultiKernelManager.default_kernel_name=l
 This works assuming that you symlinked `ilua-wrapper.sh` to `lua` as described above.
 An alternative might be to create a custom Jupyter kernel configuration (kernel.json).
 
+If the browser is not opened automatically on the notebook's URL, you might want to try
+visiting http://localhost:8888/.
+
 Please note the following restrictions/bugs:
 
 * You cannot publicly host the Jupyter Notebook as the sound is generated on the host machine.
+  Similarily, it would be hard to wrap everything in a Docker container.
 * You cannot currently interrupt an endlessly running stream without restarting the kernel
   (see this [ILua bug](https://github.com/guysv/ilua/issues/1)).
-* ILua does not work well with our custom `Stream:_tostring()` metamethods.
+* ILua does not work well with our custom `Stream:__tostring()` metamethods.
   As a workaround, invoke `tostring()` manually on Streams.
 * The output of other functions like Stream:toplot() is garbled.
 * You cannot currently output rich text or graphics.
   There is a working [workaround](https://github.com/guysv/ilua/issues/5), though.
+  At least `Stream:gnuplot()` can now plot graphs with this workaround.
+
+There are workarounds for most of these problems and they might eventually be upstreamed
+or ILua will be forked.
\ No newline at end of file
diff --git a/TODO b/TODO
index b1a5e62..294dc27 100644
--- a/TODO
+++ b/TODO
@@ -1,13 +1,14 @@
 # Bugs
 
-* Stream:foreach() cannot be interrupted
-  Perhaps C core should export an interrupted variable that we can check from Lua.
-  For Stream:play() this is solved differently.
 * The MIDIStream should be flushed when starting via Stream:play().
 * There are lots of limitations with Jupyter servers (see README).
   Perhaps it would be better to use xeus-lua, but that would require us to refactor everything
-  into a proper Lua library.
+  into a proper Lua library and it's harder to build.
   https://github.com/jupyter-xeus/xeus-lua
+* CTRL+C interruption on the REPL loop does not always work.
+  Test case: while true do end
+  This is a known bug in LuaJIT, see https://luajit.org/faq.html#ctrlc.
+  As a workaround, add checkint() to all tight loops.
 
 # Features
 
@@ -30,6 +31,8 @@
   or with a Stream:savenow() shortcut.
 * Multi-core support via GStreamer-like queue:
   https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-queue.html
+  * If it turns out, we cannot cleanly fork() after starting the Jack client,
+    it might be possible to send Streams as byte code dumps as well.
 * Stream optimizer
   * Automatic caching (finding identical subtrees).
   * Removing contraproductive caching, e.g. after plain number streams, MIDIStream, EvdevStream...
diff --git a/applause.c b/applause.c
index 70713cb..5853e32 100644
--- a/applause.c
+++ b/applause.c
@@ -42,7 +42,10 @@
 #define CMD_SERVER_IP   	"127.0.0.1"
 #define CMD_SERVER_PORT		10000
 
-#define MAX(X,Y) ((X) > (Y) ? (X) : (Y))
+#define MAX(X,Y)		((X) > (Y) ? (X) : (Y))
+
+#define likely(X)		__builtin_expect((X), 1)
+#define unlikely(X)		__builtin_expect((X), 0)
 
 static jack_client_t		*client = NULL;
 
@@ -66,9 +69,10 @@ static applause_output_port *output_ports = NULL;
 /** Number of Applause output ports */
 static int output_ports_count = DEFAULT_NUMBER_OF_OUTPUT_PORTS;
 
-static sig_atomic_t		interrupted = 0;
+static lua_State		*L_global = NULL;
 
-static sig_atomic_t		playback = 0;
+static volatile sig_atomic_t	interrupted = 0;
+static volatile sig_atomic_t	playback = 0;
 
 typedef struct applause_midi_port {
 	jack_port_t		*jack_port;
@@ -132,14 +136,57 @@ svsem_free(int id)
 }
 
 /**
- * Handler for SIGINT, SIGUSR1 and SIGUSR2 signals.
+ * Can be called from Lua code to check the interruption-flag,
+ * ie. to detect SIGINT and CTRL+C interruptions.
  *
- * SIGINT is delivered e.g. when the user presses
- * CTRL+C.
- * It sets `interrupted` which is polled in the Lua play()
- * method which allows us to interrupt long (possibly infinite)
- * sound playback.
+ * We do not simply expose the interrupted flag since LuaJIT
+ * does not support volatile variables and we cannot guarantee that
+ * the read is not optimized away.
+ * On the other hand, this function cannot raise a Lua error
+ * (and C functions via the classic C API are slow), so there is
+ * another wrapper in applause.lua.
+ */
+int
+applause_is_interrupted(void)
+{
+	if (likely(!interrupted))
+		return 0;
+
+	interrupted = 0;
+	lua_sethook(L_global, NULL, 0, 0);
+	return 1;
+}
+
+static void
+signal_hook(lua_State *L, lua_Debug *ar)
+{
+	interrupted = 0;
+	lua_sethook(L, NULL, 0, 0);
+
+	/* Avoid luaL_error -- a C hook doesn't add an extra frame. */
+	luaL_where(L, 0);
+	lua_pushfstring(L, "%sinterrupted!", lua_tostring(L, -1));
+	lua_error(L);
+}
+
+static inline void
+applause_interrupt(void)
+{
+	/*
+	 * This should raise a Lua error via the hook ASAP.
+	 * Unfortunetely, this cannot work reliably with JIT compilation
+	 * enabled (see also https://luajit.org/faq.html#ctrlc).
+	 * We therefore also set an interrupted flag that can be polled
+	 * efficiently in tight loops.
+	 */
+	interrupted = 1;
+	lua_sethook(L_global, signal_hook, LUA_MASKCALL | LUA_MASKRET | LUA_MASKCOUNT, 1);
+}
+
+/**
+ * Handler for SIGINT, SIGUSR1 and SIGUSR2 signals.
  *
+ * SIGINT is delivered e.g. when the user presses CTRL+C.
  * SIGUSR1 and SIGUSR2 are sent by parent to child
  * clients in order to control playback.
  */
@@ -147,7 +194,7 @@ static void
 signal_handler(int signum)
 {
 	switch (signum) {
-	case SIGINT: interrupted = 1; break;
+	case SIGINT: applause_interrupt(); break;
 	case SIGUSR1: playback = 1; break;
 	case SIGUSR2: playback = 0; break;
 	}
@@ -415,17 +462,12 @@ applause_push_sample(int output_port_id, double sample_double)
 	jack_default_audio_sample_t sample =
 			(jack_default_audio_sample_t)sample_double;
 
-	if (interrupted) {
-		interrupted = 0;
-		return APPLAUSE_AUDIO_INTERRUPTED;
-	}
-
 	/*
 	 * NOTE: The alternative to reporting invalid port Ids here
 	 * would be exporting output_ports_count, so the Lua code can
 	 * check it and assert()ing here instead.
 	 */
-	if (output_port_id < 1 || output_port_id > output_ports_count)
+	if (unlikely(output_port_id < 1 || output_port_id > output_ports_count))
 		return APPLAUSE_AUDIO_INVALID_PORT;
 	port = output_ports + output_port_id - 1;
 
@@ -441,7 +483,7 @@ applause_push_sample(int output_port_id, double sample_double)
 	jack_ringbuffer_write(port->buffer, (const char *)&sample,
 	                      sizeof(sample));
 
-	if (port->buffer_xrun) {
+	if (unlikely(port->buffer_xrun)) {
 		port->buffer_xrun = 0;
 		return APPLAUSE_AUDIO_XRUN;
 	}
@@ -625,8 +667,8 @@ static pthread_mutex_t lua_mutex = PTHREAD_MUTEX_INITIALIZER;
 
 /**
  * Thread monitoring the client file desciptor.
- * This will set the interrupted flag if the remote
- * end closes its write part of the socket.
+ * This will raise interrupt the current Lua script if the remote
+ * end closes its write part of the socket (just like SIGINT would).
  * That's why we need the length header at the beginning
  * of requests.
  * There seems to be no easy way to find out whether the
@@ -653,7 +695,7 @@ monitor_thread_cb(void *user_data)
 	while (poll(&pfd, 1, -1) != 1 ||
 	       !(pfd.revents & (POLLRDHUP | POLLERR | POLLHUP)));
 
-	interrupted = 1;
+	applause_interrupt();
 	return NULL;
 }
 
@@ -741,9 +783,9 @@ command_server(void *user_data)
 		pthread_mutex_lock(&lua_mutex);
 
 		/*
-		 * Start another thread monitoring client_fd and setting
-		 * the interrupted flag in case the remote end terminates
-		 * prematurely. This has the effect of ^C in SciTECO interrupting
+		 * Start another thread monitoring client_fd for raising SIGINT
+		 * in case the remote end terminates prematurely.
+		 * This has the effect of ^C in SciTECO interrupting
 		 * the current command (e.g. play()) at least when using
 		 * socat (see monitor_thread_cb()).
 		 * A monitoring thread is started instead of running do_command()
@@ -773,7 +815,7 @@ command_server(void *user_data)
 		/*
 		 * FIXME: It would be nice to catch broken sockets
 		 * (e.g. waiting for play() has been interrupted), in order
-		 * to set the interrupted flag.
+		 * to raise SIGINT.
 		 * However it seems we'd need another thread for that running
 		 * do_command(), so we can select() the fd and eventually
 		 * join the thread.
@@ -855,13 +897,6 @@ main(int argc, char **argv)
 		}
 	}
 
-	/*
-	 * Register sigint_handler() as the SIGINT handler.
-	 * This sets `interrupted`. Currently this is only polled
-	 * in the Lua play() method in order to interrupt long
-	 * sound playing.
-	 * Otherwise it is ignored.
-	 */
 	memset(&signal_action, 0, sizeof(signal_action));
 	signal_action.sa_handler = signal_handler;
 	sigaction(SIGINT, &signal_action, NULL);
@@ -871,7 +906,7 @@ main(int argc, char **argv)
 	signal_action.sa_handler = SIG_IGN;
 	sigaction(SIGPIPE, &signal_action, NULL);
 
-	L = luaL_newstate();
+	L = L_global = luaL_newstate();
 	if (!L) {
 		fprintf(stderr, "Error creating Lua state.\n");
 		exit(EXIT_FAILURE);
diff --git a/applause.h b/applause.h
index c4d9d87..fc01d49 100644
--- a/applause.h
+++ b/applause.h
@@ -2,10 +2,11 @@
 
 enum applause_audio_state {
 	APPLAUSE_AUDIO_OK = 0,
-	APPLAUSE_AUDIO_INTERRUPTED,
 	APPLAUSE_AUDIO_XRUN,
 	APPLAUSE_AUDIO_INVALID_PORT
 };
 
 enum applause_audio_state applause_push_sample(int output_port_id,
                                                double sample_double);
+
+int applause_is_interrupted(void);
diff --git a/applause.lua b/applause.lua
index 702e1ce..702913d 100644
--- a/applause.lua
+++ b/applause.lua
@@ -32,6 +32,22 @@ end
 
 cdef_include "applause.h"
 
+--- Check whether the process was interrupted (SIGINT received).
+--
+-- This checks and automatically raises an error if CTRL+C was pressed
+-- or SIGINT was received.
+-- The automatic way to handle SIGINT is **not** reliable in LuaJIT.
+-- A call to this function should therefore be added to all tight
+-- loops.
+function checkint()
+	-- This does not poll the interrupted-flag directly
+	-- since LuaJIT does not support volatile qualifiers and could
+	-- optimize the read away.
+	if C.applause_is_interrupted() ~= 0 then
+		error("SIGINT received", 2)
+	end
+end
+
 --
 -- Define C functions for benchmarking (POSIX libc)
 --
@@ -664,8 +680,7 @@ function Stream:play(first_port)
 	local old_stepmul = collectgarbage("setstepmul", 100)
 
 	local channels = self.channels
-	local state
-	self:foreach(function(frame)
+	local _, err = pcall(Stream.foreach, self, function(frame)
 		-- Loop should get unrolled automatically
 		for i = 1, channels do
 			local sample = tonumber(frame[i])
@@ -674,7 +689,7 @@ function Stream:play(first_port)
 			-- NOTE: Invalid port Ids are currently silently
 			-- ignored. Perhaps it's better to check state or
 			-- to access output_ports_count from applause.c.
-			state = C.applause_push_sample(first_port+i, sample)
+			local state = C.applause_push_sample(first_port+i, sample)
 
 			-- React to buffer underruns.
 			-- This is done here instead of in the realtime thread
@@ -683,17 +698,13 @@ function Stream:play(first_port)
 			if state == C.APPLAUSE_AUDIO_XRUN then
 				io.stderr:write("WARNING: Buffer underrun detected\n")
 			end
-
-			if state == C.APPLAUSE_AUDIO_INTERRUPTED then return true end
 		end
 	end)
 
 	collectgarbage("setpause", old_pause)
 	collectgarbage("setstepmul", old_stepmul)
 
-	if state == C.APPLAUSE_AUDIO_INTERRUPTED then
-		error("SIGINT received", 2)
-	end
+	if err then error(err, 2) end
 end
 
 --- Execute function for each frame generated by the stream.
@@ -702,21 +713,19 @@ end
 -- @func fnc
 --   Function to execute.
 --   It gets passed an array of samples, one for each channel.
--- @fixme This is not currently interruptable and therefore not suitable
--- to be executed dynamically at the command-line.
+--   If it returns true, the loop is terminated.
 function Stream:foreach(fnc)
-	local clear = table.clear
-
 	-- NOTE: This implementation is for single-channel streams
 	-- only. See also MuxStream:foreach().
+	local clear = table.clear
 	local frame = table.new(1, 0)
 	local tick = self:gtick()
 
-	while true do
+	repeat
+		checkint()
 		frame[1] = tick()
 		clear(sampleCache)
-		if not frame[1] or fnc(frame) then break end
-	end
+	until not frame[1] or fnc(frame)
 end
 
 --- Benchmark stream (time to generate all samples).
@@ -873,8 +882,6 @@ end
 -- @string[opt="full"] vbufmode Buffering mode.
 -- @int[opt] vbufsize The buffer size.
 -- @see file:setvbuf
--- @fixme This is currently allowed for infinite streams as well,
--- but there is no way to interrupt Stream:foreach().
 function Stream:pipe(prog, vbufmode, vbufsize)
 	local hnd = io.popen(prog, "w")
 	hnd:setvbuf(vbufmode or "full", vbufsize)
@@ -1282,7 +1289,8 @@ function MuxStream:foreach(fnc)
 	local channels = self.channels
 	local frame = table.new(channels, 0)
 
-	while true do
+	repeat
+		checkint()
 		clear(sampleCache)
 
 		for i = 1, channels do
@@ -1291,9 +1299,7 @@ function MuxStream:foreach(fnc)
 			-- length, if one ends all end
 			if not frame[i] then return end
 		end
-
-		if fnc(frame) then break end
-	end
+	until fnc(frame)
 end
 
 --- Mux several streams with the source stream into a multi-channel stream.
diff --git a/examples/simple.ipynb b/examples/simple.ipynb
index 1bfddff..a841d33 100644
--- a/examples/simple.ipynb
+++ b/examples/simple.ipynb
@@ -7,7 +7,9 @@
    "source": [
     "Applause supports Jupyter via [ILua](https://github.com/guysv/ilua)!\n",
     "\n",
-    "Unfortunately, there are currently some restrictions..."
+    "Unfortunately, there are currently some restrictions.\n",
+    "\n",
+    "**NOTE:** The following examples require some custom patches on top of ILua."
    ]
   },
   {
@@ -15,54 +17,28 @@
    "id": "a0620a19-0b04-4b28-8169-fafe75002bbe",
    "metadata": {},
    "source": [
-    "Manually call `tostring()` when printing streams (**FIXME**):"
+    "ILua does not require a leading `=` for printing values:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 1,
    "id": "9179293d-9b2d-47b2-bb1c-4ee8cf576606",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "\"{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}\""
+       "{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 1,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "tostring(iota(10))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "88154e0d-baae-41ed-a973-92d2c1a233cc",
-   "metadata": {},
-   "source": [
-    "You cannot currently interrupt streams, so make sure not to play infinite streams:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "04a654a8-f3e4-4168-966e-2d3e961e0bcf",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING: Buffer underrun detected\n"
-     ]
-    }
-   ],
-   "source": [
-    "Stream.SinOsc(440):sub(1, sec(5)):play()"
+    "iota(10)"
    ]
   },
   {
@@ -75,7 +51,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 2,
    "id": "98b40a38-0ea1-4d5c-97a7-3de458812cb9",
    "metadata": {},
    "outputs": [
@@ -395,9 +371,78 @@
    ]
   },
   {
+   "cell_type": "markdown",
+   "id": "3cadc448-1620-44b1-8821-7e54ef0ab1c7",
+   "metadata": {},
+   "source": [
+    "Play this sound for 5 seconds:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "d9b94d06-b5d8-42a7-93a8-25c4add1dd30",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "WARNING: Buffer underrun detected\n"
+     ]
+    }
+   ],
+   "source": [
+    "w:sub(1, sec(5)):play()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "27665672-a97d-4b2e-a976-1e4ebb204ccb",
+   "metadata": {},
+   "source": [
+    "In fact, you can play infinite streams. To interrupt them, press the \"Interrupt the kernel\" button on the toolbar.\n",
+    "\n",
+    "**WARNING:** Just like on the Applause command line, while you can always interrupt playing streams, you cannot reliably interrupt all commands. Add `checkint()` to tight loops. If everything fails, you might have to restart the kernel!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "78cd3b35-5e15-4c80-b7d2-dbe1be77856a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "WARNING: Buffer underrun detected\n"
+     ]
+    },
+    {
+     "ename": "n/a",
+     "evalue": "applause.lua:725: SIGINT received",
+     "execution_count": 5,
+     "output_type": "error",
+     "traceback": [
+      "applause.lua:725: SIGINT received",
+      "stack traceback:",
+      "\t[C]: in function 'error'",
+      "\tapplause.lua:707: in function <applause.lua:663>",
+      "\t[C]: in function 'xpcall'",
+      "\t...ies/ilua/env/lib/python3.8/site-packages/ilua/interp.lua:65: in function 'handle_execute'",
+      "\t...ies/ilua/env/lib/python3.8/site-packages/ilua/interp.lua:176: in main chunk"
+     ]
+    }
+   ],
+   "source": [
+    "w:play()"
+   ]
+  },
+  {
    "cell_type": "code",
    "execution_count": null,
-   "id": "78a2478e-819c-42e0-a47d-5682e60886e1",
+   "id": "14bd4fad-e9ce-4b5f-8e46-47a140ad47e7",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -413,7 +458,7 @@
    "file_extension": ".lua",
    "mimetype": "text/x-lua",
    "name": "lua",
-   "version": "5.1"
+   "version": "n/a"
   }
  },
  "nbformat": 4,