This is a small example on how to implement an interpreter using Clojure and the Instaparse library.
Dependencies
First we create a deps.edn file to get Rich Hickey’s Clojure, Mark Engelberg’s Instaparse, the Midje test suite by Brian Marick, and my modified version of Max Miorim’s midje-runner:
We also need to create an initial grammar in resources/clj_calculator/calculator.bnf defining a minimal grammar and a regular expression for parsing an integer:
START = <WHITESPACE?> (NUMBER | SUM | DIFF | PROD) <WHITESPACE?>
SUM = NUMBER <WHITESPACE?> <'+'> <WHITESPACE?> NUMBER
DIFF = NUMBER <WHITESPACE?> <'-'> <WHITESPACE?> NUMBER
PROD = NUMBER <WHITESPACE?> <'*'> <WHITESPACE?> NUMBER
NUMBER = #'[-+]?[0-9]+'
WHITESPACE = #'[,\ \t]+'
Transforming syntax trees
Instaparse comes with a useful transformation function for recursively transforming the abstract syntax tree we obtained from parsing.
First we write and run a failing test for transforming a string to an integer:
(facts"Test calculator"(calculate"-42")=>-42)
To pass the test we implement a calculator function which transforms the syntax tree.
Initially it only needs to deal with the nonterminal symbols START and NUMBER:
Obviously we can use the transformation function to also perform the calculations.
Here are the tests for the three possible operations of the parse tree.
A naive implementation using a blind EXPR nonterminal symbol passes the test:
START = <WHITESPACE?> (EXPR | SUM | DIFF | PROD) <WHITESPACE?>
<EXPR> = SUM | DIFF | PROD | NUMBER
SUM = EXPR <WHITESPACE?> <'+'> <WHITESPACE?> EXPR
DIFF = EXPR <WHITESPACE?> <'-'> <WHITESPACE?> EXPR
PROD = EXPR <WHITESPACE?> <'*'> <WHITESPACE?> EXPR
NUMBER = #'[-+]?[0-9]+'
WHITESPACE = #'[,\ \t]+'
However there is a problem with this grammar: It is ambiguous.
The following failing test shows that the parser could generate two different parse trees:
When parsing small strings, this might not be a problem.
However if you use an ambiguous grammar to parse a large file with a syntax error near the end, the resulting combinatorial explosion leads to a long processing time before the parser can return the syntax error.
The good thing is, that Instaparse uses the GLL parsing algorithm, i.e. it can handle a left-recursive grammar to resolve the ambiguity:
START = <WHITESPACE?> (EXPR | SUM | DIFF | PROD) <WHITESPACE?>
<EXPR> = SUM | DIFF | PROD | NUMBER
SUM = EXPR <WHITESPACE?> <'+'> <WHITESPACE?> NUMBER
DIFF = EXPR <WHITESPACE?> <'-'> <WHITESPACE?> NUMBER
PROD = EXPR <WHITESPACE?> <'*'> <WHITESPACE?> NUMBER
NUMBER = #'[-+]?[0-9]+'
WHITESPACE = #'[,\ \t]+'
This grammar is not ambiguous any more and will pass above test.
Grouping using brackets
We might want to use brackets to group expressions and influence the order expressions are applied:
For game development I have been using LWJGL3 which is a great Java library for cross-platform development.
Among other things it has bindings for OpenGL, GLFW, and STB.
Recently I discovered that it also has Nuklear bindings.
Nuklear is a small C library useful for developing GUIs for games.
It receives control input and commands to populate a GUI and converts those into render instructions.
Nuklear focuses solely on the user interface, while input and graphics backend are handled by the application.
It is therefore very flexible and can be integrated into a 3D game implemented using OpenGL, DirectX, Vulkan, or other.
LWJGL Nuklear bindings come with the GLFWDemo.java example.
In this article I have basically translated the input and graphics backend to Clojure.
I also added examples for several different controls.
I have pushed the source code to Github if you want to look at it straight away.
A big thank you to Ioannis Tsakpinis who developed LWJGL and GLFWDemo.java in particular.
And a big thank you to Micha Mettke who developed the Nuklear library.
The demo is more than 400 lines of code.
This is because it has to implement the graphics backend, input conversion, and truetype font conversion to bitmap font.
If you are rather looking for a Clojure GUI library which does not require you to do this, you might want to look at HumbleUI.
There also is Quil which seems to be more about graphics and animations.
(nsnukleartest(:import[org.lwjgl.glfwGLFW][org.lwjgl.openglGLGL20])); ...(defvertex-source"#version 410 core
uniform mat4 projection;
in vec2 position;
in vec2 texcoord;
in vec4 color;
out vec2 frag_uv;
out vec4 frag_color;
void main()
{
frag_uv = texcoord;
frag_color = color;
gl_Position = projection * vec4(position, 0, 1);
}")(deffragment-source"#version 410 core
uniform sampler2D tex;
in vec2 frag_uv;
in vec4 frag_color;
out vec4 out_color;
void main()
{
out_color = frag_color * texture(tex, frag_uv);
}")(defprogram(GL20/glCreateProgram))(defvertex-shader(GL20/glCreateShaderGL20/GL_VERTEX_SHADER))(GL20/glShaderSourcevertex-shadervertex-source)(GL20/glCompileShadervertex-shader)(when(zero?(GL20/glGetShaderivertex-shaderGL20/GL_COMPILE_STATUS))(println(GL20/glGetShaderInfoLogvertex-shader1024))(System/exit1))(deffragment-shader(GL20/glCreateShaderGL20/GL_FRAGMENT_SHADER))(GL20/glShaderSourcefragment-shaderfragment-source)(GL20/glCompileShaderfragment-shader)(when(zero?(GL20/glGetShaderifragment-shaderGL20/GL_COMPILE_STATUS))(println(GL20/glGetShaderInfoLogfragment-shader1024))(System/exit1))(GL20/glAttachShaderprogramvertex-shader)(GL20/glAttachShaderprogramfragment-shader)(GL20/glLinkProgramprogram)(when(zero?(GL20/glGetProgramiprogramGL20/GL_LINK_STATUS))(println(GL20/glGetProgramInfoLogprogram1024))(System/exit1))(GL20/glDeleteShadervertex-shader)(GL20/glDeleteShaderfragment-shader); ...(GL20/glDeleteProgramprogram); ...
The vertex shader passes through texture coordinates and fragment colors.
Furthermore it scales the input position to OpenGL normalized device coordinates (we will set the projection matrix later).
The fragment shader performs a texture lookup and multiplies the result with the fragment color value.
The Clojure code compiles and links the shaders and checks for possible errors.
An array buffer containing the position, texture coordinates, and colors are allocated.
Furthermore an element array buffer is allocated which contains element indices.
A row in the array buffer contains 20 bytes:
2 times 4 bytes for floating point “position”
2 times 4 bytes for floating point texture coordinate “texcoord”
4 bytes for RGBA color value “color”
The Nuklear library needs to be configured with the same layout of the vertex array buffer.
For this purpose a Nuklear vertex layout object is initialised using the NK_VERTEX_ATTRIBUTE_COUNT attribute as a terminator:
The null texture basically just consists of a single white pixel so that the shader term texture(tex, frag_uv) evaluates to vec4(1, 1, 1, 1).
The Nuklear null texture uses fixed texture coordinates for lookup (here: 0.5, 0.5).
I.e. it is possible to embed the null texture in a bigger multi-purpose texture to save a texture slot.
Nuklear GUI
Nuklear Context, Command Buffer, and Configuration
Finally we can set up a Nuklear context object “context”, a render command buffer “cmds”, and a rendering configuration “config”.
Nuklear even delegates allocating and freeing up memory, so we need to register callbacks for that as well.
Blending with existing pixel data is enabled and the blending equation and function are set
Culling of back or front faces is disabled
Depth testing is disabled
Scissor testing is enabled
The first texture slot is enabled
Also the uniform projection matrix for mapping pixel coordinates [0, width] x [0, height] to [-1, 1] x [-1, 1] is defined.
The projection matrix also flips the y-coordinates since the direction of the OpenGL y-axis is reversed in relation to the pixel y-coordinates.
Minimal Test GUI
Now we will add a minimal GUI just using a progress bar for testing rendering without fonts.
First we set up a few values and then in the main loop we start Nuklear input using Nuklear/nk_input_begin, call GLFW to process events, and then end Nuklear input.
We will implement the GLFW callbacks to convert events to Nuklear calls later.
We start populating the GUI by calling Nuklear/nk_begin thereby specifying the window size.
We increase the progress value and store it in a PointerBuffer object.
The call (Nuklear/nk_layout_row_dynamic context 32 1) sets the GUI layout to 32 pixels height and one widget per row.
Then a progress bar is created and the GUI is finalised using Nuklear/nk_end.
The rendering backend sets the viewport and then array buffers for the vertex data and the indices are allocated.
Then the buffers are mapped to memory resulting in the two java.nio.DirectByteBuffer objects “vertices” and “elements”.
The two static buffers are then converted to Nuklear buffer objects using Nuklear/nk_buffer_init_fixed.
Then the core method of the Nuklear library Nuklear/nk_convert is called. It populates the (dynamic) command buffer “cmds” which we initialised earlier as well as the mapped vertex buffer and index buffer.
After the conversion, the two OpenGL memory mappings are undone.
A Clojure loop then is used to get chunks of type NkDrawCommand from the render buffer.
Each draw command requires setting the texture id and the clipping region.
Then a part of the index and vertex buffer is rendered using GL11/glDrawElements.
Finally Nuklear/nk_clear is used to reset the GUI specification for the next frame and Nuklear/nk_buffer_clear is used to empty the command buffer.
GLFW/glfwSwapBuffers is used to publish the new rendered frame.
Now we finally have a widget working!
Mouse Events
Cursor Position and Buttons
The next step one can do is converting GLFW mouse events to Nuklear input.
The first callback is to process mouse cursor movement events.
The second callback converts mouse button press and release events to Nuklear input.
The progress bar is modifyable and you should now be able to change it by clicking on it.
Note that using a case statement instead of cond did not work for some reason.
Scroll Events
The Nuklear library can also be informed about scroll events.
Here is the corresponding GLFW callback to pass scroll events on to the Nuklear library.
Basically the font file is read and converted to a java.nio.DirectByteBuffer (let me know if you find a more straightforward way to do this).
The data is used to initialise an STB font info object.
The next steps I can’t explain in detail but they basically pack the glyphs into a greyscale bitmap.
Finally a white RGBA texture data is created with the greyscale bitmap as the alpha channel.
You can write out the RGBA data to a PNG file and inspect it using GIMP or your favourite image editor.
Font Texture and Nuklear Callbacks
The RGBA bitmap font can now be converted to an OpenGL texture with linear interpolation and the texture id of the NkUserFont object is set.
I don’t fully understand yet how the “width” and “query” implementations work.
Hopefully I find a way to do a unit-tested reimplementation to get a better understanding later.
On a positive note though, at this point it is possible to render text.
In the following code we add a button for stopping and a button for starting the progress bar.
Here is a screenshot with an integer and a float property.
Symbol and Image Buttons
Nuklear has several stock symbols for symbol buttons.
Furthermore one can register a texture to be used in an image button.
Each button method returns true if the button was clicked.
The combo box in the following screenshot uses one column.
Drawing Custom Widgets
It is possible to use draw commands to draw a custom widget.
There are also methods for checking if the mouse is hovering over the widget or if the mouse was clicked.
Now it is possible to move the cursor in the text box and also delete characters.
Clipboard and other Control Key Combinations
Finally one can implement some Control key combinations.
Except for undo and redo I managed to get the keyboard combinations from the GLFWDemo.java example to work.
We also implement the clipboard integration.
The following screenshot shows the text edit field with some text selected to copy to the clipboard.
Styling
You can get Nuklear styles from Nuklear/demo/common/style.c.
My favourite is the dark theme.
The style is set by populating a color table and then using nk_style_from_table to overwrite the style.
Nuklear does not seem to support nested layouts.
However as shown by Komari Spaghetti one can use groups for nesting layouts in Nuklear.
Basically you just need to set window padding to zero temporarily and disable the scroll bars.
In the following sample there are five buttons using two columns with different button sizes.
The screenshot shows the layout achieved in this case.
Type hints
In order to improve performance, one can use Clojure type hints.
This is especially effective when applied to the implementations of width and query method or the NkUserFont object.
width gets called for each string and query even gets called for each character in the GUI.
One can enable reflection warnings in order to find where type hints are needed to improve performance.
If you are using Clojure, you might be interested in nREPL which lets you connect a REPL terminal to a running Clojure program.
In the following howto I am setting up a small nREPL demo using the nREPL server and the REPL-y nREPL client.
First I set up aliases for the server and client in $HOME/.clojure/deps.edn as follows:
The following program then displays a counter which gets increased once per second.
Furthermore it starts an nREPL server on port 7888.
The program goes into the file $HOME/Documents/repltest/src/repltest/core.clj.
Now one can run the program using clj -M:nrepl -m repltest.core.
The program will print out consecutive numbers as follows:
1
2
3
4
..
.
Now you need to open a second terminal for the nREPL client.
You run the network client using clojure -M:reply.
The important thing which took me some time to find out is that you need to then switch to your applications namespace as follows:
user=>(ns repltest.core)
Now you can easily access the variables of the main program:
repltest.core=> @t
42
You can also modify the value while the main program is still running:
repltest.core=>(swap! t - 10)
32
You should see the counter decrease in the application’s output.
You can even redefine the display methods using the nREPL client.
I.e. you can do interactive development.
