What you will learn in this blog
Every good software developer knows that splitting complex software into modules is necessary to achieve good code quality. Lua is no exception to this rule.
Code reuse also dictates modularization. And of course you don't want to constantly have to reinvent the wheel for basic tasks like consuming a web service. There are modules made by other developers, ready and tested for that purpose.
Lua modules are Lua files that follow a simple set of rules that allow the module loader of Lua to locate and load them. First of all there are a naming conventions for the files and their paths. Lua module names are directly derived from the filename and relative path to the Lua search path.
As with most search paths, the
LUA_PATH is actually a semicolon-separated collection of filesystem paths. Lua scans them in the order they are listed to find a module. If the module exists in multiple paths, the module found first wins and Lua stops searching.
Under Linux a typical
LUA_PATH contains entries like these:
/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;[...]and many more. You get the idea. Under MacOS and Windows the individual elements might be different, but the concept is the same. Notice that the entries can contain the "
?" placholder and must end in "
A file that from the perspective of the
LUA_PATH can be found under
exasolvs/query_renderer.lua is treated by Lua as if you named it
Implementation-wise a module is a Lua regular table. This means you can do everything with it a regular table allows. Most of the time you will call a module's functions though. Those are regular Lua functions associated with a key in the table.
The global function
require(...) tells the module loader to locate the module, load it if it hasn't been loaded already and hand you back the table that represents the module.
Lua remembers which packages are loaded in
package.loaded. Keep that in mind, we will come back to this later.
Exasol comes with the following LuaRocks packages preinstalled:
So if you need JSON support or TCP sockets, you don't need to install anything yourself.
If you haven't done that yet, I recommend you install LuaRocks.
Make sure that the LuaRocks module path is in the
luarocks command line utility can help you with this task by displaying the exports that you need to set.
You probably want those path changes to be persistent, so depending on your OS you will have to set the environment variables in a place that is read at OS or user session start.
After that installing a package is a simple matter of issuing the following command:
sudo luarocks install <package-name>
I am assuming here that you want to install the package globally. If you want it to be visible only for your user or you do not have administrative privileges, you simply add the
brew update brew install luarocks
There is also the option to install the software by hand, but that is a little bit outside the scope of this article.
You can use LuaRocks under Windows too. Since the installation is more complex, please refer to the Windows installation instructions on the Lua homepage.
The world is big and available modules are numerous. But how exactly do you get new modules into Exasol that are not prepackaged with the Exasol distribution?
You might have noticed that Exasol's Lua implementation does not allow filesystem access — and it does so for a good reason: security.
If you are now asking yourself how on earth you are supposed to use modules if you can't install them, let me show you a neat trick using the clever and flexible design of the Lua language.
The best way to handle modules that your Lua script depends on is to package them directly with the script. This way you make sure that versions match and that the script is self-sufficient. That also makes installing and uninstalling packages trivial.
There is one small caveat that we will discuss at the end of the article. For now let's focus on how to make a neat all-in-one bundle.
If your are looking for ways to bundle modules with your Lua scripts, you will probably come across a tool with the somewhat strange name "amalg". While that certainly does not roll of the tough too nicely, this little gem is actually quite useful.
Amalg is a single-file script written in Lua that takes a set of modules and optionally a script and embeds them into a single script.
To do so, it makes clever use of the transparency of Lua's module loading mechanisms by wrapping the source code twice: first per module in a
do ... end block to avoid scope collisions and then in a loader function. Afterwards amalg directly associates that loader function with the module name in the
Let's look at an example below. I indented the code for better readability. Amalg does not do that.
do local _ENV = _ENV package.preload["remotelog"] = function( ... ) local arg = _G.arg; -- actual module implementation end end
When you run such a script, the code of the module is stored in the loader function, assigned to a key in
package.preload that happens to be the module name.
When your use
require("<module-name>"), Lua first checks if the module is already loaded in
packages.loaded. If it is not, it next checks if a loader is registered in
package.preload. In our case it now is. Lua calls the registered loader function, and returns the module reference to the caller of
As you can see, this variant works without extra filesystem access and is therefore perfectly suited for our Exasol scenario.
That being said, there is a tweak that is necessary in all Exasol versions prior to 7.1. Unlike in regular Lua installations the loader that checks
package.preload is not available by default, so we need to register a function for that purpose first.
That means you need to prepend all bundled Lua scripts in Exasol up to and including 7.0 with the following lines:
table.insert(package.loaders, function (module_name) local loader = package.preload[module_name] if not loader then error("Module " .. module_name .. " not found in package.preload.") else return loader end end )
Note that in version 5.2
package.loaders has been renamed to
package.searchers. While the new name definitely is more precise, this is a breaking change. If you want to support 5.1 and 5.2+, I suggest a version switch.
Now that we have all the pieces in place, let's look at how you actually use amalg in a real-world scenario.
You can find a complete command line reference in the project's README.
Let's focus on a typical case when creating Lua scripts. For demonstration purposes I will show a minimal example that uses Exasol's
remotelog. You might remember that module from the previous article in this series.
sudo luarocks install amalg
sudo luarocks install remotelog
main.luathat logs the Lua version number via
log = require("remotelog") log.connect("172.17.0.1", 3000) log.info(_VERSION)
amalg.lua -o bundle.lua -s main.lua remotelog
-o switch defines the output file,
-s includes a regular Lua script and the last parameter names a Lua module that should be bundled. Here you can list as many modules as you want. It is required though, that the module is in the
If you installed LuaRocks currectly and remembered to add the Lua rocks to the search path, amalg will find modules downloaded as Lua rocks.
You now have a combination of the
remotelog module and the script
main.lua in the file
The last piece of the puzzle is creating a Lua script in Exasol that uses the bundle you just created. You can find the syntax for creating Lua scripts in our doc portal.
Here is a concrete example.
CREATE LUA SCRIPT "BUNDLE" () AS table.insert(_G.package.loaders, function (module_name) local loader = package.preload[module_name] if not loader then error("Module " .. module_name .. " not found in package.preload.") else return loader end end ) -- insert the bundle here /
The command starts with
CREATE LUA SCRIPT <name> AS. Then we register the searcher function that checks
package.prepload after that you insert the Lua code amalg produced and finally you end the command single dash in a separate line.
It's not all just fun and games. Let's talk about serious matters for a moment — security updates and licenses in dependencies. If you start bundling external libraries, you have to be clear that you are responsible for what ends up in the bundle.
First of all make sure you read and understand the licenses your external dependencies have. The original authors deserve that you respect those licenses and only redistribute what you are allowed to.
Some software versions don't age as well as others and you have to be prepared for security vulnerabilities being found in your dependencies. When you bundle your software with external modules, you are also responsible for reacting to security fixes in your dependencies by creating and releasing updated versions of the bundle. Even your code did not change, you still should release bundle updates as quickly as possible once a security update in one of the bundled modules is available.
Bundling modules with your Lua scripts is easy thanks to "amalg". And it gets around the restriction that Exasol Lua scripts are not allowed to access the disk to load modules.
If you bundle modules, you become a distributor, making you responsible for checking the 3rd party module licenses and providing security updates for the bundles if vulnerabilities are found in the modules you bundle.
In the next part of our series we will discuss unit testing Lua scripts and why that is especially valuable in the context of database scripts.