Fully managed cloud. 30-day FREE full featured trial. Start Now
Showing results for 
Search instead for 
Did you mean: 

Closure function in a Lua's iterator function and a variable scope


Hi all,

I just want to share my vision and ask Exasol experts to aprove it or comment it.

Variable scope in the nested functions, classes, objects, etc are hard point in any programming language especially in a scripting language.
The behavior could be significantly diffirent even it looks the same in both.
The questions about iterators in DEV course. We have simple examples and real function in Query Wrapper lib.

In Lua docs here are definitions and examples could help a lot to understand how it works  lua.org/pil/6.1.html

My first look on the example in DEV course was misleaded. I see in the wrapper code that function each time execute a query.
I see it could not be effective. 
But it will not happens with anonymouse closure function call in return clause. Function adds only position variable.
Thus we have in memry one time executed query results and pointer ("pos").

The "Closure" function in the exaple has "i" variable and it is initialized once on the first call.
The "magic" is that we don't do it again when we call an anonymouse function in the Retrun.
Next time called function iterates because "i" variable current value is stored "somewhere" in a memory.
And only anonymouse function will be executed each time we call main function (first-class function).

So its why it is good way to program own iterator. We can store in memory only additional variables.

Thanks for comments and/or answers.

But my question when and/or how free memory? Do we need to control it or Exasol's or the wrapper lib controls it?

Exasol Wrapper git Query Wrapper ithub.com/exasol/etl-utils
From lua.org/pil/6.1.html is some explanation how "upvalue" named variable stored.

"Now, the anonymous function uses an upvalue, i, to keep its counter. However, by the time we call the anonymous function, i is already out of scope, because the function that created that variable (newCounter) has returned. Nevertheless, Lua handles that situation correctly, using the concept of closure. Simply put, a closure is a function plus all it needs to access its upvalues correctly. If we call newCounter again, it will create a new local variable i, so we will get a new closure, acting over that new variable:"

It means that the function called


Team Exasol
Team Exasol

Hi Alexey,

a few points on your topic:

  • Lua is using the garbage collector approach. Given that, you can never be sure when exactly memory gets freed, but memory allocated for some value (X=5) becomes eligible for deallocation when
    • the storing variable is assigned a new value (X=10)
    • the variable is unassigned (X=nil)
    • the program leaves the variable's scope (function / code block)
  • The scope of a closure created in the "header" of a loop ("for value in iterator(base)") is that loop.
    • Note that value_iterator and row_iterator in query wrapper will accept either a precalculated result set and an SQL string. In case of the string, the result set will be part of the closure and automatically gets unset when the loop finishes.
  • Results of query and pquery are simple handles within Lua, the actual result set data is stored across the cluster within database memory.
    • When Lua frees the handle, Exasol destroys the result set.
    • There is a limit of 250 open result set handles for Lua scripts
  • While Lua creates copies of values in almost any operation / assignment, this is not the case when copying rows of a result set ("rowData = result[rowNum]")
    • Exasol is using reference counters and object graphs for its own garbage collector
    • If your Lua script keeps such copies/references, the result set does not get closed (refer to query wrapper functions write_log_details and wrap_ps_execute)
    • If you need to keep data from the result set, make sure to copy all the single values instead of the whole row.

-- Stefan