What´s your favorite way to schedule "exa-jobs" ?

[mwellbro]

Hi all,

I was wondering: what is your preferred way of scheduling scripts on exasol without resorting to an external scheduler ?

I´m thinking along the lines "scripts that normally should run periodically while the DB instance is up", like the session_watchdog for example or any script that dumps "snapshoted" tables to somewhere ( while EXA_DB_SIZE_HOURLY is nice, a "per object analysis" does pack a lot more "argumentative punch" ).

We could of course use any external capabilities like a scheduler or a cron job on some other machine but that does create a dependency where there shouldn´t be any...

Do any of you do something along these lines ? Also: how would you/ would we bootstrap something like that "native to/on exa" ?
Perhaps there is a parameter which would allow us to start a JAR from BucketFS or anything like that ?

Looking forward to your input 🙂

[exa-ThomasM]

Hi @mwellbro 

Generally, the management node is not designed for executing jobs periodically (on a high frequency). This node is usually undersized for this purpose. In most situations, it will work out when executing a job on a lower frequency. Unfortunately, due to the undersizing of this machine, we do not recommend to schedule jobs on a high frequency, since this will cause the machine to have a higher load of RAM or disk usage. This might cause the clean executing of these jobs and furthermore when connecting to the DB also block the DB (using an active session slot).

Recommended way to schedule and execute these jobs from outside of the cluster. You will find a CLI Exaplus version within our download portal.

This might not be, what you want to hear within this thread. 😁 Sorry about that!

[mwellbro]

Thanks for your input, @exa-ThomasM !

What would you consider high frequency ? I guess once per minute is already be too much ?

The background for this thread is actually that we are scheduling everything outside our cluster, but once the scheduling system goes into "maintenance mode" ( oversimplified ) then there no longer is a watchdog "protecting" the DB instance.
It would "feel" much better to just know once the DB is running such things are taken care of ( or, for that matter, there will always be daily dumps of the exa_dba_object_sizes without the need to configure that on an external scheduler 😊 ).

Well, I´ll look forward to any further comments here, even though so far you all just tell me off....sniff...😉

[exa-Nico]

Once per minute is too much 🙈 

I think that was the frequency that caused the problems I alluded to...

[mwellbro]

"Once per minute is too much" for the fastest database around ? <pirateVoice>ARRRRR</pirateVoice> 😉

[exa-ThomasM]

@mwellbro 

Sorry for the late reply.

Yes, as @exa-Nico said, this was the frequency, which caused an issue.

If you're extending the hardware (RAM/CPU) for the management node, you could try it out and hope everything goes well. Ofc the specs are depending on the level of usage of the scripts.

[exa-Nico]

Maybe not what you want to hear @mwellbro but I have just always used exaplus + cron (like described here).

I'm not aware of any special parameters like that.... I think there has to be a client to trigger a Lua script or UDF (even if these sessions are simply permanently running and use a combination of sleep() and a schedule table to do jobs). So there is still that dependency...

[mwellbro]

Hi @exa-Nico ,

yeah, not what I wanted to hear 😉

But as per your link: "(...)One method is to install Exaplus CLI on a Linux environment(...)" - why not use the good old licence server which doesn´t have that much to do anyways ? We cover linux, I´d think it´s more than capable of running an exaplus and we´ve got cron-tab - all bases covered, I´d say 😉

We´d have created a self-sustained exa-scheduler 😊

Of course not intended for ETL/ELT or any shenanigans like that, but routine maintenance scripts ? 

[exa-Nico]

Wellll.... there have been cases where heavy cron job use on the license server have caused some problems, which also impacted the database. 

A small, separate server is better 😊

Basically, use at your own risk...

Maybe @exa-ThomasM can add something here 😄