Question Is the ROWID always assigned in an ascending order? Is it possible to determine the sequence of inserted records with the ROWID? Answer The ROWIDs of a table are managed by the DBMS. They are assigned in an ascending order per node and stay distinct within a table. For different tables, they could be the same. DML statements such as UPDATE, DELETE, TRUNCATE or MERGE might internally reorder data storage, invalidating and reassigning all the ROWIDs. Contrary to that, structural table changes such as adding a column leave the ROWIDs unchanged. Altering   distribution keys   of or   reorganizing   a table will certainly reassign ROWID values. Therefore the ROWID can't be used to determine the exact sequence or age of a record, it is designed to be a   short term   identifier for rows to be used for duplicate elimination. The ROWID pseudo column can only be used on   table   objects, not on views or subselects. If you try it on a view, you will receive an appropriate error message Additional References ROWID Syntax
View full article
This article provides steps to manage session-level use of resources when there are a high number of sessions attached to Exasol.
View full article
This article explains the difference between temporary tables, views, and CTE's.
View full article
SOL: Metadata Backup Background How it works The file contains several scripts which will copy the DDL for all objects in the database. It also contains the necessary scripts to restore the metadata. If the config file is set up correctly, "_backup._sh" performs the following tasks: Creates the specified directory. This is where all of the files will be stored Connects to the database using an Exaplus profile (must be created beforehand) Once connected to the database, the script creates a schema "BACKUP_SCRIPTS" and 2 scripts which are used in the backup process. EXPORT statements are generated using the database script "BACKUP_SYS" for several system tables which can be referenced later on. The CSV files are saved in the './ddl/' path. A database "restore_sys.sql" script is created and saved in the './ddl/' path that includes all the commands neccesary to restore the system tables on a new "SYS_OLD" schema. The script executes the database script "METADATA_BACKUP" and creates DDL for all database objects, including schemas, tables, views, users, roles. Limitations in the script are listed at the end. Specifically, this script will read data from system tables and create the necessary CREATE statements which can be executed at a later time. The script creates a new schema, stores the data into a table in this new schema, and then exports the table contents into an SQL file to prevent formatting errors. After the export, the schema is dropped. The DDL and CSV's of the old tables are compressed and saved as a .tar.gz file in the './backups' directory or on a different location if "EXTERNAL_DIR" is set on the "config" file. Limitations Creating DDL based on system tables is not perfect and has some limitations and imperfections. To safeguard incorrect DDL creation, the script also saves the system tables which are used in the script. If a DDL is not created perfectly or if you discover imperfections/errors, you can query the system tables directly and create your own DDL. invalid (outdated) views and their DCL will not be created views and functions for which the schema or object itself was renamed still contain the original object names and may cause an error on DDL execution comments at the end of view text may cause problems GRANTOR of all privileges will be the user that runs the SQL code returned by the script passwords of all users will be 'Start123', except for connections where passwords will be left empty functions with dependencies will not be created in the appropriate order UDF scripts with delimited identifiers (in- or output) will cause an error on DDL execution If authenticating using Kerberos (< 6.0.8), please alter the script BI_METADATA_BACKUPV2 lines 155 and 156 Comments containing apostrophes (single-quote) will cause an error on DDL execution Note: This solution has been updated to include new permissions/rights in version 6.1 Prerequisites Linux system Exaplus installed Database is already created and is able to be connected from the system you are running the scripts Database user, which you will connect to the database with, has the following system privileges: CREATE SCHEMA CREATE TABLE CREATE SCRIPT SELECT ANY DICTIONARY How to create a Metadata Backup Concept? Step 1 Save the attached tar.gz file to a directory of your choice. The only requirements are that the system is Linux-based and that Exaplus command line is installed. Step 2 Unzip the file: tar xf metabackup_vYYYYMMDD.tar.gz Step 3 Create an Exaplus profile with all of the connection details, including database user, password, and connection string. Details on the parameters for Exaplus can be found in the user manual. Example: /usr/opt/EXASuite-6/EXASolution-6.0.10/bin/Console/exaplus -u [YOU_USER] -p [YOUR_PASSWORD] -c [DB IP Address]:8563 -wp metadata_backup_profile Step 4 Change directory to the newly created folder: cd metabackup Step 5 Edit config file with the help of following information: Global Section: EXAPLUS = Path to EXAPLUS excluding the exaplus. For the example above, it would be ' /usr/opt/EXASuite-6/EXASolution-6.0.10/bin/Console' PROFILENAME = The name of the profile created in the previous step as ( metadata_backup_profile   ) Backup Section: EXTERNAL_DIR = The path where the metadata backup will be stored if specified. Can be an external filesystem. Should be mounted or available beforehand SYSPATH = The path where metabackup_vYYYYMMDD.tar.gz was extracted to DB_NAME = The Database Name EXAPLUS_TIMEOUT = Timeout for Exaplus (default 300 seconds). If you want to prevent long-running queries, set the timeout accordingly. Please note, for very large databases, it might take over 5 minutes to run all of the scripts, so please set the timeout higher. EXAPLUS_RECONNECT = Reconnect tries for Exaplus if the connection fails. Default value is set to '1' Step 6 Make .SH files executable chmod 755 *.sh Step 7 Run backup.sh ./backup.sh or bash backup.sh SOL: Metadata Restore Background This script will import the CSV's created in the Backup and run all of the CREATE statements. The script opens an Exaplus session, creates a schema called 'SYS_OLD' containing the same system tables that were created in the backup, and then imports the CSV's into these tables. All of the CREATE statements are executed, which restores the 'structure' of the database, however all tables are empty. Note: During execution, the owner of all objects is the user running the script. At the end of the script, the owner of all schema changes to match the correct owner. To monitor errors, profiling is enabled by default. You can search through EXA_DBA_PROFILE_LAST_DAY to find commands which were rolled back Limitations If the database is not empty, some objects may fail to be created if they already exist in the database. Objects will not be overwritten. Limitations of the create DDL script may cause errors during the CREATE statements. Please check the restore log, profiling or auditing to identify statements which were not created successfully. If restoring to a database running a different version than the database from the backup, the IMPORT of old sys tables may fail due to different columns. Prerequisites Linux system Exaplus command line Database is already created and is able to be connected from the system you are running the scripts on It is recommended to start the database with auditing ENABLED Database user will need extensive CREATE privileges. It is recommended that the user running the scripts has DBA privileges as the following commands will be carried out: CREATE SCHEMA, TABLE, VIEW, SCRIPT, CONNECTION, ETC GRANT How to apply a  Metadata Restore? Step 1 The restore script can be run from the same system you ran the backup on or a different system. If running the restore on a new system, please follow steps 1-4 found in the Backup instructions to set up the files. NOTE: When setting up your exaplus profile, please enter the information for the database you are restoring to. Step 2 Unpack the backup tar into the directory of your choice tar xf ddl-backup-DB_NAME-YYYY-MM-DD-HH-Mi-SS.tar.gz Step 3 Edit config file with the following information from: Backup Section: SYSPATH = The path where metabackup.tar.gz was extracted to DB_NAME = The Database Name Restore Section: BACKUP_RESTORE_PATH = The path that you unpacked the backup file to (should end with '/ddls') RESTORE_SYS_SQL_PATH = The path containing the restore script Step 4 Run restore.sh ./restore.sh or bash restore.sh Additional References https://www.exasol.com/support/secure/attachment/78805/metadatabackup_v20190418.tar.gz  https://community.exasol.com/t5/database-features/create-ddl-for-the-entire-database/ta-p/1417 https://www.exasol.com/support/browse/IDEA-371
View full article
Background Exasol R SDK offers functionality to interact with the Exasol database out of R programs. It extends the ORDBC and offers fast data transfer between Exasol and R ( exa.readData()   and   exa.writeData() ) and it makes it convenient to run parts of your R code in parallel in Exasol using R UDF functions behind the scenes ( exa.createScript() ) Example (Digit recognition) We will use a well-known machine learning algorithm for classifying handwritten digits. It is using a randomforest library, which is used to create decision trees for any kind of image. For a given training set and the actual result for each element, the library returns a decision tree that can be applied to other similar images, the test data. The data is stored pixel-wise (one value for each pixel). This How-To will focus on how to use the EXASolution R SDK and not focus on details of the algorithm or R. Further information on the algorithm and the data can be found   here. The example was chosen to show how easy it is to read or write data from or to EXASolution and how easy it is to deploy a given algorithm to an Exasol UDF. Prerequisites Exasol R SDK and randomForest needs to be installed on the used system. The attached files can be used to setup your Exasol database for the example: Run   01_DDL.sql      (Creates a schema, a table for train data set and a table for test data set) Place csv files ( train.csv     and   test.csv   ) at an appropriate place and change the path to those files in   02_IMPORT.sql   Run   02_IMPORT.sql     (Imports the train and test data set) How to use ExaSolution R SDK R implementation (reading the data sets from an EXASolution database) The attached file   03_RandomForest.r   shows how to use   exa.readData()   and   exa.writeData()   to interact with the database in R. As of Exasol 6.2.x, we needed to modify 1 line in 03_RandomForest.r in order to account for R version differences. Specifically, we are adding a parameter to the save command - version = 2. You are welcome to skip this modification and come back to revisit if you run into issues. Before: save(rf, file = "< path to rf.dat>") After: save(rf, version = 2, file = "< path to rf.dat>")  Additional installation notes for 03_RandomForest.r - installing needed packages. Continuing on, the general procedure is: Step 1. Connect to EXASolution and read the train and test data This requires an ODBC DSN (named "solo" in this example) Step 2. Read train and test data from the database and prepare them for further processing Test data was enriched by an image id Train data was enriched by a label showing the actual handwritten digit Step 3. Use randomForest() to create a decision tree and print prediction In this example, the decision forest is kept and tested on the test data Extract the predictions for test data and print them Step 4. Write decision tree to Exasol We will use a bucket called binary to store the decision forest Detailed information on EXABucket FS and buckets can be found in https://community.exasol.com/t5/environment-management/how-to-create-an-exabucketfs-service-and-bucket/ta-p/1515** This item does not apply to Exasol 6+ and is only left in for archival purposes.   Important Deprecation Notice:  In Version 5 it is not possible to store binary data in EXASOL database. One possibility is to serialize the decision forest and store it in a table. SInce VARCHAR type in EXASOL is limited to 2 million characters, the resulting string should not exceed 2 million characters.  curl -v -X PUT -T rf.dat http: //w:<write_pw>@<ip of one cluster node>:<port>/binary/rf.dat Step 5. Deploying a UDF The other file ( 05_PREDICTINDB.sql ) deploys the functionality to a UDF in EXASOL. We create a variadic UDF receiving an arbitrary amount of input parameters (all pixels) The output is the predicted digit. Load forest from the bucket Create a dataframe for all input columns (pixels) Use   predict()   and emit the predictions Step 6. Using UDF within EXASolution Now it is possible to use the deployed function within EXASolution like any other built-in function: select rf.predict(Model_str, PIXEL0,PIXEL1,PIXEL2,PIXEL3,PIXEL4,PIXEL5,PIXEL6,PIXEL7,PIXEL8,PIXEL9,PIXEL10,PIXEL11,PIXEL12,PIXEL13,PIXEL14,PIXEL15,PIXEL16,PIXEL17,PIXEL18,PIXEL19,PIXEL20,PIXEL21,PIXEL22,PIXEL23,PIXEL24,PIXEL25,PIXEL26,PIXEL27,PIXEL28,PIXEL29,PIXEL30,PIXEL31,PIXEL32,PIXEL33,PIXEL34,PIXEL35,PIXEL36,PIXEL37,PIXEL38,PIXEL39,PIXEL40,PIXEL41,PIXEL42,PIXEL43,PIXEL44,PIXEL45,PIXEL46,PIXEL47,PIXEL48,PIXEL49,PIXEL50,PIXEL51,PIXEL52,PIXEL53,PIXEL54,PIXEL55,PIXEL56,PIXEL57,PIXEL58,PIXEL59,PIXEL60,PIXEL61,PIXEL62,PIXEL63,PIXEL64,PIXEL65,PIXEL66,PIXEL67,PIXEL68,PIXEL69,PIXEL70,PIXEL71,PIXEL72,PIXEL73,PIXEL74,PIXEL75,PIXEL76,PIXEL77,PIXEL78,PIXEL79,PIXEL80,PIXEL81,PIXEL82,PIXEL83,PIXEL84,PIXEL85,PIXEL86,PIXEL87,PIXEL88,PIXEL89,PIXEL90,PIXEL91,PIXEL92,PIXEL93,PIXEL94,PIXEL95,PIXEL96,PIXEL97,PIXEL98,PIXEL99,PIXEL100,PIXEL101,PIXEL102,PIXEL103,PIXEL104,PIXEL105,PIXEL106,PIXEL107,PIXEL108,PIXEL109,PIXEL110,PIXEL111,PIXEL112,PIXEL113,PIXEL114,PIXEL115,PIXEL116,PIXEL117,PIXEL118,PIXEL119,PIXEL120,PIXEL121,PIXEL122,PIXEL123,PIXEL124,PIXEL125,PIXEL126,PIXEL127,PIXEL128,PIXEL129,PIXEL130,PIXEL131,PIXEL132,PIXEL133,PIXEL134,PIXEL135,PIXEL136,PIXEL137,PIXEL138,PIXEL139,PIXEL140,PIXEL141,PIXEL142,PIXEL143,PIXEL144,PIXEL145,PIXEL146,PIXEL147,PIXEL148,PIXEL149,PIXEL150,PIXEL151,PIXEL152,PIXEL153,PIXEL154,PIXEL155,PIXEL156,PIXEL157,PIXEL158,PIXEL159,PIXEL160,PIXEL161,PIXEL162,PIXEL163,PIXEL164,PIXEL165,PIXEL166,PIXEL167,PIXEL168,PIXEL169,PIXEL170,PIXEL171,PIXEL172,PIXEL173,PIXEL174,PIXEL175,PIXEL176,PIXEL177,PIXEL178,PIXEL179,PIXEL180,PIXEL181,PIXEL182,PIXEL183,PIXEL184,PIXEL185,PIXEL186,PIXEL187,PIXEL188,PIXEL189,PIXEL190,PIXEL191,PIXEL192,PIXEL193,PIXEL194,PIXEL195,PIXEL196,PIXEL197,PIXEL198,PIXEL199,PIXEL200,PIXEL201,PIXEL202,PIXEL203,PIXEL204,PIXEL205,PIXEL206,PIXEL207,PIXEL208,PIXEL209,PIXEL210,PIXEL211,PIXEL212,PIXEL213,PIXEL214,PIXEL215,PIXEL216,PIXEL217,PIXEL218,PIXEL219,PIXEL220,PIXEL221,PIXEL222,PIXEL223,PIXEL224,PIXEL225,PIXEL226,PIXEL227,PIXEL228,PIXEL229,PIXEL230,PIXEL231,PIXEL232,PIXEL233,PIXEL234,PIXEL235,PIXEL236,PIXEL237,PIXEL238,PIXEL239,PIXEL240,PIXEL241,PIXEL242,PIXEL243,PIXEL244,PIXEL245,PIXEL246,PIXEL247,PIXEL248,PIXEL249,PIXEL250,PIXEL251,PIXEL252,PIXEL253,PIXEL254,PIXEL255,PIXEL256,PIXEL257,PIXEL258,PIXEL259,PIXEL260,PIXEL261,PIXEL262,PIXEL263,PIXEL264,PIXEL265,PIXEL266,PIXEL267,PIXEL268,PIXEL269,PIXEL270,PIXEL271,PIXEL272,PIXEL273,PIXEL274,PIXEL275,PIXEL276,PIXEL277,PIXEL278,PIXEL279,PIXEL280,PIXEL281,PIXEL282,PIXEL283,PIXEL284,PIXEL285,PIXEL286,PIXEL287,PIXEL288,PIXEL289,PIXEL290,PIXEL291,PIXEL292,PIXEL293,PIXEL294,PIXEL295,PIXEL296,PIXEL297,PIXEL298,PIXEL299,PIXEL300,PIXEL301,PIXEL302,PIXEL303,PIXEL304,PIXEL305,PIXEL306,PIXEL307,PIXEL308,PIXEL309,PIXEL310,PIXEL311,PIXEL312,PIXEL313,PIXEL314,PIXEL315,PIXEL316,PIXEL317,PIXEL318,PIXEL319,PIXEL320,PIXEL321,PIXEL322,PIXEL323,PIXEL324,PIXEL325,PIXEL326,PIXEL327,PIXEL328,PIXEL329,PIXEL330,PIXEL331,PIXEL332,PIXEL333,PIXEL334,PIXEL335,PIXEL336,PIXEL337,PIXEL338,PIXEL339,PIXEL340,PIXEL341,PIXEL342,PIXEL343,PIXEL344,PIXEL345,PIXEL346,PIXEL347,PIXEL348,PIXEL349,PIXEL350,PIXEL351,PIXEL352,PIXEL353,PIXEL354,PIXEL355,PIXEL356,PIXEL357,PIXEL358,PIXEL359,PIXEL360,PIXEL361,PIXEL362,PIXEL363,PIXEL364,PIXEL365,PIXEL366,PIXEL367,PIXEL368,PIXEL369,PIXEL370,PIXEL371,PIXEL372,PIXEL373,PIXEL374,PIXEL375,PIXEL376,PIXEL377,PIXEL378,PIXEL379,PIXEL380,PIXEL381,PIXEL382,PIXEL383,PIXEL384,PIXEL385,PIXEL386,PIXEL387,PIXEL388,PIXEL389,PIXEL390,PIXEL391,PIXEL392,PIXEL393,PIXEL394,PIXEL395,PIXEL396,PIXEL397,PIXEL398,PIXEL399,PIXEL400,PIXEL401,PIXEL402,PIXEL403,PIXEL404,PIXEL405,PIXEL406,PIXEL407,PIXEL408,PIXEL409,PIXEL410,PIXEL411,PIXEL412,PIXEL413,PIXEL414,PIXEL415,PIXEL416,PIXEL417,PIXEL418,PIXEL419,PIXEL420,PIXEL421,PIXEL422,PIXEL423,PIXEL424,PIXEL425,PIXEL426,PIXEL427,PIXEL428,PIXEL429,PIXEL430,PIXEL431,PIXEL432,PIXEL433,PIXEL434,PIXEL435,PIXEL436,PIXEL437,PIXEL438,PIXEL439,PIXEL440,PIXEL441,PIXEL442,PIXEL443,PIXEL444,PIXEL445,PIXEL446,PIXEL447,PIXEL448,PIXEL449,PIXEL450,PIXEL451,PIXEL452,PIXEL453,PIXEL454,PIXEL455,PIXEL456,PIXEL457,PIXEL458,PIXEL459,PIXEL460,PIXEL461,PIXEL462,PIXEL463,PIXEL464,PIXEL465,PIXEL466,PIXEL467,PIXEL468,PIXEL469,PIXEL470,PIXEL471,PIXEL472,PIXEL473,PIXEL474,PIXEL475,PIXEL476,PIXEL477,PIXEL478,PIXEL479,PIXEL480,PIXEL481,PIXEL482,PIXEL483,PIXEL484,PIXEL485,PIXEL486,PIXEL487,PIXEL488,PIXEL489,PIXEL490,PIXEL491,PIXEL492,PIXEL493,PIXEL494,PIXEL495,PIXEL496,PIXEL497,PIXEL498,PIXEL499,PIXEL500,PIXEL501,PIXEL502,PIXEL503,PIXEL504,PIXEL505,PIXEL506,PIXEL507,PIXEL508,PIXEL509,PIXEL510,PIXEL511,PIXEL512,PIXEL513,PIXEL514,PIXEL515,PIXEL516,PIXEL517,PIXEL518,PIXEL519,PIXEL520,PIXEL521,PIXEL522,PIXEL523,PIXEL524,PIXEL525,PIXEL526,PIXEL527,PIXEL528,PIXEL529,PIXEL530,PIXEL531,PIXEL532,PIXEL533,PIXEL534,PIXEL535,PIXEL536,PIXEL537,PIXEL538,PIXEL539,PIXEL540,PIXEL541,PIXEL542,PIXEL543,PIXEL544,PIXEL545,PIXEL546,PIXEL547,PIXEL548,PIXEL549,PIXEL550,PIXEL551,PIXEL552,PIXEL553,PIXEL554,PIXEL555,PIXEL556,PIXEL557,PIXEL558,PIXEL559,PIXEL560,PIXEL561,PIXEL562,PIXEL563,PIXEL564,PIXEL565,PIXEL566,PIXEL567,PIXEL568,PIXEL569,PIXEL570,PIXEL571,PIXEL572,PIXEL573,PIXEL574,PIXEL575,PIXEL576,PIXEL577,PIXEL578,PIXEL579,PIXEL580,PIXEL581,PIXEL582,PIXEL583,PIXEL584,PIXEL585,PIXEL586,PIXEL587,PIXEL588,PIXEL589,PIXEL590,PIXEL591,PIXEL592,PIXEL593,PIXEL594,PIXEL595,PIXEL596,PIXEL597,PIXEL598,PIXEL599,PIXEL600,PIXEL601,PIXEL602,PIXEL603,PIXEL604,PIXEL605,PIXEL606,PIXEL607,PIXEL608,PIXEL609,PIXEL610,PIXEL611,PIXEL612,PIXEL613,PIXEL614,PIXEL615,PIXEL616,PIXEL617,PIXEL618,PIXEL619,PIXEL620,PIXEL621,PIXEL622,PIXEL623,PIXEL624,PIXEL625,PIXEL626,PIXEL627,PIXEL628,PIXEL629,PIXEL630,PIXEL631,PIXEL632,PIXEL633,PIXEL634,PIXEL635,PIXEL636,PIXEL637,PIXEL638,PIXEL639,PIXEL640,PIXEL641,PIXEL642,PIXEL643,PIXEL644,PIXEL645,PIXEL646,PIXEL647,PIXEL648,PIXEL649,PIXEL650,PIXEL651,PIXEL652,PIXEL653,PIXEL654,PIXEL655,PIXEL656,PIXEL657,PIXEL658,PIXEL659,PIXEL660,PIXEL661,PIXEL662,PIXEL663,PIXEL664,PIXEL665,PIXEL666,PIXEL667,PIXEL668,PIXEL669,PIXEL670,PIXEL671,PIXEL672,PIXEL673,PIXEL674,PIXEL675,PIXEL676,PIXEL677,PIXEL678,PIXEL679,PIXEL680,PIXEL681,PIXEL682,PIXEL683,PIXEL684,PIXEL685,PIXEL686,PIXEL687,PIXEL688,PIXEL689,PIXEL690,PIXEL691,PIXEL692,PIXEL693,PIXEL694,PIXEL695,PIXEL696,PIXEL697,PIXEL698,PIXEL699,PIXEL700,PIXEL701,PIXEL702,PIXEL703,PIXEL704,PIXEL705,PIXEL706,PIXEL707,PIXEL708,PIXEL709,PIXEL710,PIXEL711,PIXEL712,PIXEL713,PIXEL714,PIXEL715,PIXEL716,PIXEL717,PIXEL718,PIXEL719,PIXEL720,PIXEL721,PIXEL722,PIXEL723,PIXEL724,PIXEL725,PIXEL726,PIXEL727,PIXEL728,PIXEL729,PIXEL730,PIXEL731,PIXEL732,PIXEL733,PIXEL734,PIXEL735,PIXEL736,PIXEL737,PIXEL738,PIXEL739,PIXEL740,PIXEL741,PIXEL742,PIXEL743,PIXEL744,PIXEL745,PIXEL746,PIXEL747,PIXEL748,PIXEL749,PIXEL750,PIXEL751,PIXEL752,PIXEL753,PIXEL754,PIXEL755,PIXEL756,PIXEL757,PIXEL758,PIXEL759,PIXEL760,PIXEL761,PIXEL762,PIXEL763,PIXEL764,PIXEL765,PIXEL766,PIXEL767,PIXEL768,PIXEL769,PIXEL770,PIXEL771,PIXEL772,PIXEL773,PIXEL774,PIXEL775,PIXEL776,PIXEL777,PIXEL778,PIXEL779,PIXEL780,PIXEL781,PIXEL782,PIXEL783) from rf.test cross join rfmodels.models; PRED 0 0 3 0 3 3 0 0 1 7
View full article
This article summarizes Exasol's basic resource management which includes CPU, RAM, Network, Disks and Runtime, Priority Groups.
View full article
Problem You have successfully created a LDAP user, but the user cannot log in. There might be a configuration problem with your LDAP server, the distinguished name might be wrong or any other database independent problem. Solution Try to connect/bind to your LDAP server with the ldapsearch command line tool. If you cannot connect with ldapsearch, the database won't be able to connect the LDAP user. Here is a simple example: ldapsearch -v -x -D "uid=Ben,ou=People,dc=myserv,dc=com" -w my_password -H ldap://my_ldap_server.com  
View full article
Questions What data can you read using subconnections? What data can you insert using subconnections? How to establish parallel connections? What about transactions in parallel mode? What about transactions in parallel mode? Answer You can use subconnections to read or insert data in parallel from and into the EXASOL server. This will increase performance considerably. Using this interface you can read/write data in parallel from different physical hosts or from different processes or in one process from different threads. What data can you read using subconnections? You need a statement that produces a large result set. The result set can be then retrieved in parts from the parallel connections. For small results this interface has no advantage. What data can you insert using subconnections? Like when reading data, using this interfaces only makes sense if you want to insert large amounts of data. You can simply insert rows into a table in EXASOL using a prepared insert statement. The data coming through the parallel connections will be put together by the server into the right table. How to establish parallel connections? First you need a normal JDBC connection. On this you can use the method   EnterParallel()   to start operating with subconnections. You will receive connection strings for all new parallel connections you can start now. Start the subconnections with auto commit off. After this you can start reading or sending data, nearly like in a normal connection. Attention: You can specify the maximum number of subconnections in   EnterParallel() . This number may be reduced by the server because only one subconnection is allowed per database node. You have to establish the subconnections by using all connection strings received from   Get Worker Hosts() . Subconnections can only be used after all connections have been established. What about transactions in parallel mode? Start the subconnections with auto commit off. Commits should be made only on the main connection after the subconections have inserted all data and they have closed the prepared statements. An Java example is attached. In the example a main connection reads the connection strings for the subconnections from the server. For each subconnection a thread is started that inserts a few rows. Commit is executed on the main connection. Then other threads read the data from the table.
View full article
This article describes how you convert Exasol's GEOMETRY types to GeoJSON
View full article
This article give suggestions on how to minimize transaction conflicts.
View full article
This article gives general information about Exasol's indices.
View full article
Exasol Virtual Schemas are an abstraction layer that makes external data sources accessible in our data analytics platform through regular SQL commands. This is not limited to RDBM. In this project I show how you can use Virtual Schemas to connect to the openweather REST API.
View full article
Estimating work duration when doing reorganizations within Exasol.
View full article
Background You can access to database backups virtually. The scenario will be like this: Extract tables/data from backup In case the database is configured to use all available memory, the DB RAM needs to be decreased before creating a VR instance. If your system has enough free memory available proceed with step 2 Reducing the DB RAM requires a short downtime of the production database Prerequisites Online backup, Virtual Restore access is not supported for remote backups. Enough free disk space for an EXAStorage DATA volume (used for metadata) Unused Main Memory (DB RAM) The VR instance needs the same amount of active nodes as the backup VR instance database port, e.g. TCP 9563 (please be aware this port should be allowed through the firewall) Recommendation at least 4GiB per node for a small amount of data and max. 5% of the memory of the nodes Amount of VR DB RAM depends on the amount of data to be restored Calculate DB RAM for the VR instance: In this example: each node has 31GiB of Main Memory The amount of data is very small (small table), thus 4GiB DB RAM per node is sufficient (see Prerequisites) The database is running on 4 active nodes and has 112GiB DB RAM   Amount of active nodes * 4GiB = DB RAM VR instance: 4 * 4GiB = 16GiB DB RAM VR instance   How to virtual-access on database backups Step 1: Reducing the DB RAM to allocate RAM for the VR instance DB RAM - DB RAM VR instance = reduced DB RAM: 112GiB - 16GiB = 96GiB   Step 1.1: Shutdown database Select the database and click Shutdown from Services >  EXASolution . Skip this step if the database is already shut down.         Step 1.2: Edit/Adjust DB RAM Wait for the database to be shut down. Then edit database settings: Click on the "Edit" Button Set the DB RAM to 96GiB Apply changes Start the database   Step 2: Create DATA volume for the VR instance Go to the database properties:   Note down amount of active database nodes: Step 2.1: Create EXAStorage DATA volume Click on EXAStorage and then click on "Add Volume": Redundancy 1 Allowed Users e.g. Admin Read-only Users: None Priority 10 Volume Type: DATA Volume Size: 20GiB Nodes List: Active database nodes Number of Master Nodes: Same amount as active database nodes Block Size: None Disk: e.g. d03_storage   Step 3: Create EXAStorage DATA volume   EXASolution - Add DB name: VR Active Nodes: Same amount as active database nodes Node List: Active database nodes EXAStorage DATA Volume: Choose the previously created volume ID   Network Interfaces: Same as the active database Connection Port: e.g. 9563 it must differ from production   DB RAM: 16GiB Step 4: Create EXAStorage DATA volume Click on the newly created database Click on "Backups" Click on "Show foreign database backups" Select backup you want to restore (Ensure backup dependencies are fulfilled) Restore Type: Virtual Access Click on Restore, this will start the VR instance Wait for the database to become online Step 5: Example Import/Export data IMPORT: CREATE OR REPLACE TABLE SCHEMA . TABLE AS SELECT * FROM ( import from EXA at ' CONNECTION - STRING :PORT' USER 'myuser' IDENTIFIED BY "mypass" table SCHEMA . TABLE ); EXPORT: EXPORT SCHEMA . TABLE INTO EXA at ' CONNECTION - STRING :PORT' USER "myuser" IDENTIFIED BY "mypass" TABLE SCHEMA . TABLE ; Step 6: Cleanup VR database instance Shutdown VR instance Delete VR instance Delete EXAStorage DATA volume of the VR instance Afterward, increase DB RAM for production
View full article
SQL statement to add locking information to your session system tables using the EXA_SQL_LAST_DAY statistics.
View full article
Question Does Exasol index Null values? Answer Exasol's indices contain NULL values. If you add a row with a NULL value in an index column, you will see an increase in MEM_OBJECT_SIZE in system table EXA_USER_INDICES. If there are a lot of NULLs, they will be compressed inside the index. The NULLs of the index have rather few application scenarios as filters and joins typically omit NULLs (even before accessing the index). The NULLs are only relevant in some corner cases like UNION processing or so-called "OR NULL" equi-joins. Additional References https://www.exasol.com/support/browse/EXASOL-2197 https://community.exasol.com/t5/database-features/null-in-exasol/ta-p/363
View full article
This article talks about the use of UNION ALL at its benefits.
View full article
This article gives an example of using the Preprocessor function to automatically handle unsupported functions in Exasol
View full article
Question With Exasol's automated disk and memory management, it its not always easy to answer questions like How much data is stored in the database? How much disk space do I use? How much memory is in use? How much disk space is left? And how much more data can I actually store? Answer This article addresses a few of those questions by giving an overview of the relations between the different values one can read from Exasol's system tables or web management interface (EXAoperation). Overview Have a look at the following diagram showing the general relationships between many of the keywords used when talking about data sizes in Exasol:       Notes: The ratio between   RAW_   and   MEM_OBJECT_SIZE   is what Exasol commonly defines as   compression ratio The distinction between   RAW   and   MEM   sizes also applies to system data: Everything is   compressed in RAM   and mapped to/from disk. Usually there is no need to fit all of the data into DB RAM. Systems with a large amount of /passive data/ may perform well with 10% of   DB_RAM   compared to overall   MEM_OBJECT_SIZE A typical "first guess" estimation for required   DB_RAM   is about 10% of the expected   RAW_OBJECT_SIZE While   deleted blocks   in storage are reused by later operations, as a general rule, storage volumes   do not shrink   unless forced to do so. In   DB_RAM , the ratio between active data and temp data is flexible, but temp is hard-limited at 80%. Information Location Object Level Metric Location Comment RAW_OBJECT_SIZE System table   EXA_DBA_OBJECT_SIZES Theoretical value, never actually required MEM_OBJECT_SIZE System table   EXA_DBA_OBJECT_SIZES   Index size AUXILIARY_SIZE   in System table   EXA_DBA_INDICES   Statistics + Audit Size STATISTICS_SIZE   in System table   EXA_STATISTICS_OBJECT_SIZES   HDD_READ / HDD_WRITE N/A   Database Level Metric Location Comment RAW_OBJECT_SIZE System table   EXA_DB_SIZE_LAST_DAY   MEM_OBJECT_SIZE System table   EXA_DB_SIZE_LAST_DAY   Index Size System table   EXA_DB_SIZE_LAST_DAY   Statistics+Audit Size System table   EXA_DB_SIZE_LAST_DAY   HDD_READ / HDD_WRITE System Table   EXA_MONITOR_LAST_DAY   DB_RAM Size System table   EXA_SYSTEM_EVENTS You can assume that the database will always "use" all that RAM and does not yield to others. RECOMMENDED_DB_RAM_SIZE System table   EXA_DB_SIZE_LAST_DAY   TEMP_DB_RAM_SIZE System Table   EXA_MONITOR_LAST_DAY == (DB_RAM:temp + Temporary:used) DB_RAM:active data N/A   DB_RAM:temp see Node Level below   Persistent:size STORAGE_SIZE   in   EXA_DB_SIZE_LAST_DAY   Persistent:committed see Node Level below ~= ( MEM_OBJECT_SIZE + AUXILIARY_SIZE + STATISTICS_SIZE ) Persistent:deleted see Node Level below size = committed * (100 / USE )   Node Level EXASOL 6.0.0   introduces a new system table for more detailed storage usage:   EXA_VOLUME_USAGE Please refer to the user manual or column comments for details on the columns. Metric Location Comment Persistent:size sum(VOLUME_SIZE) Size is always synchronized across nodes Persistent:committed data sum(COMMIT_DATA)   Temporary:used sum(SWAP_DATA)   Temporary:unused sum(UNUSED_DATA)     Storage Level Metric Location Comment Persistent Volume Size EXAStorage -> (Labels: <dbname>_persistent) -> Size == Persistent:size from Database level Temporary Volume Size EXAStorage -> (Labels: <dbname>_temporary) -> Size   Exists Slave Segment? EXAStorage -> persistent -> Redundancy Gives total number of copies: 1 == no redundancy Free Disk Space EXAStorage -> Space on Disks -> sum(sum:free)   Free database (disk) space See  how-to-monitor-free-database-disk-space      
View full article
Problem JSON data that is stored in EXASOL tables can be accessed through UDFs. JSON documents can be accessed through path expressions. To understand what path expressions are, let us have a look at the following JSON document: { "name" : "Bob" , "age" : 37, "address" :{ "street" : "Example Street 5" , "city" : "Berlin" }, "phone" :[{ "type" : "home" , "number" : "030555555" },{ "type" : "mobile" , "number" : "017777777" }], "email" :[ "bob@example.com" , "bobberlin@example.com" ] } The JSON document contains five fields: “name” (a string), “age” (an integer), “address” (a document), “phone” (an array containing documents) and “email” (an array containing strings). Path expressions start with a dollar sign ($) representing the root document. The dot operator (.) is used to select a field of a document, the star in box brackets ( [*] ) selects and unnests all elements of an array. The following path expression finds all phone numbers: $.phone[*].number This expression is evaluated as follows: path step result $ { "name" : "Bob" , "age" : 37, "address" :{ "street" : "Example Street 5" , "city" : "Berlin" }, "phone" :[{ "type" : "home" , "number" : "030555555" },{ "type" : "mobile" , "number" : "017777777" }], "email" :[ "bob@example.com" , "bobberlin@example.com" ] } $.phone [{ "type" : "home" , "number" : "030555555" },{ "type" : "mobile" , "number" : "017777777" }] $.phone[*] { "type" : "home" , "number" : "030555555" } { "type" : "mobile" , "number" : "017777777" } $.phone[*].number "030555555" "017777777" Solution This solution presents a generic Python UDF json_table to access field values in JSON documents through path expressions. The json_table function has the following form: select json_table( <json string or column >, < path expression>, < path expression>, ... ) emits (< column_name > <data_type>, < column_name > <data_type>, ...) The JSON_TABLE UDF attached to this solution takes a VARCHAR containing JSON data as a first parameter and one or more path expressions: create or replace python scalar script json_table(...) emits(...) as The function can be called in a SELECT query. The EMITS clause has to be used to define the output column names and their data types. SELECT json_table('{ " name " : "Bob" , "age" : 37, "address" :{ "street" : "Example Street 5" , "city" : "Berlin" }, "phone" :[{ " type " : "home" , " number " : "030555555" },{ " type " : "mobile" , " number " : "017777777" }], "email" :[ "bob@example.com" , "bobberlin@example.com" ]} ',' $.phone[*]. number ') EMITS (phone VARCHAR (50)); When the JSON data is stored in a table, the first parameter of JSON_TABLE contains the column name: CREATE TABLE example_table (column_a INT , json_data VARCHAR (2000000)); -- INSERT INTO example_table VALUES (1, '{ " name ": "Bob",…' ); ( as above) SELECT json_table(json_data, '$.phone[*]. number ' ) EMITS (phone VARCHAR (50)) FROM example_table; It is possible to use both the json_table UDF and normal columns of the table within the SELECT clause: SELECT column_a, json_table(json_data, '$.phone[*]. number ' ) EMITS (phone VARCHAR (50)) FROM example_table; When a row in the input table consists of n phone numbers within the JSON column, there will be n output rows for that tuple. The value of column_a is constant for all those rows: COLUMN_A PHONE 1 030555555 1 017777777 The following table shows some more valid path expressions: path result $.name "Bob" $.address { "street" : "Example Street 5" , "city" : "Berlin" } $.address.city "Berlin" $.email [ "bob@example.com" , "bobberlin@example.com" ] $.email[*] "bob@example.com" "bobberlin@example.com" This query converts the JSON data into column values: SELECT json_table(json_data, '$. name ' , '$.age' , '$.address.city' ) EMITS ( name VARCHAR (500), age INT , city VARCHAR (500)) FROM example_table; NAME AGE CITY Bob 37 Berlin When unnesting an array, the values from different levels stay the same for every array element: SELECT json_table(json_data, '$. name ' , '$.age' , '$.address.city' , '$.email[*]' ) EMITS ( name VARCHAR (500), age INT , city VARCHAR (500), email VARCHAR (500)) FROM example_table; NAME AGE CITY EMAIL Bob 37 Berlin bob@example.com Bob 37 Berlin bobberlin@example.com The result of unnesting more than one array is the cross product of those arrays: SELECT json_table(json_data, '$. name ' , '$.age' , '$.address.city' , '$.email[*]' , '$.phone[*]. type ' , '$.phone[*]. number ' ) EMITS ( name VARCHAR (500), age INT , city VARCHAR (500), email VARCHAR (500), phone_type VARCHAR (50), phone_number VARCHAR (50)) FROM example_table; NAME AGE CITY EMAIL PHONE_TYPE PHONE_NUMBER Bob 37 Berlin bob@example.com home 030555555 Bob 37 Berlin bob@example.com mobile 017777777 Bob 37 Berlin bobberlin@example.com home 030555555 Bob 37 Berlin bobberlin@example.com mobile 017777777 Details and limitations: When the JSON input is not a single document but an array, the elements of the array can be accessed via $ [*] It is recommended to use the correct data types in the EMITS clause. Otherwise, casting is done which can lead to type-conversion errors. When accessing non-atomic values, i.e. arrays without unnesting them, or documents, they are returned as a VARCHAR containing the result in JSON format. Accessing multi-dimensional arrays is not supported. This means, at most one [*] can be used in a path expression. Additional References https://docs.exasol.com/advanced_analytics/accessing_json_data_udfs.htm https://docs.exasol.com/sql_references/functions/alphabeticallistfunctions/json_value.htm https://community.exasol.com/t5/data-science/parsing-json-data-with-python/ta-p/1247 https://community.exasol.com/t5/discussion-forum/sneakpreview-on-version-7-json-functions/m-p/1710
View full article
Top Contributors