Environment Management
Manage the environment around the database, such as Cloud, Monitoring, Exaoperation and scalability
cancel
Showing results for 
Search instead for 
Did you mean: 
Backup Synchronization Plugin Installation Overview The backup synchronization process involves synchronization of data and metadata between clusters on different networks. This document provides you with the steps on how to install the Backup Synchronization plugin on two clusters using XML-RPC. XML-RPC lets you establish communication between the two clusters quickly and easily. Exasol’s XML-RPC support is implemented using the xmlrpclib (http://docs.python.org/library/xmlrpclib.html) library that is included with Python 2.2 and later. Recommendations Knowledge of XML-RPC Knowledge of SSH Authentication Plugin Installation You can follow the below steps to install the backup synchronization plugin on clusters. As an example, let us consider two clusters EDU01 and EDU02. Step 1: Uninstall Older Plugins Versions on All Clusters Any older plugin installed on the clusters must be uninstalled before you can install the latest plugin. The following sample script deactivates and uninstalls the older plugins on the two clusters EDU01 and EDU02.       Sample Script Sample Script to Uninstall Plugin on EDU001from xmlrpclib import Server as xmlrpc import ssl from pprint import pprint as pp from base64 import b64encode server = xmlrpc( 'https: //FTPbackup:***PW***@10.60.101.10/cluster1/' ) pp(server.showPluginList()) # [ 'Administration.BackupSync-1.0.1' ] pname = 'Administration.BackupSync-1.0.1' nlist = server.getNodeList() pp([[node] + server.callPlugin(pname, node, 'STATUS' , '') for node in nlist]) ##[[ 'n0011' , 0, 'activated' ], ## [ 'n0012' , 0, 'activated' ], ## [ 'n0013' , 0, 'activated' ]] pp([[node] + server.callPlugin(pname, node, 'DEACTIVATE' , '') for node in nlist]) pp([[node] + server.callPlugin(pname, node, 'UNINSTALL' , '') for node in nlist]) exit()  Sample Script to Uninstall Plugin on EDU02from xmlrpclib import Server as xmlrpc import ssl from pprint import pprint as pp from base64 import b64encode server = xmlrpc( 'https: //FTPbackup:***PW***@10.60.102.10/cluster1/' ) pp(server.showPluginList()) pname = 'Administration.BackupSync-1.0.1' nlist = server.getNodeList() pp([[node] + server.callPlugin(pname, node, 'DEACTIVATE' , '') for node in nlist]) pp([[node] + server.callPlugin(pname, node, 'UNINSTALL' , '') for node in nlist]) exit()    Note: All credentials and files names used in the sample scripts are example credentials and file names. Please replace them with the correct one for your clusters. Step 2: Remove Plugin from EXAoperation Once you have deactivated and uninstalled the previously installed plugin, you must remove the entry of the plugin from EXAoperation for all the clusters. Doing this will clear the folders directories previously created.  You can follow the below steps to remove the plugin from EXAoperation: Log in to EXAoperation for the desired cluster and navigate to Software under Configuration On the right-hand side, select the checkbox next to the plugin you want to remove. Click Delete to remove the plugin. Step 3: Upload Latest Plugin to EXAoperation You can receive the latest Backup Synchronization Plugin by contacting the Exasol support team. To be able to use the Backup Synchronization Plugin, it must be first uploaded to EXAoperation and then installed on the clusters. The lasted plugin must be uploaded through EXAoperation on all clusters (in this example, it must be uploaded to EDU01 and EDU02) by following these steps: In EXAoperation, navigate to Software under Configuration On the right-hand side of the screen, click Choose File next to Software Update File and click Submit. Once the upload is complete, you can see the plugin is listed in the below section of the screen.  Step 4: Install the Latest Plugin After the latest plugin is uploaded through EXAoperation (refer to step 3), you must install it on both the clusters (EDU01 and EDU02). The following sample scripts will install the latest plugins on the clusters.  Sample Script to Install Plugin on EDU01nlist = server.getNodeList() pp(server.showPluginFunctions(pname)) { 'ACTIVATE' : 'Activate plugin.' , 'DEACTIVATE' : 'Deactivate plugin.' , 'GETLOG' : 'Return the log output.' , 'INSTALL' : 'Install plugin.' , 'SSHKEY' : 'Prepare public key for ssh.' , 'STATUS' : 'Show status of plugin (not installed, activated, not activated).' , 'UNINSTALL' : 'Install plugin.' , 'UPLOAD_CONFIG' : 'Upload configuration.' , 'UPLOAD_KEY' : 'Upload public key for ssh' } pp([[node] + server.callPlugin(pname, node, 'INSTALL' , '') for node in nlist]) pp([[node] + server.callPlugin(pname, node, 'STATUS' , '') for node in nlist]) Sample Script to Install Plugin on EDU02nlist = server.getNodeList() pp([[node] + server.callPlugin(pname, node, 'INSTALL' , '') for node in nlist]) pp([[node] + server.callPlugin(pname, node, 'STATUS' , '') for node in nlist]) Step 5: Create a User in EXAoperation You must create a new user with admin role. The user account will be used to access the Exasol backup via FTP. You can use any existing user account with admin role, however, creating a new user for this purpose makes it easy to segregate specific tasks to users. Note: This user must be created on both the clusters. In this case, a user is created through EXAoperation for EDU01 and EDU02 clusters. To create a new user: In EXAoperation, navigation to Configuration > Access Management. Click Users tab and click Add. Complete the following information for the new user. For example: Login: FTPBackup Title: 4FTPbackup Description: 4FTPbackup Identification by: Internal Password: Enter the desired password Click Add. A user is created. The new user created will have the role of ‘User’ by default. To set the role as ‘Administrator’ for this new user, click the Roles tab and select Administrator from the Roles dropdown list. Click Apply. Step 6: Create Archive Volume from EXAoperation An archive volume must be created on all the clusters (in this example, an archive volume must be created on EDU01 and EDU02) for storing the database backups. As a rule of thumb, for up to 1-3 nodes in a cluster which are also the source nodes, at least one archive node on the destination cluster is recommended. For more information on Archive volume creation, see Create Archive Volume. Step 7: Create and Upload Configuration File To establish a connection between the clusters, you need to create a configuration file which contains the following information: The database names The start cycles – Unix-like cron format Local URL to the backup Remote URL to the backup Remote nodes and SSH The configuration file enables you to establish a connection between two clusters. However, if you are connecting to more than one cluster, then you need an exclusive configuration file to connect to each cluster. For example, you need a configuration file to establish a connection between EDU01 and EDU02. Now, if you have a third cluster EDU03, and you want to establish a connection between EDU02 and EDU03, or from EDU03 to EDU01. You must create a new configuration file for each of these connections. The following are sample configuration files for the clusters:  Sample Configuration File for EDU01 #Filename on SupportHost: /home/ssh/client.cfg Connection1 { DATABASE_NAME = exa_db1 START_CYCLE = */2 * * * * LOCAL_URL = ftp: //FTPbackup:**PW**@%s/v0003 REMOTE_URL = ftp: //FTPbackup:**PW**@%s/v0002 REMOTE_NODES = 10.60.102.11; 10.60.102.12 Verbose = true SSH = client }  Sample Configuration File for EDU02#Filename on SupportHost: /home/ssh/server.cfg Connection1 { Verbose = true SSH = server } Note: The connection name in the configuration files must be the same for the source and destination clusters. Setting the value for Verbose to true in the configuration file enables verbose logging – which captures more details on  syncing of files – in the EXACluster Monitoring Service.  If this not enabled only errors are logged. This option is disabled by default.   The configuration files created must be uploaded to the clusters. The following sample scripts can be used to upload the configuration files:  Sample Script to Upload Configuration File on EDU01nlist = server.getNodeList() config = b64encode(open( '/home/ssh/client.cfg' ).read()) pp([[node] + server.callPlugin(pname, node, 'UPLOAD_CONFIG' , config) for node in nlist])  Sample Script to Upload Configuration File on EDU02nlist = server.getNodeList() config = b64encode(open( '/home/ssh/server.cfg' ).read()) pp([[node] + server.callPlugin(pname, node, 'UPLOAD_CONFIG' , config) for node in nlist])  Step 8: Create SSH Key SSH key can be used to establish secure connections between the clusters. The following sample script generates the SSH key on the client machine with respect to the connection specified in the configuration file.  SSH Keygen Sample Scriptnlist = server.getNodeList() pp(server.showPluginFunctions(pname)) pp([[node] + server.callPlugin(pname, node, 'SSHKEY' , 'Connection1' ) for node in nlist]) >>> pp([[node] + server.callPlugin(pname, node, 'SSHKEY' , 'Connection1' ) for node in nlist]) [[ 'n0011' , 0, 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA8V3Au7gr7jmoWIZlbTTNb/ 3Lkku44mlxeC/gTHHvFjgYQSjtFvWZl7i3NIQqrJk4ApQDcqBTRLT8/VNT4PHWyRt+3I ImmGH0D3V9rl+NmCQVjJh/sSKttI5cMR3P6JSg76mhaIjkKddnILHIJVW3R1Q2g+bgr5 R1qaCXQghb9M/mdHdbfTkk7zI41tAchlZrjbcRfRwOMAYOGSHIdegB1qs1kMBbEivcS9 3sKCyXG46dLchQspIeShdwHFjCJDDRYGIWiH4N6M2P50PjGM4lQTyFrJzAD89LYV IfMiN6d+2XTCYCy7W0uezp7OqwBsp2UY31omw9jtSqDn3g5KOIZQ== root@n0011.c0001.exacluster.local']] >>> Note: The ssh key generated can be used only with the connection it is linked to.  A public key is created for each node.  This public key must be copied and uploaded to the remote server to be able to use SSH authentication. Step 9: Upload SSH Key The public key generated for the node must be uploaded to the remote server to be able to authenticate using SSH. Create a file (for example - /home/ssh/connection1_ssh_key_file.key) and copy the key generated in the previous step for the node into this file. The sample script below uploads SSH key for the specific connection to the remote server (in this example, the SSH key is uploaded to EDU02).  Sample Script to Upload SSH Key on EDU02nlist = server.getNodeList() # Upload KEY key = b64encode(open( '/home/ssh/Connection1_ssh_key_file.key' ).read ()) pp([[node] + server.callPlugin(pname, node, 'UPLOAD_KEY' , key) for node in nlist]) Activate SSH key for "Connection1" pp([[node] + server.callPlugin(pname, node, 'SSHKEY' , 'Connection1' ) for node in nlist]) Step 10: Activate Connection between the two clusters Once you have uploaded the SSH key file, the connections on the clusters must be activated to be able to establish a secure connection between them.  Sample Script to Activate Connection on EDU01nlist = server.getNodeList() # Upload KEY key = b64encode(open( '/home/ssh/Connection1_ssh_key_file.key' ).read ()) pp([[node] + server.callPlugin(pname, node, 'UPLOAD_KEY' , key) for node in nlist]) # Activate SSH key for "Connection1" pp([[node] + server.callPlugin(pname, node, 'ACTIVATE' , 'Connection1' ) for node in nlist])  Step 11: Deactivate Connection between two cluster In case you want to deactivate this connection, you can follow the below sample script: Sample Script to Deactivate Connection on EDU01nlist = server.getNodeList() pp([[node] + server.callPlugin(pname, node, 'DEACTIVATE' , 'Connection1' ) for node in [ 'n0011' , 'n0012' ]])   
View full article
Synopsis This article depicts the steps required to start a cluster when all nodes are powered off and how to shut a cluster down using the EXAoperation XML-RPC interface. The python snippets are mere examples for the usage of the XML-RPC function calls and provided as-is. Please refer to the EXAoperation manual for details and further information on XML-RPC. Alphabetical list of referenced XML-RPC calls Function Context Description callPlugin Cluster Execute a call to an EXAoperation plugin getEXAoperationMaster Cluster Return the node of the current EXAoperation master node getDatabaseConnectionState Database instance Get connection state of an EXASolution instance getDatabaseConnectionString Database instance Return the connection string of an EXASolution instance as used by EXAplus and EXASolution drivers getDatabaseList Cluster Lists all database instances defined on the cluster getDatabaseOperation Database instance Get current operation of an EXASolution instance getDatabaseState Database instance Get the runtime state of an EXASolution instance getHardwareInformation Cluster Reports information about your system's hardware as provided by   dmidecode getNodeList Cluster Lists all defined cluster nodes except for license server(s) getServiceState Cluster List the cluster services and their current runtime status logEntries Logservice Fetch messages collected y a preconfigured EXAoperation logservice shutdownNode Cluster Shutdown (and power off) a cluster node startDatabase Database instance Start an EXASolution instance startEXAStorage Storage service Start the EXAStorage service startupNode Cluster Cold start a cluster node stopDatabase Database instance Stop an EXASolution instance stopEXAStorage Storage service Stop the EXAStorage service   Establishing the connection to EXAoperation To send XML-RPC requests to EXAoperation, please connect to the EXAoperation HTTP or HTTPS listener and provide the base URL matching to the context of a function call as described in the EXAoperation manual (chapter "XML-RPC interface") and listed in the table below. The code examples in this article are written in Python (tested in versions 2.7 and 3.4). import sys if sys.version_info[0] > 2: # Importing the XML-RPC library in python 3 from xmlrpc.client import ServerProxy else : # Importing the XML-RPC library in python 2 from xmlrpclib import ServerProxy # define the EXAoperation url cluster_url = "https: //user:password@license-server/cluster1" # create a handle to the XML-RPC interface cluster = ServerProxy(cluster_url) Startup of a cluster 1. Power-on the license server and wait for EXAoperation to start License servers are the only nodes able to boot from the local hard disk. All other (database/compute) nodes receive their boot images via PXE. Hence, you need to have at least one license server up and running to kick-start the rest of the cluster. Physically Power-on the license server and wait until the EXAoperation interfaces are connectible. cluster_url = "https: //user:password@license-server/cluster1" while True: try : cluster = ServerProxy(cluster_url) if cluster.getNodeList(): print( "connected\n" ) break except: continue 2. Start the database/compute nodes Please note that The option to power-on the database nodes using startupNode() is only usable if the nodes are equipped with an out-of-band management interface (like HP iLO or Dell iDRAC) and if this interface is configured in EXAoperation. Virtualized environments (such as vSphere) provide means to automate the startup of servers on a sideband channel. for node in cluster.getNodeList(): cluster.startupNode(node) The function getNodeList returns the list of database nodes currently configured in EXAoperation but it does not provide information about the availability in the cluster. You may check if a node is online by querying the node's hardware inventory. for node in cluster.getNodeList(): if 'dmidecode' in cluster.getHardwareInformation(node): print( "node {} is online\n" .format(node)) else : print( "node {} is offline\n" .format(node)) The boot process itself can be monitored by following the messages in an appropriate logservice. Look for messages like 'Boot process finished after XXX seconds' for every node. logservice_url = "https: //user:password@license-server/cluster1/logservice1" logservice = ServerProxy(logservice_url) logservice.logEntries() It is vital that all cluster nodes are up and running before you proceed with the next steps. 3. Start the EXAStorage service EXAStorage provides volumes as persistence layer for EXASolution databases. This service does not start on boot automatically. The startEXAStorage function returns 'OK' on success or an exception in case of a failure. cluster_url = "https: //user:password@license-server/cluster1" storage_url = "https: //user:password@license-server/cluster1/storage" cluster = ServerProxy(cluster_url) storage = ServerProxy(storage_url) # start the Storage service storage.startEXAStorage() # check the runtime state of all services cluster.getServiceState() The getServiceState call returns a list of all cluster services. Ensure that all of them indicate the runtime state 'OK' before you proceed. [[ 'Loggingd' , 'OK' ], [ 'Lockd' , 'OK' ], [ 'Storaged' , 'OK' ], [ 'DWAd' , 'OK' ]] 4. Start the EXASolution instances Iterate over the EXASolution instances and start them: for db in cluster.getDatabaseList(): instance_url = "https: //user:password@license-server/cluster1/db_{}" .format(db) instance = ServerProxy(instance_url) instance.startDatabase() while True: if 'Yes' == instance.getDatabaseConnectionState(): print( "database {} is accepting connections at {}\n" .format( db, instance.getDatabaseConnectionString())) break Again, you may monitor the database startup process by following an appropriate logservice. Wait for messages indicating that the given database is accepting connections. 5. Start services from EXAoperation plugins Some third-party plugins for EXAoperation may require further attention. This example shows how to conditionally start the VMware tools Daemon. plugin = 'Administration.vmware-tools' # Restart the service on the license server # to bring it into the correct PID namespace cluster.callPlugin(plugin, 'n0010' , 'STOP' ) cluster.callPlugin(plugin, 'n0010' , 'START' ) for node in cluster.getNodeList(): if 'vmtoolsd is running' not in cluster.callPlugin(plugin, node, 'STATUS' )[1]: cluster.callPlugin(plugin, node, 'START' ) Shutdown of a cluster The shutdown of a cluster includes all actions taken for the startup in reverse order. To prevent unwanted effects and possible data loss, it's commendable to perform additional checks on running service operations. Example license_server_id = "n0010" exaoperation_master = cluster.getEXAoperationMaster() if exaoperation_master != license_server_id: print( "node {} is the current EXAoperation master but it should be {}\n" .format( exaoperation_master, license_server_id)) If the license server is not the EXAoperation master node, please log into EXAoperation and move EXAoperation to the license server before you continue. 1. Shutdown of the EXASolution instances Iterate over the EXASolution instances, review their operational state and stop them. for db in cluster.getDatabaseList(): instance_url = "https: //user:password@license-server/cluster1/db_{}" .format(db) instance = ServerProxy(instance_url) state = instance.getDatabaseState() if 'running' == state: operation = instance.getDatabaseOperation() if 'None' == operation: instance.stopDatabase() while True: if 'setup' == instance.getDatabaseState(): print( "database {} stopped\n" .format(db)) break else : print( "Database {} is currently in operation state {}\n" .format(db, operation)) else : print( "Database {} is currently in runtime state {}\n" .format(db, state)) 2. Shutdown of the EXAStorage service Please assure yourself that all databases are shut down properly before stopping EXAStorage! cluster_url = "https: //user:password@license-server/cluster1" storage_url = "https: //user:password@license-server/cluster1/storage" cluster = ServerProxy(cluster_url) storage = ServerProxy(storage_url) storage.stopEXAStorage() cluster.getServiceState() The state of the Storaged will switch to 'not running': [[ 'Loggingd' , 'OK' ], [ 'Lockd' , 'OK' ], [ 'Storaged' , 'not running' ], [ 'DWAd' , 'OK' ]] 3. Shutdown of the cluster nodes and of the license server(s) at last for node in cluster.getNodeList(): cluster.shutdownNode(node) license_servers = [ 'n0010' ,] for ls in license_servers: cluster.shutdownNode(ls) The last call triggers the shutdown of the license server(s) and therefore terminate all EXAoperation instances.
View full article
Certified Hardware List
The hardware certified by Exasol can be found in the link below:

Certified Hardware List

If your preferred hardware is not certified, refer to our Certification Process for more information on this process.
Top Contributors