My name is Joel Wittenmyer and I am a Data Architect Specialist at Allianz Life of North America. I have been a fan of Exasol since 2010, when I designed a National Medical Records Data Warehouse with Exasol as it's centerpiece. More recently, I spent @10 years fighting to get Exsasol into Allianz Life as a platform to support reporting and analytics. We went into production with it about a year ago. I am still a huge fan. I completed all the training on the old site, and look forward to completing it on the new site. We are using the Query_Wrapper from the Github repository, with some enhancements, to refresh our test instance each day. We are also in the process of using the Python script to copy backup files from the cluster to a NAS location so that our enterprise backup solution can grab them. In addition to our Enterprise Business Intelligence department's use as a reporting platform, we have a team of data scientists that use Exasol, and it serves as a point of distribution for data analysts throughout the company for data they use in their analytics.
Thank you Exasol!
Hi Joel, it took us 2 years to make the decision to abandon SQL Server in 2016. But in 2020 there is a lot more choice... I would be very interested in the usage of the query_wrapper Github repository and the data refreshment strategy your are using to fill your test instance daily. Are you using Python + XML-RPC interface for automating the backup?
from what I understand the query_wrapper sort of allows you to dynamically create the sql statements in a more elegant way than do simple string concatenation. I am very interesting in understanding the pros of this approach in contrast to just creating the SQL using a template engine such as Jinja or Lua string concat. So I will be very thankful if you could let me know a little bit more about the use case - what is the query_wrapper doing? - exporting parts of the prod data to test, doing data masking, anonymization, etc... We are using our ETL tool (I want to replace it, with something like Prefect.io, mara ETL, Apache Airflow ... something open source) to schedule and orchestrate our data pipeline. The data ingestion from the source systems is all driven by dynamic SQL - we have a few tables that govern, what source tables to ingest, how to ingest (delta / full-load), and have manually develloped views that map the source to our persistent staging (views are necessary as we do a renaming of columns to enforce standard naming convention across all source systems, if I did this project again - I would not alter the column names in the persistent staging - I would do that later on top in a view layer.).
It's impressive to fight for 10 years and not give up on your goal there. I have recently joined a company and there are somethings I wish were different. Do you have any advice to help keep up the perseverance to get the change and get the rest of the teams on board as well?
Well, the only way we succeeded was by gaining the support of the VP of Enterprise Business Intelligence. He reports directly to the CFO. He made the case that in order to be able to deliver on his commitments, he needed a different way of operating and IT was not interested in helping him. He and I have a great relationship, and I've been an ardent student of database internals ever since I worked for Oracle. I told him Exasol was what he needed, and we fought off every dirty trick you can imagine (and some you probably wouldn't believe IT people could come up with) to just get it in the door and test it. After people saw what it was capable of, they shrugged off the Fear, Uncertainty, and Doubt and embraced it. In summary, we got 'C' level support. Without the CFO to run interference, our Enterprise Architecture group wouldn't have let us get close enough to Exasol to smell it 🙂