Airflow @ AWS 2.02

drumcircle
Rising Star

 

Has anyone succeeded with the new 2.02 airflow exposed by MWAA on AWS?

Kateryna posted a generous cookbook which worked on the last version.

https://katerynababich.medium.com/exasol-plugin-for-aws-airflow-mwaa-ca60faa09bfe

 

1 ACCEPTED SOLUTION

Accepted Solutions

drumcircle
Rising Star

My colleague got this to work.

Following the recommendations here:

	https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html#configuring-dag-plugins-changed
	"New: Imports in plugins. Importing operators, sensors, hooks added in plugins using airflow.{operators,sensors,hooks}.<plugin_name> is no longer supported. These extensions should be imported as regular Python modules. In v2.0+, the recommended approach is to place them in the DAGs directory and create and use an .airflowignore file to exclude them from being parsed as DAGs."

 

The changes from the example are:

	exasol_sensor.py, exasol_operator.py, and exasol_hook.py are placed in the dags folder.
	exasol_sensor.py, exasol_operator.py, and exasol_hook.py are added to the .airflowignore file.

	The imports in exasol_sensor.py and exasol_operator.py are changed to:
		from exasol_hook import ExasolHook

	The imports in sample_dag_exasol_plugin.py are changed to:
		from exasol_sensor import ExasolSensor
		from exasol_operator import ExasolOperator
		
	No plugins.zip file is necessary any more.
	A requirements.txt with this line is necessary:
		pyexasol

View solution in original post

2 REPLIES 2

exa-Franz
Team Exasol
Team Exasol

Hi @drumcircle, what a coincidence 😉 I read the same article end of last week, and used it as a template for a Hello World Example for the airflow-exasol-provider. But I ran it onto a Airflow 2.1.4 instance that I installed from the scratch, which worked quite nice. I read over the documentation of MWAA. And now I was searching on how to install a provider and just found out it is already preinstalled for Exasol:
https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html

Let me know how it goes, I would be interested in your use case if you can share some details, I see more and more Exasol customers use it 😉
Have a good start into the week!

Best,
Franz

EDIT: I just saw that the version preinstalled for the Exasol Airflow provider on MWAA is not the current one. I assume if you want to install the newest (v 2.0.1), you have to follow that path:
https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html
https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html

But I am not sure how to deal with any conflicts that might arise, as you have also the one already preinstalled (v 1.1.1).

drumcircle
Rising Star

My colleague got this to work.

Following the recommendations here:

	https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html#configuring-dag-plugins-changed
	"New: Imports in plugins. Importing operators, sensors, hooks added in plugins using airflow.{operators,sensors,hooks}.<plugin_name> is no longer supported. These extensions should be imported as regular Python modules. In v2.0+, the recommended approach is to place them in the DAGs directory and create and use an .airflowignore file to exclude them from being parsed as DAGs."

 

The changes from the example are:

	exasol_sensor.py, exasol_operator.py, and exasol_hook.py are placed in the dags folder.
	exasol_sensor.py, exasol_operator.py, and exasol_hook.py are added to the .airflowignore file.

	The imports in exasol_sensor.py and exasol_operator.py are changed to:
		from exasol_hook import ExasolHook

	The imports in sample_dag_exasol_plugin.py are changed to:
		from exasol_sensor import ExasolSensor
		from exasol_operator import ExasolOperator
		
	No plugins.zip file is necessary any more.
	A requirements.txt with this line is necessary:
		pyexasol

View solution in original post