We released version 5.2.0 of the Transformers Extension on Github and Pypi.
The Exasol Transformers Extension allows you to use pre-trained machine learning models from
Hugging Face directly in your Exasol instance.
It lets you install the models via the transformers-api directly into Exasol’s filesystem BucketFS, and use them on your data using provided UDF’s.
Summary
Since our last announcement, our AI Functions set has been completed. AI_ANSWER, AI_CLASSIFY, AI_SENTIMENT, AI_TRANSLATE and AI_EXTRACT_ENTITIES are now available.
Additionally, the upload of models to the BucketFS has gotten a rework to allow the use of bigger models.
We also added a new script which installs all available UDF’s in Database.
Lastly, the transformers package was updated from version 4 to version 5.
BREAKING CHANGES:
In transformers version 5, the pipeline for question-answering has been removed. For our AI_ANSWER_EXTEDNED UDF, we replaced it with the text-generation pipeline. You will not be able to use you old question-answering models with this new implementation, switch to text-generation models instead. Make sure these models support using a chat_template(instruct models).
(The translation pipeline was also removed, but we decided to maintain it ourselves for
now, so the AI_TRANSLATE_EXTENDED UDF is unaffected.)