where to set awsS3PartSize?

drumcircle
Rising Star

Is awsS3PartSize a global database parameter?  Where is it set?  Docs are unclear.

https://docs.exasol.com/loading_data/load_data_amazon_s3.htm

2 REPLIES 2

mwellbro
Xpert

Hi @drumcircle ,

I´d guess the "The value of the awsS3PartSize parameter is consistent across all your queries. You cannot configure it separately for different queries." section from the docs means what you think it means, aka. its a global DB parameter - also, the "camel case" notation would seem to support that notion.
Setting it would be done in the "Extra Database Parameters" in ExaOperation - if the assumption is wrong then the DB won´t start ( which I believe is always the case when you try to set a non-existent parameter ).
Format would probably be ( in case you´d want to double up to 20MB) :

-awsS3PartSize=20971520

Cheers,
Malte

mwellbro
Xpert

Hi again,

since I prefer to give "tested advise" when possible:

mwellbro_0-1632691900069.png

So you can put it down there - if you choose the value too small ( I did for test purposes ) you´ll be greeted by:

mwellbro_1-1632692017598.png

In my test, I´ve uploaded about 315MB from an AWS m5.large over to S3 from a one-node-cluster - didn´t see
much change in exec time when setting it from standard to 80MB:

mwellbro_2-1632692172749.png

 

...and using CloudWatch to showcase the drop in needed packages wasn´t much fun so I won´t be
attaching the corresponding screenshots - might try a metric stream next time to get the wanted detail granularity.

So there you have it. "tried and true" 😉

Cheers,
Malte