How to speed up backup

danila
SQL-Fighter

I'm writing backup to remote Microsoft Windows FTP server. Currently it takes around 19-20 hours to create full (level 0) backup which weights a bit more than 3 TB.

Our network setup now is exactly lie described here: https://docs.exasol.com/administration/on-premise/installation/prepareenvironment/minimum_network_se...

Which network card should I change to 10GBit/s in order to speed up the backup?
Is it only the license node's network card responsible for it?

1 ACCEPTED SOLUTION

mwellbro
Xpert

Hi Danila,

for the NIC-topic I´ll defer to someone more versed in that than myself - depending on your use case the following might be worth knowing when it comes to speeding up backup´s : 

It is possible ( though certainly not recommended for a production system ) to switch off the validation of the backup taken - might be interesting if you are using those backups in non-critical areas.

In order to make use of this add the following to the "Extra Database Parameters" in ExaOperation ( on the Exasolution Instance )

-suppressValidationOfBackup=1

Cheers,
Malte

View solution in original post

6 REPLIES 6

jwarlander
Xpert

Since, as per the documentation, "All nodes must be able to reach the remote target", I do assume that all data nodes will be writing backup data concurrently corresponding to each node's local data -- and in that case, I'd say that definitely the data node(s) would need the 10GBit/s speeds..

As for the license node, someone from Exasol can probably answer this better, but it's probably just accessing the remote volume for management purposes?

Either way..

  • What can the FTP server be expected to handle in terms of transfer rate?
  • Do you have end-to-end network performance to support 10Gbit/s?
  • How powerful are the Exasol data nodes?

Our backup lands at about 4.7TB, but it's hard to compare as we run on AWS EC2 instances and store backups in S3; still, I see it took 8 hours for the last full backup to finish.

Supposedly we should get 20GBit/s in network bandwidth, so probably a lot of the time spent is more on the Exasol side, processing the backup data that needs to be sent to S3.

mwellbro
Xpert

Not to be nosy and only if you´re able to disclose this, but what instance class were your data nodes using for the 4,7TB / 8hrs backup and how many nodes were in the cluster , @jwarlander ?

Don´t have much experience in that area myself so I´m always on the lookout for data to compare against 😁

jwarlander
Xpert

No worries! 😁

We run a cluster of 4 x m5.16xlarge (each with 64 vCPU and 256GB RAM).

mwellbro
Xpert

Hi Danila,

for the NIC-topic I´ll defer to someone more versed in that than myself - depending on your use case the following might be worth knowing when it comes to speeding up backup´s : 

It is possible ( though certainly not recommended for a production system ) to switch off the validation of the backup taken - might be interesting if you are using those backups in non-critical areas.

In order to make use of this add the following to the "Extra Database Parameters" in ExaOperation ( on the Exasolution Instance )

-suppressValidationOfBackup=1

Cheers,
Malte

View solution in original post

danila
SQL-Fighter

Thanks. I'll try out this flag on test system.

exa-MathiasHo
Community Manager
Community Manager

Hi @danila,

Just checking: Did you have success with your test? 

 

Best regards,

Mathias