Remote archive volumes on


Has anyone had any success using  as the destination for a remote archive volume? (It's S3 compatible)

I'm getting a generic "Connection Problems" error (with archive options set to "s3,noverifypeer") and I'm curious if it's something inherent to



tried to upload logs as a test, and got an error. Tcpdump told me this:

exasol-> minio

POST /support/exacluster_debuginfo-2021-03-17_14-31-40.tar.gz?uploadId=bac6de06-a82a-4d6e-8806-c269aac58c99 HTTP/1.1
host: XXX
accept: */*
content-type: application/octet-stream
x-amz-date: 20210317T113149Z
x-amz-content-sha256: XXX
authorization: AWS4-HMAC-SHA256 Credential=minio/20210317/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=XXX
content-length: 52



HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 473
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Server: MinIO
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 166D1DE4D56BBBFC
X-Xss-Protection: 1; mode=block
Date: Wed, 17 Mar 2021 11:31:49 GMT

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema.</Message><Key>exacluster_debuginfo-2021-03-17_14-31-40.tar.gz</Key><BucketName>support</BucketName><Resource>/support/exacluster_debuginfo-2021-03-17_14-31-40.tar.gz</Resource><Region>us-east-1</Region><RequestId>166D1DE4D56BBBFC</RequestId><HostId>f1cdbcc7-75aa-4c9f-8321-3401acb04a9f</HostId></Error>

so, almost everything works, except the last package...


Hi @bykva ,

1) Was there any content in the <CompleteMultiPartUpload> XML tag? It should look something like this:



2) I don't know if this is the cause of your problem but in order to make Minio+Exasol work for me I had to patch Minio so that the region HTTP header was in lowercase like this:

x-amz-bucket-region: us-east-1

When it was uppercase Exasol did not recognize it:

X-Amz-Bucket-Region: us-east-1


Hope that helps,



1) bykva_0-1616075229034.png

that's the root case - there is no data between tags 😞

2) no, i've read your post and fixed http header already.


Also - one more question: What is the name of your bucket in Minio? "<BucketName>support</BucketName>" suggests to me that Minio is getting the bucketname from the URL instead of the domain. It needs to get it from the domain in order to work with Exasol.   So if your bucketname is "abc" and your Minio host is "" then you need to create a DNS alias "" and set the MINIO_DOMAIN ENV to "".  Then you tell Exasol the remote volume is at "" and Minio will interpret the "abc" as the bucketname.


1) Are all the previous responses "200 OK" ?

There should be 5 total requests from Exasol->Minio:



POST /support/exacluster_debuginfo-2021-03-18_09-34-05.tar.gz?uploads

PUT /support/exacluster_debuginfo-2021-03-18_09-34-05.tar.gz?partNumber=1&uploadId=274b2d53-2323-4c7f-98db-235612538a46

POST /support/exacluster_debuginfo-2021-03-18_09-34-05.tar.gz?uploadId=274b2d53-2323-4c7f-98db-235612538a46

 Make sure they all return "200 OK"

By the way I find the "mc" utility useful for debugging Minio (

Once you have installed that you can debug the connection via "mc admin trace -all -v <alias>"


2)  I see in the example that you posted above that the header is still "X-Amz-Bucket-Region" instead of "x-amz-bucket-region"






unfortunately, all HEAD requests get non-200 answers
try1: (with haproxy http-response add-header, original minio code)




Full trace:



try2: (recompilled minio with x-amz patch) 




Full trace:



About x-amz-... header: we have such installation: exasol <-> haproxy <-> minio. on haproxy there is an option
http-response add-header x-amz-bucket-region us-east-1

and i run tcpdump on the minio server, that's why you see X-Amz-... in my screenshots. but as u suggest, i've tried to run with a patched minio.

About support: for testing purpose i try to store logs on minio: support -> Store Debug Information into Archive Volume. i use minio remote archive which has url: and docker env vars:
-e MINIO_REGION_NAME=us-east-1 \
-e \

DNS: -> minio -> minio


If HEAD is giving an error it usually means either the bucket is not correctly being recognized (400) or the permissions are not correct (403)...but there are various other things that can cause these errors. The code that specifies them is here

1) Try setting the bucket permissions to "public" and see if that helps "mc policy set public <alias>/<bucketname>"

2) If "" is your full url then I assume you have a bucket named "exasol" created in Minio?

3) Try debugging the HEAD directly from curl until you can get that running without an error.  In one window watch the trace via " mc admin trace -insecure -all -v <alias>"  and in another window run "curl -X HEAD -k https://localhost:9000 -H "host:"



1) tried to set public permissions, head is now 200 OK

2) i've created exasol and support buckets in minio. when there was no "support bucket" i got a clear error in xml "no support bucket". and i can upload files to those buckets via mc and web-interface.

3) this debug gives me exactly same result as in tcpdump :((
exasol->minio (really, no data between tags)
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema.</Message><Key>support/exacluster_debuginfo-2021-03-19_17-26-02.tar.gz</Key><BucketName>exasol</BucketName><Resource>/support/exacluster_debuginfo-2021-03-19_17-26-02.tar.gz</Resource><Region>us-east-1</Region><RequestId>166DC491ACEDBE9A</RequestId><HostId>f1cdbcc7-75aa-4c9f-8321-3401acb04a9f</HostId></Error>


Unfortunately I have to give up 😞

I can't think of anything that would cause that XML to be empty and I haven't been able to reproduce that in my set up.

Maybe look at the XMLs of the previous requests and see if there's anything suspicious in them.

What version of Exasol are you running?



anyway, many thanks to you!

i'm using version 6.2.13 both for EXASolution and EXAClusterOS

Community Manager
Community Manager
I have alerted the team internally as well, hope we come up with a solution
Connecting Customers, Partners, Prospects and Exasolians is my passion. Apart from that I cycle, listen to music, and try to understand what all those technical discussions really mean...


sorry, any updates about my problem?


Thanks Charlie, that got me on the right track.

I'm happy to say we finally got it working although it required recompiling with a minor patch. It turns out that Exasol expects the X-Amz-Bucket-Region HTTP header name to be lowercase while sends it with initial-caps. Thankfully is fairly easy to patch/recompile.

Here are the steps we took to get Exasol to recognize as a remote archive volume (first three thanks to Charlie):

  1. Enable SSL in
  2. Have listen on 443 (Exasol will ignore any port specified)
  3. Assuming that your server is and it has a bucket named backups, create a DNS alias which also resolves to
  4. In the startup set the MINIO_DOMAIN ENV variable to  This will cause to extract the bucket name from the virtual host passed to it instead of extracting it from the URL path (which is the default)
  5. In the startup set the MINIO_REGION_NAME ENV variable to us-east-1. This will cause to include that in all HTTP response headers.
  6. Get the source code from and apply the patch below. See the repository's Dockerfile for how to rebuild it: 
    --- /cmd/api-headers.go
    +++ /cmd/api-headers.go
    @@ -51,7 +51,8 @@ func setCommonHeaders(w http.ResponseWriter) {
        // Set `x-amz-bucket-region` only if region is set on the server
        // by default minio uses an empty region.
        if region := globalServerRegion; region != "" {
    -       w.Header().Set(xhttp.AmzBucketRegion, region)
    +       h := strings.ToLower(xhttp.AmzBucketRegion)
    +       w.Header()[h] = append(w.Header()[h], region)
        w.Header().Set(xhttp.AcceptRanges, "bytes")
  7. In the Exasol interface for adding a remote archive volume
    • Set the Archive URL to
    • Specify the access and secret keys in the username/password fields
    • Specify an Option of s3 (and any other applicable options)




tried it once but never got it to run.

My discoveries so far:

  • you have to enable SSL in
  • you have to run in on port 443 -> exasol will ignore the port number on the provided url
  • you will have to specify the bucket you want your data stored in prior to the hostname as Exasol strips the first part from the url and uses it as bucket
  • Take a backup to the not working remote volume to see more information in the log
    pddserver(1.0): Backup error. Backup could not be written successfully: [Remote volume error: Unable to retrieve S3 region from AWS response]:​

That's the farthest I have come.


That's the trace

 "host": "mybucket.",
 "api": "errorResponseHandler",
 "request": {
  "time": "2020-08-25T09:38:23.9855559Z",
  "proto": "HTTP/1.1",
  "method": "HEAD",
  "path": "/",
  "headers": {
   "Accept": "*/*",
   "Content-Length": "0",
   "Host": "mybucket."
 "response": {
  "time": "2020-08-25T09:38:23.9855559Z",
  "headers": {
   "Accept-Ranges": "bytes",
   "Content-Length": "231",
   "Content-Type": "application/xml",
   "Server": "MinIO/RELEASE.2020-08-25T00-21-20Z",
   "Vary": "Origin"
  "body": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>XMinioUnknownAPIRequest</Code><Message>Unknown API request at /</Message><Resource>/</Resource><RequestId></RequestId><HostId>4055faa6-8d98-4752-9af6-d9bb88510912</HostId></Error>",
  "statusCode": 400
 "callStats": {
  "rx": 27,
  "tx": 374,
  "duration": 0,
  "timeToFirstByte": 0