Remote archive volumes on min.io

PeterK
Xpert

Has anyone had any success using https://min.io/  as the destination for a remote archive volume? (It's S3 compatible)

I'm getting a generic "Connection Problems" error (with archive options set to "s3,noverifypeer") and I'm curious if it's something inherent to min.io.

14 REPLIES 14

Charlie
Xpert

Hi,

tried it once but never got it to run.

My discoveries so far:

  • you have to enable SSL in min.io
  • you have to run in on port 443 -> exasol will ignore the port number on the provided url
  • you will have to specify the bucket you want your data stored in prior to the hostname as Exasol strips the first part from the url and uses it as bucket
    https://mybucket.192.168.188.53​
  • Take a backup to the not working remote volume to see more information in the log
    pddserver(1.0): Backup error. Backup could not be written successfully: [Remote volume error: Unable to retrieve S3 region from AWS response]:​

That's the farthest I have come.

Edit:

That's the min.io trace

{
 "host": "mybucket.192.168.188.53",
 "api": "errorResponseHandler",
 "request": {
  "time": "2020-08-25T09:38:23.9855559Z",
  "proto": "HTTP/1.1",
  "method": "HEAD",
  "path": "/",
  "headers": {
   "Accept": "*/*",
   "Content-Length": "0",
   "Host": "mybucket.192.168.188.53"
  }
 },
 "response": {
  "time": "2020-08-25T09:38:23.9855559Z",
  "headers": {
   "Accept-Ranges": "bytes",
   "Content-Length": "231",
   "Content-Type": "application/xml",
   "Server": "MinIO/RELEASE.2020-08-25T00-21-20Z",
   "Vary": "Origin"
  },
  "body": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>XMinioUnknownAPIRequest</Code><Message>Unknown API request at /</Message><Resource>/</Resource><RequestId></RequestId><HostId>4055faa6-8d98-4752-9af6-d9bb88510912</HostId></Error>",
  "statusCode": 400
 },
 "callStats": {
  "rx": 27,
  "tx": 374,
  "duration": 0,
  "timeToFirstByte": 0
 }
}

Regards

PeterK
Xpert

Thanks Charlie, that got me on the right track.

I'm happy to say we finally got it working although it required recompiling min.io with a minor patch. It turns out that Exasol expects the X-Amz-Bucket-Region HTTP header name to be lowercase while min.io sends it with initial-caps. Thankfully min.io is fairly easy to patch/recompile.

Here are the steps we took to get Exasol to recognize min.io as a remote archive volume (first three thanks to Charlie):

  1. Enable SSL in min.io
  2. Have min.io listen on 443 (Exasol will ignore any port specified)
  3. Assuming that your min.io server is minio.yourdomain.com and it has a bucket named backups, create a DNS alias backups.minio.yourdomain.com which also resolves to min.io.
  4. In the min.io startup set the MINIO_DOMAIN ENV variable to minio.yourdomain.com.  This will cause min.io to extract the bucket name from the virtual host passed to it instead of extracting it from the URL path (which is the default)
  5. In the min.io startup set the MINIO_REGION_NAME ENV variable to us-east-1. This will cause min.io to include that in all HTTP response headers.
  6. Get the min.io source code from https://github.com/minio/minio.git and apply the patch below. See the repository's Dockerfile for how to rebuild it: 
    --- /cmd/api-headers.go
    +++ /cmd/api-headers.go
    @@ -51,7 +51,8 @@ func setCommonHeaders(w http.ResponseWriter) {
        // Set `x-amz-bucket-region` only if region is set on the server
        // by default minio uses an empty region.
        if region := globalServerRegion; region != "" {
    -       w.Header().Set(xhttp.AmzBucketRegion, region)
    +       h := strings.ToLower(xhttp.AmzBucketRegion)
    +       w.Header()[h] = append(w.Header()[h], region)
        }
        w.Header().Set(xhttp.AcceptRanges, "bytes")
  7. In the Exasol interface for adding a remote archive volume
    • Set the Archive URL to https://backups.minio.yourdomain.com
    • Specify the access and secret keys in the username/password fields
    • Specify an Option of s3 (and any other applicable options)

Enjoy!

bykva
Padawan

tried to upload logs as a test, and got an error. Tcpdump told me this:

exasol-> minio

POST /support/exacluster_debuginfo-2021-03-17_14-31-40.tar.gz?uploadId=bac6de06-a82a-4d6e-8806-c269aac58c99 HTTP/1.1
host: XXX
accept: */*
content-type: application/octet-stream
x-amz-date: 20210317T113149Z
x-amz-content-sha256: XXX
authorization: AWS4-HMAC-SHA256 Credential=minio/20210317/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=XXX
content-length: 52

<CompleteMultipartUpload>
</CompleteMultipartUpload>

minio->exasol

HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 473
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Server: MinIO
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 166D1DE4D56BBBFC
X-Xss-Protection: 1; mode=block
Date: Wed, 17 Mar 2021 11:31:49 GMT

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema.</Message><Key>exacluster_debuginfo-2021-03-17_14-31-40.tar.gz</Key><BucketName>support</BucketName><Resource>/support/exacluster_debuginfo-2021-03-17_14-31-40.tar.gz</Resource><Region>us-east-1</Region><RequestId>166D1DE4D56BBBFC</RequestId><HostId>f1cdbcc7-75aa-4c9f-8321-3401acb04a9f</HostId></Error>

so, almost everything works, except the last package...

PeterK
Xpert

Hi @bykva ,

1) Was there any content in the <CompleteMultiPartUpload> XML tag? It should look something like this:

<CompleteMultipartUpload>
  <Part>
    <PartNumber>1</PartNumber>
    <ETag>"784d6a1f37a2b6b54123dba6ffb86f1b"</ETag>
  </Part>
</CompleteMultipartUpload>

 

2) I don't know if this is the cause of your problem but in order to make Minio+Exasol work for me I had to patch Minio so that the region HTTP header was in lowercase like this:

x-amz-bucket-region: us-east-1

When it was uppercase Exasol did not recognize it:

X-Amz-Bucket-Region: us-east-1

 

Hope that helps,

Peter

bykva
Padawan

1) bykva_0-1616075229034.png

that's the root case - there is no data between tags 😞

2) no, i've read your post and fixed http header already.

PeterK
Xpert

1) Are all the previous responses "200 OK" ?

There should be 5 total requests from Exasol->Minio:

HEAD /

GET /

POST /support/exacluster_debuginfo-2021-03-18_09-34-05.tar.gz?uploads

PUT /support/exacluster_debuginfo-2021-03-18_09-34-05.tar.gz?partNumber=1&uploadId=274b2d53-2323-4c7f-98db-235612538a46

POST /support/exacluster_debuginfo-2021-03-18_09-34-05.tar.gz?uploadId=274b2d53-2323-4c7f-98db-235612538a46

 Make sure they all return "200 OK"

By the way I find the "mc" utility useful for debugging Minio (https://docs.min.io/docs/minio-client-complete-guide.html)

Once you have installed that you can debug the connection via "mc admin trace -all -v <alias>"

 

2)  I see in the example that you posted above that the header is still "X-Amz-Bucket-Region" instead of "x-amz-bucket-region"

 

Regards,

Peter

 

PeterK
Xpert

Also - one more question: What is the name of your bucket in Minio? "<BucketName>support</BucketName>" suggests to me that Minio is getting the bucketname from the URL instead of the domain. It needs to get it from the domain in order to work with Exasol.   So if your bucketname is "abc" and your Minio host is "company.com" then you need to create a DNS alias "abc.company.com" and set the MINIO_DOMAIN ENV to "company.com".  Then you tell Exasol the remote volume is at "https://abc.company.com" and Minio will interpret the "abc" as the bucketname.

bykva
Padawan

unfortunately, all HEAD requests get non-200 answers
try1: (with haproxy http-response add-header, original minio code)
exasol->minio

bykva_5-1616156101007.png


minio->exasol

bykva_4-1616155855833.png


Full trace:

bykva_1-1616155797707.png

 


try2: (recompilled minio with x-amz patch) 
exasol->minio

bykva_3-1616155838715.png


minio->exasol

bykva_2-1616155823707.png


Full trace:

bykva_0-1616155777572.png

 




About x-amz-... header: we have such installation: exasol <-> haproxy <-> minio. on haproxy there is an option
http-response add-header x-amz-bucket-region us-east-1

and i run tcpdump on the minio server, that's why you see X-Amz-... in my screenshots. but as u suggest, i've tried to run with a patched minio.

About support: for testing purpose i try to store logs on minio: support -> Store Debug Information into Archive Volume. i use minio remote archive which has url: exasol.s3.XXX.ru and docker env vars:
-e MINIO_REGION_NAME=us-east-1 \
-e MINIO_DOMAIN=s3.XXXX.ru \

DNS:
exasol.s3.XXX.ru -> minio
s3.XXX.ru -> minio

PeterK
Xpert

If HEAD is giving an error it usually means either the bucket is not correctly being recognized (400) or the permissions are not correct (403)...but there are various other things that can cause these errors. The code that specifies them is here https://github.com/minio/minio/blob/master/cmd/web-handlers.go#L2348

1) Try setting the bucket permissions to "public" and see if that helps "mc policy set public <alias>/<bucketname>"

2) If "exasol.s3.prod.xxxx.ru" is your full url then I assume you have a bucket named "exasol" created in Minio?

3) Try debugging the HEAD directly from curl until you can get that running without an error.  In one window watch the trace via " mc admin trace -insecure -all -v <alias>"  and in another window run "curl -X HEAD -k https://localhost:9000 -H "host: exasol.s3.prod.xxxx.ru"