AutoDL not working with MinIO backend (migrating from Google Cloud) {google-cloud-merged branch}

[ObadaS]

When changing the .env file to use MinIO instead of Google Cloud on an existing deployment, then trying to download a file that's stored in MinIO (the files were copied from the Google cloud buckets into the MinIO buckets), we get this error.

Traceback:

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/app/codalab/apps/web/views.py" in get
  1293.         if not submission.detailed_results_ready:

File "/app/codalab/apps/web/models.py" in detailed_results_ready
  1529.         if self.detailed_results_file and self.detailed_results_file.url and self.detailed_results_file.file:

File "/usr/local/lib/python2.7/site-packages/django/db/models/fields/files.py" in _get_file
  51.             self._file = self.storage.open(self.name, 'rb')

File "/usr/local/lib/python2.7/site-packages/django/core/files/storage.py" in open
  38.         return self._open(name, mode)

File "/usr/local/lib/python2.7/site-packages/storages/backends/s3boto.py" in _open
  371.         f = self.file_class(name, mode, self)

File "/usr/local/lib/python2.7/site-packages/storages/backends/s3boto.py" in __init__
  64.         self.key = storage.bucket.get_key(self._storage._encode_name(name))

File "/usr/local/lib/python2.7/site-packages/boto/s3/bucket.py" in get_key
  193.         key, resp = self._get_key_internal(key_name, headers, query_args_l)

File "/usr/local/lib/python2.7/site-packages/boto/s3/bucket.py" in _get_key_internal
  200.                                                 query_args=query_args)

File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py" in make_request
  671.             retry_handler=retry_handler

File "/usr/local/lib/python2.7/site-packages/boto/connection.py" in make_request
  1071.                           retry_handler=retry_handler)

File "/usr/local/lib/python2.7/site-packages/boto/connection.py" in _mexe
  927.                 request.authorize(connection=self)

File "/usr/local/lib/python2.7/site-packages/boto/connection.py" in authorize
  377.         connection._auth_handler.add_auth(self, **kwargs)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in add_auth
  755.                                                          **kwargs)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in add_auth
  574.         string_to_sign = self.string_to_sign(req, canonical_request)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in string_to_sign
  514.         sts.append(self.credential_scope(http_request))

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in credential_scope
  496.         region_name = self.determine_region_name(http_request.host)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in determine_region_name
  690.         return region_name

Exception Type: UnboundLocalError at /my/competition/submission/43690/input.zip
Exception Value: local variable 'region_name' referenced before assignment

I am pretty sure that MinIO support was added later, starting (maybe) with this commit which is not present in the current branch for autodl.

[Didayolo]

Hi @ObadaS, can you put also the command you are trying to run? And a quick summary of what we are trying to achieve here? Thank you.

[ObadaS]

@Didayolo I updated the original post with more details