py-pa / django-minio-storage Goto Github PK
View Code? Open in Web Editor NEWA django storage driver for minio
License: MIT License
A django storage driver for minio
License: MIT License
It seems like a unnecessary potential for adding a lot of latency huge amounts of stat-calls between django and minio.. Some times the URL function can even be called many times for the same file object on one page.
def url(self, name):
# type: (str) -> str
if self.exists(name):
url = self.client.presigned_get_object(self.bucket_name, name)
parsed_url = urlparse(url)
it is not common for Django storage drivers to does IO inside their url() functions, they usually generate the URLs early and let's the browser fail by getting a 404 if it happens... Djangos own FileSystemStorage does not verify file existence in it's url function.
If a application really needs to check for file existence before generating URLs it can be added to the application layer logic instead.
Hello
It would be really nice to add a storage class that supports manifesting static files.
When using Disk Cache I see no improvement in the speed of collectstatic
.
I wonder why because inside the Minio container I can see the files available locally.
And I'm running collectstatic
without changing anything.
Minio Server: 2019-04-09T01:22:30Z
PYTHON_VERSION=3.7.3
Django==2.2
django-minio-storage==0.2.2
- django [required: >=1.8, installed: 2.2]
- pytz [required: Any, installed: 2019.1]
- sqlparse [required: Any, installed: 0.3.0]
- minio [required: >=4.0.3, installed: 4.0.14]
- certifi [required: Any, installed: 2019.3.9]
- python-dateutil [required: Any, installed: 2.8.0]
- six [required: >=1.5, installed: 1.12.0]
- pytz [required: Any, installed: 2019.1]
- urllib3 [required: Any, installed: 1.24.1]
That seems to be a short path to ensure that seek(0) is called on the file object which currently should cause exceptions when streams from minio are used directly and pass when temp files are used
Hi,
Minio API supports region when using a cloud provider that supports if (AWS, GCP), as per:
https://docs.minio.io/docs/python-client-api-reference#make_bucket
I think it is just a matter of adding a new ENV and pass it to:
https://github.com/py-pa/django-minio-storage/blob/master/minio_storage/storage.py#L57
if it is defined. I can try to do a PR but I'm not really skilled in python.
I have too many upcoming changes which depends on my current pull requests here..
I created a new github org today to handle the current list of forks of I and some other people need on pypi ( https://github.com/py-pa/ ) .
I moved my personal fork to https://github.com/py-pa/django-minio-storage , will set things up during the next days.
For the time being py-pa
has three main goals:
pypi.python.org
jazzband
OR collaborate within the py-pa
org (after jotting down some simple ground rules to clarify how org wide or project wide maintenance should work)..Just installed the app, seems fine but who can i use it for my media files?
i'm only getting it to work for my static files.
this is what my config looks like:
#S3 Storage
STATIC_ROOT = './static_files/'
STATIC_URL = '/static/'
MEDIA_ROOT = './media_files/'
MEDIA_URL = '/media/'
DEFAULT_FILE_STORAGE = "minio_storage.storage.MinioMediaStorage"
STATICFILES_STORAGE = "minio_storage.storage.MinioStaticStorage"
MINIO_STORAGE_ENDPOINT = '127.0.0.1:9000'
MINIO_STORAGE_ACCESS_KEY = 'W3XRS0AK3J1PXHOQPIPK'
MINIO_STORAGE_SECRET_KEY = 'aF72q5G4ZFhfOwoThzFhyzh3aqZoiZBPpse1HVWO'
MINIO_STORAGE_USE_HTTPS = False
MINIO_STORAGE_MEDIA_BUCKET_NAME = 'media'
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET = True
MINIO_STORAGE_STATIC_BUCKET_NAME = 'static'
MINIO_STORAGE_AUTO_CREATE_STATIC_BUCKET = True
MINIO_STORAGE_STATIC_USE_PRESIGNED = True
MINIO_STORAGE_MEDIA_USE_PRESIGNED = True
urls.py:
...
if settings.DEBUG:
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
i also thought about to set:
MINIO_STORAGE_MEDIA_URL = 'http://127.0.0.1:9000/media'
MINIO_STORAGE_STATIC_URL = 'http://127.0.0.1:9000/static'
but it seems they are getting ignored.
Any idea what could be the issue here?
the bucket(s) are getting created but the url django calls for media files is 127.0.0.1:8000, to the local django webserver and not the minio server like it does for static files (127.0.0.1:9000)
many many thanks in advance.
I'm considering using this library for production. Shall I bite the bullet? ๐
DEFAULT_FILE_STORAGE = "minio_storage.storage.MinioMediaStorage"
STATICFILES_STORAGE = "minio_storage.storage.MinioStaticStorage"
MINIO_STORAGE_MEDIA_BUCKET_NAME = 'media'
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET = True
MINIO_STORAGE_STATIC_BUCKET_NAME = 'static'
MINIO_STORAGE_AUTO_CREATE_STATIC_BUCKET = True
MINIO_STORAGE_ENDPOINT = 'localhost:9000'
MINIO_STORAGE_ACCESS_KEY = 'KBP6WXGPS387090EZMG8'
MINIO_STORAGE_SECRET_KEY = 'DRjFXylyfMqn2zilAr33xORhaYz5r9e8r37XPz3A'
MINIO_STORAGE_USE_HTTPS = False
python manage.py collectstatic --no-input --settings=uihspot.config.settings.local
I'm running locally.
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
collected = self.collect()
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
handler(path, prefixed_path, storage)
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 354, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 260, in delete_file
if self.storage.exists(prefixed_path):
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/minio_storage/storage.py", line 95, in exists
self.client.stat_object(self.bucket_name, self._sanitize_path(name))
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/minio/api.py", line 1045, in stat_object
object_name=object_name)
File "/Users/el/.pyenv/versions/uih-spot/lib/python3.6/site-packages/minio/api.py", line 1900, in _url_open
object_name).get_exception()
minio.error.NoSuchBucket: NoSuchBucket: message: The specified bucket does not exist.
In my requirements.txt
django-minio-storage
works well with minio==2.2.1
, but from this time 30May2017 19:33 GMT+7. minio==2.2.2
by now and I got bug
Hi, I found a bug where running Django's collectstatic
command twice:
Traceback (most recent call last):
File "manage.py", line 6, in <module>
execute_from_command_line(sys.argv)
File "/project/env/lib/python/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/project/env/lib/python/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/project/env/lib/python/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/project/env/lib/python/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/project/env/lib/python/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
collected = self.collect()
File "/project/env/lib/python/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
handler(path, prefixed_path, storage)
File "/project/env/lib/python/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 354, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/project/env/lib/python/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 263, in delete_file
target_last_modified = self.storage.get_modified_time(prefixed_path)
File "/project/env/lib/python/site-packages/django/core/files/storage.py", line 230, in get_modified_time
dt = self.modified_time(name)
File "/project/src/minio_storage/storage.py", line 231, in modified_time
return datetime.datetime.fromtimestamp(info.last_modified)
TypeError: an integer is required (got type time.struct_time)
This bug also appeared when running the tests as is:
py35-django111 inst: /Users/maikel/projects/python-packages/django-minio-storage/.tox/dist/django-minio-storage-0.1.1.dev75+g67e6be7.zip
py35-django111 installed: certifi==2017.7.27.1,chardet==3.0.4,coverage==4.4.1,Django==1.11.6,django-minio-storage==0.1.1.dev75+g67e6be7,idna==2.6,minio==2.2.5,mock==2.0.0,pbr==3.1.1,py==1.4.34,pytest==3.1.1,pytest-cov==2.4.0,pytest-django==3.1.2,pytest-pythonpath==0.7.1,pytz==2017.2,requests==2.18.4,six==1.11.0,urllib3==1.22
py35-django111 runtests: PYTHONHASHSEED='1380463971'
py35-django111 runtests: commands[0] | pytest
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.5.2, pytest-3.1.1, py-1.4.34, pluggy-0.4.0
Django settings: django_minio_storage_tests.settings (from ini file)
rootdir: /Users/maikel/projects/python-packages/django-minio-storage, inifile: tox.ini
plugins: pythonpath-0.7.1, django-3.1.2, cov-2.4.0
collected 34 items
tests/test_app/tests/bucket_tests.py ...
tests/test_app/tests/delete_tests.py .
tests/test_app/tests/retrieve_tests.py FF.....F...........
tests/test_app/tests/upload_tests.py ...........
===================================================================================================== FAILURES =====================================================================================================
_______________________________________________________________________________ RetrieveTestsWithRestrictedBucket.test_accessed_time _______________________________________________________________________________
tests/test_app/tests/retrieve_tests.py:61: in test_accessed_time
self.assertIsInstance(self.media_storage.accessed_time(self.new_file),
minio_storage/storage.py:187: in accessed_time
return self.modified_time(name)
minio_storage/storage.py:200: in modified_time
return datetime.datetime.fromtimestamp(info.last_modified)
E TypeError: a float is required
_______________________________________________________________________________ RetrieveTestsWithRestrictedBucket.test_created_time ________________________________________________________________________________
tests/test_app/tests/retrieve_tests.py:65: in test_created_time
self.assertIsInstance(self.media_storage.created_time(self.new_file),
minio_storage/storage.py:194: in created_time
return self.modified_time(name)
minio_storage/storage.py:200: in modified_time
return datetime.datetime.fromtimestamp(info.last_modified)
E TypeError: a float is required
_______________________________________________________________________________ RetrieveTestsWithRestrictedBucket.test_modified_time _______________________________________________________________________________
tests/test_app/tests/retrieve_tests.py:57: in test_modified_time
self.assertIsInstance(self.media_storage.modified_time(self.new_file),
minio_storage/storage.py:200: in modified_time
return datetime.datetime.fromtimestamp(info.last_modified)
E TypeError: a float is required
======================================================================================= 3 failed, 31 passed in 1.68 seconds ========================================================================================
ERROR: InvocationError: '/Users/maikel/projects/python-packages/django-minio-storage/.tox/py35-django111/bin/pytest'
_____________________________________________________________________________________________________ summary ______________________________________________________________________________________________________
ERROR: py35-django111: commands failed
Hi there
The url returned by MinioStorage should be urlencoded when base_url is not None
, similar to when it is not provided.
The access I make inside docker is the service name: port number. The user needs to access minio-storage. How should I set it?
Packages:
Django==1.11.5
django-minio-storage==0.1.0 (installed from pypi)
minio==2.2.4
When saving file, MinioStorage checks if file exists and is supposed to return false if minio returns NoSuchKey exception. With packages installed above, this causes uncaught exception in Django and the file is not saved. The problem is that NoSuchKey is not a ResponseError so checking if error.code == "NoSuchKey" is not working. Problem can be fixed by importing the except NoSuchKey from minio.error instead.
This is what happens right now if no settings are defined and an instance is created where create_minio_client_from_settings is called in the constructor.
File "/opt/foo/foo/gdpr/storage.py", line 13, in __init__
client = create_minio_client_from_settings()
File "/usr/local/lib/python3.5/site-packages/minio_storage/storage.py", line 385, in create_minio_client_from_settings
secure=secure)
File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 143, in __init__
is_valid_endpoint(endpoint)
File "/usr/local/lib/python3.5/site-packages/minio/helpers.py", line 301, in is_valid_endpoint
if hostname[-1] == '.':
IndexError: string index out of range
def url(self, name, expires=timedelta(days=7)):
# type: (str) -> str
# NOTE: Here be dragons, when a external base_url is used the code
# below is both using "internal" minio clint APIs and somewhat
# subverting how minio/S3 expects urls to be generated in the first
# place.
#
if self.presign_urls:
url = self.client.presigned_get_object(self.bucket_name, name, expires=timedelta(days=7))
if self.base_url is not None:
parsed_url = urlparse(url)
path = parsed_url.path.split(self.bucket_name, 1)[1]
url = '{0}{1}?{2}{3}{4}'.format(
self.base_url, path, parsed_url.params,
parsed_url.query, parsed_url.fragment)
There seems to be a problem with file names beginning with ".". This should be addressed.
I can not bring up your image and your dockerfile does not mention anythin on export
Am I miss something?
$ docker info
Containers: 53
Running: 6
Paused: 0
Stopped: 47
Images: 544
Server Version: 1.12.3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 602
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null overlay bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.30-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.951 GiB
Name: moby
ID: 4CKI:3OK6:CB3O:VLCE:WARA:YJWA:WCUM:4XZF:ESGR:6LZ5:5BED:H5NM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 53
Goroutines: 78
System Time: 2016-11-18T11:16:22.527800156Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: tiras
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
My compose
$ docker-compose -version
docker-compose version 1.9.0-rc4, build 181a4e9
Your compose file
minio:
image: "minio/minio"
command: /export
environment:
MINIO_ACCESS_KEY: key2000
MINIO_SECRET_KEY: secret2000
ports:
- "9000:9000"
My problem:
Recreating uihportal_minio_1
Attaching to uihportal_minio_1
minio_1 | minio: <ERROR> โ/exportโ is not a minio sub-command. See โminio --helpโ.
uihportal_minio_1 exited with code 1
Hy i am using minio as my static and media storage for django application.
These are my django settings
DEFAULT_FILE_STORAGE = 'minio_storage.storage.MinioMediaStorage'
STATICFILES_STORAGE = 'minio_storage.storage.MinioStaticStorage'
MINIO_STORAGE_ENDPOINT = 'mino server url'
MINIO_STORAGE_ACCESS_KEY = os.environ.get('MINIO_STORAGE_ACCESS_KEY')
MINIO_STORAGE_SECRET_KEY = os.environ.get('MINIO_STORAGE_SECRET_KEY')
MINIO_STORAGE_MEDIA_BUCKET_NAME = \
os.environ.get('MINIO_STORAGE_MEDIA_BUCKET_NAME')
MINIO_STORAGE_STATIC_BUCKET_NAME = \
os.environ.get('MINIO_STORAGE_STATIC_BUCKET_NAME')
MINIO_STORAGE_MEDIA_USE_PRESIGNED = True
MINIO_STORAGE_USE_HTTPS = True
When I access django admin. I am getting this response
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied.</Message>
<Key/>
<BucketName/>
<Resource>/static/admin/css/base.css</Resource>
<RequestId>3L137</RequestId>
<HostId>3L137</HostId>
</Error>
Let me know what i am doing wrong
To what extent should the exceptions be translated to IOError and things which django file handling code might be expecting.
Currently, minio is not considered stable or production-ready by its developers, this should be in the README.
Is there any way to specify presigned url expiry time. If i want my object to be accessed only within 1 hour. How do i do that ?
Pick something arbitrarily random instead so that tests can be run even if another minio instance is running on the default port.
An exception is raised...
I haven't looked into the details of how features but it seems to be that the MINIO_PARTIAL_URL
setting can be made implicit if MINIO_PARTIAL_URL_BASE
is set?
If the need for two settings goes away MINIO_PARTIAL_URL
can become the base url setting itself.
It can default to None
or which disables the feature.
@fazpu ?
storage.listdir shall return tuple of (dirs, files)
Now it just returns flat iterable. Not tuple.
https://docs.djangoproject.com/en/2.0/_modules/django/core/files/storage/#Storage.listdir
For example, when user upload a video, I might need to process it with ffmpeg. So firstly, I need to save video to local disk. How to do this?
I saw the Settings of minio:9000 and IP:9153. Is there any difference
Hey there,
I have been using your package and I noticed that the version installed using pip install django-minio-storage
is not up to date with the GitHub repository.
This lead to confusion since I didn't recheck the version installed after the command and confused me. ๐
I resorted at the end to installing it using the command pipenv install git+https://github.com/py-pa/django-minio-storage.git#egg=django-minio-storage
wich was suggested to me after I asked this SO question. Doing so led to some problems concerning my packages and the requirements.txt
file.
I'd like also to note that I noticed several good changes between the version in PyPI and your repository, so I think synchronizing them will help other people use your package more efficiently.
Thank you in advance
https://cl.ly/323p1z151Z3i
https://cl.ly/1H2q2c012K2J
I have them in directory.
(eneos-factory) root@eneos-factory-dev:~/eneos-factory# ls static/rest_framework/img/
glyphicons-halflings.png glyphicons-halflings-white.png grid.png
django-minio-storage==0.1.0
minio==2.2.1 # via django-minio-storage
django==1.11
settings.py:
DEFAULT_FILE_STORAGE = "minio_storage.storage.MinioMediaStorage"
STATICFILES_STORAGE = "minio_storage.storage.MinioStaticStorage"
MINIO_STORAGE_ENDPOINT = "eneos-factory.c0d1um.io"
MINIO_STORAGE_ACCESS_KEY = "KBP6WXGPS387090EZMG8"
MINIO_STORAGE_SECRET_KEY = "DRjFXylyfMqn2zilAr33xORhaYz5r9e8r37XPz3A"
MINIO_STORAGE_USE_HTTPS = True
MINIO_STORAGE_MEDIA_BUCKET_NAME = "media"
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET = True
MINIO_STORAGE_STATIC_BUCKET_NAME = "static"
MINIO_STORAGE_AUTO_CREATE_STATIC_BUCKET = True
# Static files (CSS, JavaScript, Images)
STATIC_URL = '/static/'
# Media configuration
MEDIA_URL = '/media/'
# Proxy server settings: Hey, Django, whenever you get an "X_Forwarded_Proto" in the HTTP request,
# it means a HTTPS request was sent to the proxy server by the end users
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
site-enable:
upstream app {
server localhost:8000;
}
upstream minio {
server localhost:9000;
}
server {
listen 80;
server_name eneos-factory.c0d1um.io;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name eneos-factory.c0d1um.io;
add_header Strict-Transport-Security "max-age=31536000";
ssl_certificate /etc/letsencrypt/live/eneos-factory.c0d1um.io/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/eneos-factory.c0d1um.io/privkey.pem;
location / {
proxy_pass_request_headers on;
proxy_set_header Host $host;
proxy_pass http://app;
}
location /static {
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_pass http://minio;
}
location /media {
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_pass http://minio;
}
location ~ /.well-known {
allow all;
}
}
Where am I wrong?
I saw a bucket-does not exist kind of error instead of a created bucket when developing earlier..
This is not really verified since I don't remember the details but it should be investigated.
I have django 1.4.2 installed for project ..and I want to use django-minio-storage when I am installing it also upgrades my django version. when I revert django version its giving me "No Module named deconstruct"
as its not easy to upgrade django in one from 1.4.2 to 1.11.11. I want to make it work for 1.4.2 please help do that.
I'm having trouble collecting staticfiles into minio server. The issue seems to be the exists
method in storage.py
which is not catching the thrown exception properly when the file does not exists. Here is the log:
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
collected = self.collect()
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
handler(path, prefixed_path, storage)
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 354, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 260, in delete_file
if self.storage.exists(prefixed_path):
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/minio_storage/storage.py", line 95, in exists
self.client.stat_object(self.bucket_name, self._sanitize_path(name))
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/minio/api.py", line 935, in stat_object
object_name=object_name)
File "/home/arash/.virtualenvs/lyan/lib/python3.5/site-packages/minio/api.py", line 1781, in _url_open
object_name).get_exception()
minio.error.NoSuchKey: NoSuchKey: message: The specified key does not exist.
I use self-signed cert by using ssl
openssl genrsa -out private.key 1024
and
openssl req -new -x509 -days 3650 -key private.key -out public.crt -subj "/C=TH/ST=state/L=location/O=organization/CN=domain"
Then put the in .minio
$ pwd
/Users/el/.minio/certs
Sarits-MacBook-Air-2:certs el$ ls
private.key public.crt
Sarits-MacBook-Air-2:uih-portal el$ minio server ~/Pictures/
Endpoint: https://192.168.11.145:9000 https://127.0.0.1:9000
AccessKey: KBP6WXGPS387090EZMG8
SecretKey: DRjFXylyfMqn2zilAr33xORhaYz5r9e8r37XPz3A
Region: us-east-1
SqsARNs:
Browser Access:
https://192.168.11.145:9000 https://127.0.0.1:9000
Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
$ mc config host add myminio https://192.168.11.145:9000 KBP6WXGPS387090EZMG8 DRjFXylyfMqn2zilAr33xORhaYz5r9e8r37XPz3A
Object API (Amazon S3 compatible):
Go: https://docs.minio.io/docs/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/java-client-quickstart-guide
Python: https://docs.minio.io/docs/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
2016/11/22 10:46:35 http: TLS handshake error from [::1]:60412: remote error: tls: unknown certificate authority
I follow your examples as below:
from minio import Minio
from minio.error import ResponseError
MINIO_STORAGE_ACCESS_KEY = "KBP6WXGPS387090EZMG8"
MINIO_STORAGE_SECRET_KEY = "DRjFXylyfMqn2zilAr33xORhaYz5r9e8r37XPz3A"
client = Minio('localhost:9000',
access_key=MINIO_STORAGE_ACCESS_KEY,
secret_key=MINIO_STORAGE_SECRET_KEY,
secure=True)
# Put an object 'my-objectname' with contents from 'my-filepath'
try:
client.fput_object('kimbucket', 'joker.jpg', '/Users/el/Pictures/Joker-Wallpaper-09.jpg')
except ResponseError as err:
print(err)
$ python fput_object.py
Traceback (most recent call last):
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/connectionpool.py", line 595, in urlopen
chunked=chunked)
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/connectionpool.py", line 352, in _make_request
self._validate_conn(conn)
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/connectionpool.py", line 831, in _validate_conn
conn.connect()
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/connection.py", line 289, in connect
ssl_version=resolved_ssl_version)
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 376, in wrap_socket
_context=self)
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 747, in __init__
self.do_handshake()
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 983, in do_handshake
self._sslobj.do_handshake()
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 628, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "fput_object.py", line 33, in <module>
client.fput_object('kimbucket', 'joker.jpg', '/Users/el/Pictures/Joker-Wallpaper-09.jpg')
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/minio-2.0.2-py3.5.egg/minio/api.py", line 564, in fput_object
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/minio-2.0.2-py3.5.egg/minio/api.py", line 1434, in _do_put_object
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/minio-2.0.2-py3.5.egg/minio/api.py", line 1698, in _url_open
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/minio-2.0.2-py3.5.egg/minio/api.py", line 1644, in _get_bucket_region
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/minio-2.0.2-py3.5.egg/minio/api.py", line 1673, in _get_bucket_location
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/poolmanager.py", line 248, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/Users/el/.pyenv/versions/3.5.1/lib/python3.5/site-packages/urllib3/connectionpool.py", line 621, in urlopen
raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
How can I use self-signed on development computer?
Recently I've been doing some work using this library in combination with django management commands to make exporting large amounts of data easy. For several days I was stuck with an error that seemed to be related to the number of calls I was making to minio to fetch stored images. After fetching 28,231 objects from minio, I was receiving this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
At first I believed this to be because the underlying sockets were getting stuck in TIME_WAIT, but had some difficulty testing that hypothesis. After some playing around with solutions, it seems to have actually been a garbage collection issue. Even though the library calls self.file.close()
and self.file.release_conn()
on the underlying file object, the reference to the HTTPConnection that is used to read from minio stays around and given enough requests are made, ties up the system so that no further requests can be made.
My solution to this was to manually set the filefields file
property to None
. I have created a custom storage that uses the ReadOnlyMinioObjectFile
, but this issue seemed to be present on both classes. The solution I used:
# image is a model object, and data is a filefield on that object that uses minio as it's storage
with image.data.open('rb') as file_data:
data_bytes = file_data.read()
file_data.file = None # Without this line, the process will fail after enough connections
My use case is definitely an edge case and this problem probably wouldn't affect most users, but I thought it'd still be a good idea to put this out there in case anyone else ran into it.
The release has just been announced.
https://docs.djangoproject.com/en/1.10/releases/1.10/
Storage.generate_filename()
method, to be investigated as well.Whenever I try to do something with this module, django breaks:
No module named 'minio.policy'
at line .../minio_storage/storage.py:16
Did the original minio API change or something?
It was working flawlessly a few weeks ago.
I don't have time to figure out which of the involved parties I should blame so I'll create an issue about it because I ran out of django_minio time for today....
E0509 17:00:46.331 django.request:135 Internal Server Error: /cms/images/chooser/upload/
Traceback (most recent call last):
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
response = self._get_response(request)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/views/decorators/cache.py", line 43, in _cache_controlled
response = viewfunc(request, *args, **kw)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/wagtail/wagtailadmin/decorators.py", line 27, in decorated_view
return view_func(request, *args, **kwargs)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/wagtail/wagtailadmin/utils.py", line 133, in wrapped_view_func
return view_func(request, *args, **kwargs)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/wagtail/wagtailimages/views/chooser.py", line 132, in chooser_upload
form.save()
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/wagtail/wagtailadmin/forms.py", line 387, in save
return super(BaseCollectionMemberForm, self).save(commit=commit)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/forms/models.py", line 451, in save
self.instance.save()
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/base.py", line 806, in save
force_update=force_update, update_fields=update_fields)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/base.py", line 836, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/base.py", line 922, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/base.py", line 961, in _do_insert
using=using, raw=raw)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/query.py", line 1061, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
for sql, params in self.as_sql():
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1051, in as_sql
for obj in self.query.objs
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1051, in <listcomp>
for obj in self.query.objs
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1050, in <listcomp>
[self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1000, in pre_save_val
return field.pre_save(obj, add=True)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/fields/files.py", line 297, in pre_save
file.save(file.name, file.file, save=False)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/fields/files.py", line 96, in save
setattr(self.instance, self.field.name, self.name)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/fields/files.py", line 378, in __set__
self.field.update_dimension_fields(instance, force=True)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/db/models/fields/files.py", line 481, in update_dimension_fields
width = file.width
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/files/images.py", line 19, in width
return self._get_image_dimensions()[0]
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/files/images.py", line 29, in _get_image_dimensions
self._dimensions_cache = get_image_dimensions(self, close=close)
File "/home/thomasf/.virtualenvs/some_project/lib/python3.6/site-packages/django/core/files/images.py", line 45, in get_image_dimensions
file.seek(0)
io.UnsupportedOperation: seek
It seems that minio >= 4.0.0
removed the minio.policy
module. This module should be update to either follow the new minio implementation or to restrict the valid minio versions.
File "/usr/local/lib/python3.5/site-packages/minio_storage/storage.py", line 16, in <module>
from minio.policy import Policy
ImportError: No module named 'minio.policy'
something like this
levalex/django-minio-storage@8f67bb3
There is issue with import.
Quick fix for issue I had is here.
Write tests for ReadOnlyMinioObjectFile
Document the existence of multiple file class types, why they are useful
and how to use them.
I have Django and Minio servers running on the same server (VM).
When Django performs static files collection or uploading files from app to Minio, I would like it to (for performance reason) communicate with Minio server via localhost
. Of course, these files are still served over the internet via the public domain URL.
If this is the intent, are configurations below correct?
Assuming that minio is running on the default port 9000 and the public domain name is: my.site.com
MINIO_STORAGE_ENDPOINT = 'my.site.com'
MINIO_STORAGE_MEDIA_URL = 'http://localhost:9000/media'
MINIO_STORAGE_STATIC_URL = 'http://localhost:9000/static'
Hello, I want to use minio along with django-minio-storage for storing uploaded media files for my users.
The thing is that I don't want all these uploaded files to be seen by all users but I want to have an access check before downloading each file. Only if the check passes the file will be actually downloaded (for example, a simple check is to only allow each user to download the file he has uploaded).
The easy way to do this is to do the check in django and, if the check passes connect to minio through django, download the file and pass it to the user. As you probably understand this is a major no-no since you should never serve static or media files through your django applications for reasons that should be all too obvious.
In any case, the proper way to handle this using normal media files (along with an http server that supports the X-Sendfile functionality) is with django-sendfile (https://github.com/johnsensible/django-sendfile), i.e you create a get_file
method that will do the actual permission check and if it passes it will send the X-Sendfile header to nginx to actually return the file (or else it will return an 403-Access denied). This combines the best of both worlds, do any permission check you want in django and then notify the nginx (web server) to actually serve the file without any more effort on the django app.
I wasn't able to find something similar for django-minio-storage. From what I could understand, in minio each bucket can have a special access control that can be either public/download/upload or none. This seems like all or nothing; is there a way to have the fine-grained control I need using minio and django-minio-storage?
Please keep in mind that I don't want to actually serve the file through my django app only to do the check in django and if it passes serve it through minio; but only to users that pass the check.
I have a configuration in docker
MINIO_STORAGE_ENDPOINT = 'minio:9000'.
This creates static error links for me "minio:9000/static/img/----"
I can correct it by sending MINIO_STORAGE_MEDIA_URL , but this creates another error in the future.
I will explain: As ENDPOINT is used to generate the image source, this causes an error if the token probiene from a source not set in ENDPOINT.
"The request signature we calculated does not match the signature you provided. Check your key and signing method.
"
one solution is to use MINIO_STORAGE_STATIC_URL to generate the token (if not None), another solution is to use another variable to generate the tokens, something like MINIO_STORAGE_URL_CONNECTION
NOTE: I am using a domain and https to generate all the django files,
https:// cdn.domain.com
It throws exception that it fails to read enough data not reaching right file size.
After install django-minio-storage and configure, i get this error:
Django Version: 1.11.4
Python Version: 3.6.2
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'minio_storage',
'system']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
41. response = get_response(request)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/contrib/admin/options.py" in wrapper
551. return self.admin_site.admin_view(view)(*args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
149. response = view_func(request, *args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
57. response = view_func(request, *args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/contrib/admin/sites.py" in inner
224. return view(request, *args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/contrib/admin/options.py" in add_view
1508. return self.changeform_view(request, None, form_url, extra_context)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapper
67. return bound_func(*args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
149. response = view_func(request, *args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/utils/decorators.py" in bound_func
63. return func.__get__(self, type(self))(*args2, **kwargs2)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/contrib/admin/options.py" in changeform_view
1408. return self._changeform_view(request, object_id, form_url, extra_context)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/contrib/admin/options.py" in _changeform_view
1448. self.save_model(request, new_object, form, not add)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/contrib/admin/options.py" in save_model
979. obj.save()
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/base.py" in save
807. force_update=force_update, update_fields=update_fields)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/base.py" in save_base
837. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/base.py" in _save_table
923. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/base.py" in _do_insert
962. using=using, raw=raw)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/manager.py" in manager_method
85. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/query.py" in _insert
1076. return query.get_compiler(using=using).execute_sql(return_id)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in execute_sql
1106. for sql, params in self.as_sql():
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in as_sql
1059. for obj in self.query.objs
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in <listcomp>
1059. for obj in self.query.objs
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in <listcomp>
1058. [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in pre_save_val
1008. return field.pre_save(obj, add=True)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/fields/files.py" in pre_save
296. file.save(file.name, file.file, save=False)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/db/models/fields/files.py" in save
94. self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/core/files/storage.py" in save
53. name = self.get_available_name(name, max_length=max_length)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/django/core/files/storage.py" in get_available_name
77. while self.exists(name) or (max_length and len(name) > max_length):
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/minio_storage/storage.py" in exists
95. self.client.stat_object(self.bucket_name, self._sanitize_path(name))
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/minio/api.py" in stat_object
913. object_name=object_name)
File "/Volumes/HDD2/Proyectos/python/netflix/lib/python3.6/site-packages/minio/api.py" in _url_open
1774. object_name).get_exception()
Exception Type: NoSuchKey at /admin/system/movie/add/
Exception Value: NoSuchKey: message: The specified key does not exist.
i check in minio server via web and the dir static and media is there, any idea.
Currently, there is a problem with unicode file names.
Explain that some exceptions are wrapped as MinIOError and other might be directly raised to django.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.