Light

awsdocs / amazon-s3-user-guide Goto Github PK

This guide has been archived. Please see https://github.com/awsdocs/amazon-s3-userguide for an open source version of the Amazon S3 docs.

License: Other

amazon-s3-user-guide's Introduction

This guide has been archived

This guide has been archived. Please see https://github.com/awsdocs/amazon-s3-userguide which combines information from the three retired Amazon S3 guides: Amazon S3 Developer Guide, Console User Guide, and Getting Started Guide.

Amazon S3 User Guide

The open source version of the Amazon S3 Console User Guide. You can submit feedback & requests for changes by submitting issues in this repo or by making proposed changes & submitting a pull request.

License Summary

The documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International License. See the LICENSE file.

The sample code within this documentation is made available under a modified MIT license. See the LICENSE-SAMPLECODE file.

amazon-s3-user-guide's People

Contributors

Stargazers

Watchers

amazon-s3-user-guide's Issues

How clear ClientError in Flask API server?

Describe the bug
A clear and concise description of what the bug is.

This bug occurs in Flask API server to image download. I search this error at Google. I find same issue at this repo. it is here.

boto/botocore#1413

This issue is specific time. this error occur in Flask API server run after about 1hours. I don't know this issue. Help me please.

Steps to reproduce
If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples.

My code is it.

@app.route('/inspection', methods=['GET', 'POST'])
def image_inspection():
    if request.method == "GET":
        check_srl = request.args.get('assetCheckSrl')
        origin_url = request.args.get('origin').replace('"','')
        inspection_url = request.args.get('inspection').replace('"','')
        origin_path = origin_url.split('com/')[1]
        inspection_path = inspection_url.split('com/')[1]

        print("check_srl :" , check_srl)
        print("origin_url :" , origin_url) # https://XXXX.s3.ap-northeast-2.amazonaws.com/XXXX/asset/init/1110_asset_201125173650.jpg
        print("inspection_url :" , inspection_url) # https://XXXX-s3.s3.ap-northeast-2.amazonaws.com/XXXX/asset/init/1110_asset_201125173855.jpg
        print("origin_path :" , origin_path) # XXXX/asset/init/1110_asset_201125173650.jpg
        print("inspection_path :" , inspection_path) # XXXX/asset/init/1110_asset_201125173855.jpg

        origin_image = image_origin_object(origin_path)
        time.sleep(1)
        inspection_image = image_inspection_object(inspection_path)
        ~~~~~~~~~~~~~~~~~~~~~~
        ~~~~~~~~~~~~~~~~~~~~~~
        ~~~~~~~~~~~~~~~~~~~~~~
        compare_url = result_save(result_feature['result']['result_path'], inspection_url, inspection_path)

        return jsonify(result_feature)

def image_origin_object(origin_path):
    
    origin_image = './images/' + origin_path.split('init/')[1]
    s3.Bucket(BUCKET_NAME).download_file(origin_path, origin_image)
    
    return origin_image

def image_inspection_object(inspection_path):

    inspection_image = './images/' + inspection_path.split('init/')[1]
    print('inspection_path >>> : ', inspection_path)
    print('inspection_image >>> : ', inspection_image)
    s3.Bucket(BUCKET_NAME).download_file(inspection_path, inspection_image)

    return inspection_image

def result_save(file_name, inspection_url, inspection_path):

    s3_client = boto3.client('s3')
    inspection_url = inspection_url.replace('init','compare')
    KEY = inspection_path.replace('init','compare')
    response = s3_client.upload_file(file_name, BUCKET_NAME, KEY)

    return inspection_url

I referenced boto3 document

https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3-example-download-file.html

Expected behavior
A clear and concise description of what you expected to happen.

I make Flask API server for image processing in AWS EC2. AWS S3 Bucket is public setting. AWS EC2 configure already set my KEY, region.

Debug logs
Full stack trace by adding

XX.XX.236.140 - - [26/Nov/2020 04:28:55] "GET /inspection?assetCheckSrl=89
&origin=https://XXXX.s3.ap-northeast-2.amazonaws.com/XXXX/asset/init/1110_asset_201125173650.jpg
&inspection=https://XXXX-s3.s3.ap-northeast-2.amazonaws.com/XXXX/asset/init/1110_asset_201125173855.jpg HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ubuntu/environment/XXXX-XXXX-IMAGE/app.py", line 37, in image_inspection
    origin_image = image_origin_object(origin_path)
  File "/home/ubuntu/environment/XXXX-XXXX-IMAGE/app.py", line 57, in image_origin_object
    s3.Bucket(BUCKET_NAME).download_file(origin_path, origin_image)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 246, in bucket_download_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/s3transfer/download.py", line 343, in _submit
    **transfer_future.meta.call_args.extra_args
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/environment/venv/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

S3 event notification limits

Should this page detail how many S3 event notifications are supported?

https://github.com/awsdocs/amazon-s3-user-guide/blob/master/doc_source/setup-event-notification-destination.md

I vaguely recall it was around ~1000 concurrent events for S3, but I cannot easily find it :/

Overcoming problems deleting S3 buckets

Is there a place for tips for overcoming problems deleting buckets?

Example: a bucket created using Elastic Beanstalk had a generated policy that precluded bucket deletion using normal methods. Using a tip from Stack Overflow I deleted the policy and then was able to delete the bucket using the described procedures.

Why S3 bucket can't enfoce by tags?

We need to enforce S3 bucket based on Some Tags. While checking S3 Action condition Keys there is no aws:RequestTag option. Is there a reason for that?

Can anyone help me to fix this problem?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.