Comments (5)
I agree, the race condition is out of our hands, sorry.
I'd also recommend opening the object with a context manager:
with pdf_s3file.open('wb') as fout:
…
The way you do it now with DataSheet(practice, pdf_s3file.open('wb'))
relies on garbage collector to correctly terminate and clean up the object, which is risky in Python. Especially when cloud storages / external APIs are involved.
from smart_open.
I don't see smart_open anywhere in the code sample that you've provided. Where does smart_open come into this?
Please provide a reproducible code sample and stack traces, i fyou can.
from smart_open.
My mistake, I checked and just realized I mistakenly posted this under smart-open but it's actually a dependency of the s3path library. I can move the request over there instead if needed. It looks like smart-open is the culprit however. I put the relevant import for s3path at the top.
from s3path import S3Path, PureS3Path, register_configuration_parameter
register_configuration_parameter(
S3Path(f"/{bucket_name }"),
parameters={
'ServerSideEncryption': 'AES256'
})
def s3_get_path_obj(did, filename='', pure=False):
bname = f"/{bucket_name}"
"""Returns either PureS3Path or S3Path object.
Parameters:
did: Integer of NPI for practice
optional:
filename: String of filename if not directory.
pure: Boolean - Whether to use S3PurePath or S3Path"""
if pure:
s3obj = PureS3Path(bname,'practice',did,filename)
else:
s3obj = S3Path(bname,'practice',did,filename)
return s3obj
Stack trace:
Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 330, in _get
return client.get_object(Bucket=bucket, Key=key, Range=range_string)
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\botocore\client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\botocore\client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\werkzeug\wsgi.py", line 538, in __next__
data = self.file.read(self.buffer_size)
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 610, in read
self._fill_buffer(size)
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 713, in _fill_buffer
bytes_read = self._buffer.fill(self._raw_reader)
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\bytebuffer.py", line 152, in fill
new_bytes = source.read(size)
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 473, in read
self._open_body()
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 438, in _open_body
response = _get(
File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 338, in _get
raise wrapped_error from error
OSError: unable to access bucket: 'smd-set-repo' key: 'practice/1619049558/1619049558.pdf' version: None error: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
from smart_open.
I don't think there's anything smart_open could be doing differently here. boto3 is telling us the object at the specified key does not exist.
If you're seeing different behavior based on timing, then it's possible you're dealing with a race condition, but it's up to you to deal with that (out of scope for smart_open).
from smart_open.
That was it! I just refactored my code to put in the context safety convention. Not sure why I wasn't using it, I've seen it used many times before but probably rushed this.
I managed to generate about 15 different reports with the context manager without any exceptions.
Thank you!
edit: for context if anyone else finds this, this is what I'm doing in Flask:
pdf_s3file = datasheet_3.generate_pdf(did)
customer = Customer(did)
try:
with pdf_s3file.open('rb') as f:
return send_file(f,
attachment_filename=pdf_s3file.name,mimetype='application/pdf')
except Exception as e:
error_msg = f"Error generating datasheet 3 for {pdf_s3file.name}: {e}"
flash(error_msg)
return _render_dashboard_(customer)
from smart_open.
Related Issues (20)
- Copiyng and decompressing huge files on the fly HOT 3
- Slow performance due to lack of buffering for GzipFile.write HOT 6
- S3 multipart uploads to text streams are committed on exception HOT 3
- Error when opening docstring HOT 1
- GCS permission denied 'storage.buckets.get' when using 'open' HOT 2
- python 3.11 support?
- Support for type annotations HOT 3
- Suggeted - allowing cache mechanism for files
- Getting OSError in s3 when permission for kms:Decrypt are missing HOT 4
- S3 open fails on files that contain '@' in their path HOT 5
- Writing to FTP fails with error "503 ASCII (Text) data type is not supported for file transfer operations. Please configure your FTP client to use IMAGE (Binary) type and try again" HOT 1
- Test failures with urllib3 2.0.4 HOT 4
- Compatibility issue with soundfile HOT 1
- Add OAuth2 support HOT 1
- pip install for version 3.0.0 failing HOT 14
- Feature request: zstandard compression HOT 1
- Incompatibility with moto 5 HOT 8
- Version 7.0.0 issue - import botocore error HOT 5
- Inconsistent python_requires minimum version HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from smart_open.