Giter VIP home page Giter VIP logo

Comments (5)

piskvorky avatar piskvorky commented on June 23, 2024 2

I agree, the race condition is out of our hands, sorry.

I'd also recommend opening the object with a context manager:

with pdf_s3file.open('wb') as fout:
    …

The way you do it now with DataSheet(practice, pdf_s3file.open('wb')) relies on garbage collector to correctly terminate and clean up the object, which is risky in Python. Especially when cloud storages / external APIs are involved.

from smart_open.

mpenkov avatar mpenkov commented on June 23, 2024

I don't see smart_open anywhere in the code sample that you've provided. Where does smart_open come into this?

Please provide a reproducible code sample and stack traces, i fyou can.

from smart_open.

atomey avatar atomey commented on June 23, 2024

My mistake, I checked and just realized I mistakenly posted this under smart-open but it's actually a dependency of the s3path library. I can move the request over there instead if needed. It looks like smart-open is the culprit however. I put the relevant import for s3path at the top.


from s3path import S3Path, PureS3Path, register_configuration_parameter

register_configuration_parameter(
    S3Path(f"/{bucket_name }"),
    parameters={
        'ServerSideEncryption': 'AES256'
    })

def s3_get_path_obj(did, filename='', pure=False):
    bname = f"/{bucket_name}" 
    """Returns either PureS3Path or S3Path object.
    
    Parameters:
            did: Integer of NPI for practice
            optional:
            filename: String of filename if not directory.
            pure: Boolean - Whether to use S3PurePath or S3Path"""
    if pure:
        s3obj = PureS3Path(bname,'practice',did,filename)
    else:
        s3obj = S3Path(bname,'practice',did,filename)
    return s3obj

Stack trace:

Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 330, in _get
    return client.get_object(Bucket=bucket, Key=key, Range=range_string)
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\botocore\client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\botocore\client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\werkzeug\wsgi.py", line 538, in __next__
    data = self.file.read(self.buffer_size)
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 610, in read
    self._fill_buffer(size)
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 713, in _fill_buffer
    bytes_read = self._buffer.fill(self._raw_reader)
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\bytebuffer.py", line 152, in fill
    new_bytes = source.read(size)
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 473, in read
    self._open_body()
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 438, in _open_body
    response = _get(
  File "C:\Users\adame\.virtualenvs\SET-7TiNCuEs\Lib\site-packages\smart_open\s3.py", line 338, in _get
    raise wrapped_error from error
OSError: unable to access bucket: 'smd-set-repo' key: 'practice/1619049558/1619049558.pdf' version: None error: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

from smart_open.

mpenkov avatar mpenkov commented on June 23, 2024

I don't think there's anything smart_open could be doing differently here. boto3 is telling us the object at the specified key does not exist.

If you're seeing different behavior based on timing, then it's possible you're dealing with a race condition, but it's up to you to deal with that (out of scope for smart_open).

from smart_open.

atomey avatar atomey commented on June 23, 2024

That was it! I just refactored my code to put in the context safety convention. Not sure why I wasn't using it, I've seen it used many times before but probably rushed this.

I managed to generate about 15 different reports with the context manager without any exceptions.

Thank you!

edit: for context if anyone else finds this, this is what I'm doing in Flask:

pdf_s3file = datasheet_3.generate_pdf(did)

customer = Customer(did)
try:
    with pdf_s3file.open('rb') as f:
        return send_file(f,
                        attachment_filename=pdf_s3file.name,mimetype='application/pdf')
except Exception as e:
    error_msg = f"Error generating datasheet 3 for {pdf_s3file.name}: {e}"
    flash(error_msg)
    return _render_dashboard_(customer)

from smart_open.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.