async-retriever bug about async-retriever HOT 7 CLOSED

Thanks for reporting this issue. Unfortunately, I cannot reproduce this error. I used the same env file that you mentioned and tried it on macOS and Linux, and they both worked fine without any issue. I compared the package versions that were installed in the env that I created with yours, and they were exactly the same. Maybe you're not activating the correct env?

As an aside, your mamba version is way outdated, the latest version is 1.1.0 which is their production version. So, I'd suggest updating yours.

from async-retriever.

Thanks Taher, I think I might have figured out the issue. In my working folder there is a cache/ folder with aiohttp_cache.sqllite. It must have become corrupted. I realized if I used the environment in another folder it worked fine, but not in my working folder. After deleting the .sqllite file - it works. Just an FYI.

from async-retriever.

Ah, ok. Yeah, it's possible that a cache that was created with a previous version is not compatible with a newer version. Thanks for letting me know. I am going to close it for now, please reopen if the issue persists.

from async-retriever.

Taher - I'd been bumping into this problem for a while not understanding what the issue was. Previously I'd been pinning the package versions to an older version to get it to work. Wondering if there is a way you can clear the cache or ? if there is an inconsistency automatically, otherwise won't this happen again? Or maybe raising an error that informs the user.

from async-retriever.

I see. If that's something that happened to you several times, we should investigate. I never had this issue before, and I rarely delete caches for exactly catching these types of issues for testing purposes. There were some instances of cache corruption due to some network and sqlite issues that I had to delete, but never the issue that you had.

That said, can you tell me more about this issue, and what were the circumstances that you had this issue other than what you've already said? For example, if you know the version of aiohttp-client-cache (the package that async-retriever uses) that the existing cache was generated with. Based on the traceback that you submitted, it seems that a cached response that has been loaded does not have .method but when I checked aiohttp-client-cache, this method has been there over the past couple of major versions.

Technically, URLs, keywords, and pickled responses are stored. So, my only guess is that, there can be a situation where there's a breaking change in aiohttp-client-cache that leads to loading a pickled object that's not compatible with the current version. But, I never bumped into this issue.

from async-retriever.

Regarding automatic handling of these types of errors, it's difficult to generalize and catch. I think it'd be best if users handle them on a case-by-case basis.

For deleting a certain cached request and response, you can use the delete_url_cache, please take a look at README of async-retriever for an example. But if you want to delete the whole cache, one way can be something like this:

try:
    # some functions
except AttributeError as e:
    if "CachedResponse" in str(e):
        Path("cache", "aiohttp_cache.sqlite").unlink()
    # retry

Or you can simply disable caching altogether:

import os

os.environ["HYRIVER_CACHE_DISABLE"] = "true"

from async-retriever.

@rmcd-mscb The issue seems to be related to a bug in aiohttp-client-cache. I was able to reproduce your error. After installing the latest version of aiohttp-client-cache which is v0.8.1. The issue was fixed. Please give it a try and let me know it solves the issue.

from async-retriever.