Comments (6)
teocns
commented on August 22, 2024
1
Facing the same issue. Something's off with module parser: running pytest tests/subdir passes
(.venv) user:~/proj/proj-agent $ pytest tests/agent/ PASSED
(.venv) user:~/proj/proj-agent [j1] [!1] $ pytest tests/agent/
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.11.0, pytest-8.2.1, pluggy-1.5.0
rootdir: /Users/user/proj/proj-agent
configfile: pytest.ini
plugins: anyio-4.3.0, Faker-25.6.0, asyncio-0.23.7, mock-3.14.0, xdist-3.6.1
asyncio: mode=Mode.STRICT
collected 7 items
tests/agent/tools/test_registry.py ... [ 42%]
tests/agent/tools/test_tool.py .... [100%]
===================================================================== 7 passed in 0.27s =====================================================================
whereas running bare pytest without a path specified - thus project root - will have dill tests fail:
(.venv) user:~/proj/proj-agent [j1] $ pytest -k test_tool FAIL
(.venv) user:~/proj/proj-agent [j1] $ pytest -k test_tool
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.11.0, pytest-8.2.1, pluggy-1.5.0
rootdir: /Users/user/proj/proj-agent
configfile: pytest.ini
plugins: anyio-4.3.0, Faker-25.6.0, asyncio-0.23.7, mock-3.14.0, xdist-3.6.1
asyncio: mode=Mode.STRICT
collected 128 items / 124 deselected / 4 selected
tests/agent/tools/test_tool.py F.FF [100%]
========================================================================= FAILURES ==========================================================================
_____________________________________________________________________ test_tool_wrapper _____________________________________________________________________
fn = <function fn.<locals>._fn at 0x187c70c20>
def test_tool_wrapper(fn):
> otool = Tool.wrap(fn)
tests/agent/tools/test_tool.py:19:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
proj/agent/tools/tool.py:64: in wrap
payload=dill.dumps(obj),
.venv/lib/python3.11/site-packages/dill/_dill.py:280: in dumps
dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
.venv/lib/python3.11/site-packages/dill/_dill.py:252: in dump
Pickler(file, protocol, **_kwds).dump(obj)
.venv/lib/python3.11/site-packages/dill/_dill.py:420: in dump
StockPickler.dump(self, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:487: in dump
self.save(obj)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1985: in save_function
_save_with_postproc(pickler, (_create_function, (
.venv/lib/python3.11/site-packages/dill/_dill.py:1112: in _save_with_postproc
pickler._batch_setitems(iter(source.items()))
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <dill._dill.Pickler object at 0x187c27e90>, obj = <_io.TextIOWrapper name="<_io.FileIO name=6 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>
save_persistent_id = True
def save(self, obj, save_persistent_id=True):
self.framer.commit_frame()
# Check for persistent id (defined by a subclass)
pid = self.persistent_id(obj)
if pid is not None and save_persistent_id:
self.save_pers(pid)
return
# Check the memo
x = self.memo.get(id(obj))
if x is not None:
self.write(self.get(x[0]))
return
rv = NotImplemented
reduce = getattr(self, "reducer_override", None)
if reduce is not None:
rv = reduce(obj)
if rv is NotImplemented:
# Check the type dispatch table
t = type(obj)
f = self.dispatch.get(t)
if f is not None:
f(self, obj) # Call unbound method with explicit self
return
# Check private dispatch table if any, or else
# copyreg.dispatch_table
reduce = getattr(self, 'dispatch_table', dispatch_table).get(t)
if reduce is not None:
rv = reduce(obj)
else:
# Check for a class with a custom metaclass; treat as regular
# class
if issubclass(t, type):
self.save_global(obj)
return
# Check for a __reduce_ex__ method, fall back to __reduce__
reduce = getattr(obj, "__reduce_ex__", None)
if reduce is not None:
> rv = reduce(self.proto)
E TypeError: cannot pickle 'EncodedFile' object
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:578: TypeError
The pickled subject being a function referenced from within a fixture's inner scope
@pytest.fixture
def fn():
def _fn(*args):
import functools
if not args:
return 0
return functools.reduce(lambda x, y: x + y, args)
return _fn
Other observations
- Adding
__init__.pytotests/and subdirs make tests fail with the same error no mater what path is specified - ...
from pytest.
qthequartermasterman
commented on August 22, 2024
This is also probably relevant. If I extend the example to include dill.loads, and then disable capture with pytest -s, I get a recursion error. Again, this code works as expected with vanilla doctest and in a script.
class MyClass:
"""Running a file containing this class with `python -m pytest this_file.py --doctest-modules` will fail with `TypeError: cannot pickle 'EncodedFile' object`.
Examples:
>>> def template_function():
... return "Hello, World!"
>>> import dill
>>> string = dill.dumps(template_function)
>>> dill.loads(string)()
'Hello, World!'
"""010 >>> string = dill.dumps(template_function)
011 >>> dill.loads(string)()
UNEXPECTED EXCEPTION: RecursionError('maximum recursion depth exceeded')
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest test_client.MyClass[3]>", line 1, in <module>
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 286, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 272, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 419, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
[Previous line repeated 955 more times]
RecursionError: maximum recursion depth exceeded
from pytest.
RonnyPfannschmidt
commented on August 22, 2024
Does it work when IO capture is disabled? pytest -s
from pytest.
qthequartermasterman
commented on August 22, 2024
Does it work when IO capture is disabled?
pytest -s
I get a different error during unpickling that also only occurs with pytest. You can see my second post in this issue for those details.
from pytest.
RonnyPfannschmidt
commented on August 22, 2024
Based on the traceback dill pickles random objects
We might be able to give better errors but we cant fix dill doing things that are fragile
from pytest.
RonnyPfannschmidt
commented on August 22, 2024
You need to disable assertion rewrite for at least the module
And then hope
It's generally unsafe to serialize inner functions in any way
Dill is playing with fire there and it breaks whenever someone throws in objects that aren't serializable
from pytest.
Related Issues (20)
- Weird collection stats when filtering from `pytest_collection_modifyitems` hook HOT 1
- Too tight a type constraint on ExceptionInfo.errisinstance HOT 3
- Cache: Crash on Windows when using pytest-xdist HOT 5
- It would be good to have fixtures with dinamically changed / different scopes HOT 2
- CLI flag for supporting env variables from .env file HOT 3
- Erroneous quotes in `tmp_path_retention_policy` example in docs
- Tests not collected if parametrized with `itertools` objects and imported in multiple files HOT 3
- Doctests in __main__ are ignored HOT 3
- [TODO] Upload JUnit reports to Codecov HOT 2
- No tests are run when PyTest encounters some error condition on some of them HOT 1
- Warn on test arguments with default values? HOT 1
- [8.1.x][8.3.x] `testpaths` not taken into account for initial conftests loading when set through configuration file? HOT 4
- Surprisingly, pytest doesn't collect warnings when importing the package for plugins HOT 4
- Support navigating exceptions chains in pdb
- short test summary and -vv
- --failed-first and --new-first don't play well together
- Please make something working HOT 1
- Enable test cases to be executed in a coroutine manner HOT 1
- pytest collection appears to stall/slow down/jam up when some third-party libraries are used; add function to ignore specific modules HOT 2
- Regression in testdir fixture HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest.