Comments (5)
Yeah, unfortunately that's by design, but maybe there's a workaround.
What's happening is the presence of list[int]
causes iterable args to check their first member. And the Foo
instance does exist at that point (after __new__
), but isn't initialized yet. Is that why __iter__
is raising an error?
Workarounds for Foo
:
- use
list
without a subscript - hack
__iter__
to return without error if uninitialized
Fixes for multimethod:
- catch
TypeError
on callingiter
; detecting whether an object is iterable has never been perfect - switch from detecting
Iterable
toCollection
; wouldn't help if the class is also a collection - special case methods named
__init__
- use a different type checker for each position; in this case
arg
would be iterated, but notself
They all seem a little hacky to me at first glance. The root problem here is dispatching on a dynamically typed language; there's no "right" way to determine an arg is-a list[int]
. Only checking the first member is already cheating.
from multimethod.
So . . . since this was a blocking issue for me (in this and in three other overload decorator packages . . .), I went ahead and wrote my own decorator from scratch. Obviously, it's far less tested than yours and it was a huge pain to write, but it does seem to work everywhere.
In the process of doing this, I think I encountered a similar problem as you did. I'm still not sure I quite understand quite where/why the problem occurs in multimethod
, but I think I see the problem it breaks on: namely, if one of the argument types is of type types.GenericAlias
(e.g. list[int]
), there's no easy way to tell whether the type of an argument matches it (e.g. an argument [0,1,2]
of type list
doesn't match it).
I think the right answer here is not to try to inspect the argument before you know it's actually a list
.
My solution is, if the required argument is of type types.GenericAlias
, to simply check that the argument satisfies it using the types themselves. E.g., for an argument that's supposed to be a list[int]
, it checks that it's (1) a list
. Only the argument actually is a list
does it then (2) check that each of its elements is an int
.
I'm maybe not explaining it well, but it should be pretty obvious. And, in case it's not, here's some actual code:
def generic_match( tgen:types.GenericAlias, targ:type,arg:typing.Any ) -> bool:
assert type(arg) == targ
if targ != tgen.__origin__:
return False
if tgen.__origin__ == tuple: #note: I didn't test this branch yet
return tuple(map(type,arg)) == tgen.__args__
elif tgen.__origin__ == list :
assert len(tgen.__args__) == 1
telems:type = tgen.__args__[0]
for elem in arg:
if not issubclass( type(elem), telems ): return False
return True
else:
assert False #Not implemented
Note this is incomplete; I think there probably would have to be some recursive testing here too.
Of course, the immediate downside of this approach is that this can have extremely bad performance (e.g. try doing a thousand overloaded calls, where each time you have to check a list with a million elements . . .).
There are various ways around that—for example, just checking that the first element, if there is one, is the right type and hoping for the best on the rest. The tradeoff is that you can get false matches. Or, you can just check that the generic type matches and don't check any of the contents. This is simplest and fastest, but in addition to giving false matches, you lose all ability to disambiguate subtypes of the generic. For example, my decorator can disambiguate all of these overloads:
def foo( arg:int ): #...
def foo( arg:list[int ] ): #...
def foo( arg:list[float] ): #...
. . . which I think is pretty neat. Note the different types on the list
.
Sorry for rambling. Hopefully this gives some ideas. Maybe if I understood the Python typesystem a bit better I could come up with something clearer. Also, I can probably work up a gist or something if this is unclear (but I'm not going to do so pre-emptively since I think it's fairly straightforward and you've suffered enough of my code already :) ).
from multimethod.
65b0266 has a fix for your example. It only uses the recursive type checker on params that use subscripts.
I think the right answer here is not to try to inspect the argument before you know it's actually a list.
The catch is making that performant. The dispatch works in 2 phases::
- determine the types of the arguments - uncached
- compare the types using
issubclass
- cached by type
So if the argument is Foo()
, step 1 has to know in advance whether matching Iterable[...]
is a possibility in step 2. If so step 1 must resolve the arg to Foo[...]
, otherwise it could stop at just Foo
.
from multimethod.
(Thanks for looking into the fix. I confirm this fixes at least this use-case—at least enough to re-enable this as an option in my code.)
I indeed ran into performance issues with my implementation. However, I found that adding a small cache basically fixes it.
My key for caching works somewhat differently though, I think: I just hash the types of the arguments (expanding list
s and tuple
s, as necessary; maybe could expand other types, e.g. anything that's iterable?). Also, the value in the cache is the original wrapped function.
I'm not sure whether hashing the types and doing a table lookup vs. checking them against cached types is faster—though if I had to guess, the former has the edge. For the record, since the patch fixes the issue, I can compare the implementations; mine is about 5% faster. This does not necessarily mean the caching method is better fundamentally; just that there's room to be gained.
from multimethod.
fc5b004 added a type checker that is adaptive to the registered parameters. It should do minimal iterating on arguments, while still supporting Unions, nested subscripts, etc.
from multimethod.
Related Issues (20)
- Regression in 1.10 regarding typing.Literal HOT 6
- Clarify if `@multimethod` works with position-or-keyword arguments HOT 1
- Clarify "Positional-only" in Readme HOT 1
- Issues with thread safety
- Is there a way to dispatch superclass methods? HOT 1
- F811 redefinition of unused '__init__' from line 33 HOT 2
- Thread safety fix HOT 1
- Doesn't work properly with pydantic types HOT 2
- Using >1 different Union raises TypeError HOT 1
- Make __call__ faster to step through in the debugger HOT 2
- multimethod does not recognize correctly 'collections.abc.Container' types HOT 7
- TypeError: type 'dict_keys' is not an acceptable base type
- Strange behaviour with pandas function application methods HOT 1
- multimethod "forgets" keyword only arguments HOT 2
- Documentation isn't updated? HOT 5
- Overload fails on Optional arguments HOT 1
- flake8 reports redefinition when using multimethod HOT 2
- Classmethod fails on the uninitialized class object HOT 2
- Different names for the multimethod in a class does not work with staticmethod and classmethod decorators HOT 1
- PEP about multiple dispatch and @typing.overload
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from multimethod.