David R. MacIverWhile we're talking about weird Python edge cases, what do you think this does?
```
class Foo:
def __getitem__(self, i):
return i
def __len__(self):
return 5
print(list(Foo()))
```
The answer is so counter-intuitive to me that I have independently discovered it twice, about a year apart, and posted outraged comments about it in discord both times, and the second time had to be reminded that this was not the first time I'd discovered the behaviour.
It's possible it's also not the first time I've posted it on mastodon.
CF Bolz-Tereick@DRMacIver hmmmmm, if it were the obvious [0, 1, 2, 3, 4] you wouldn't be outraged. so let me guess, it's an infinite loop? a MemoryError?
@cfbolz Correct, yes. It will hang until it eventually runs out of memory.
Now, do you know why?
Jonathan Hartley@DRMacIver @cfbolz Wild guess: the iteration doesn't consult `__len__`, but just keeps retrieving incremental indexes waiting for one to raise an exception.
UPDATE: my language is sloppy, but you get what I'm arm-waving about.
@tartley @DRMacIver yeah, it probably waits for an IndexError?
@cfbolz @tartley Yup, exactly that. Default iteration is like:
```
i = 0
while True:
try:
yield x[i]
except IndexError:
break
i += 1
```
Instead of:
```
for i in range(len(x)):
yield x[i]
```
In some sense you should never be able to tell the difference in correct code, so you'll rarely notice this, but I've at least twice managed to write incorrect code that doesn't raise for out of bounds access
@DRMacIver @tartley I think it's a bit of a general pattern in python, that the language mostly doesn't really use more than one special method in combination (with some exceptions of course)
@DRMacIver @tartley also, re the hang: is it actually not even ctrl-c-able? that's *really* annoying?!
@cfbolz @DRMacIver oh darn I hadn't expected that. I confess I don't know what determines that. Now you've nerd sniped me into figuring that out...
David Beazley@tartley @cfbolz @DRMacIver It's probably running a tight loop in some C code somewhere outside of the main eval loop where ctrl-c checking happens.
@dabeaz @tartley @DRMacIver nope, it was actually my own fault. ctrl-c works fine in CPython, but actually not in pypy3 😬 (it's my default python, so I tried there)
@tartley @dabeaz @DRMacIver I mean, you can definitely get CPython to enter an uninterrupted loop with the approach below. you get bonus points for telling me why operator.index works as `__getitem__` despite not taking a self?! (I don't know the answer to that). and why ctrl-c doesn't work here.
Jason Madden@cfbolz @tartley @dabeaz @DRMacIver `opeartor.index` is a builtin function, which means it isn't a descriptor like a regular function. In effect, it's the same as this: ``__getitem__ = staticmethod(lambda i: i)``
@ossmkitty @tartley @dabeaz @DRMacIver yeah, exactly, thank! I was just hunting that down too! basically it's because index.__get__ does not exist.
so this works for anything that's callable with one argument but without __get__. things I just tried:
danzin@cfbolz @ossmkitty @tartley @dabeaz @DRMacIver Ohoho, those look just like the kind of weirdness that can uncover interpreter and extension bugs!
Thank you for your contribution to my fuzzing habit :)
@cfbolz @tartley @DRMacIver Whoa. That's freaky. I don't think I've ever seen the __index__ method before. Couldn't tell you want it does or what purpose it serves.
@dabeaz @tartley @DRMacIver it's very similar to __int__ and used for sequence indexing? the main point iirc was to forbid indexing into with floats 🤷♀️
@dabeaz @tartley @DRMacIver anyway, it's a super old feature, from python 2.5 times: https://peps.python.org/pep-0357/ (so you likely saw it at some point and suppressed the memory ;-))
PEP 357 – Allowing Any Object to be Used for Slicing | peps.python.org
This PEP proposes adding an nb_index slot in PyNumberMethods and an __index__ special method so that arbitrary objects can be used whenever integers are explicitly needed in Python, such as in slice syntax (from which the slot gets its name).
@dabeaz @tartley @DRMacIver bit mean, but it's in section '4.12 Conversion Protocols' of Python Distilled, no? 😬😂
@cfbolz @tartley @DRMacIver Could be. I'll refer to my PTSD comment from earlier). Arg. The surface area is too damn big and there are corner cases everywhere!
@cfbolz @tartley @DRMacIver Since we're on the general topic of WTF?!? Here's one of my favorite examples to show and discuss in class:
@dabeaz @tartley @DRMacIver wow, that's amazing/horrible! NamedTuple isn't even a class at all, but a function?!
@DRMacIver @dabeaz @tartley inheriting from something that's not a type just means calling it, I think. That's definitely in the 'weird corner cases' category again though,so I don't remember the details 😬
@dabeaz @tartley @DRMacIver collections.namedtuple has long been a private enemy of PyPy because it used to use really bad anti-patterns in the way it does meta programming. Nowadays it's fixed I think, but for a couple of CPython releases we were basically constantly annoyed with that code
@cfbolz @tartley @DRMacIver One of the things I love about this is asking students what they think the answer ought to be first and maybe taking a poll before revealing anything.
@dabeaz @tartley @DRMacIver heh, that's a slightly mean exercise.
My main question is why we still use named tuples at all, now that data classes are a thing
Pozorvlak@cfbolz @dabeaz @tartley @DRMacIver Interestingly/horribly, this hasn't always been the case: the first Python interpreter I tried this on was Python 3.7, when typing.NamedTuple was a class. The isinstance call still returned False, of course, because everything about named tuples is cursed.
@pozorvlak @cfbolz @tartley @DRMacIver Wait, NamedTuple started as a class and they changed it to a function? Good god.
Tin Tvrtković@dabeaz @cfbolz @tartley @DRMacIver hold on, this works nowadays? I remember having to commit crimes to get namedtuples to work in cattrs
@tintvrtkovic @cfbolz @tartley @DRMacIver I think this NamedTuple inheritance thing has "worked" since at least 3.6.
@DRMacIver @tartley great, isn't it? In your example ctrl C works because signals are checked at the start of every python function. In my example, the __getitem__ is implemented in C, so no python code ever runs that can be interrupted. So it gets killed by the kernel oom killer eventually 🤷♀️
Tony Garnock-Jones@DRMacIver wtf. Soooo in pep 234 we have discussion of whether IndexError should terminate iteration, some negative commentary, but no clear decision... and in the implementation... :-(
@tonyg TBF these are not exactly contradictory. This isn't the normal iterator protocol, it's the fallback protocol for C-style iteration when you don't have an iterator.
Still hate it though.
@tonyg I think if an iterator raises IndexError (instead of StopIteration) it gets propagated, but actually I don't know this for sure and I'm afraid to check.
@DRMacIver Seems reasonable - just this fallback thing that's, er, underdocumented is less than reasonable...
This, for example, produces 0 thru 4 when iterated over, and checking the pep and a few cursory google searches did not explain this satisfactorily:
class Foo:
def __getitem__(self, i):
if i < 5: return i
raise IndexError()