Defining iterator in a separate class no longer works in 3.13

[antonio-rojas]

Bug report

Bug description:

Defining an interator for a class in a separate class no longer works properly in 3.13. With the following test_iter.py:

class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]

With Python 3.13.1 one gets:

>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    [x for x in X]
                ^
TypeError: 'list2iterator' object is not iterable

With Python 3.12.7 it works:

>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
[1, 2, 3]

Bisected to bcc7227e

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux

[Eclips4]

cc @efimov-mikhail @markshannon @JelleZijlstra

[ronaldoussoren]

https://docs.python.org/3/library/stdtypes.html#iterator-types says that iterators must also implement __iter__ (returning self). That's a long standing requirement on iterators, and not one introduced in 3.13.

[antonio-rojas]

Alright, thanks

[Eclips4]

@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12:
example.py:

class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]


target = list2([1, 2, 3])
for i in target:
    print(i)


[print(i) for i in target]

3.12.8:

eclips4@suffering ~/tmp> python3.12 example.py
1
2
3
1
2
3

3.13.1:

1                                                                                  
2
3
Traceback (most recent call last):
  File "/Users/eclips4/tmp/example.py", line 23, in <module>
    [print(i) for i in target]
                       ^^^^^^
TypeError: 'list2iterator' object is not iterable

Though it probably shouldn't work, it is definitely a regression.

Bytecode for [x for x in y] 3.12.8:

  0           0 RESUME                   0

  1           2 LOAD_NAME                0 (y)
              4 GET_ITER
              6 LOAD_FAST_AND_CLEAR      0 (x)
              8 SWAP                     2
             10 BUILD_LIST               0
             12 SWAP                     2
        >>   14 FOR_ITER                 4 (to 26)
             18 STORE_FAST               0 (x)
             20 LOAD_FAST                0 (x)
             22 LIST_APPEND              2
             24 JUMP_BACKWARD            6 (to 14)
        >>   26 END_FOR
             28 SWAP                     2
             30 STORE_FAST               0 (x)
             32 RETURN_VALUE
        >>   34 SWAP                     2
             36 POP_TOP
             38 SWAP                     2
             40 STORE_FAST               0 (x)
             42 RERAISE                  0
ExceptionTable:
  10 to 26 -> 34 [2]

3.13.1:

   0           RESUME                   0

   1           LOAD_NAME                0 (y)
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (x)
               SWAP                     2
       L1:     BUILD_LIST               0
               SWAP                     2
               GET_ITER  - problematic instruction
       L2:     FOR_ITER                 4 (to L3)
               STORE_FAST_LOAD_FAST     0 (x, x)
               LIST_APPEND              2
               JUMP_BACKWARD            6 (to L2)
       L3:     END_FOR
               POP_TOP
       L4:     SWAP                     2
               STORE_FAST               0 (x)
               RETURN_VALUE

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (x)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [2]

[ronaldoussoren]

@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12: example.py:

Right, the code from the OP is broken according to the data model, but we still did a change that's not backward compatible. I'm not deep enough into this to fully understand the ramifications.

[efimov-mikhail]

This behavior change is a direct consequence of change in bytecode generation in my PR. I don't think this is a bug.

[terryjreedy]

The exact example worked in the CPython implementation of 3.12, but that may be an accident, as it should not have by the language definition. It might have already failed in 3.12 in other implementations. In any case, adding the following exposes the defect in list2iterable even in 3.12.

target2 = iter(target)  # A list2iterator object
target3 = iter(target2)  # Fails in 3.12.8 
# TypeError: 'list2iterator' object is not iterable

If there is still a question about changing anything, Guido might remember the original design discussions.

[pochmann3]

Relevant bit from glossary: iterator:

CPython implementation detail: CPython does not consistently apply the requirement that an iterator define __iter__().

[terryjreedy]

Right. A defective iterator may or may not work, so skip the requirement at your own risk. Int caching is another implementation detail. Implementation details are 1. subject to change and 2. are sometimes noted to discourage the filing of issues. I think the iterator note followed a discussion and decision that the inconsistency should not be treated as a bug. But the note also reaffirms the requirement. Removing it, at least for for-loops, would have required other implementations and future CPythons to allow the omission. All that is needed for list2iterator is def __iter__(self): return self. This is not a great burden. Its presence would make list2iterator also work for iter(list2iterator(somelist)).

[rhettinger]

FWIW, the OP's code also works in PyPy3, so this is more than a C implementation detail.

[serhiy-storchaka]

I think that the original change was not needed snd should be reverted. It just slows dowsn everyone's code.