There are a few situations where seemingly harmless use of a borrowed
reference can lead to problems. These all have to do with implicit
invocations of the interpreter, which can cause the owner of a
reference to dispose of it.
The first and most important case to know about is using
Py_DECREF() on an unrelated object while borrowing a
reference to a list item. For instance:
This function first borrows a reference to list[0], then
replaces list[1] with the value 0, and finally prints
the borrowed reference. Looks harmless, right? But it's not!
Let's follow the control flow into PyList_SetItem(). The list
owns references to all its items, so when item 1 is replaced, it has
to dispose of the original item 1. Now let's suppose the original
item 1 was an instance of a user-defined class, and let's further
suppose that the class defined a __del__() method. If this
class instance has a reference count of 1, disposing of it will call
its __del__() method.
Since it is written in Python, the __del__() method can execute
arbitrary Python code. Could it perhaps do something to invalidate
the reference to item in bug()? You bet! Assuming
that the list passed into bug() is accessible to the
__del__() method, it could execute a statement to the effect of
"del list[0]", and assuming this was the last reference to that
object, it would free the memory associated with it, thereby
invalidating item.
The solution, once you know the source of the problem, is easy:
temporarily increment the reference count. The correct version of the
function reads:
This is a true story. An older version of Python contained variants
of this bug and someone spent a considerable amount of time in a C
debugger to figure out why his __del__() methods would fail...
The second case of problems with a borrowed reference is a variant
involving threads. Normally, multiple threads in the Python
interpreter can't get in each other's way, because there is a global
lock protecting Python's entire object space. However, it is possible
to temporarily release this lock using the macro
Py_BEGIN_ALLOW_THREADS, and to re-acquire it using
Py_END_ALLOW_THREADS. This is common around blocking I/O
calls, to let other threads use the processor while waiting for the I/O to
complete. Obviously, the following function has the same problem as
the previous one: