recussiveness of _thread.RLock doesn't work if lock acquired inside signal handler inside lock.acquire

[xBZZZZ]

Bug report

Bug description:

rlocktesting.py:

from _thread import start_new_thread,RLock
from select import select
from os import write
from signal import signal,SIGALRM,alarm

lock=RLock()

def threadfunc():
	write(1,b"thread: trying to acquire lock\n")
	lock.acquire()
	write(1,b"thread: acquired lock and doing unrelated stuff for 3 seconds\n")
	select((),(),(),3)
	write(1,b"thread: trying to release lock\n")
	lock.release()
	write(1,b"thread: released lock\n")

def SIGALRMhandler(signum,frame):
	write(1,b"SIGALRM: trying to acquire lock\n")
	lock.acquire()
	write(1,b"SIGALRM: acquired lock, returning\n")

signal(SIGALRM,SIGALRMhandler)
start_new_thread(threadfunc,())
alarm(2)
write(1,b"main: doing unrelated stuff for 1 second\n")
select((),(),(),1)
write(1,b"main: trying to acquire lock\n")
lock.acquire()#global scope
write(1,b"main: acquired lock\nEND OF USELESS PROGRAM\n")

running:

$ python3 rlocktesting.py
main: doing unrelated stuff for 1 second
thread: trying to acquire lock
thread: acquired lock and doing unrelated stuff for 3 seconds
main: trying to acquire lock
SIGALRM: trying to acquire lock
thread: trying to release lock
thread: released lock
SIGALRM: acquired lock, returning

it gets stuck forever, pressing ctrl+c:

^CTraceback (most recent call last):
  File "/tmp/rlocktesting.py", line 28, in <module>
    lock.acquire()#global scope
    ~~~~~~~~~~~~^^
KeyboardInterrupt

I excpeted main thread to successfully acquire lock twice (first inside SIGALRMhandler, second continuing inside global scope)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-144706: docs note about combining RLock and signal handlers #144736
• [3.14] gh-144706: Warn against using synchronization primitives within signal handlers (GH-144736) #144767
• [3.13] gh-144706: Warn against using synchronization primitives within signal handlers (GH-144736) #144768

[ZeroIntensity]

See #126434 for a similar issue:

The general problem is that Python can execute signal handling code asynchronously at many places that don't expect to support reentrancy.

You generally cannot acquire locks in signal handlers, regardless of language.

[xBZZZZ]

You generally cannot acquire locks in signal handlers

the acquire locks in signal handlers part works, it's the recursiveness of RLock that doesn't work

if lock released inside SIGALRMhandler, continuing lock.acquire()#global scope doesn't get stuck

rlocktesting with release.py:

from _thread import start_new_thread,RLock
from select import select
from os import write
from signal import signal,SIGALRM,alarm

lock=RLock()

def threadfunc():
	write(1,b"thread: trying to acquire lock\n")
	lock.acquire()
	write(1,b"thread: acquired lock and doing unrelated stuff for 3 seconds\n")
	select((),(),(),3)
	write(1,b"thread: trying to release lock\n")
	lock.release()
	write(1,b"thread: released lock\n")

def SIGALRMhandler(signum,frame):
	write(1,b"SIGALRM: trying to acquire lock\n")
	lock.acquire()
	write(1,b"SIGALRM: acquired lock, releasing lock\n")
	lock.release()
	write(1,b"SIGALRM: released lock\n")

signal(SIGALRM,SIGALRMhandler)
start_new_thread(threadfunc,())
alarm(2)
write(1,b"main: doing unrelated stuff for 1 second\n")
select((),(),(),1)
write(1,b"main: trying to acquire lock\n")
lock.acquire()#global scope
write(1,b"main: acquired lock\nEND OF USELESS PROGRAM\n")

running:

$ python3.13 'rlocktesting with release.py'
main: doing unrelated stuff for 1 second
thread: trying to acquire lock
thread: acquired lock and doing unrelated stuff for 3 seconds
main: trying to acquire lock
SIGALRM: trying to acquire lock
thread: trying to release lock
thread: released lock
SIGALRM: acquired lock, releasing lock
SIGALRM: released lock
main: acquired lock
END OF USELESS PROGRAM

[robsdedude]

I was able to reproduce the behavior on my machine. The RLock does not grant re-entrance when acquired from the main thread as well as the signal handler concurrently. I find the assumption reasonable that one would be able to recurse into an RLock from the main-thread and a signal handler since the docs state:

Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.

Besides, only the main thread of the main interpreter is allowed to set a new signal handler.

-- https://docs.python.org/3/library/signal.html#signals-and-threads

[ZeroIntensity]

Locks in the threading module are interruptible, meaning they respond to signals (which is what allows you to interrupt them with CTRL+C):

cpython/Modules/_threadmodule.c

Lines 1101 to 1103 in 3e0322f

	PyLockStatus r = _PyRecursiveMutex_LockTimed(
	&self->lock, timeout,
	_PY_LOCK_PYTHONLOCK | _PY_LOCK_HANDLE_SIGNALS | _PY_LOCK_DETACH);

(Notice the _PY_LOCK_HANDLE_SIGNALS flag.)

This is the cause of your deadlock. Let's take a look at _PyRecursiveMutex_LockTimed for some context:

cpython/Python/lock.c

Lines 405 to 419 in 3e0322f

	PyLockStatus
	_PyRecursiveMutex_LockTimed(_PyRecursiveMutex *m, PyTime_t timeout, _PyLockFlags flags)
	{
	PyThread_ident_t thread = PyThread_get_thread_ident_ex();
	if (recursive_mutex_is_owned_by(m, thread)) {
	m->level++;
	return PY_LOCK_ACQUIRED;
	}
	PyLockStatus s = _PyMutex_LockTimed(&m->mutex, timeout, flags);
	if (s == PY_LOCK_ACQUIRED) {
	_Py_atomic_store_ullong_relaxed(&m->thread, thread);
	assert(m->level == 0);
	}
	return s;
	}

CPython's recursive mutex is implemented using a non-recursive mutex that is shielded by a check for the current thread. With that in mind, here's the flow of calls:

1. The subthread acquires the lock, which forces the SIGALRM handler to wait when it attempts to acquire it. When there's no other thread holding the lock, _PyMutex_LockTimed takes a fast path that doesn't wait (and thus can't handle signals).
2. The main thread reaches the lock.acquire call and dives into _PyRecursiveMutex_LockTimed. It sees that the recursive mutex is not held by the main thread, so it begins waiting for the subthread to release it.
3. The main thread is then interrupted by the alarm signal, which triggers the signal handler.
4. The signal handler does the same dance; it goes to _PyRecursiveMutex_LockTimed, sees that the lock isn't owned by the main thread, and begins waiting. Now both the signal handler and the main program are waiting on the same non-recursive lock!
5. The subthread releases the lock, allowing the signal handler to take it and yield control to the main thread.
6. The main thread is still waiting on the non-recursive lock, because it still thinks that the lock is owned by the subthread
7. Deadlock!

This is just a quirk of signal handling. Please don't mix signals and locks.

[robsdedude]

This is just a quirk of signal handling. Please don't mix signals and locks.

Do you reckon it's worth adding some note like this to the docs?

[ZeroIntensity]

Sure, if you can find a good place for it. There might be a note like this already, though.

[xBZZZZ]

There might be a note like this already, though.

can't find note:

• https://docs.python.org/3.13/library/threading.html (the page that documents RLock) only mentions signals like this:

  Changed in version 3.2: Lock acquisition can now be interrupted by signals on POSIX if the underlying threading implementation supports it.

• https://docs.python.org/3.13/library/signal.html doesn't mention RLock

[ZeroIntensity]

Reopening as a docs issue