I am simultaneously excited for the implementation techniques that #python free threading is going to open up for framework authors and deeply concerned about the way I see discussions going around it, as it seems application authors will be speedrunning the “you can fit so many untestable data races in here” to “you can fit so many intractable mutex deadlocks in here” pipeline that java previewed for us all in the early aughts
Discussion
Loading...
Post
@glyph Why Threads Are A Bad Idea (for most purposes), John Ousterhout: https://www.cc.gatech.edu/classes/AY2010/cs4210_fall/papers/ousterhout-threads.pdf and https://ieeexplore.ieee.org/document/1631937
@cratermoon I am … familiar. I have probably cited this article 1000 times. most recently, https://mastodon.social/@glyph/115345123317563644
@glyph Does it make that much of a difference for pure Python code? You might not hit the performance wall that'd push you towards a multi-process solution as quickly, but you already needed to deal with execution in other threads being interleaved with your own thread.
I think a bigger problem might end up being updated versions of extensions claiming to support nogil but not quite getting it right (either not locking their own data structures, or incorrectly relying on a borrowed reference to some PyObject).
@jamesh code that was wrong before will still be wrong, code that was correct before will still be correct. but a lot of wrong code will probably have been working most of the time in prod, only occasionally behaving a bit weird, and the huge explosion of potential switch points will reveal plenty of bugs that were accidentally, stochastically protected by the GIL’s accidental block-of-bytecode boundaries
@glyph I would have thought there were enough opportunities for thread switches for that it would be hard to rely on the GIL from pure Python code.
That seems quite different to extensions, where they would traditionally know that no other thread could be touching the interpreter, and exactly where that lock would be released.
@jamesh basically I think that the same problems apply in C and in Python, but in C the consequences are probably a lot more immediate and dramatic (segfaults, etc)
@glyph It's 1986 and the Erlang folks are like... you don't have to do it like that
It's 2025 and the Elixir people are still absolutely insufferable about it
(it's me, I'm being insufferable about it)
@glyph
Prequel
It's 1999 and I'm kicking myself for not keeping practiced on COBOL
@glyph this might be a good time to remind folks of https://pypi.org/project/diagnose/ whose Breakpoints allow you to pause and resume threads from some other thread.
@glyph 😂 … I think in 1994 I was having conversations about how it was impossible to debug multithreaded shared-state code in FORTRAN.
Bonfire Dinteg Labs
This is a bonfire demo instance for testing purposes. This is not a production site. There are no backups for now. Data, including profiles may be wiped without notice. No service or other guarantees expressed or implied.
Automatic federation enabled