Web lists-archives.com

cygtls crash causing twisted 13.2 app to livelock?




Hi folks,
I've had a buildbot worker running on cygwin for the last few
years.  The python program occasionally appears to busy-hang
after forking a git, regardless of version of windows or
up-to-dateness or 32/64 bitness of cygwin.
More details at http://trac.buildbot.net/ticket/3670
Today it happened at the same time on both win7 and win10,
so I'm starting to get motivated to understand it.
Chances are it's just a bug in twisted 13.2 (which is
pretty old), but I did see something interesting:

strace of the livelocked python shows the following output repeatedly:

-- Process 8232, exception c0000005 at 0000000180042C51
   55 2120509 [flasio] python 9276 exception::handle: In
cygwin_except_handler exception 0xC0000005 at 0x180042C51 sp 0x275CB60
   32 2120541 [flasio] python 9276 exception::handle: In
cygwin_except_handler signal 11 at 0x180042C51
   72 2120613 [flasio] python 9276 _cygtls::inside_kernel: pc
0x180042C51, h 0x180040000, inside_kernel 0
   40 2120653 [flasio] python 9276 seterrno_from_nt_status:
/home/corinna/src/cygwin/cygwin-2.6.0/cygwin-2.6.0-1.x86_64/src/newlib-cygwin/winsup/cygwin/fhandler_mailslot.cc:132
status 0xC0000034 -> windows error 2
   30 2120683 [flasio] python 9276 geterrno_from_win_error: windows
error 2 == errno 2
   30 2120713 [flasio] python 9276 sig_send: sendsig 0xC4, pid 9276,
signal 11, its_me 1
   28 2120741 [flasio] python 9276 sig_send: wakeup 0x42C
   32 2120773 [sig] python 9276 sigpacket::process: signal 11 processing
   19 2120792 [sig] python 9276 sigpacket::process: signal 11 blocked
   18 2120810 [sig] python 9276 sigpacket::process: returning -1
   22 2120832 [sig] python 9276 wait_sig: signalling pack.wakeup 0x42C
   24 2120856 [flasio] python 9276 sig_send: Waiting for pack.wakeup 0x42C
   30 2120886 [flasio] python 9276 sig_send: returning 0x0 from
sending signal 11

Attaching with gdb and doing 'info threads' and 'bt N' on each thread
until I found a related-looking thread showed:

#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51e415f in WaitForSingleObjectEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x000000018011f027 in sig_send(_pinfo*, siginfo_t&, _cygtls*) ()
from /usr/bin/cygwin1.dll
#3  0x000000018005c7c1 in exception::handle(_EXCEPTION_RECORD*, void*,
_CONTEXT*, _DISPATCHER_CONTEXT*) () from /usr/bin/cygwin1.dll
#4  0x00007ffbd845666d in ntdll!.chkstk () from
/cygdrive/c/Windows/SYSTEM32/ntdll.dll
#5  0x00007ffbd83d3c00 in ntdll!RtlWalkFrameChain () from
/cygdrive/c/Windows/SYSTEM32/ntdll.dll
#6  0x00007ffbd845577a in ntdll!KiUserExceptionDispatcher ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#7  0x0000000180042c51 in _cfree () from /usr/bin/cygwin1.dll
#8  0x00000001801deab3 in fhandler_pipe::~fhandler_pipe() () from
/usr/bin/cygwin1.dll
#9  0x00000001800621f5 in flush_async_io(void*) () from /usr/bin/cygwin1.dll
#10 0x0000000180044753 in cygthread::callfunc(bool) () from /usr/bin/cygwin1.dll
#11 0x0000000180044cea in cygthread::stub(void*) () from /usr/bin/cygwin1.dll
#12 0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*,
void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#13 0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*,
void*), void*) ()
   from /usr/bin/cygwin1.dll
#14 0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#15 0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll

An exception in chkstk in a thread that looks like it's cygwin's baby
seemed interesting enough to report.

I'll attach the output of cygcheck and gdb.
(gdb) info threads
  Id   Target Id         Frame
  9    Thread 14024.0x264c 0x00007ffbd84553e1 in ntdll!DbgBreakPoint ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  8    Thread 14024.0x3e34 0x00007ffbd845538a in ntdll!ZwWaitForWorkViaWorkerFactory ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  7    Thread 14024.0x1064 0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  6    Thread 14024.0x1f60 0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  5    Thread 14024.0x2088 0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  4    Thread 14024.0x1e64 0x00007ffbd8453dda in ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  3    Thread 14024.0xa14 0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
  2    Thread 14024.0x3628 0x00007ffbd845388a in ntdll!ZwReadFile ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
* 1    Thread 14024.0x708 0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) thread 1
[Switching to thread 1 (Thread 14024.0x708)]
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51e415f in WaitForSingleObjectEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x0000000180129b0b in muto::acquire(unsigned int) () from /usr/bin/cygwin1.dll
#3  0x0000000180042c3d in _cfree () from /usr/bin/cygwin1.dll
#4  0x00000001800dd36d in path_conv::~path_conv() () from /usr/bin/cygwin1.dll
#5  0x00000001801deabb in fhandler_pipe::~fhandler_pipe() () from /usr/bin/cygwin1.dll
#6  0x000000018012eaa5 in close () from /usr/bin/cygwin1.dll
#7  0x0000000180117a4b in _sigfe () from /usr/bin/cygwin1.dll
#8  0x0000000180117a95 in _sigbe () from /usr/bin/cygwin1.dll
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 2
[Switching to thread 2 (Thread 14024.0x3628)]
#0  0x00007ffbd845388a in ntdll!ZwReadFile () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd845388a in ntdll!ZwReadFile () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51de784 in ReadFile () from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x000000018011e703 in wait_sig(void*) () from /usr/bin/cygwin1.dll
#3  0x0000000180044753 in cygthread::callfunc(bool) () from /usr/bin/cygwin1.dll
#4  0x0000000180044cea in cygthread::stub(void*) () from /usr/bin/cygwin1.dll
#5  0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*, void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#6  0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*, void*), void*) ()
   from /usr/bin/cygwin1.dll
#7  0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#8  0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#9  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 3
[Switching to thread 3 (Thread 14024.0xa14)]
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51e415f in WaitForSingleObjectEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x0000000180044d21 in cygthread::stub(void*) () from /usr/bin/cygwin1.dll
#3  0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*, void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#4  0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*, void*), void*) ()
   from /usr/bin/cygwin1.dll
#5  0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#6  0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#7  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 4
[Switching to thread 4 (Thread 14024.0x1e64)]
#0  0x00007ffbd8453dda in ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd8453dda in ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd520918f in WaitForMultipleObjectsEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x00007ffbd520908e in WaitForMultipleObjects ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#3  0x00000001800459fc in cygwait(void*, _LARGE_INTEGER*, unsigned int) ()
   from /usr/bin/cygwin1.dll
#4  0x0000000180143f56 in semaphore::_wait() () from /usr/bin/cygwin1.dll
#5  0x0000000180117a4b in _sigfe () from /usr/bin/cygwin1.dll
#6  0x00000003fe4942c8 in libpython2.7!PyThread_acquire_lock () from /usr/bin/libpython2.7.dll
#7  0x00000003fe499f32 in libpython2.7!PyObject_GC_Del () from /usr/bin/libpython2.7.dll
#8  0x00000003fe4564df in libpython2.7!PyEval_EvalFrameEx () from /usr/bin/libpython2.7.dll
#9  0x00000003fe45778a in libpython2.7!PyEval_EvalCodeEx () from /usr/bin/libpython2.7.dll
#10 0x00000003fe455535 in libpython2.7!PyEval_EvalFrameEx () from /usr/bin/libpython2.7.dll
#11 0x00000003fe45778a in libpython2.7!PyEval_EvalCodeEx () from /usr/bin/libpython2.7.dll
#12 0x00000003fe455535 in libpython2.7!PyEval_EvalFrameEx () from /usr/bin/libpython2.7.dll
#13 0x00000003fe45778a in libpython2.7!PyEval_EvalCodeEx () from /usr/bin/libpython2.7.dll
#14 0x00000003fe3c7925 in libpython2.7!PyFrame_Fini () from /usr/bin/libpython2.7.dll
#15 0x00000003fe393151 in libpython2.7!PyObject_Call () from /usr/bin/libpython2.7.dll
#16 0x00000003fe4510a7 in libpython2.7!PyEval_EvalFrameEx () from /usr/bin/libpython2.7.dll
#17 0x00000003fe455653 in libpython2.7!PyEval_EvalFrameEx () from /usr/bin/libpython2.7.dll
#18 0x00000003fe455653 in libpython2.7!PyEval_EvalFrameEx () from /usr/bin/libpython2.7.dll
#19 0x00000003fe45778a in libpython2.7!PyEval_EvalCodeEx () from /usr/bin/libpython2.7.dll
#20 0x00000003fe3c7847 in libpython2.7!PyFrame_Fini () from /usr/bin/libpython2.7.dll
#21 0x00000003fe393151 in libpython2.7!PyObject_Call () from /usr/bin/libpython2.7.dll
#22 0x00000003fe3a620c in libpython2.7!PyInstance_New () from /usr/bin/libpython2.7.dll
#23 0x00000003fe393151 in libpython2.7!PyObject_Call () from /usr/bin/libpython2.7.dll
#24 0x00000003fe44e117 in libpython2.7!PyEval_CallObjectWithKeywords ()
   from /usr/bin/libpython2.7.dll
#25 0x00000003fe49a2f2 in libpython2.7!PyObject_GC_Del () from /usr/bin/libpython2.7.dll
#26 0x000000018013f439 in pthread::thread_init_wrapper(void*) () from /usr/bin/cygwin1.dll
#27 0x00000001800bd841 in pthread_wrapper () from /usr/bin/cygwin1.dll
#28 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 5
[Switching to thread 5 (Thread 14024.0x2088)]
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51e415f in WaitForSingleObjectEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x000000018011f027 in sig_send(_pinfo*, siginfo_t&, _cygtls*) () from /usr/bin/cygwin1.dll
#3  0x000000018005c7c1 in exception::handle(_EXCEPTION_RECORD*, void*, _CONTEXT*, _DISPATCHER_CONTEXT*) () from /usr/bin/cygwin1.dll
#4  0x00007ffbd845666d in ntdll!.chkstk () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#5  0x00007ffbd83d3c00 in ntdll!RtlWalkFrameChain () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#6  0x00007ffbd845577a in ntdll!KiUserExceptionDispatcher ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#7  0x0000000180042c51 in _cfree () from /usr/bin/cygwin1.dll
#8  0x00000001801deab3 in fhandler_pipe::~fhandler_pipe() () from /usr/bin/cygwin1.dll
#9  0x00000001800621f5 in flush_async_io(void*) () from /usr/bin/cygwin1.dll
#10 0x0000000180044753 in cygthread::callfunc(bool) () from /usr/bin/cygwin1.dll
#11 0x0000000180044cea in cygthread::stub(void*) () from /usr/bin/cygwin1.dll
#12 0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*, void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#13 0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*, void*), void*) ()
   from /usr/bin/cygwin1.dll
#14 0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#15 0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#16 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 6
[Switching to thread 6 (Thread 14024.0x1f60)]
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51e415f in WaitForSingleObjectEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x0000000180044d21 in cygthread::stub(void*) () from /usr/bin/cygwin1.dll
#3  0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*, void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#4  0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*, void*), void*) ()
   from /usr/bin/cygwin1.dll
#5  0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#6  0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#7  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 7
[Switching to thread 7 (Thread 14024.0x1064)]
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd845386a in ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd51e415f in WaitForSingleObjectEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x0000000180044d21 in cygthread::stub(void*) () from /usr/bin/cygwin1.dll
#3  0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*, void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#4  0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*, void*), void*) ()
   from /usr/bin/cygwin1.dll
#5  0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#6  0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#7  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 8
[Switching to thread 8 (Thread 14024.0x3e34)]
#0  0x00007ffbd845538a in ntdll!ZwWaitForWorkViaWorkerFactory ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bit
Undefined command: "bit".  Try "help".
(gdb) bt
#0  0x00007ffbd845538a in ntdll!ZwWaitForWorkViaWorkerFactory ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd83f93a5 in ntdll!EtwEventRegister () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#2  0x0000000180045733 in _cygtls::call2(unsigned int (*)(void*, void*), void*, void*) ()
   from /usr/bin/cygwin1.dll
#3  0x00000001800457e4 in _cygtls::call(unsigned int (*)(void*, void*), void*) ()
   from /usr/bin/cygwin1.dll
#4  0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#5  0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#6  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) thread 9
[Switching to thread 9 (Thread 14024.0x264c)]
#0  0x00007ffbd84553e1 in ntdll!DbgBreakPoint () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffbd84553e1 in ntdll!DbgBreakPoint () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffbd847c9a1 in ntdll!DbgUiRemoteBreakin ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#2  0x00007ffbd5e52d92 in KERNEL32!BaseThreadInitThunk ()
   from /cygdrive/c/Windows/system32/KERNEL32.DLL
#3  0x00007ffbd83c9f64 in ntdll!RtlUserThreadStart ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#4  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Attachment: cygcheck.out
Description: Binary data

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple