Web lists-archives.com

Puzzling Git backtrace




I'm trying to assist a Bitbucket Server customer who is seeing some
"git-upload-pack" processes "hang" on their server.

While investigating, we had them connect gdb to their processes (which
are 2.10.0 built from source using a simple unzip-and-run-make
approach) and get the backtraces for them. The output that they're
seeing makes no sense to me, though, so I'm throwing this out to the
list just to see if anyone has any idea how the processes could end up
like this.

One "family" of "hung" processes looks like this in ps:

32432  7811 atlbitb+ Mar22 /bin/git upload-pack /path/to/repository
32433 32432 atlbitb+ Mar22 git-upload-pack /path/to/repository

Looks normal enough. The backtrace for process 32432 also looks like I'd expect:

#0  0x00007fa39842fc9c in __libc_waitpid (pid=pid@entry=32433,
stat_loc=stat_loc@entry=0x7fffb42a4e4c, options=options@entry=0) at
../sysdeps/unix/sysv/linux/waitpid.c:31
#1  0x00000000005098f2 in wait_or_whine (pid=32433, argv0=0xd6e0c0
"git-upload-pack", in_signal=in_signal@entry=0) at run-command.c:229
#2  0x000000000050a949 in finish_command
(cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:537
#3  0x000000000050a9c1 in run_command (cmd=cmd@entry=0x7fffb42a4ea0)
at run-command.c:558
#4  0x000000000050aa5e in run_command_v_opt_cd_env
(argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40, dir=dir@entry=0x0,
env=env@entry=0x0) at run-command.c:578
#5  0x000000000050aa79 in run_command_v_opt
(argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40) at
run-command.c:563
#6  0x0000000000405ad0 in execv_dashed_external (argv=0x7fffb42a5140)
at git.c:569
#7  run_argv (argv=0x7fffb42a4f10, argcp=0x7fffb42a4f1c) at git.c:596
#8  cmd_main (argc=2, argc@entry=3, argv=0x7fffb42a5140,
argv@entry=0x7fffb42a5138) at git.c:665
#9  0x0000000000404d9d in main (argc=3, argv=0x7fffb42a5138) at common-main.c:40

It's in "wait_or_whine" for "git-upload-pack".

When they attached to 32433 and printed its backtrace, though, things
go a little sideways:

(gdb) attach 32433
Attaching to program: /usr/bin/git, process 32433
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols
from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f79d1aca240 in ?? ()
(gdb) bt
#0  0x00007f79d1aca240 in ?? ()
#1  0x000000000045179e in mktree_line (allow_missing=4,
nul_term_line=0, len=<optimized out>, buf=<optimized out>) at
builtin/mktree.c:103
#2  cmd_mktree (ac=<optimized out>, av=<optimized out>,
prefix=<optimized out>) at builtin/mktree.c:173
#3  0x0000000000000000 in ?? ()

So ps shows "git-upload-pack", and the parent process shows it's
waiting for "git-upload-pack", but gdb shows it's attaching to "git",
not "git-upload-pack", and the stack trace on the child shows it's
running "git mktree". I've searched through the source code and I
don't see any path through upload-pack.c that could result in it
essentially exec'ing "git mktree" over the top of itself, or even
spawning a "git mktree" of its own.

Has anyone seen anything like this before? Any thoughts on how what
should be a "git-upload-pack" could possibly end up being a "git
mktree" instead?

Best regards,
Bryan