[PATCH 0/2] Multi-pack-index: Fix "too many file descriptors" bug
- Date: Mon, 29 Apr 2019 09:18:54 -0700 (PDT)
- From: "Derrick Stolee via GitGitGadget" <gitgitgadget@xxxxxxxxx>
- Subject: [PATCH 0/2] Multi-pack-index: Fix "too many file descriptors" bug
Thanks to Jeff H for finding the problem with the multi-pack-index regarding
many packs. Specifically: if we open too many packs, the close_one_pack()
method cannot find the packs from the multi-pack-index to close.
Jeff already fixed the problem explicitly in 'git multi-pack-index verify'
which would hit this issue 100% of the time we had 2000+ packs. This issue
could still happen in 'git rev-list --all --objects' if there are enough
packs containing commits and trees. This series fixes the issue.
The basic solution is to add packs from the multi-pack-index into the
packed_git struct as they are opened. To avoid performance issues, add a
multi_pack_index bit to the packed_git struct. Midx-aware algorithms can
then ignore those packs.
There was a very subtle issue that happens during a 'git repack': we clear
the multi-pack-index after possibly reading some packs from it, thus leaving
some packs in the packed_git struct but having a NULL multi_pack_index in
the object store. This informs the change to close_midx().
I'm based on a recent 'master' commit that contains the following three
branches due to nearby changes causing conflicts if I pick only Jeff's
change as a base:
Derrick Stolee (2):
midx: pass a repository pointer
midx: add packs to packed_git linked list
builtin/multi-pack-index.c | 2 +-
builtin/pack-objects.c | 2 +-
midx.c | 42 +++++++++++++++++++++++++-------------
midx.h | 7 ++++---
object-store.h | 9 ++------
packfile.c | 30 ++++++++-------------------
sha1-name.c | 6 ++++++
7 files changed, 51 insertions(+), 47 deletions(-)
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-182/derrickstolee/many-packs-v1