Web lists-archives.com

Re: fatal: Out of memory, getdelim failed under NFS mounts




Am 10.08.2017 um 20:56 schrieb Junio C Hamano:
> René Scharfe <l.s.r@xxxxxx> writes:
> 
>> I doubt the type of file system matters.  The questions are: How much
>> main memory do you have, what is git trying to cram into it, is there
>> a way to reduce the memory footprint or do you need to add more RAM?
>>
>>> any recommendations on how to pin point the "offender"? ;)
>> Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
>> good start, I think, to find out which of the different activities
>> that pull is doing causes the out-of-memory error.
>>
>> "free" and "ulimit -a" can help you find out how much memory you can
>> use.
>>
>> Also: What does "wc -L .git/FETCH_HEAD .git/packed-refs" report?
>> getdelim() is used mostly to read lines from files like these and in
>> the admittedly unlikely case that they are *really* long such an
>> error would be expected.
> 
> There is only one getdelim() call, which was introduced in v2.5.0
> timeframe, and it is used like this:
> 
> 	r = getdelim(&sb->buf, &sb->alloc, term, fp);
> 
> 	if (r > 0) {
> 		sb->len = r;
> 		return 0;
> 	}
> 	assert(r == -1);
> 
> 	/*
> 	 * Normally we would have called xrealloc, which will try to free
> 	 * memory and recover. But we have no way to tell getdelim() to do so.
> 	 * Worse, we cannot try to recover ENOMEM ourselves, because we have
> 	 * no idea how many bytes were read by getdelim.
> 	 *
> 	 * Dying here is reasonable. It mirrors what xrealloc would do on
> 	 * catastrophic memory failure. We skip the opportunity to free pack
> 	 * memory and retry, but that's unlikely to help for a malloc small
> 	 * enough to hold a single line of input, anyway.
> 	 */
> 	if (errno == ENOMEM)
> 		die("Out of memory, getdelim failed");
> 
> So the function is returning -1 and leaving ENOMEM in errno on
> Yaroslav's system.
> 
> I wonder if we are truly hitting out of memory, though.  The same
> symptom could bee seen if getdelim() does not touch errno when it
> returns -1, but some other system call earlier set it to ENOMEM,
> for example.

That can happen when the end of a file is reached; getdelim() returns
-1 and leaves errno unchanged.  So we need to set errno to 0 just
before that call.

Still *some* function must have run into a memory shortage in that
scenario, so adding/assigning more should help, right?

> If the same version of Git is recompiled there without HAVE_GETDELIM
> defined, would it still die with out of memory (presumably inside
> the call to strbuf_grow() in the strbuf_getwholeline() function)?

It's worth a try, if possible.

René