Web lists-archives.com

Re: Gitlab Evaluation & Migration

On Thu, Feb 28, 2019 at 9:43 PM Ben Cooksley <bcooksley@xxxxxxx> wrote:
On Fri, Mar 1, 2019 at 3:13 AM Nate Graham <nate@xxxxxxx> wrote:
>  ---- On Wed, 27 Feb 2019 23:02:03 -0700 Ben Cooksley <bcooksley@xxxxxxx> wrote ----
>  > In terms of server load, it would be nice if the setup of forks was
>  > still something the developer had to initiate rather than being done
>  > automatically for every repository touched by kdesrc-build (I say this
>  > mainly as if we had 50 people fork just half of the mainline
>  > repositories we have, that's ~450GB of space used up - a massive
>  > scalability issue)
> This seems like a challenge that needs to be addressed regardless of whether or not kdesrc-build does it automatically, because people creating tons and tons of forks is guaranteed to happen anyway if we move to Gitlab. It seems non-optimal if having more people able to submit merge requests results in the potential to blow up our servers.

We have a little over 1,000 mainline repositories, so in the above
example we'd be talking about 25,000 forks being created - and i'd be
expecting quite a bit more than 50 people to use kdesrc-build. To use
another scenario, if the metric of half the repositories being
involved (or even a quarter) held true with say 300 users, you're now
looking at 75,000 - 150,000 forked repositories (and probably around
1.4TB - 2.7TB of space used) courtesy of an automated tool.

It would take quite a while for us to reach 150,000 forked
repositories on Gitlab if humans were to be creating these manually,
however if an automated tool is going to be creating them as part of
it's workflow, then we will hit it much more quickly (and is a
phenomenal waste of resources given virtually all of those forks will
never be utilised)

I wonder if advanced filesystem features like ZFS deduplication may help in this situation.

I certainly do expect a number of forks to be created yes, but i'd
rather they be useful forks where someone at least intends on working
on something, rather than ones created automatically by software "just
in case" someone decides to work on a project.

> Nate