Web lists-archives.com

Re: Is Access to Salsa restricted to a certain number of queries per time and host?




On Wed, Jul 11, 2018 at 01:34:29PM +0200, Julien Cristau wrote:
> On 07/11/2018 10:18 AM, Andreas Tille wrote:
> > Hi,
> > 
> > I'm running a daily cron job on host blends.debian.net to gather machine
> > readable data from all blends packages.  The cron job fetches only the
> > following files
> > 
> >     debian/changelog
> >     debian/control
> >     debian/copyright
> >     debian/README.Debian
> >     debian/upstream/edam
> >     debian/upstream/metadata
> > 
> > (if the latter two exist) from about 2000 repositories.  These data are
> > consumed in UDD from where they are used in the Blends web sentinel.  The
> > script which is running can be found in Git[1].
> > 
> > Unfortunately the cron job seems to stop with
> > 
> > Traceback (most recent call last):
> >   File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 251, in wrapped_f
> >     return f(*args, **kwargs)
> >   File "/usr/lib/python3/dist-packages/gitlab/mixins.py", line 48, in get
> >     server_data = self.gitlab.http_get(path, **kwargs)
> >   File "/usr/lib/python3/dist-packages/gitlab/__init__.py", line 728, in http_get
> >     streamed=streamed, **kwargs)
> >   File "/usr/lib/python3/dist-packages/gitlab/__init__.py", line 706, in http_request
> >     response_body=result.content)
> > gitlab.exceptions.GitlabHttpError: 429: b'Retry later\n'
> > 
> > During handling of the above exception, another exception occurred:
> > 
> > Traceback (most recent call last):
> >   File "/srv/blends.debian.org/misc/machine_readable/fetch-machine-readable_salsa.py", line 106, in <module>
> >     project = gl.projects.get(pr.attributes['id']) # without this extra get repository_tree() fails
> >   File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 253, in wrapped_f
> >     raise error(e.error_message, e.response_code, e.response_body)
> > gitlab.exceptions.GitlabGetError: 429: b'Retry later\n'
> > 
> You could probably save yourself some trouble by not polling repos that
> have had no activity since you last looked at them.

Or by setting up a webhook[0] so the relevant repos can notify you when
there's a commit that changes the files you care about.

[0]: https://salsa.debian.org/help/user/project/integrations/webhooks.md

Cheers,
-- 
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7  2D23 DFE6 91AE 331B A3DB