Stream: Coq devs & plugin devs

Topic: Too many pipelines


view this post on Zulip Gaëtan Gilbert (Aug 24 2020 at 09:21):

What should we do about the recurring too many pipeline jobs error?
The limit is documented here https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlab-cicd
"Max jobs in active pipelines gitlab.com:500 for Free tier, unlimited otherwise self-hosted:unlimited"

view this post on Zulip Emilio Jesús Gallego Arias (Aug 24 2020 at 09:25):

I understand we could apply for some open source program and get that limit lifted too?

view this post on Zulip Gaëtan Gilbert (Aug 24 2020 at 09:32):

If you mean https://about.gitlab.com/solutions/open-source/program/ then "All public projects on GitLab.com automatically receive Gold functionality at the project level. Apply to this program if you need Gold functionality at the group level."
job limits are project level

view this post on Zulip Emilio Jesús Gallego Arias (Aug 25 2020 at 09:35):

Ok, just an idea, maybe we could just write to them? Documentation on that was always a bit imprecise, I see they have reworded it a bit tho.

view this post on Zulip Gaëtan Gilbert (Aug 25 2020 at 17:20):

I made https://gitlab.com/gitlab-org/gitlab/-/issues/240895
it sounds like we should apply for the program

view this post on Zulip Emilio Jesús Gallego Arias (Aug 25 2020 at 17:24):

That was my understanding last time I looked.

view this post on Zulip Emilio Jesús Gallego Arias (Aug 25 2020 at 17:24):

That we needed to apply

view this post on Zulip Jason Gross (Aug 25 2020 at 18:19):

I suspect we probably don't want GitHub actions, as this will result in the CI taking 25x more real time to run (since we only get 20 concurrent jobs)

view this post on Zulip Gaëtan Gilbert (Aug 25 2020 at 18:40):

do we really get 500 jobs on gitlab? O_O

view this post on Zulip Théo Zimmermann (Aug 25 2020 at 19:01):

20 concurrent jobs is already pretty good. On GitLab, I feel like we frequently get that as well, but sometimes we also get a single free runner and most of our jobs are run by our own runners. If I'm not mistaken, GitHub also supports setting up your own runners.

view this post on Zulip Gaëtan Gilbert (Aug 27 2020 at 12:57):

https://forum.gitlab.com/t/not-treated-as-open-source-for-pipeline-job-limit/41938

view this post on Zulip Gaëtan Gilbert (Sep 01 2020 at 09:42):

Emilio Jesús Gallego Arias said:

That we needed to apply

ok shall we do it?
is there any blocker?

view this post on Zulip Emilio Jesús Gallego Arias (Sep 01 2020 at 11:22):

Not that I'm aware

view this post on Zulip Matthieu Sozeau (Sep 02 2020 at 15:44):

I applied, we'll see

view this post on Zulip Jason Gross (Sep 07 2020 at 02:22):

Note that they replied at https://gitlab.com/gitlab-org/gitlab/-/issues/240895#note_402263876:
"""
Ah apologies I didn't realise the project was public.
I'm going to ask around for you to best advise on the next steps.
As mentioned previously this isn't the best process to get the right answers for you. It certainly appears your project is on the free tier going by the behavior but I'm not able to say why this isn't the case sadly.
To ensure you're getting to speak to the right people to help with this further I would recommend either creating a thread about this on our support forum or to contact our Open Source team's email at opensource@gitlab.com. Hope this helps you get it sorted!
"""
So perhaps someone should email?

view this post on Zulip Gaëtan Gilbert (Sep 23 2020 at 13:11):

@Matthieu Sozeau any response to the application? we're still seeing the issue eg https://gitlab.com/coq/coq/-/pipelines/193464370

view this post on Zulip Matthieu Sozeau (Sep 23 2020 at 13:14):

Nope, not yet. They said "allow 10 days for a response" and I sent it last wednesday

view this post on Zulip Matthieu Sozeau (Sep 30 2020 at 08:00):

I just signed the papers to the Gold plan, so we should benefit from it anytime now

view this post on Zulip Théo Zimmermann (Oct 01 2020 at 07:15):

Does not seem to be the case yet: screenshot

view this post on Zulip Théo Zimmermann (Oct 05 2020 at 15:44):

Still the case today :frown:

view this post on Zulip Théo Zimmermann (Oct 05 2020 at 15:49):

And we still get the "job activity exceeded" failure: https://gitlab.com/coq/coq/-/pipelines/197875878

view this post on Zulip Théo Zimmermann (Oct 15 2020 at 14:33):

We're still on the free plan with recurrent "job activity exceeded" failures.

view this post on Zulip Théo Zimmermann (Oct 19 2020 at 12:57):

This is still an issue.

view this post on Zulip Gaëtan Gilbert (May 21 2021 at 16:50):

@Matthieu Sozeau can you ping gitlab about this?

view this post on Zulip Matthieu Sozeau (May 21 2021 at 16:51):

Yes, sure

view this post on Zulip Matthieu Sozeau (May 26 2021 at 09:52):

Done

view this post on Zulip Matthieu Sozeau (May 26 2021 at 09:52):

(I pinged them again)

view this post on Zulip Jason Gross (May 26 2021 at 21:05):

As a kludge, can coqbot restart pipelines that fail with this error?

view this post on Zulip Théo Zimmermann (May 27 2021 at 07:09):

The reason these pipelines fail is that we go over the limits for the Free plan (which should have been lifted if GitLab had really granted us the Gold OSS plan). We believe that the fact that restarting manually allows the pipeline to run despite this limitation is a GitLab bug and we are fortunate that this bug still exists. Therefore, we should avoid abusing it by systematically restarting the failed pipelines, which would put us well beyond the limits for the Free plan.

view this post on Zulip Théo Zimmermann (May 27 2021 at 07:11):

The plan to alleviate this problem was rather to reduce the number of jobs that are systematically run for each pipeline, with a dual mode, where, by default, we run a more lightweight pipeline, then upon request and before merging a PR, we run the full pipeline. This is not yet implemented because I had other priorities that interfered.

view this post on Zulip Emilio Jesús Gallego Arias (May 27 2021 at 09:23):

Indeed, @Théo Zimmermann gave a good summary, the current state of the job limit is a bit strange as even the direction / scope of the bug is not clear. Moreover, all CI systems are under attack due to crypto currency mining, so things may get quite worse for us, but fortunately our choice of using gitlab means we could provide a backup if needed.

view this post on Zulip Matthieu Sozeau (May 27 2021 at 10:36):

Don't we have new workers available for this actually?

view this post on Zulip Gaëtan Gilbert (May 27 2021 at 10:38):

this is not related to workers

view this post on Zulip Théo Zimmermann (May 27 2021 at 10:40):

Even if we had an infinite number of workers that we own, we would still need to switch to our own GitLab (e.g. Inria's GitLab) or get the Gold plan to avoid this limit on the number of concurrent jobs.

view this post on Zulip Matthieu Sozeau (May 27 2021 at 10:42):

I see

view this post on Zulip Matthieu Sozeau (Jun 02 2021 at 15:05):

Sadly, they seem to be overwhelmed... no response still after a week

view this post on Zulip Matthieu Sozeau (Jun 07 2021 at 18:15):

image.png Finally !

view this post on Zulip Théo Zimmermann (Jun 07 2021 at 18:31):

Wonderful! But given the actual start date, they could have adapted the end date :stuck_out_tongue:

view this post on Zulip Matthieu Sozeau (Jun 07 2021 at 18:50):

I guess they'll renew easily, as we're linked to the Inria account

view this post on Zulip Matthieu Sozeau (Jun 07 2021 at 18:54):

I can't seem to find a way to check that the pipeline limit has been unlifted, but we'll know soon enough :)

view this post on Zulip Théo Zimmermann (Jun 09 2021 at 10:47):

Indeed, the last known pipeline activity limit errors were on June 3rd.

view this post on Zulip Matthieu Sozeau (Jun 10 2021 at 14:05):

Hurray!

view this post on Zulip Gaëtan Gilbert (Sep 07 2021 at 10:35):

I'm getting a prompt to renew on https://gitlab.com/coq/coq/-/pipelines
Screenshot_20210907_123410.png
it needs customer credentials, @Matthieu Sozeau can you handle it?

view this post on Zulip Théo Zimmermann (Sep 07 2021 at 12:21):

In another thread, Matthieu answered that the renewal was supposed to be automatic (but that he would check with GitLab support).

view this post on Zulip Matthieu Sozeau (Sep 07 2021 at 13:22):

Done, I sent a renewal request, apparently they're in the process of automating this but it does not work yet for Gitlab for OSS projects

view this post on Zulip Matthieu Sozeau (Sep 18 2021 at 12:45):

The renewal was signed today, so we should get no interuptions


Last updated: Oct 21 2021 at 20:02 UTC