Gaëtan Gilbert (Aug 24 2020 at 09:21): Gaëtan Gilbert (Aug 24 2020 at 09:21):What should we do about the recurring too many pipeline jobs error?
The limit is documented here https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlab-cicd
"Max jobs in active pipelines gitlab.com:500 for Free tier, unlimited otherwise self-hosted:unlimited"
Emilio Jesús Gallego Arias (Aug 24 2020 at 09:25):I understand we could apply for some open source program and get that limit lifted too?
Gaëtan Gilbert (Aug 24 2020 at 09:32):If you mean https://about.gitlab.com/solutions/open-source/program/ then "All public projects on GitLab.com automatically receive Gold functionality at the project level. Apply to this program if you need Gold functionality at the group level."
job limits are project level
Emilio Jesús Gallego Arias (Aug 25 2020 at 09:35):Ok, just an idea, maybe we could just write to them? Documentation on that was always a bit imprecise, I see they have reworded it a bit tho.
Gaëtan Gilbert (Aug 25 2020 at 17:20):I made https://gitlab.com/gitlab-org/gitlab/-/issues/240895
it sounds like we should apply for the program
Emilio Jesús Gallego Arias (Aug 25 2020 at 17:24):That was my understanding last time I looked.
Emilio Jesús Gallego Arias (Aug 25 2020 at 17:24):That we needed to apply
Jason Gross (Aug 25 2020 at 18:19):I suspect we probably don't want GitHub actions, as this will result in the CI taking 25x more real time to run (since we only get 20 concurrent jobs)
Gaëtan Gilbert (Aug 25 2020 at 18:40):do we really get 500 jobs on gitlab? O_O
Théo Zimmermann (Aug 25 2020 at 19:01):20 concurrent jobs is already pretty good. On GitLab, I feel like we frequently get that as well, but sometimes we also get a single free runner and most of our jobs are run by our own runners. If I'm not mistaken, GitHub also supports setting up your own runners.
Gaëtan Gilbert (Aug 27 2020 at 12:57):https://forum.gitlab.com/t/not-treated-as-open-source-for-pipeline-job-limit/41938
Gaëtan Gilbert (Sep 01 2020 at 09:42):Emilio Jesús Gallego Arias said:
That we needed to apply
ok shall we do it?
is there any blocker?
Emilio Jesús Gallego Arias (Sep 01 2020 at 11:22):Not that I'm aware
Matthieu Sozeau (Sep 02 2020 at 15:44):I applied, we'll see
Jason Gross (Sep 07 2020 at 02:22):Note that they replied at https://gitlab.com/gitlab-org/gitlab/-/issues/240895#note_402263876:
"""
Ah apologies I didn't realise the project was public.
I'm going to ask around for you to best advise on the next steps.
As mentioned previously this isn't the best process to get the right answers for you. It certainly appears your project is on the free tier going by the behavior but I'm not able to say why this isn't the case sadly.
To ensure you're getting to speak to the right people to help with this further I would recommend either creating a thread about this on our support forum or to contact our Open Source team's email at opensource@gitlab.com. Hope this helps you get it sorted!
"""
So perhaps someone should email?
Gaëtan Gilbert (Sep 23 2020 at 13:11):@Matthieu Sozeau any response to the application? we're still seeing the issue eg https://gitlab.com/coq/coq/-/pipelines/193464370
Matthieu Sozeau (Sep 23 2020 at 13:14):Nope, not yet. They said "allow 10 days for a response" and I sent it last wednesday
Matthieu Sozeau (Sep 30 2020 at 08:00):I just signed the papers to the Gold plan, so we should benefit from it anytime now
Théo Zimmermann (Oct 01 2020 at 07:15):Does not seem to be the case yet: screenshot
Théo Zimmermann (Oct 05 2020 at 15:44):Still the case today :frown:
Théo Zimmermann (Oct 05 2020 at 15:49):And we still get the "job activity exceeded" failure: https://gitlab.com/coq/coq/-/pipelines/197875878
Théo Zimmermann (Oct 15 2020 at 14:33):We're still on the free plan with recurrent "job activity exceeded" failures.
Théo Zimmermann (Oct 19 2020 at 12:57):This is still an issue.
Gaëtan Gilbert (May 21 2021 at 16:50):@Matthieu Sozeau can you ping gitlab about this?
Matthieu Sozeau (May 21 2021 at 16:51):Yes, sure
Matthieu Sozeau (May 26 2021 at 09:52):Done
Matthieu Sozeau (May 26 2021 at 09:52):(I pinged them again)
Jason Gross (May 26 2021 at 21:05):As a kludge, can coqbot restart pipelines that fail with this error?
Théo Zimmermann (May 27 2021 at 07:09):The reason these pipelines fail is that we go over the limits for the Free plan (which should have been lifted if GitLab had really granted us the Gold OSS plan). We believe that the fact that restarting manually allows the pipeline to run despite this limitation is a GitLab bug and we are fortunate that this bug still exists. Therefore, we should avoid abusing it by systematically restarting the failed pipelines, which would put us well beyond the limits for the Free plan.
Théo Zimmermann (May 27 2021 at 07:11):The plan to alleviate this problem was rather to reduce the number of jobs that are systematically run for each pipeline, with a dual mode, where, by default, we run a more lightweight pipeline, then upon request and before merging a PR, we run the full pipeline. This is not yet implemented because I had other priorities that interfered.
Emilio Jesús Gallego Arias (May 27 2021 at 09:23):Indeed, @Théo Zimmermann gave a good summary, the current state of the job limit is a bit strange as even the direction / scope of the bug is not clear. Moreover, all CI systems are under attack due to crypto currency mining, so things may get quite worse for us, but fortunately our choice of using gitlab means we could provide a backup if needed.
Matthieu Sozeau (May 27 2021 at 10:36):Don't we have new workers available for this actually?
Gaëtan Gilbert (May 27 2021 at 10:38):this is not related to workers
Théo Zimmermann (May 27 2021 at 10:40):Even if we had an infinite number of workers that we own, we would still need to switch to our own GitLab (e.g. Inria's GitLab) or get the Gold plan to avoid this limit on the number of concurrent jobs.
Matthieu Sozeau (May 27 2021 at 10:42):I see
Matthieu Sozeau (Jun 02 2021 at 15:05):Sadly, they seem to be overwhelmed... no response still after a week
Matthieu Sozeau (Jun 07 2021 at 18:15):image.png Finally !
Théo Zimmermann (Jun 07 2021 at 18:31):Wonderful! But given the actual start date, they could have adapted the end date :stuck_out_tongue:
Matthieu Sozeau (Jun 07 2021 at 18:50):I guess they'll renew easily, as we're linked to the Inria account
Matthieu Sozeau (Jun 07 2021 at 18:54):I can't seem to find a way to check that the pipeline limit has been unlifted, but we'll know soon enough :)
Théo Zimmermann (Jun 09 2021 at 10:47):Indeed, the last known pipeline activity limit errors were on June 3rd.
Matthieu Sozeau (Jun 10 2021 at 14:05):Hurray!
Gaëtan Gilbert (Sep 07 2021 at 10:35):I'm getting a prompt to renew on https://gitlab.com/coq/coq/-/pipelines
Screenshot_20210907_123410.png
it needs customer credentials, @Matthieu Sozeau can you handle it?
Théo Zimmermann (Sep 07 2021 at 12:21):In another thread, Matthieu answered that the renewal was supposed to be automatic (but that he would check with GitLab support).
Matthieu Sozeau (Sep 07 2021 at 13:22):Done, I sent a renewal request, apparently they're in the process of automating this but it does not work yet for Gitlab for OSS projects
Matthieu Sozeau (Sep 18 2021 at 12:45):The renewal was signed today, so we should get no interuptions
Last updated: Oct 21 2021 at 20:02 UTC