What should we do about the recurring too many pipeline jobs error?
The limit is documented here https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlab-cicd
"Max jobs in active pipelines gitlab.com:500 for Free tier, unlimited otherwise self-hosted:unlimited"
I understand we could apply for some open source program and get that limit lifted too?
If you mean https://about.gitlab.com/solutions/open-source/program/ then "All public projects on GitLab.com automatically receive Gold functionality at the project level. Apply to this program if you need Gold functionality at the group level."
job limits are project level
Ok, just an idea, maybe we could just write to them? Documentation on that was always a bit imprecise, I see they have reworded it a bit tho.
I made https://gitlab.com/gitlab-org/gitlab/-/issues/240895
it sounds like we should apply for the program
That was my understanding last time I looked.
That we needed to apply
I suspect we probably don't want GitHub actions, as this will result in the CI taking 25x more real time to run (since we only get 20 concurrent jobs)
do we really get 500 jobs on gitlab? O_O
20 concurrent jobs is already pretty good. On GitLab, I feel like we frequently get that as well, but sometimes we also get a single free runner and most of our jobs are run by our own runners. If I'm not mistaken, GitHub also supports setting up your own runners.
Emilio Jesús Gallego Arias said:
That we needed to apply
ok shall we do it?
is there any blocker?
Not that I'm aware
I applied, we'll see
Note that they replied at https://gitlab.com/gitlab-org/gitlab/-/issues/240895#note_402263876:
Ah apologies I didn't realise the project was public.
I'm going to ask around for you to best advise on the next steps.
As mentioned previously this isn't the best process to get the right answers for you. It certainly appears your project is on the free tier going by the behavior but I'm not able to say why this isn't the case sadly.
To ensure you're getting to speak to the right people to help with this further I would recommend either creating a thread about this on our support forum or to contact our Open Source team's email at firstname.lastname@example.org. Hope this helps you get it sorted!
So perhaps someone should email?
@Matthieu Sozeau any response to the application? we're still seeing the issue eg https://gitlab.com/coq/coq/-/pipelines/193464370
Nope, not yet. They said "allow 10 days for a response" and I sent it last wednesday
I just signed the papers to the Gold plan, so we should benefit from it anytime now
Does not seem to be the case yet: screenshot
Still the case today :frown:
And we still get the "job activity exceeded" failure: https://gitlab.com/coq/coq/-/pipelines/197875878
We're still on the free plan with recurrent "job activity exceeded" failures.
This is still an issue.
@Matthieu Sozeau can you ping gitlab about this?
(I pinged them again)
As a kludge, can coqbot restart pipelines that fail with this error?
The reason these pipelines fail is that we go over the limits for the Free plan (which should have been lifted if GitLab had really granted us the Gold OSS plan). We believe that the fact that restarting manually allows the pipeline to run despite this limitation is a GitLab bug and we are fortunate that this bug still exists. Therefore, we should avoid abusing it by systematically restarting the failed pipelines, which would put us well beyond the limits for the Free plan.
The plan to alleviate this problem was rather to reduce the number of jobs that are systematically run for each pipeline, with a dual mode, where, by default, we run a more lightweight pipeline, then upon request and before merging a PR, we run the full pipeline. This is not yet implemented because I had other priorities that interfered.
Indeed, @Théo Zimmermann gave a good summary, the current state of the job limit is a bit strange as even the direction / scope of the bug is not clear. Moreover, all CI systems are under attack due to crypto currency mining, so things may get quite worse for us, but fortunately our choice of using gitlab means we could provide a backup if needed.
Don't we have new workers available for this actually?
this is not related to workers
Even if we had an infinite number of workers that we own, we would still need to switch to our own GitLab (e.g. Inria's GitLab) or get the Gold plan to avoid this limit on the number of concurrent jobs.
Sadly, they seem to be overwhelmed... no response still after a week
image.png Finally !
Wonderful! But given the actual start date, they could have adapted the end date :stuck_out_tongue:
I guess they'll renew easily, as we're linked to the Inria account
I can't seem to find a way to check that the pipeline limit has been unlifted, but we'll know soon enough :)
Indeed, the last known pipeline activity limit errors were on June 3rd.
I'm getting a prompt to renew on https://gitlab.com/coq/coq/-/pipelines
it needs customer credentials, @Matthieu Sozeau can you handle it?
In another thread, Matthieu answered that the renewal was supposed to be automatic (but that he would check with GitLab support).
Done, I sent a renewal request, apparently they're in the process of automating this but it does not work yet for Gitlab for OSS projects
The renewal was signed today, so we should get no interuptions
Last updated: Oct 21 2021 at 20:02 UTC