Stream: Coq devs & plugin devs

Topic: git process errors


view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:23):

I've seen multiple failures of jobs running on pyrolyse because of git errors. Could it be due to multiple jobs running at the same time and a lack of proper compartmentalization? How do we work around this issue?

view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:23):

https://gitlab.com/coq/coq/-/jobs?scope=finished

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:25):

I have seen that too, no idea.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:25):

I guess the runner backend is not the best one?

view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:26):

On the most recent jobs, it seems that the majority of the jobs sent to pyrolyse are failing for this reason.

view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:27):

Roquableu doesn't seem to hit this issue. Maybe I should update the GitLab runner on pyrolyse?

view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:28):

It might not be even worth keeping pyrolyse active until we fix that.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:29):

Do they differ on their choice of runner backend?

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:29):

If the problem is the runner backend should be easy to solve

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:29):

[famous last words]

view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:30):

By runner backend do you mean the GitLab runner software or do you mean something else?

view this post on Zulip Théo Zimmermann (Apr 27 2021 at 15:31):

They were not installed at the same time (pyrolyse setup is older) so they could differ in particular in the version used.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:42):

I mean an option on the runner config

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:42):

let me grab the exact name

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:43):

Ok, I mean the "executor"

view this post on Zulip Emilio Jesús Gallego Arias (Apr 27 2021 at 15:43):

https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-executors

view this post on Zulip Théo Zimmermann (Apr 28 2021 at 08:02):

Log for a failed pyrolyse job says "Using Docker executor"

view this post on Zulip Théo Zimmermann (Apr 28 2021 at 08:03):

And same on roquableu

view this post on Zulip Enrico Tassi (Apr 28 2021 at 08:04):

pyrolise has also some other funky setup which makes dune find its binary as a dune file and error because it's binary and not text, it happened many times in the opam coq archive.

view this post on Zulip Enrico Tassi (Apr 28 2021 at 08:04):

(so using proper system images could help there too)

view this post on Zulip Théo Zimmermann (Apr 28 2021 at 09:09):

Enrico Tassi said:

(so using proper system images could help there too)

What do you mean by this? And do you understand what's going on? I'm willing to intervene on pyrolyse but I need to understand what the issues are and how to solve them.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 28 2021 at 12:41):

Oh, if that is not the executor then I need to read more, thanks for checking @Théo Zimmermann

view this post on Zulip Théo Zimmermann (Apr 28 2021 at 17:40):

So FWIW, pyrolyse is running gitlab-runner 11.2.0, roquableu is running gitlab-runner 11.9.2 and the shared runners are gitlab-runner 13.11.0-rc1. We can imagine that updating the gitlab runner on pyrolyse to a more recent version should help.

view this post on Zulip Théo Zimmermann (Apr 28 2021 at 17:41):

Oh actually, this is the first time I see this, but it also failed on roquableu with the same error: https://gitlab.com/coq/coq/-/jobs/1219206323

view this post on Zulip Théo Zimmermann (May 07 2021 at 07:25):

Pyrolyse will soon be upgraded to a newer system (from Ubuntu 16.04 currently) and then I will reinstall the latest version of the gitlab-runner. Expect some downtimes.

view this post on Zulip Maxime Dénès (May 07 2021 at 08:40):

Btw do we have some IaC to manage our runners?

view this post on Zulip Gaëtan Gilbert (May 07 2021 at 08:40):

what's iac?

view this post on Zulip Maxime Dénès (May 07 2021 at 08:41):

infra as code

view this post on Zulip Maxime Dénès (May 07 2021 at 08:41):

for this kind of upgrade, typically, you'd perform the OS upgrade and relaunch an Ansible playbook or something like that

view this post on Zulip Maxime Dénès (May 07 2021 at 08:43):

AFAIK we already have 6+ runners, so it would probably be already interesting to automate the configuration

view this post on Zulip Maxime Dénès (May 07 2021 at 08:44):

it would also help have a uniform gitlab runner version, etc

view this post on Zulip Théo Zimmermann (May 07 2021 at 08:54):

No we don't, and yes, that would be great to have if someone wants to invest time in learning to use some tooling for this kind of thing.

view this post on Zulip Maxime Dénès (May 07 2021 at 08:56):

I could give it a try at some point, sounds reasonably easy

view this post on Zulip Théo Zimmermann (May 07 2021 at 14:40):

FTR, pyrolyse was upgraded but for some reason I don't have access to it yet. According to some Puppet config, @Maxime Dénès and @Pierre-Marie Pédrot do (if ever you want to take care of reinstalling the GitLab runner). Otherwise, I will take care of that next week when my access to the machine is fixed.

view this post on Zulip Théo Zimmermann (May 10 2021 at 14:27):

Even on Ubuntu 20.04, the apt package for gitlab-runner is stuck at 11.2.0. I'll try another installation method.

view this post on Zulip Théo Zimmermann (May 10 2021 at 14:56):

I've installed the latest version. But I can't seem to make it connect to GitLab. If someone with access wants to look into it, you are welcome.


Last updated: Oct 16 2021 at 07:02 UTC