Théo Zimmermann (Apr 27 2021 at 15:23): Théo Zimmermann (Apr 27 2021 at 15:23):I've seen multiple failures of jobs running on pyrolyse because of git errors. Could it be due to multiple jobs running at the same time and a lack of proper compartmentalization? How do we work around this issue?
Théo Zimmermann (Apr 27 2021 at 15:23):https://gitlab.com/coq/coq/-/jobs?scope=finished
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:25):I have seen that too, no idea.
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:25):I guess the runner backend is not the best one?
Théo Zimmermann (Apr 27 2021 at 15:26):On the most recent jobs, it seems that the majority of the jobs sent to pyrolyse are failing for this reason.
Théo Zimmermann (Apr 27 2021 at 15:27):Roquableu doesn't seem to hit this issue. Maybe I should update the GitLab runner on pyrolyse?
Théo Zimmermann (Apr 27 2021 at 15:28):It might not be even worth keeping pyrolyse active until we fix that.
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:29):Do they differ on their choice of runner backend?
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:29):If the problem is the runner backend should be easy to solve
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:29):[famous last words]
Théo Zimmermann (Apr 27 2021 at 15:30):By runner backend do you mean the GitLab runner software or do you mean something else?
Théo Zimmermann (Apr 27 2021 at 15:31):They were not installed at the same time (pyrolyse setup is older) so they could differ in particular in the version used.
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:42):I mean an option on the runner config
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:42):let me grab the exact name
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:43):Ok, I mean the "executor"
Emilio Jesús Gallego Arias (Apr 27 2021 at 15:43):https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-executors
Théo Zimmermann (Apr 28 2021 at 08:02):Log for a failed pyrolyse job says "Using Docker executor"
Théo Zimmermann (Apr 28 2021 at 08:03):And same on roquableu
Enrico Tassi (Apr 28 2021 at 08:04):pyrolise has also some other funky setup which makes dune find its binary as a dune file and error because it's binary and not text, it happened many times in the opam coq archive.
Enrico Tassi (Apr 28 2021 at 08:04):(so using proper system images could help there too)
Théo Zimmermann (Apr 28 2021 at 09:09):Enrico Tassi said:
(so using proper system images could help there too)
What do you mean by this? And do you understand what's going on? I'm willing to intervene on pyrolyse but I need to understand what the issues are and how to solve them.
Emilio Jesús Gallego Arias (Apr 28 2021 at 12:41):Oh, if that is not the executor then I need to read more, thanks for checking @Théo Zimmermann
Théo Zimmermann (Apr 28 2021 at 17:40):So FWIW, pyrolyse is running gitlab-runner 11.2.0, roquableu is running gitlab-runner 11.9.2 and the shared runners are gitlab-runner 13.11.0-rc1. We can imagine that updating the gitlab runner on pyrolyse to a more recent version should help.
Théo Zimmermann (Apr 28 2021 at 17:41):Oh actually, this is the first time I see this, but it also failed on roquableu with the same error: https://gitlab.com/coq/coq/-/jobs/1219206323
Théo Zimmermann (May 07 2021 at 07:25):Pyrolyse will soon be upgraded to a newer system (from Ubuntu 16.04 currently) and then I will reinstall the latest version of the gitlab-runner. Expect some downtimes.
Maxime Dénès (May 07 2021 at 08:40):Btw do we have some IaC to manage our runners?
Gaëtan Gilbert (May 07 2021 at 08:40):what's iac?
Maxime Dénès (May 07 2021 at 08:41):infra as code
Maxime Dénès (May 07 2021 at 08:41):for this kind of upgrade, typically, you'd perform the OS upgrade and relaunch an Ansible playbook or something like that
Maxime Dénès (May 07 2021 at 08:43):AFAIK we already have 6+ runners, so it would probably be already interesting to automate the configuration
Maxime Dénès (May 07 2021 at 08:44):it would also help have a uniform gitlab runner version, etc
Théo Zimmermann (May 07 2021 at 08:54):No we don't, and yes, that would be great to have if someone wants to invest time in learning to use some tooling for this kind of thing.
Maxime Dénès (May 07 2021 at 08:56):I could give it a try at some point, sounds reasonably easy
Théo Zimmermann (May 07 2021 at 14:40):FTR, pyrolyse was upgraded but for some reason I don't have access to it yet. According to some Puppet config, @Maxime Dénès and @Pierre-Marie Pédrot do (if ever you want to take care of reinstalling the GitLab runner). Otherwise, I will take care of that next week when my access to the machine is fixed.
Théo Zimmermann (May 10 2021 at 14:27):Even on Ubuntu 20.04, the apt package for gitlab-runner is stuck at 11.2.0. I'll try another installation method.
Théo Zimmermann (May 10 2021 at 14:56):I've installed the latest version. But I can't seem to make it connect to GitLab. If someone with access wants to look into it, you are welcome.
Last updated: Sep 15 2024 at 13:02 UTC