Stream: Dune devs & users

Topic: Error with Dune and Nix in CI


view this post on Zulip Karl Palmskog (Dec 16 2021 at 10:02):

bringing up the following issue here to see if Dune experts may be able to weigh in: https://github.com/coq/opam-coq-archive/issues/1995

File "nix/store/2w9yyr6c3py70kwsv8zvvx5afsfx3zx3-dune-2.8.5/bin/dune", line 1, characters 0-0:
Error: Invalid dune file
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

(This is not related to Dune for Coq though, but maybe some bad interaction between Nix and Dune)

view this post on Zulip Gaëtan Gilbert (Dec 16 2021 at 10:13):

can you run nix in a more verbose way to see what command it's running?

view this post on Zulip Karl Palmskog (Dec 16 2021 at 10:16):

not sure, but you can see a failing job here which has some more hints: https://gitlab.com/coq/opam-coq-archive/-/jobs/1889653040

copying path '/nix/store/qdh5bjaz7xsd6imsl10f3nfcqylpyvxv-ocaml4.12.0-opam-core-2.0.8' from 'https://cache.nixos.org'...
Info: Creating file dune-project with this contents:
| (lang dune 2.8)
File "nix/store/2w9yyr6c3py70kwsv8zvvx5afsfx3zx3-dune-2.8.5/bin/dune", line 1, characters 0-0:
Error: Invalid dune file

view this post on Zulip Théo Zimmermann (Dec 16 2021 at 10:16):

The command that is executed is nix-shell --run "dune exec --profile=release -- archive2web released extra-dev > coq-packages.json".

view this post on Zulip Théo Zimmermann (Dec 16 2021 at 10:17):

The error we see here happen within a successful nix-shell AFAICT.

view this post on Zulip Théo Zimmermann (Dec 16 2021 at 10:17):

Thus it is the inner dune exec --profile=release -- archive2web released extra-dev > coq-packages.json

view this post on Zulip Théo Zimmermann (Dec 16 2021 at 10:17):

And any change to verbosity of the Dune command should help.

view this post on Zulip Karl Palmskog (Dec 16 2021 at 10:17):

yeah, it's from here: https://github.com/coq/opam-coq-archive/blob/c1a5336ebb0cfe0bbf3bab985f5157f1cf1a3fc0/.gitlab-ci.yml#L153-L163

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 12:54):

The problem is that in this call, dune can't find the "root" for the project, thus it assumes the current dir is a root and creates the dune-proejct

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 12:54):

for example this behavior has been reverted in 3.0

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 12:54):

so dune exec is being called from the wrong directory

view this post on Zulip Karl Palmskog (Dec 16 2021 at 13:05):

could it be due to this file being out of date: https://github.com/coq/opam-coq-archive/blob/master/dune-project

(lang dune 1.9)
(name opam-coq-archive)

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:53):

Not sure, the problem is that dune exec is called from the wrong directory.

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:54):

Yeah that's the problem dune exec --profile=release -- archive2web released extra-dev

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:54):

this is called from the root of the build, it should be something such as cd $COQ_ARCHIVE_DIR && dune exec

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:55):

or use the --root option

view this post on Zulip Enrico Tassi (Dec 16 2021 at 13:55):

how can this depend on the CI worker, or the OS version?

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:55):

This problem depends on two things:

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:56):

if your opam switch is in _opam dune will not scan it

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:56):

but here opam exec puts the cwd at the root of the switch build [weird, but it is this way]

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:56):

so check pwd in that nix-shell call

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:56):

and adjust using --root or cd

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:57):

As mentioned Dune 3.0 have slightly saner rules for when a workspace can be detected

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:58):

but in this case the command would likely fail "cannot find dune workspace"

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:59):

The problems reproducing this bug are IMHO due to a) the reproduction calls dune inside the right directory or b) the reproduction calls dune from the wrong directory, but the opam switch was not initialized yet so the bin/dune clash is avoided, then dune will pick the workspace after scan

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:59):

[which is a bit worrying and thus removed in 3.0]

view this post on Zulip Emilio Jesús Gallego Arias (Dec 16 2021 at 13:59):

This change had to wait for 3.0 as it is not backwards-compatible

view this post on Zulip Enrico Tassi (Dec 16 2021 at 14:13):

this still does not explain why the bug pops up just now


Last updated: Jan 30 2023 at 18:04 UTC