Stream: Coq devs & plugin devs

Topic: macOS CI


view this post on Zulip Théo Zimmermann (Sep 30 2020 at 15:55):

macOS tests are frequently failing during the system package installation phase these days with errors like the following (I had to scroll up a lot to see it):

fatal: unable to access 'https://github.com/Homebrew/homebrew-core/': transfer closed with outstanding read data remaining
Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core failed!

view this post on Zulip Théo Zimmermann (Oct 01 2020 at 07:48):

It apparently is a GitHub issue that is actively being discussed (and worked on?) at https://github.com/Homebrew/homebrew-core/issues/61772.

view this post on Zulip Théo Zimmermann (Oct 09 2020 at 14:36):

The issue still exists and makes many of our pull request checks fail. The Homebrew team has no intention to propose a solution despite the issue affecting lots of users and CI pipelines. They blame GitHub and have closed and locked the issue.

view this post on Zulip Théo Zimmermann (Oct 09 2020 at 14:37):

Given their lack of care for this problem but also the various issues that we have had in the past (e.g

view this post on Zulip Théo Zimmermann (Oct 09 2020 at 14:39):

pinning), I wonder whether we should not try to remove our dependency on Homebrew, e.g. by switching to MacPorts (@Michael Soegtrop has some experience with it).

view this post on Zulip Théo Zimmermann (Oct 09 2020 at 14:41):

Another option (bigger change) would be to try to create the bundle from a Nix build. I'm not even sure it would work, but it would probably be the best possible option with respect to reproducibility.

view this post on Zulip Paolo Giarrusso (Oct 09 2020 at 19:14):

this issue is probably genuinely due to GitHub, but their support to users generates the same sentiment in most users.

view this post on Zulip Paolo Giarrusso (Oct 09 2020 at 19:15):

As a partial workaround, they are a bit less aggressive against threads outside of their bug tracker (they have a discourse), which they consider as their TODO list with an “inbox 0” policy.

view this post on Zulip Paolo Giarrusso (Oct 09 2020 at 19:16):

regarding Nix, do you have a local expert, _and_ can you install Nix on your macOS CI machine? Creating a /nix “partition” is somewhat invasive (tho they have a wizard for it)

view this post on Zulip Paolo Giarrusso (Oct 09 2020 at 19:17):

and can you build binaries that are _not_ installed in /nix, given that binaries really like absolute paths?

view this post on Zulip Théo Zimmermann (Oct 12 2020 at 07:57):

Regarding Nix, no, we have knowledgeable people on Nix but not on Nix + macOS. My hope for this solution resided in the understanding that macOS bundles are sorts of isolated filesystems that you just mount when you wish to use them. Maybe I'm mistaken though.

view this post on Zulip Maxime Dénès (Oct 12 2020 at 08:54):

Isn't Vincent knowledgeable in Nix + macOS? We could ask him for advice if needed.

view this post on Zulip Théo Zimmermann (Oct 12 2020 at 09:01):

Partly yes. I remember him saying that he was no macOS expert and he used a macOS machine just because that's what IMDEA had given him.

view this post on Zulip Paolo Giarrusso (Oct 12 2020 at 09:01):

for most Mac users, a .dmg is a filesystem that you mount (at a random place) to copy the contained folder (e.g. CoqIDE.app) into /Applications/CoqIDE-8.12.app. But it’s true that many apps can be run from there, and the app can even be renamed (?), so you’re probably right?

view this post on Zulip Paolo Giarrusso (Oct 12 2020 at 09:02):

so the question might indeed reduce to “can you create a correct .dmg from Nix”

view this post on Zulip Théo Zimmermann (Oct 12 2020 at 09:03):

Yes, and I haven't found answers to this question with a quick Google search so I don't think that it's a common practice.

view this post on Zulip Théo Zimmermann (Oct 12 2020 at 09:03):

But let's not forget that I was also suggesting a simpler solution that could be to switch to Mac Ports.

view this post on Zulip Michael Soegtrop (Oct 12 2020 at 09:25):

Théo Zimmermann said:

pinning), I wonder whether we should not try to remove our dependency on Homebrew, e.g. by switching to MacPorts (Michael Soegtrop has some experience with it).

The coq platform supports both MacPorts and homebrew. I did quite a few pushes to opam conf packages to add depext lines for MacPorts. For what is needed for core Coq all should be merged upstream meanwhile.

I personally use MacPorts and I would say it is fairly stable. Afaik they have their own infrastructure but it is not said that this is more stable than GitHub longterm.

Probably for our CI we should have our own servers for whatever our CI needs - not only for stability reasons but also to control updates.

view this post on Zulip Théo Zimmermann (Oct 21 2020 at 21:05):

This is still an issue, after more than 3 weeks. Homebrew installs fail almost 25% of the time.

view this post on Zulip Théo Zimmermann (Oct 21 2020 at 21:06):

We should really look into migrating to MacPorts.

view this post on Zulip Emilio Jesús Gallego Arias (Oct 21 2020 at 22:59):

How about Nix? It seems to me that given the status of the OSX package system is the most convenient for users.

view this post on Zulip Théo Zimmermann (Oct 22 2020 at 08:16):

Any user can already install Coq on macOS through Nix. If you are talking about using Nix to generate the macOS bundle in our CI, then yes, this is also a solution that I proposed earlier in this thread, although I am not sure how practical it is.

view this post on Zulip Théo Zimmermann (Oct 28 2020 at 08:14):

Now it's failing more than half of the time!

view this post on Zulip Théo Zimmermann (Nov 05 2020 at 13:53):

It looks like the outstanding problem might have been fixed but the macOS CI is still failing (now all the time) because of some other issue that happened in the meantime:

fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Error: Fetching /usr/local/Homebrew/Library/Taps/local/homebrew-openssl failed!
Fetching /usr/local/Homebrew/Library/Taps/local/homebrew-python2 failed!
fatal: invalid upstream 'origin/master'
fatal: invalid upstream 'origin/master'

Last updated: Oct 21 2021 at 21:03 UTC