Michael Soegtrop (Jan 30 2021 at 15:12): Michael Soegtrop (Jan 30 2021 at 15:12):I analyzed the CI failures on the v8.13 branch. Apparently tar got broken on Cygwin 32 with the update to tar 1.33. It has issues extracting symlinks to files which are extracted later.
It is rather complicated to use old package versions in cygwin. So the question is what we shall do about this. The options I see are:
1.) Change the package which has issues. As far as I can tell this applies only to two empty test result files in mathcomp. On option would be to upload a fixed version (symlinks replaces with copies) and use this until this is fixed in cygwin.
2.) Switch off 32 bit CI / support until this is fixed
3.) Compile tar from sources in 32 bit Windows Coq platform
4.) Use some hacks to get the old binary version (which is not guaranteed to exist forever)
The cygwin maintainers are aware of the issue since January 11, but not much did happen since then (which is rare).
@Enrico Tassi : since this only affects mathcomp: what is your opinion?
Michael Soegtrop (Jan 30 2021 at 15:29):P.S.: I think I found a solution for installing the old tar. Essentially one can download the binary archive from the cygwin repo server and just extract it to /. This seems to work in local tests (still running).
Enrico Tassi (Jan 30 2021 at 17:19):Hum, these symlinks are just for the test suite, which we don't necessarily run, so any quick hack to install a non broken tar version would be ok to me (or even a broken one, as long as it does not fail).
What worries me the most is that, while cygwin is a life saver on windows, as a distribution it looks like debian experimental: no testing. I don't know how their update process works, but breaking tar seems a super severe issue to me, how is that it was not spot/fixed before it reached final users?
I don't know if it is just a feeling, or if things break as often as it seems. My worry is that if they break during a release, then it becomes really painful. It seems that cygwin mirrors keep 2 version of each package. Maybe we can devise a system where one can hint "install version X if available", hoping that by the time X is removed, X+2 fixed the problem of version X+1. I don't know setup.exe well enough to know if this is feasible.
Michael Soegtrop (Jan 30 2021 at 17:27):@Enrico Tassi : I don't know how their update process works, but breaking tar seems a super severe issue to me, how is that it was not spot/fixed before it reached final users?
I think they do reasonable testing - cygwin breaks rarely - I would say there were no more than 5 issues since 8.6. I think the reason this was not spotted before is that it happens only on 32 bit Windows and also most tar files do work fine, e.g. in the whole process of installing cygwin and the platform only the mathcomp file didn't work. As far as I can tell it fails if the file one symlinks to is not already there, which seems to be rare.
Btw.: afaik the bug is in the original gnu code - it likely wouldn't work on 32 bit Linux either, but it might also depend on the compiler. It was just unsafe C code.
Théo Zimmermann (Jan 30 2021 at 17:48):Note that we've discussed before with Michael to have some form of caching of cygwin so that we don't risk having such failures during a release process (and builds are reproducible).
Michael Soegtrop (Feb 01 2021 at 09:31):We should indeed have such a cache. As it looks there is just another Cygwin update which breaks things - as it looks an update of GNU binutils leads to random (not reproducible) assembler failures. Cygwin pushed the GNU changes through quickly because it claims to fix an important security issue.
Michael Soegtrop (Feb 01 2021 at 09:33): Enrico Tassi (Feb 01 2021 at 09:46):The problem with a cache is that it has to be maintained. So I would prefer any other option. Apparently one cannot install a specific version of a package from the command line, so my proposal is not going to work either.
Enrico Tassi (Feb 01 2021 at 09:47):I found this thing, but I don't know if it is reliable https://github.com/transcode-open/apt-cyg
Enrico Tassi (Feb 01 2021 at 09:50):and it does not support versions either, so it does not work out of the box
Enrico Tassi (Feb 01 2021 at 09:57):Hum, I found a note https://cygwin.com/package-server.html which says you can create just an overlay repository. I wonder where we could host it.
Michael Soegtrop (Feb 01 2021 at 10:16):My current solution is that I just download and untar the old version in the setup script - as it looks the repo servers keep the current + 2 older versions. This is not 100% guaranteed to work, one might have to run a post install script in addition, but if it works for tar, it should work for most things.
See (https://github.com/coq/platform/blob/v8.13/shell_scripts/init_cygwin_fixes.sh) - I have currently two PRs in CI to fix the binutils issue in the same way.
For release branches I would set up a private repo, on master I would test the latest version and fix it in this way as needed.
Enrico Tassi (Feb 01 2021 at 10:31):Ah, I see that is fine! I would just add a a fallaback if the file does not exist anymore: if we release these script they may stop working when the file disappears (and maybe, in the meanwhile, the package was fixed so the hack is not needed anymore).
Michael Soegtrop (Feb 01 2021 at 10:41):I think it is fine as is - if the file is removed from there it will likely be removed from all repos soon. I made sure to have a local copy in case. I expect that this will be fixed quickly. For the tar fix there is already an update in testing (the testing version is harder to install, though - it contains cygwin.dll as well, which cannot be replaced from within cygwin).
There seem to be preview versions of cygwin - I don't know the details, but I think it would make sense to have a CI with this run regularly enough to detect issues before they are released - at least if they don't do a quick release.
Btw.: I am quite sure that always 3 versions are kept, because in the UI setup one can select 3 versions of every package. Not sure why they don't have this in the command line.
Enrico Tassi (Feb 01 2021 at 10:44):What I mean is that if a user downloads a platform script today which works thanks to the patch, then the very same script may stop working tomorrow simply because cygwin moved to version x+4. If these scripts are part of a released tag, then this is a bit annoying. If the scripts are from a branch, then he can pull and be back on track (assuming we fixed it).
I don't know if you expect users to download a tag an run the scripts from there.
Enrico Tassi (Feb 01 2021 at 10:46):Of course there are other reasons for scripts to break, but this one is easy to foresee: at some point the wget line will fail.
Michael Soegtrop (Feb 01 2021 at 10:47):Ah indeed. At least we should give a reasonable error message. But I think it is still better to just fail in a way we can track then doing something like just seeing if the latest cygwin works.
Anyway I expect this to be fixed before we do a release.
Talking of this: I would like to do a tag now since in US courses are starting and I want to have a fallback install method in case the DMG or Windows installer does not work (I tested the DMG yesterday and it works fine - it just has plain Coq and CoqIDE but for SF courses this is sufficient).
Last updated: Jun 05 2023 at 11:01 UTC