For more efficient dune-based CI, https://gitlab.com/gasche/gitlab-ocaml-ci-example shows how to save and restore the
_build folder*1 across CI runs. I wonder how far one could get; I’m hoping this is easier since AFAIK from the manual the cache data are immutable, but I’m not sure if there’s any cache metadata.
I guess this should work for
~/.cache/dune as well, or would you expect issues?
Could one share the cache folder via NFS, to support _parallel_ builds? Dune couldn’t hard link files between the cache and the build tree; and one would need to disable the cache daemon (since it uses a Unix domain socket). Does the cache need any synchronization? File locking and NFS aren’t friends IIRC.
*1 In particular on Gitlab, with some workarounds for gitlab bugs — but similar ideas should apply elsewhere.
We have discussed a plan for actually having our workers to keep a cache locally, so indeed that could speed CI runs, however sharing the cache seems more tricky, I guess that in this case the preferred method is to actually have the cache daemon act as a server
I mean, I’ve seen some plans for distributed caching in https://dune.build/blog/dune-retreat-2020/, but that sounds like a much harder problem (and complex solution) than “have the server listen over TCP rather than Unix domain sockets”. And it’s necessary in certain scenarios.
But if I’m willing to, say, prefetch “everything” into a local cache (say with
rsync), I can imagine simpler solutions on the dune side.
Indeed, all I know is that improved cache is very much into the dune roadmap as I'm pretty sure large industrial OCaml users do need that in other to replace some of their tooling with Dune, but I am not an expert; I'd suggest discussion in the Dune bug tracker to see how we can achieve this.
Coq for now will be more modest and just equip our own workers with a local cache.
By the way this is the cache implementation https://github.com/ocaml/dune/pull/4443
Last updated: Oct 16 2021 at 07:02 UTC