Stream: Dune devs & users

Topic: Caching/sharing ~/.cache/dune on Gitlab/via NFS


view this post on Zulip Paolo Giarrusso (Apr 04 2021 at 16:41):

For more efficient dune-based CI, https://gitlab.com/gasche/gitlab-ocaml-ci-example shows how to save and restore the _build folder*1 across CI runs. I wonder how far one could get; I’m hoping this is easier since AFAIK from the manual the cache data are immutable, but I’m not sure if there’s any cache metadata.

*1 In particular on Gitlab, with some workarounds for gitlab bugs — but similar ideas should apply elsewhere.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 04 2021 at 17:17):

We have discussed a plan for actually having our workers to keep a cache locally, so indeed that could speed CI runs, however sharing the cache seems more tricky, I guess that in this case the preferred method is to actually have the cache daemon act as a server

view this post on Zulip Paolo Giarrusso (Apr 04 2021 at 23:53):

I mean, I’ve seen some plans for distributed caching in https://dune.build/blog/dune-retreat-2020/, but that sounds like a much harder problem (and complex solution) than “have the server listen over TCP rather than Unix domain sockets”. And it’s necessary in certain scenarios.

view this post on Zulip Paolo Giarrusso (Apr 04 2021 at 23:59):

But if I’m willing to, say, prefetch “everything” into a local cache (say with rsync), I can imagine simpler solutions on the dune side.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 05 2021 at 15:55):

Indeed, all I know is that improved cache is very much into the dune roadmap as I'm pretty sure large industrial OCaml users do need that in other to replace some of their tooling with Dune, but I am not an expert; I'd suggest discussion in the Dune bug tracker to see how we can achieve this.

Coq for now will be more modest and just equip our own workers with a local cache.

view this post on Zulip Emilio Jesús Gallego Arias (Apr 07 2021 at 17:17):

By the way this is the cache implementation https://github.com/ocaml/dune/pull/4443


Last updated: Oct 16 2021 at 07:02 UTC