Gabriel Scherer (May 16 2021 at 12:47): Gabriel Scherer (May 16 2021 at 12:47):@Théo Zimmermann I just wrote a script to generate types-Zulip archives using Gitlab CI + Pages. Would you be willing to eat my own dogfood by trying the script for a second instance?
Gabriel Scherer (May 16 2021 at 12:48):https://gitlab.com/gasche/zulip-archive-gitlab-ci
Théo Zimmermann (May 16 2021 at 18:42):Wow, that's great @Gabriel Scherer! I'll definitely try to use it for the Coq Zulip and let you know if I run in any trouble.
Théo Zimmermann (May 24 2021 at 15:51):Thanks to @Gabriel Scherer, we know have a public archive at https://coq.gitlab.io/zulip-archive/.
Théo Zimmermann (May 24 2021 at 15:51):However, this is not yet updated automatically. The documentation suggests to set up a scheduled pipeline manually, but AFAICS, this can only be set up by modifying .gitlab-ci.yml.
Théo Zimmermann (May 24 2021 at 15:52):@Gabriel Scherer Shouldn't the template include some code for scheduled pipelines?
Yannick Forster (May 24 2021 at 17:46):That's great! Small teething problem: I'm getting 404 errors when clicking on timestamps of messages in a topic overview. There's a -gitlab-ci in the URL which likely shouldn't be there.
Gabriel Scherer (May 24 2021 at 19:23):@Théo Zimmermann no, you can create a scheduled pipeline manually from the web interface
Gabriel Scherer (May 24 2021 at 19:23):actually I don't know how to add the to .gitlab-ci.yml -- pull requests welcome!
Gabriel Scherer (May 24 2021 at 19:23):the manual way is in the "CI/CD > Schedules" corner of the repository menu
Gabriel Scherer (May 24 2021 at 19:24):I tried to document this in the README, but apparently I failed
Théo Zimmermann (May 24 2021 at 19:26):Oh no, you're right and I read too fast: I was looking for this under Settings.
Théo Zimmermann (May 24 2021 at 19:30):OK, so I've now set up a scheduled pipeline to run every hour.
Théo Zimmermann (May 24 2021 at 19:33):Yannick Forster said:
That's great! Small teething problem: I'm getting 404 errors when clicking on timestamps of messages in a topic overview. There's a
-gitlab-ciin the URL which likely shouldn't be there.
Now fixed, by re-running a pipeline (after I changed the name of the repo).
Théo Zimmermann (May 24 2021 at 19:35):I guess we should add a link to https://coq.inria.fr/community and then make an announcement on #Announcements. WDYT @Cyril Cohen?
Gabriel Scherer (May 24 2021 at 19:40):I would welcome a last PR to add a link to your archives to the "Known Deployments" list of https://gitlab.com/gasche/zulip-archive-gitlab-ci . (One reason to maintain that is that communities are terrible about advertising their Zulip chats outside their core users, so any cross-links from one to another is a win.)
Karl Palmskog (Sep 10 2021 at 09:25):@Théo Zimmermann isn't it time to announce the Zulip archive link more widely and even put a link on the Coq website? The hourly updates seem to work fine to me.
Théo Zimmermann (Sep 10 2021 at 09:26):Right, I had announced in June that this would be done "in a few days".
Théo Zimmermann (Sep 10 2021 at 09:27):Would you mind either opening a PR adding a link or opening an issue on the www repo and assigning it to me?
Karl Palmskog (Sep 10 2021 at 09:30):OK, I can do a www PR. But I think it should also be put on the Discourse.
Karl Palmskog (Sep 10 2021 at 09:50):any help to get the "other PR" unstuck also appreciated
Théo Zimmermann (Sep 10 2021 at 10:13):I'm not so sure that it should be announced on Discourse given that it's purely related to Zulip and was already announced there...
Karl Palmskog (Sep 10 2021 at 10:17):well, how about just putting a message in the Zulip announcement thread with the link: https://coq.discourse.group/t/creation-of-a-zulip-chatroom/839
Yannick Forster (Sep 14 2021 at 14:45):For convenience, I would find it very helpful if there would be a link from the archive to the actual discussion in Zulip. Is that possible?
Théo Zimmermann (Sep 14 2021 at 14:48):Here is the tool we use: https://github.com/zulip/zulip-archive
Théo Zimmermann (Sep 14 2021 at 14:49):And here is the way we deploy it: https://gitlab.com/gasche/zulip-archive-gitlab-ci
Théo Zimmermann (Sep 14 2021 at 14:49):I don't know more that this. Feel free to look around / open issues / report what you found :smile:
Yannick Forster (Sep 14 2021 at 16:16):Thanks! In fact, one can go to the actual Zulip discussion by clicking on the profile picture of the poster of a message. That wasn't very obvious too me, especially since the profile pictures are not displayed properly, but I guess it's good enough.
Yannick Forster (Sep 14 2021 at 16:18):Two more questions: Is it intended that the profile pictures are not shown? Is it intended that we just have black on white? The Lean zulip archive looks a lot more modern and accessible. (I'm asking because it might be a bug or it might be just that nobody yet found the time to giving it a nicer look and feel)
Théo Zimmermann (Sep 14 2021 at 21:05):I wouldn't know. We should ask @Gabriel Scherer.
Gabriel Scherer (Sep 14 2021 at 21:19):It's possible to customize the HTML&CSS of the generated archive, and the Learn archive does it, but I never worked on doing this within zulip-archive-gitlab-ci . @Yannick Forster if you (or someone you know) would be willing to give it a try, Merge Requests are warmly welcome.
Gabriel Scherer (Sep 14 2021 at 21:20):Note: older versions of the zulip-archive tool used Jekyll, which made customization somewhat easy (drop Mustache templates in a _layout directory), but newer versions of the script have a more ad-hoc and simpler customization mechanism. The Lean archive still uses an older version of the tool, so their approach is not directly reusable / future-proof.
Théo Zimmermann (Feb 22 2023 at 10:07):Hello @Gabriel Scherer, since this morning, the Zulip archive pipeline fails consistently with:
Traceback (most recent call last):
File "/builds/coq/zulip-archive/zulip-archive/archive.py", line 42, in <module>
import zulip
ModuleNotFoundError: No module named 'zulip'
Here is an example: https://gitlab.com/coq/zulip-archive/-/jobs/3812590329
Do you know what this means?
Gaëtan Gilbert (Feb 22 2023 at 10:18):probably related to the "error: externally-managed-environment" a bit earlier in the log
Gabriel Scherer (Feb 22 2023 at 10:55):I have no idea what this means, sorry. Can I let you investigate?
(I wondered if this was due to an upstream change in the zulip-archive repo, but I am pulling from a feature branch that shouldn't evolve in time so this is (possible but) less likely.)
Gaëtan Gilbert (Feb 22 2023 at 11:03):it's a change in debian/ubuntu's python packaging
you have to use a "virtual env" (I don't know what that is btw) to use pip, you can't just pip install anymore
Théo Zimmermann (Feb 22 2023 at 11:34):OK, thanks for the diagnosis. I think I can fix it by switching to Nix to install the required dependencies (I do not know how to use pip / venv even if I kind of know what this is, it's like a local opam switch).
Théo Zimmermann (Feb 22 2023 at 11:51):Fixed with: https://gitlab.com/coq/zulip-archive/-/commit/62eb7182b3f5957e1fb26736006c0d9497572c7b
Théo Zimmermann (Feb 22 2023 at 11:53):Possibly another fix would have been to replace debian:testing by debian:stable or something like that (but it would have broken at the next Debian stable release).
Théo Zimmermann (Mar 12 2023 at 20:36):@Gabriel Scherer I've noticed that the pipelines at https://gitlab.com/gasche/types-chat-archive/-/pipelines keep failing since this day. FWIW, my fix at the URL above should work there as well. Let me know if you want me to do a PR.
Paolo Giarrusso (Mar 12 2023 at 20:42):Do we still need a separate archive? It seems Zulip itself allows anon browsing now
Gabriel Scherer (Mar 12 2023 at 20:52):@Théo Zimmermann Thanks! For some reason I had missed the conversation above (I don't stay connected on the Coq Zulip, but was hoping for a notification in case of further reply.) Yes, a PR would be very kind of you.
Théo Zimmermann (Mar 12 2023 at 20:53):Should I do it on https://gitlab.com/gasche/zulip-archive-gitlab-ci/ or directly on https://gitlab.com/gasche/types-chat-archive?
Gabriel Scherer (Mar 12 2023 at 20:54):Actually I guess I should just get motivated and push myself -- to both repos indeed.
Gabriel Scherer (Mar 12 2023 at 20:54):@Paolo Giarrusso two reasons to keep the archive around:
Théo Zimmermann (Mar 12 2023 at 20:55):Paolo Giarrusso said:
Do we still need a separate archive? It seems Zulip itself allows anon browsing now
Indeed, we have turned all the public streams into web-public so Zulip allows anonymous browsing. However, at the time it was introduced, this was not web-search-indexed. I don't know what the status is today, but I have kept the archive because the cost was small.
Gabriel Scherer (Mar 12 2023 at 20:55):Hm, looking at the patch, I see that you sneakily nixified the script :-p
Gabriel Scherer (Mar 12 2023 at 20:58):Nix sounds like it is going to be harder to maintain in the future for me given that I know nothing about it yet. I should either get a standard-distro fix, or decide that you are now the maintainer of the archive script. (I don't know which is easier for other potential users.)
Théo Zimmermann (Mar 12 2023 at 21:01):This was the only fix I was able to write quickly with my knowledge. At least, it works, but I agree this is not ideal.
Gabriel Scherer (Mar 12 2023 at 21:10):Looks like the Python people decided to break a common workflow to force users to move to virtual environments, and distros are gradually migrating towards following the new rule (PEP 668).
Théo Zimmermann (Mar 12 2023 at 21:16):When I looked for, discovered and learned Nix in 2016, this was because I never wanted to learn again the quirks and changing moods of yet another application-specific package manager. This has worked pretty well for me, but this also means that I cannot be of help here ;-)
Gabriel Scherer (Mar 12 2023 at 21:28):https://gitlab.com/gasche/types-chat-archive/-/merge_requests/1 looks like it fixes the issue
Théo Zimmermann (Mar 12 2023 at 21:29):BTW, looking at the pipeline history at https://gitlab.com/coq/zulip-archive/-/pipelines, I noticed that there are two cases: when the pipeline takes about 2 minutes and when it takes about 30. The second case apparently corresponds to a cache miss, when Zulip archive has to rebuild the whole archive. This apparently never happens in the types-chat-archive pipelines. Cache misses are becoming way more common lately and are causing some random failures (failures always happen after ~21 minutes, because of some timeout, probably because of some large thing to fetch). Any idea what could cause cache misses? I suppose this is a GitLab CI issue.
Gabriel Scherer (Mar 12 2023 at 21:38):I pushed my pip-level fix to the types-chat archive and the upstream repo, so that should fix it.
Gabriel Scherer (Mar 12 2023 at 21:38):I must say that I have no clue about the cache/timeout issue. I wondered if nix was somehow interacting badly with the cache, but it looks like you also had the issue before moving to nix.
Théo Zimmermann (Mar 12 2023 at 21:39):Yes, sorry I should have clarified that I checked for this.
Gabriel Scherer (Mar 12 2023 at 21:39):(thanks @Gaëtan Gilbert for letting me know about the pip+debian issue)
Gabriel Scherer (Mar 12 2023 at 21:39):In any case, it's bedtime for me, so I will wonder about this -- maybe -- tomorrow. Have fun, and thanks all for the help.
Théo Zimmermann (Mar 12 2023 at 21:42):It's also bedtime for me, I will also wonder about this tomorrow, or later. Anyway, I'm sure I will remember to think about it because I receive an email each time a pipeline fails, and I've been receiving too many of these emails lately!
Gaëtan Gilbert (Mar 28 2023 at 13:11):BTW this pip break will probably get us when we try to regenerate the docker image for CI some day
Last updated: Sep 25 2023 at 14:01 UTC