Guillaume Melquiond (Oct 04 2022 at 14:12): Guillaume Melquiond (Oct 04 2022 at 14:12):I am a bit worried that our storage usage did not go down, despite Gaetan changing the setting to not keep the latest artifacts. In fact, the usage has grown by 20GB since last evening. We have two weeks (until October 20th) to understand how to bring this value down to under 250GB.
Théo Zimmermann (Oct 04 2022 at 14:14):Could it be that some clean-up mechanism has not triggered yet?
Guillaume Melquiond (Oct 04 2022 at 14:44):I seem to remember that they run a cron job every hour to clean job artifacts. But I fear that, because they were once marked as "don't expire", the cron job will not consider them.
Maxime Dénès (Oct 04 2022 at 15:04):Yes, maybe not retroactively indeed.
Maxime Dénès (Oct 04 2022 at 15:07):We can let it run for 24h with the new setting, to see if there's not another mechanism which catches up
Théo Zimmermann (Oct 04 2022 at 15:16):I've tried deleting a few stale branches (that were not deleted by coqbot for some reason) to see if that helps.
Guillaume Melquiond (Oct 04 2022 at 15:32):150GB have been freed. (No idea if it is related to the stale branches being removed or if it is a cron job finally kicking in.)
Théo Zimmermann (Oct 04 2022 at 15:33):More likely the latter.
Théo Zimmermann (Oct 04 2022 at 15:33):We will see if it continues going down in the next few hours / days.
Gaëtan Gilbert (Oct 04 2022 at 16:13):Guillaume Melquiond said:
I am a bit worried that our storage usage did not go down, despite Gaetan changing the setting to not keep the latest artifacts. In fact, the usage has grown by 20GB since last evening. We have two weeks (until October 20th) to understand how to bring this value down to under 250GB.
isn't the enforcement date in february?
Gaëtan Gilbert (Oct 04 2022 at 16:14):https://gitlab.com/coq/coq/-/jobs/2258609513 (open PR with no recent activity) still says "They will not be deleted"
maybe we need to delete all the pr branches to force a reset
Guillaume Melquiond (Oct 04 2022 at 16:31):Gaëtan Gilbert said:
isn't the enforcement date in february?
I am pretty sure that yesterday, the enforcement date was October 20. But the text has been modified since then and there are no date anymore, just "the new limits on Premium / Ultimate will be applicable when they upgrade" (no idea what "upgrade" means). They have also added "these changes do not apply to community programs". So, perhaps it is no longer an issue for us.
Guillaume Melquiond (Oct 04 2022 at 16:40):Actually, the sentence is still there, but now in a double-negative way: "storage limits will not be applied before 2022-10-19"
Guillaume Melquiond (Oct 04 2022 at 16:42):And there is also the sentence "Community programs will have the limits applicable for GitLab Ultimate", so this is still relevant for us.
Guillaume Melquiond (Oct 04 2022 at 16:48):So, if I understand correctly, they initially intended to enforce the limits on October 20. But since then, they have noticed that job logs count toward artifact storage, yet they cannot be removed (?). And since they are not confident they can implement a solution in the next two weeks, the enforcement date has become vague.
Gaëtan Gilbert (Oct 05 2022 at 15:17):it would be nice to actually analyze what uses storage instead of deleting random stuff and hoping it's enough
for instance we keep most artifacts for 1 month, it's probably too much tbh, but how much do we need to reduce the limit?
Maxime Dénès (Oct 05 2022 at 15:21):Ok, we can try to collect data next week
Maxime Dénès (Oct 05 2022 at 15:21):It is a bit less pressing than the CI minutes
Jason Gross (Oct 06 2022 at 11:45):Btw, the bug minimizer relies on having artifacts available for both the base of the PR (whatever version of master it's based on) and the tip of the PR. So it'd be nice to not delete artifacts from master too eagerly
Théo Zimmermann (Oct 06 2022 at 12:00):Hum, I think that the actions that we've applied on artifact removal do not affect how artifacts from master are removed because they were already not preserved very long when they do not correspond to the tip of the branch anymore.
Gaëtan Gilbert (Oct 06 2022 at 12:06):not very long = 1 month?
Gaëtan Gilbert (Nov 16 2022 at 16:40):our current usage is currently 2.5 TiB acccording to the interface
Gaëtan Gilbert (Nov 16 2022 at 16:40):(the vast majority as artifacts)
Théo Zimmermann (Nov 16 2022 at 17:05):artifacts or logs, right? They are not distinguished
Gaëtan Gilbert (Jan 31 2023 at 15:19):the UI for storage quotas got updated at some point Screenshot_20230131_161902.png
still can't tell how much is logs vs other artifacts though
Last updated: Feb 01 2023 at 16:03 UTC