Guillaume Melquiond (Oct 03 2022 at 18:31): Guillaume Melquiond (Oct 03 2022 at 18:31):By the way, I am noticing that we are currently using 4TB (!) of storage on gitlab.com. So, runners might not be our only issue.
Guillaume Melquiond (Oct 03 2022 at 18:35):To put things into perspective, in three weeks (if I understand correctly), the cost will be $6 per GB, so let us be ready for a massive bill.
Maxime Dénès (Oct 03 2022 at 18:38):@Guillaume Melquiond Do you know what this space is used for? Do we store artifacts for all builds?
Maxime Dénès (Oct 03 2022 at 18:39):And don't we have a retention policy?
Guillaume Melquiond (Oct 03 2022 at 18:39):It seems we are using 3894.7 GiB for job artifacts.
Guillaume Melquiond (Oct 03 2022 at 18:40):https://gitlab.com/groups/coq/-/usage_quotas
Gaëtan Gilbert (Oct 03 2022 at 18:41):build:base is 300MB, then we have all the other stuff so multiple GB / PR I guess
Gaëtan Gilbert (Oct 03 2022 at 18:49):also artifacts for the latest commit on a branch are never deleted
with our coqbot system that means artifacts for open PRs
so for instance this 11month old job still has 270MB of artifacts and then there's the other jobs in the pipeline https://gitlab.com/coq/coq/-/jobs/1739082131
Maxime Dénès (Oct 03 2022 at 18:50):yeah that's what I meant by retention policy
Gaëtan Gilbert (Oct 03 2022 at 18:51):that's actually a setting https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs so maybe we should unset it
Gaëtan Gilbert (Oct 04 2022 at 12:24):also I'm unchecking the "Keep artifacts from most recent successful jobs" setting, I don't think it's especially useful to have for us
Pierre-Marie Pédrot (Oct 04 2022 at 12:41):We should keep them for the bench though.
Gaëtan Gilbert (Oct 04 2022 at 12:45):Pierre-Marie Pédrot said:
We should keep them for the bench though.
why?
Pierre-Marie Pédrot (Oct 04 2022 at 12:54):@Gaëtan Gilbert I want the bench artifacts to have a more precise look at the timings. There are HTML files to display visually the results, and furthermore I've got a bunch of scripts to parse this and extract relevant data.
Théo Zimmermann (Oct 04 2022 at 12:55):Pierre-Marie Pédrot said:
Gaëtan Gilbert I want the bench artifacts to have a more precise look at the timings. There are HTML files to display visually the results, and furthermore I've got a bunch of scripts to parse this and extract relevant data.
Gaëtan is not removing artifact for all branches, he's just disabling artifacts for stale branches AFAIU.
Gaëtan Gilbert (Oct 04 2022 at 12:56):I would say enabling normal expiration times rather than disabling but I guess not a lot of difference
Guillaume Melquiond (Oct 04 2022 at 14:12):I am a bit worried that our storage usage did not go down, despite Gaetan changing the setting to not keep the latest artifacts. In fact, the usage has grown by 20GB since last evening. We have two weeks (until October 20th) to understand how to bring this value down to under 250GB.
Théo Zimmermann (Oct 04 2022 at 14:14):Could it be that some clean-up mechanism has not triggered yet?
Guillaume Melquiond (Oct 04 2022 at 14:44):I seem to remember that they run a cron job every hour to clean job artifacts. But I fear that, because they were once marked as "don't expire", the cron job will not consider them.
Maxime Dénès (Oct 04 2022 at 15:04):Yes, maybe not retroactively indeed.
Maxime Dénès (Oct 04 2022 at 15:07):We can let it run for 24h with the new setting, to see if there's not another mechanism which catches up
Théo Zimmermann (Oct 04 2022 at 15:16):I've tried deleting a few stale branches (that were not deleted by coqbot for some reason) to see if that helps.
Guillaume Melquiond (Oct 04 2022 at 15:32):150GB have been freed. (No idea if it is related to the stale branches being removed or if it is a cron job finally kicking in.)
Théo Zimmermann (Oct 04 2022 at 15:33):More likely the latter.
Théo Zimmermann (Oct 04 2022 at 15:33):We will see if it continues going down in the next few hours / days.
Gaëtan Gilbert (Oct 04 2022 at 16:13):Guillaume Melquiond said:
I am a bit worried that our storage usage did not go down, despite Gaetan changing the setting to not keep the latest artifacts. In fact, the usage has grown by 20GB since last evening. We have two weeks (until October 20th) to understand how to bring this value down to under 250GB.
isn't the enforcement date in february?
Gaëtan Gilbert (Oct 04 2022 at 16:14):https://gitlab.com/coq/coq/-/jobs/2258609513 (open PR with no recent activity) still says "They will not be deleted"
maybe we need to delete all the pr branches to force a reset
Guillaume Melquiond (Oct 04 2022 at 16:31):Gaëtan Gilbert said:
isn't the enforcement date in february?
I am pretty sure that yesterday, the enforcement date was October 20. But the text has been modified since then and there are no date anymore, just "the new limits on Premium / Ultimate will be applicable when they upgrade" (no idea what "upgrade" means). They have also added "these changes do not apply to community programs". So, perhaps it is no longer an issue for us.
Guillaume Melquiond (Oct 04 2022 at 16:40):Actually, the sentence is still there, but now in a double-negative way: "storage limits will not be applied before 2022-10-19"
Guillaume Melquiond (Oct 04 2022 at 16:42):And there is also the sentence "Community programs will have the limits applicable for GitLab Ultimate", so this is still relevant for us.
Guillaume Melquiond (Oct 04 2022 at 16:48):So, if I understand correctly, they initially intended to enforce the limits on October 20. But since then, they have noticed that job logs count toward artifact storage, yet they cannot be removed (?). And since they are not confident they can implement a solution in the next two weeks, the enforcement date has become vague.
Gaëtan Gilbert (Oct 05 2022 at 15:17):it would be nice to actually analyze what uses storage instead of deleting random stuff and hoping it's enough
for instance we keep most artifacts for 1 month, it's probably too much tbh, but how much do we need to reduce the limit?
Maxime Dénès (Oct 05 2022 at 15:21):Ok, we can try to collect data next week
Maxime Dénès (Oct 05 2022 at 15:21):It is a bit less pressing than the CI minutes
Jason Gross (Oct 06 2022 at 11:45):Btw, the bug minimizer relies on having artifacts available for both the base of the PR (whatever version of master it's based on) and the tip of the PR. So it'd be nice to not delete artifacts from master too eagerly
Théo Zimmermann (Oct 06 2022 at 12:00):Hum, I think that the actions that we've applied on artifact removal do not affect how artifacts from master are removed because they were already not preserved very long when they do not correspond to the tip of the branch anymore.
Gaëtan Gilbert (Oct 06 2022 at 12:06):not very long = 1 month?
Gaëtan Gilbert (Nov 16 2022 at 16:40):our current usage is currently 2.5 TiB acccording to the interface
Gaëtan Gilbert (Nov 16 2022 at 16:40):(the vast majority as artifacts)
Théo Zimmermann (Nov 16 2022 at 17:05):artifacts or logs, right? They are not distinguished
Gaëtan Gilbert (Jan 31 2023 at 15:19):the UI for storage quotas got updated at some point Screenshot_20230131_161902.png
still can't tell how much is logs vs other artifacts though
Last updated: Apr 20 2024 at 08:02 UTC