<gchristensen>
Equinix Metal asked me yesterday to review our account for any machines we're not really using. There were probably a half dozen machines or so that I expected their spot market algorithm would have taken back by now, so I released those, plus a couple of ARM machines that I think we could get back without trouble in the future. Just an FYI
<lukegb>
👍
<gchristensen>
they've also asked if we could make our infra more dynamic, placing and revoking spot bids based on demand
eyJhb has quit [Quit: Clever message]
eyJhb has joined #nixos-infra
cole-h has joined #nixos-infra
jonringer has joined #nixos-infra
cole-h has quit [Ping timeout: 240 seconds]
<jonringer>
gchristensen: before the 21.11 release, we should probably come up with a solution to sunsetting binary caches
<gchristensen>
oh?
<jonringer>
sunsetting packages that are in EOL nixpkgs
<jonringer>
like really old unstable, and EOL releases
<gchristensen>
oh, like pruning the cache?
<gchristensen>
or packages' nix expressions in the nixpkgs.git repository
<jonringer>
yea, IIRC, we have retained everything since the cache was started
<jonringer>
the s3 bucket
<gchristensen>
ah
<jonringer>
"really old unstable" being like 18months
<gchristensen>
not sure we need to wait (or schedule it) for 21.11, unless there is something about 21.11 you're thinking about?
<jonringer>
nothing in particular, but coming up with a reasonable solution for doing this in the present and future I would assume take some time
<samueldr>
what is the goal?
<jonringer>
assume would take*
<gchristensen>
gotcha
<gchristensen>
yeah, it is going to be hard
<jonringer>
The goal is just to be better stewards of resources, and if we wanted, could dedicated that storage in other ways
<jonringer>
I was thinking of something like, we retain the past two releases (3 during the one month transition period), and X months of unstable
<gchristensen>
so retaining old releases wouldn't even be a problem exactly
<gchristensen>
we could keep the newest channel version of every release and prune everything between and keep a lot of history, but lose a lot of other builds we probably don't need
<jonringer>
how to organize this, I have no idea.
<gchristensen>
my bet is we'll need a lot of ram
<jonringer>
Can always download more :)
<gchristensen>
create a map of all the store paths in the cache and their dependencies, register GC roots, and then delete the oldest unrooted paths
<samueldr>
old channels still being cached has been shown as an example of... not sure what, but an example that NixOS is... healthy? useful? not a toy?
<samueldr>
also been used in some examples to find older binaries quickly if you need them
<gchristensen>
yeah, but also it is useful for archaeology :)
<samueldr>
e.g. you need an old firefox
<gchristensen>
or postgers
<samueldr>
meh, no one ever needs an old postgres
<samueldr>
;)
<jonringer>
yea, it's nice as a "wow, look a these old things". But people shouldn't be encouraged to use those as they likely have CVEs and other issues
<samueldr>
they're not encouraged to use those, though, are they?
<samueldr>
without the cache it still would work (hopefully) by building, but building is expensive
<samueldr>
what I aim to say is other than "cleaning up the old stuff" I see no advantage in cleaning up the old stuff
<jonringer>
Well, seen some blog posts about dumpster diving through old releases
<gchristensen>
one advantage is cost
<jonringer>
anyway, I don't think having an "infinitely increasing cache" is sustainable on principle
<gchristensen>
if the foundation had to pay for the cache tomorrow, we'd be okay for a few months but we'd need to prune
<samueldr>
yeah, that's part of "cleaning up the old stuff" for me
<gchristensen>
yea
<gchristensen>
I wouldn't want to delete all the old history
<gchristensen>
but there are a lot of artifacts we could easily lose and nobody would ever know
<samueldr>
yeah
<gchristensen>
but back to the main point
<gchristensen>
it would be really good to be ready and able to clean up the cache in a controlled way so we don't get in to a scenario where we're feeling like we have to "just do it now"
<samueldr>
yes, planning, being ready for that is good
<gchristensen>
a related topic is our rate of growth appears to be increasing
<gchristensen>
not unexpected: (linux,darwin) -aarch64, packages
<lukegb>
I asked the question before but if we end up with "bad bits" in the binary cache it would be nice to have a written-down process for obliterating them
<gchristensen>
yeah
<lukegb>
(bad bits here being things like software that has a do-not-distribute-ever license that ends up being built and cached due to... whatever reason)
<gchristensen>
yup
<gchristensen>
ideally a process with poka yoke :P
<gchristensen>
("oops scaled S3 down by a factor of ten via typo" errors and whatnot ...)
<jonringer>
lol
<jonringer>
anyway, thats part of the reason why I brought up the 21.11 release, if we did want to have "buckets" that can be phased out altogether with related packages, then that would be the time to implement such a conversion
<MichaelRaskin>
Given … the internet it would surely be nice if FODs mostly survived
<jonringer>
similar to other jobsets, not a lot of the staging-next packages are super relevant after it's been merged to master
<MichaelRaskin>
(But indeed, too many binary builds retained)
<samueldr>
would be nice to have a "project" to scan all revisions of Nixpkgs for FODs URLs and pipe them back into archive.org or something like that
<gchristensen>
MichaelRaskin: agreed, I think those can be largely identified from the narinfo luckily
<jonringer>
nix show-derivation differientiates between eval drvs and FOD drvs
<gchristensen>
samueldr: I think they're going to software heritage these days
<samueldr>
something like that I guess :)
<samueldr>
and why not both? can't have our eggs in one basket!
<gchristensen>
:)
<MichaelRaskin>
About binaries — after one year, first + last + highest-success-rate-of-each-month for each Hydra jobset sounds like a lot, but also much less than now
<MichaelRaskin>
But a ton of Hydra DB crunching…
<samueldr>
yeah, multiple "stories", use cases, for the question about collecting the cache
<lukegb>
Hydra DB crunching isn't too bad since it can be done offline
<gchristensen>
one caveat is hydra's db doesn't have *all* the logs
<MichaelRaskin>
Oops
<gchristensen>
most, though :)
<gchristensen>
a good 80% solution
<MichaelRaskin>
Are the missing bits the oldest?
<gchristensen>
yea
<samueldr>
would be nice to gather some info, e.g. unrooted paths
<gchristensen>
I can provide a manifest of everything in the cache
<roberth>
I'm experiencing a slow https://cache.nixos.org, both from my home in NL and a server in DE
<roberth>
hmm that was a bit euro-centric; Netherlands and Germany
<roberth>
78.63 MiB download is taking minutes
<roberth>
it's still "going", still 55 MiB after restarting `nix-build`
<roberth>
this is on an idle server in a data center
supersandro2000 has quit [Killed (card.freenode.net (Nickname regained by services))]