samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs:
cole-h has quit [Quit: Goodbye]
jonringer has quit [Ping timeout: 250 seconds]
<gchristensen> Equinix Metal asked me yesterday to review our account for any machines we're not really using. There were probably a half dozen machines or so that I expected their spot market algorithm would have taken back by now, so I released those, plus a couple of ARM machines that I think we could get back without trouble in the future. Just an FYI
<lukegb> 👍
<gchristensen> they've also asked if we could make our infra more dynamic, placing and revoking spot bids based on demand
eyJhb has quit [Quit: Clever message]
eyJhb has joined #nixos-infra
cole-h has joined #nixos-infra
jonringer has joined #nixos-infra
cole-h has quit [Ping timeout: 240 seconds]
<jonringer> gchristensen: before the 21.11 release, we should probably come up with a solution to sunsetting binary caches
<gchristensen> oh?
<jonringer> sunsetting packages that are in EOL nixpkgs
<jonringer> like really old unstable, and EOL releases
<gchristensen> oh, like pruning the cache?
<gchristensen> or packages' nix expressions in the nixpkgs.git repository
<jonringer> yea, IIRC, we have retained everything since the cache was started
<jonringer> the s3 bucket
<gchristensen> ah
<jonringer> "really old unstable" being like 18months
<gchristensen> not sure we need to wait (or schedule it) for 21.11, unless there is something about 21.11 you're thinking about?
<jonringer> nothing in particular, but coming up with a reasonable solution for doing this in the present and future I would assume take some time
<samueldr> what is the goal?
<jonringer> assume would take*
<gchristensen> gotcha
<gchristensen> yeah, it is going to be hard
<jonringer> The goal is just to be better stewards of resources, and if we wanted, could dedicated that storage in other ways
<jonringer> I was thinking of something like, we retain the past two releases (3 during the one month transition period), and X months of unstable
<gchristensen> so retaining old releases wouldn't even be a problem exactly
<gchristensen> we could keep the newest channel version of every release and prune everything between and keep a lot of history, but lose a lot of other builds we probably don't need
<jonringer> how to organize this, I have no idea.
<gchristensen> my bet is we'll need a lot of ram
<jonringer> Can always download more :)
<gchristensen> create a map of all the store paths in the cache and their dependencies, register GC roots, and then delete the oldest unrooted paths
<samueldr> old channels still being cached has been shown as an example of... not sure what, but an example that NixOS is... healthy? useful? not a toy?
<samueldr> also been used in some examples to find older binaries quickly if you need them
<gchristensen> yeah, but also it is useful for archaeology :)
<samueldr> e.g. you need an old firefox
<gchristensen> or postgers
<samueldr> meh, no one ever needs an old postgres
<samueldr> ;)
<jonringer> yea, it's nice as a "wow, look a these old things". But people shouldn't be encouraged to use those as they likely have CVEs and other issues
<samueldr> they're not encouraged to use those, though, are they?
<samueldr> without the cache it still would work (hopefully) by building, but building is expensive
<samueldr> what I aim to say is other than "cleaning up the old stuff" I see no advantage in cleaning up the old stuff
<jonringer> Well, seen some blog posts about dumpster diving through old releases
<gchristensen> one advantage is cost
<jonringer> anyway, I don't think having an "infinitely increasing cache" is sustainable on principle
<gchristensen> if the foundation had to pay for the cache tomorrow, we'd be okay for a few months but we'd need to prune
<samueldr> yeah, that's part of "cleaning up the old stuff" for me
<gchristensen> yea
<gchristensen> I wouldn't want to delete all the old history
<gchristensen> but there are a lot of artifacts we could easily lose and nobody would ever know
<samueldr> yeah
<gchristensen> but back to the main point
<gchristensen> it would be really good to be ready and able to clean up the cache in a controlled way so we don't get in to a scenario where we're feeling like we have to "just do it now"
<samueldr> yes, planning, being ready for that is good
<gchristensen> a related topic is our rate of growth appears to be increasing
<gchristensen> not unexpected: (linux,darwin) -aarch64, packages
<lukegb> I asked the question before but if we end up with "bad bits" in the binary cache it would be nice to have a written-down process for obliterating them
<gchristensen> yeah
<lukegb> (bad bits here being things like software that has a do-not-distribute-ever license that ends up being built and cached due to... whatever reason)
<gchristensen> yup
<gchristensen> ideally a process with poka yoke :P
<gchristensen> ("oops scaled S3 down by a factor of ten via typo" errors and whatnot ...)
<jonringer> lol
<jonringer> anyway, thats part of the reason why I brought up the 21.11 release, if we did want to have "buckets" that can be phased out altogether with related packages, then that would be the time to implement such a conversion
<MichaelRaskin> Given … the internet it would surely be nice if FODs mostly survived
<jonringer> similar to other jobsets, not a lot of the staging-next packages are super relevant after it's been merged to master
<MichaelRaskin> (But indeed, too many binary builds retained)
<samueldr> would be nice to have a "project" to scan all revisions of Nixpkgs for FODs URLs and pipe them back into or something like that
<gchristensen> MichaelRaskin: agreed, I think those can be largely identified from the narinfo luckily
<jonringer> nix show-derivation differientiates between eval drvs and FOD drvs
<gchristensen> samueldr: I think they're going to software heritage these days
<samueldr> something like that I guess :)
<samueldr> and why not both? can't have our eggs in one basket!
<gchristensen> :)
<MichaelRaskin> About binaries — after one year, first + last + highest-success-rate-of-each-month for each Hydra jobset sounds like a lot, but also much less than now
<MichaelRaskin> But a ton of Hydra DB crunching…
<samueldr> yeah, multiple "stories", use cases, for the question about collecting the cache
<lukegb> Hydra DB crunching isn't too bad since it can be done offline
<gchristensen> one caveat is hydra's db doesn't have *all* the logs
<MichaelRaskin> Oops
<gchristensen> most, though :)
<gchristensen> a good 80% solution
<MichaelRaskin> Are the missing bits the oldest?
<gchristensen> yea
<samueldr> would be nice to gather some info, e.g. unrooted paths
<gchristensen> I can provide a manifest of everything in the cache
<roberth> I'm experiencing a slow, both from my home in NL and a server in DE
<roberth> hmm that was a bit euro-centric; Netherlands and Germany
<roberth> 78.63 MiB download is taking minutes
<roberth> it's still "going", still 55 MiB after restarting `nix-build`
<roberth> this is on an idle server in a data center
supersandro2000 has quit [Killed ( (Nickname regained by services))]
supersandro2000 has joined #nixos-infra
cole-h has joined #nixos-infra