samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs: https://logs.nix.samueldr.com/nixos-infra/
cole-h has quit [Ping timeout: 265 seconds]
cole-h has joined #nixos-infra
cole-h has quit [Ping timeout: 240 seconds]
jonringer has quit [Ping timeout: 276 seconds]
vcunat has joined #nixos-infra
vcunat has quit [Ping timeout: 260 seconds]
vcunat has joined #nixos-infra
<vcunat> Aaand my daily reminder of the serious Hydra issue :-) Just trying luck in case someone who can fix it appears here.
<vcunat> Now we can't even have binaries for the default kernel: https://hydra.nixos.org/job/nixpkgs/trunk/linux.x86_64-linux
<vcunat> which is a problem that will propagate to the stable release soon (as soon as kernel patch update gets picked in there)
<niksnut> yeah, the only big-parallel machine (wendy) was commented out
<niksnut> I've re-enabled it
<vcunat> I think that will be too little for the long term.
<vcunat> I assume the comment-out was related to https://github.com/NixOS/nixos-org-configurations/issues/146
<vcunat> Well, certainly better having this old machine than than having none.
<vcunat> But I wonder how it will fare with 16 big-parallel builds scheduled on it now...
<niksnut> load average: 127.14, 93.85, 44.77
<vcunat> Recently when I saw Wendy attempting 5-6 chromium builds at once, most of them timed out. But chromium is kind of extreme.
<niksnut> chromium probably needs its own system feature
<vcunat> "very-big-parallel" :-)
<vcunat> Theoretically it would be ideal to have the chromium build split up into multiple derivations, perhaps even partially runnable in parallel (i.e. on different machines).
<vcunat> But I'm certainly not motivated enough to attempt such a change.
<niksnut> fwiw the packet importer is showing a lot of aarch64 big-parallel machines, but no x86_64
<vcunat> Yes, there is no big-parallel tag just for x86_64/i686 -linux. That's the issue here. (but I don't get why)
<niksnut> in that case I wonder what we were using before...
<vcunat> I believe some (or maybe all) of the hash-named machines had the big-parallel tag until a few days ago.
<lukegb> We _should_ have big-parallel packet machines for x86-64
<lukegb> We at least did at some point
<lukegb> Did we get rid of some when <gchristensen> Equinix Metal asked me yesterday to review our account for any machines we're not really using. There were probably a half dozen machines or so that I expected their spot market algorithm would have taken back by now, so I released those, plus a couple of ARM machines that I think we could get back without trouble in the future. Just an FYI
<lukegb> How did https://hydra.nixos.org/build/142840331 fail with "Unsupported system type" halfway through a build
<vcunat> lukegb: it didn't
<vcunat> logs aren't versioned
<lukegb> _ugh_
<vcunat> if the same derivation is attempted again, they get overwritten
<vcunat> (same-hash derivation)
<vcunat> Old designd decision in nix, I suppose. I mean, if everything is perfectly deterministic, why would you need to version the logs? ;-)
<MichaelRaskin> _log_ determinism was actually better back then, without parallel builds
vcunat has quit [Ping timeout: 260 seconds]
vcunat has joined #nixos-infra
vcunat has quit [Ping timeout: 260 seconds]
vcunat has joined #nixos-infra
vcunat has quit [Ping timeout: 260 seconds]
vcunat has joined #nixos-infra
jonringer has joined #nixos-infra
cole-h has joined #nixos-infra
<gchristensen> I'll take a look at the importer to see why we have no x86's
<gchristensen> I didn't destroy so many x86 machines, I wonder
<gchristensen> I found the problem.
<gchristensen> we should probably take wendy back out of big-parallel once these machines are up
<gchristensen> # cat /var/lib/hydra-packet-import/machines | grep big-pa | grep x86
<gchristensen> root@78612761.packethost.net x86_64-linux,i686-linux /var/lib/hydra/queue-runner/.ssh/id_buildfarm_rsa 1 3 kvm,nixos-test,big-parallel - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUtFUWw5LzJuNjhzcXFyK3VGS3AwekRPdUNnT09saWN1UU16dkl2dmIxOTI=
<gchristensen> root@ded15ecc.packethost.net x86_64-linux,i686-linux /var/lib/hydra/queue-runner/.ssh/id_buildfarm_rsa 1 3 kvm,nixos-test,big-parallel - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUdwb3ZVcGRjQlhyWGFOZk9xM3FFNVpDd2RuQVNQMkRXWHNFTEJlNEVBWUs=
<vcunat> Nice, I see (at least) two of those up and running big-parallel jobs.
<vcunat> And wendy unblocked the current situation by itself (herself?) over the few hours already.
<gchristensen> fwiw machines have a random chance (30%) of becoming big-parallel on startup, unless they're explicitly pinned to being big-parallel
<gchristensen> this is because machines are on the spot market, so we can't really guarantee specific # of availability of a given machine
<gchristensen> this is why we never noticed that typo before :)
<vcunat> :-)
<lukegb> gchristensen++
<lukegb> I had actually looked at that file before but assumed the typo was deliberate xD
supersandro2000 has quit [Quit: The Lounge - https://thelounge.chat]
vcunat has quit [Quit: Leaving.]
supersandro2000 has joined #nixos-infra
NinjaTrappeur has quit [*.net *.split]
JJJollyjim has quit [*.net *.split]
Ox4A6F1 has quit [Ping timeout: 245 seconds]
JJJollyjim has joined #nixos-infra
NinjaTrappeur has joined #nixos-infra
domenkozar[m] has quit [Ping timeout: 246 seconds]
roberth has quit [Ping timeout: 245 seconds]
garbas[m] has quit [Ping timeout: 245 seconds]
JJJollyjim has quit [Ping timeout: 276 seconds]
Ericson2314 has joined #nixos-infra
garbas[m] has joined #nixos-infra
roberth has joined #nixos-infra
domenkozar[m] has joined #nixos-infra
thefloweringash has joined #nixos-infra
ryantm has joined #nixos-infra
nh2[m] has joined #nixos-infra
zimbatm[m] has joined #nixos-infra
ryantm has quit [Ping timeout: 246 seconds]
roberth has quit [Ping timeout: 245 seconds]
garbas[m] has quit [Ping timeout: 245 seconds]
Ericson2314 has quit [Ping timeout: 245 seconds]
thefloweringash has quit [Ping timeout: 258 seconds]
nh2[m] has quit [Ping timeout: 276 seconds]
zimbatm[m] has quit [Ping timeout: 276 seconds]
JJJollyjim has joined #nixos-infra
Ox4A6F1 has joined #nixos-infra
colemickens has joined #nixos-infra
ryantm has joined #nixos-infra
roberth has joined #nixos-infra
Ericson2314 has joined #nixos-infra
thefloweringash has joined #nixos-infra
nh2[m] has joined #nixos-infra
garbas[m] has joined #nixos-infra
zimbatm[m] has joined #nixos-infra
supersandro2000 is now known as Guest72470
Guest72470 has quit [Killed (orwell.freenode.net (Nickname regained by services))]
supersandro2000 has joined #nixos-infra