<gchristensen>
oh crap I can't log in to hydra anymore
<gchristensen>
srhb: can you trigger a fresh eval?
<vcunat>
I just did.
<vcunat>
You can't? :-)
<gchristensen>
I stopped paying Google for my google apps account
<gchristensen>
Aug 03 16:55:06 ceres hydra-evaluator[14013]: starting evaluation of jobset ‘nixos:trunk-combined’ (last checked 1649 s ago)
<gchristensen>
Aug 03 16:55:12 ceres hydra-evaluator[14013]: jobset is unchanged, skipping
<vcunat>
What does "unchanged" mean? I incidentally just pushed some commits to master.
<gchristensen>
vcunat: trigger once more?
<gchristensen>
I'll get myself a new account today, but not this morning
<aminechikhaoui>
gchristensen you can trigger evals from the backend
* vcunat
running
<gchristensen>
I see the error
<gchristensen>
oh I really see the error
<gchristensen>
this is thrilling
<gchristensen>
[1378804.091126] EXT4-fs warning (device md126): ext4_dx_add_entry:2190: Directory (ino: 4202498) index full, reach max htree level :2
<gchristensen>
[1378804.091127] EXT4-fs warning (device md126): ext4_dx_add_entry:2194: Large directory feature is not enabled on this filesystem
<vcunat>
Ah, so too many files in /nix/store
justanotheruser has quit [Quit: WeeChat 2.4]
justanotheruser has joined #nixos-dev
<aminechikhaoui>
it's weird, why would it happen just now, GC should be running in a timer
<vcunat>
In any case you probably want to set the option.
<vcunat>
tune2fs -O large_dir
<vcunat>
(I think, but on unmounted FS)
<vcunat>
I suspect dir_index might also be good for such large dirs, if there isn't already.
<gchristensen>
gotcha, I don't think I can do that type of maintenance
<gchristensen>
I've raised an issue with tier 2 support ;)
<gchristensen>
so I think this is newly happening because: the old hydra server (chef)'s disk was 440G and the new hydra server (ceres)'s disk is 860G. the GC would then be leaving many more files around than before.
<gchristensen>
I'm going to run a more aggressive GC than normal to get things moving until we can coordinate a the tuning parameter change with niksnut
<averell>
is there already an issue about moving to split-tiered store paths ala xy/z..., like git, or is this just too rare for real users?
<gchristensen>
I doubt we'll be able to move to that format, and may not be necessary if one of these fancy pants filesystems finally takes hold over ext4
<simpson>
This is usually a one-time event though.
<simpson>
And now you know. (And hopefully it'll get written down somewhere.)
<gchristensen>
having a "nixos in production" guide with a subsection of "so you wanna run a build server"
<niksnut>
oh yeah, we used to have to set some ext4 options in the past
<niksnut>
I thought those were the default now
<gchristensen>
niksnut: running GC now (with the appropriate $((...)) incantation to make 400G free.
<vcunat>
right, more aggressive GC should probably work around it, too
<tilpner>
ryantm: AFAICT Dec. 13 with 6024dc1d97212130c19d3ff5ce6b1d102837eee6
<ajs124>
I have some questions about #65602 and related spotify issues. Basically: If upstream doesn't seem to support alsa anymore, can we just drop it as well? If yes, do we document that somewhere? And if yes (again) how and where?
<tilpner>
The subset should probably be a variable used in both executions
<gchristensen>
seems like a fine idea
<gchristensen>
when I run this by hand I wished the printed text was nicer, like ncurses or something. but 99% of the times I run it, it is in buildkite where it just spews text logs anyway
<tilpner>
It's fine with RUST_LOG=r13y=debug
<gchristensen>
will my laptop finish checking bash (where it started with only 5 builds remaining) or will my larger system finish first, where it had to do all 94 from scratch. they both started at the same time.
<tilpner>
Huh, only 94?
<tilpner>
I started my Hetzner on hello, and it's at 15/172
<gchristensen>
bash is inside stdenv, so I think it has quite a lot fewer steps
<tilpner>
For 30€/mo you'd expect the thing to be able to build faster than a laptop
<tilpner>
Buuut, zfs and mirrored HDDs aren't that fast
<tilpner>
So it's often slower, and I just build there because it's quieter
<gchristensen>
aye
vcunat has joined #nixos-dev
<gchristensen>
4/5 vs. 92/94
<tilpner>
Huh, it just printed an "out" tuple with two hashes
<tilpner>
Where'd those come from?
<gchristensen>
ah! you found an unreproducible path
<tilpner>
yay ._.
<qyliss>
gchristensen: does r13y attempt to reproduce fixed-output derivations?
<gchristensen>
those hashes are narA and narB, where narA came from cache.nixos.org and narB came from you
<gchristensen>
yes
<qyliss>
nice
<gchristensen>
yes :)
drakonis_ has joined #nixos-dev
drakonis1 has quit [Ping timeout: 268 seconds]
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 264 seconds]
<gchristensen>
setting large_dir with tune2fs made it unbootable, working on alternative options
<vcunat>
hmm, that's weird
<gchristensen>
turns out it was just Grub unhappy with that option