sphalerite changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | NixOS 19.03 released! https://discourse.nixos.org/t/nixos-19-03-release/2652 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html https://r13y.com | 19.03 RMs: samueldr,sphalerite | https://logs.nix.samueldr.com/nixos-dev
johnny101m has joined #nixos-dev
johnny101m2 has quit [Ping timeout: 250 seconds]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 245 seconds]
justan0theruser is now known as justanotheruser
orivej has quit [Ping timeout: 258 seconds]
FRidh has joined #nixos-dev
orivej has joined #nixos-dev
<FRidh> hydra when evaluating:
<FRidh> error: opening lock file '/nix/store/p647jqsd6d4vhabixfkd8rckgm3b292p-0aade9c0203f.drv.lock': No space left on device
<FRidh> (use '--show-trace' to show detailed location information)
justanotheruser has quit [Ping timeout: 258 seconds]
init_6 has joined #nixos-dev
justanotheruser has joined #nixos-dev
simpson has quit [Ping timeout: 252 seconds]
alunduil has quit [Ping timeout: 252 seconds]
alunduil_ has joined #nixos-dev
alunduil_ is now known as alunduil
zimbatm has quit [Ping timeout: 252 seconds]
zimbatm has joined #nixos-dev
_e has quit [Ping timeout: 252 seconds]
_e has joined #nixos-dev
drakonis_ has quit [Ping timeout: 246 seconds]
drakonis_ has joined #nixos-dev
simpson has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
vcunat has joined #nixos-dev
<vcunat> gchristensen: ^^ Hydra evaluator out of space for about 24h now; I don't expect it will get fixed by itself.
<niksnut> looks fine to me
<niksnut> /dev/disk/by-label/root 864G 453G 367G 56% /
<vcunat> > error: opening file '/nix/store/7pwvz1asmmr0wmk3pqm7lciapgw9mzsb-unit-systemd-binfmt.service.drv': No space left on device
<{^_^}> error: syntax error, unexpected $undefined, expecting ')', at (string):255:21
<vcunat> Weird... I assume /nix/store is on the same drive, right?
<vcunat> Actually, why does opening a file even need drive space?
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 244 seconds]
drakonis has quit [Ping timeout: 252 seconds]
drakonis has joined #nixos-dev
<averell> if you open a new file for writing? also, it might have run out of inodes even with that space left?
<vcunat> Right, of course... evaluation is supposed to create lots of new *.drv files.
drakonis has quit [Ping timeout: 264 seconds]
drakonis has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 250 seconds]
avn has quit [Remote host closed the connection]
orivej has joined #nixos-dev
drakonis has joined #nixos-dev
justanotheruser has quit [Ping timeout: 258 seconds]
orivej has quit [Ping timeout: 248 seconds]
justanotheruser has joined #nixos-dev
aanderse-znc has joined #nixos-dev
aanderse has quit [Ping timeout: 264 seconds]
<gchristensen> vcunat: is it resolved now?
<FRidh> no
<gchristensen> hrm
<gchristensen> plenty of free inodes and space
<gchristensen> Aug 03 16:09:27 ceres hydra-evaluator[14013]: evaluation of jobset ‘nixpkgs:cross-trunk’ succeeded
<gchristensen> I'm thinking the system is okay, but perhaps the failed evaluation needs to succeed?
<gchristensen> restart*
orivej has joined #nixos-dev
aanderse-znc is now known as aanderse
orivej has quit [Ping timeout: 245 seconds]
init_6 has quit []
<srhb> gchristensen: Just restarted trunk-combined, same thing: https://hydra.nixos.org/jobset/nixos/trunk-combined#tabs-errors
orivej has joined #nixos-dev
<gchristensen> love debugging this stuff
<gchristensen> oh crap I can't log in to hydra anymore
<gchristensen> srhb: can you trigger a fresh eval?
<vcunat> I just did.
<vcunat> You can't? :-)
<gchristensen> I stopped paying Google for my google apps account
<gchristensen> Aug 03 16:55:06 ceres hydra-evaluator[14013]: starting evaluation of jobset ‘nixos:trunk-combined’ (last checked 1649 s ago)
<gchristensen> Aug 03 16:55:12 ceres hydra-evaluator[14013]: jobset is unchanged, skipping
<vcunat> What does "unchanged" mean? I incidentally just pushed some commits to master.
<gchristensen> vcunat: trigger once more?
<gchristensen> I'll get myself a new account today, but not this morning
<aminechikhaoui> gchristensen you can trigger evals from the backend
* vcunat running
<gchristensen> I see the error
<gchristensen> oh I really see the error
<gchristensen> this is thrilling
<gchristensen> [1378804.091126] EXT4-fs warning (device md126): ext4_dx_add_entry:2190: Directory (ino: 4202498) index full, reach max htree level :2
<gchristensen> [1378804.091127] EXT4-fs warning (device md126): ext4_dx_add_entry:2194: Large directory feature is not enabled on this filesystem
<vcunat> Ah, so too many files in /nix/store
justanotheruser has quit [Quit: WeeChat 2.4]
justanotheruser has joined #nixos-dev
<aminechikhaoui> it's weird, why would it happen just now, GC should be running in a timer
<vcunat> In any case you probably want to set the option.
<vcunat> tune2fs -O large_dir
<vcunat> (I think, but on unmounted FS)
<vcunat> I suspect dir_index might also be good for such large dirs, if there isn't already.
<gchristensen> gotcha, I don't think I can do that type of maintenance
<gchristensen> I've raised an issue with tier 2 support ;)
<gchristensen> so I think this is newly happening because: the old hydra server (chef)'s disk was 440G and the new hydra server (ceres)'s disk is 860G. the GC would then be leaving many more files around than before.
<gchristensen> I'm going to run a more aggressive GC than normal to get things moving until we can coordinate a the tuning parameter change with niksnut
<averell> is there already an issue about moving to split-tiered store paths ala xy/z..., like git, or is this just too rare for real users?
<gchristensen> I doubt we'll be able to move to that format, and may not be necessary if one of these fancy pants filesystems finally takes hold over ext4
<simpson> This is usually a one-time event though.
<simpson> And now you know. (And hopefully it'll get written down somewhere.)
<gchristensen> having a "nixos in production" guide with a subsection of "so you wanna run a build server"
<niksnut> oh yeah, we used to have to set some ext4 options in the past
<niksnut> I thought those were the default now
<gchristensen> niksnut: running GC now (with the appropriate $((...)) incantation to make 400G free.
<vcunat> right, more aggressive GC should probably work around it, too
<ivan> ryantm: release notes say 2.2
<tilpner> ryantm: AFAICT Dec. 13 with 6024dc1d97212130c19d3ff5ce6b1d102837eee6
<ajs124> I have some questions about #65602 and related spotify issues. Basically: If upstream doesn't seem to support alsa anymore, can we just drop it as well? If yes, do we document that somewhere? And if yes (again) how and where?
<{^_^}> https://github.com/NixOS/nixpkgs/pull/65602 (by angristan, 4 days ago, merged): spotify: ->
<vcunat> Well, if you find noone to support alsa (for that package) it won't work whether we admit it or not.
<vcunat> Release notes might be a good place, if it's considered important enough: https://nixos.org/nixos/manual/release-notes.html#sec-release-19.03-incompatibilities
drakonis has quit [Quit: WeeChat 2.4]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 250 seconds]
<ajs124> Hm, ok. I guess that's how it'll be then. I still kind of feel bad for introducing a regression.
<vcunat> We got some more big evals, including trunk-combined. Thanks, Graham.
<gchristensen> yay :)
drakonis_ has joined #nixos-dev
vcunat has quit [Ping timeout: 252 seconds]
drakonis has quit [Ping timeout: 276 seconds]
<gchristensen> tilpner!
drakonis has joined #nixos-dev
<pie_> Is there an overview somewhere of how the networking configuration stuff is implemented?
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 264 seconds]
<tilpner> gchristensen!
<gchristensen> your PR is great!
<tilpner> :)
<gchristensen> mind if I push a patch or two to your branch?
<tilpner> Probably not, about what? (enabled maintainer edit)
<gchristensen> fixing an unused import in src/report/mod.rs and updating check.sh to use the new CLI interface
<tilpner> Sure, though I'm not sure the CLI is final yet
<tilpner> I was also wondering whether I should extract eval into its own command
<tilpner> That would create a file with a build target per line
<tilpner> Which would be passed to check and report
FRidh has quit [Quit: Konversation terminated!]
<tilpner> This would allow for easy control over what (and how many) things are built
<gchristensen> that could be interesting
<gchristensen> oh cool
<tilpner> Might be helpful for debugging
<gchristensen> I'd be inclined to merge this PR? it is already a huge improvement, and the CLI can change again :)
<tilpner> No, not yet. It's WIP for a reason
<tilpner> Which is that currently eval doesn't have a way to push old results into "results"
<tilpner> / results.push(elem)
<gchristensen> ahh okay :)
<gchristensen> I'm still running the first ./check.sh on it, so hadn't gotten that far in my test
<pie_> (pr?)
<{^_^}> grahamc/r13y.com#9 (by tilpner, 2 hours ago, open): Refactor, add CLI
<pie_> ah, yay
<gchristensen> every time I test r13y, glibc and perl are always the last two to run.
<tilpner> gchristensen: Marked ready, but let's run a full check before merging
<gchristensen> okay doing a test on bash
<gchristensen> specifically: RUST_LOG=r13y=trace cargo run -- --max-cores 8 --max-cores-per-job 2 --subset nixpkgs:bash --rev 70503758fb4b37107953dfb03ad7c0cf36ad0435 --sha256 15g8xckhzpp84p6gv526hb6c1r286qvn8i14w8msw6172jy3kj3c check
<tilpner> The subset should probably be a variable used in both executions
<gchristensen> seems like a fine idea
<gchristensen> when I run this by hand I wished the printed text was nicer, like ncurses or something. but 99% of the times I run it, it is in buildkite where it just spews text logs anyway
<tilpner> It's fine with RUST_LOG=r13y=debug
<gchristensen> will my laptop finish checking bash (where it started with only 5 builds remaining) or will my larger system finish first, where it had to do all 94 from scratch. they both started at the same time.
<tilpner> Huh, only 94?
<tilpner> I started my Hetzner on hello, and it's at 15/172
<gchristensen> bash is inside stdenv, so I think it has quite a lot fewer steps
<tilpner> For 30€/mo you'd expect the thing to be able to build faster than a laptop
<tilpner> Buuut, zfs and mirrored HDDs aren't that fast
<tilpner> So it's often slower, and I just build there because it's quieter
<gchristensen> aye
vcunat has joined #nixos-dev
<gchristensen> 4/5 vs. 92/94
<tilpner> Huh, it just printed an "out" tuple with two hashes
<tilpner> Where'd those come from?
<gchristensen> ah! you found an unreproducible path
<tilpner> yay ._.
<qyliss> gchristensen: does r13y attempt to reproduce fixed-output derivations?
<gchristensen> those hashes are narA and narB, where narA came from cache.nixos.org and narB came from you
<gchristensen> yes
<qyliss> nice
<gchristensen> yes :)
drakonis_ has joined #nixos-dev
drakonis1 has quit [Ping timeout: 268 seconds]
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 264 seconds]
<gchristensen> setting large_dir with tune2fs made it unbootable, working on alternative options
<vcunat> hmm, that's weird
<gchristensen> turns out it was just Grub unhappy with that option
<niksnut> this issue seems documented on only 1 place on the internet (with no solution): https://wiki.archlinux.org/index.php/GRUB#error:_unknown_filesystem
<vcunat> eww, so a separate boot partition?
johnny101m has quit [Read error: Connection reset by peer]
johnny101m2 has joined #nixos-dev
<gchristensen> yeah
<gchristensen> ah tilpner the report doesn't work
drakonis_ has joined #nixos-dev
drakonis1 has quit [Ping timeout: 276 seconds]
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 250 seconds]
vcunat has quit [Ping timeout: 252 seconds]
drakonis1 has quit [Ping timeout: 244 seconds]
drakonis1 has joined #nixos-dev