<infinisil>
Idea: Once mkDerivation gets redesigned with a development mode making every phase its own derivation, let's implement that by allowing arbitrary "checkpoints" throughout the build script
<infinisil>
This means you could e.g. do "some-command; checkpoint; some-other-command", and this would create one derivation for before the checkpoint, and one after the checkpoint
<infinisil>
And the result would be that as long as you don't change any command before the checkpoint, it can reuse the results of the build up until then
<marek>
ok, about my previous error, looks like when I use fido2 as a buildInput it fails, I suspect because fido2 requires openssh back and create an infinite recursion
q3k has quit [Remote host closed the connection]
q3k has joined #nixos-dev
<Mic92>
marek: do you mean libfido2?
<marek>
Mic92: yes, indeed
<marek>
but to be honest, I'm not sure where the recrsion is, I thought fetchurl, but it probably isn't the case
<Mic92>
marek: I am not sure where this error is comming from I cannot reproduce this, when evaluating locally
<marek>
I get: error: infinite recursion encountered, at undefined position
<marek>
I figure out it's because of libfido2 in buildInputs of openssh
<marek>
I suspect somethig libfido2 depednds on depends on openssh, but I'm missing something
<marek>
I did try to override libfido2 for fetchurl, but that did not work
<Mic92>
marek: that made it fail for me
<marek>
but I can build openssh with that change fine and it evaluates for me
<marek>
gchristensen: could this be related to just how ofborg does the eval? ^
<Mic92>
marek: it is udev (systemd)
__monty__ has joined #nixos-dev
<tilpner>
Hey Mic92, you attended the last RFC meeting. Can I ask you about the status of RFC 55 (retire-inactive)? I didn't expect there to be cause for months of discussion, so I'm wondering if there's anything I can help with, or otherwise clarify :)
<tilpner>
(Alternatively, if there are meeting notes beyond "in discussion" and you could point me at them, that'd be great too!)
<marek>
Mic92: how did you find out?
<Mic92>
marek: I just removed udev from buildInputs and re-evaluated it.
<marek>
ha :) ok
<marek>
from libfido2 right?
<Mic92>
marek: systemd depends on gnupg and some other crypto stuff.
<Mic92>
marek: yes
<marek>
what would be the correct way to fix it? override in libfido2?
<Mic92>
marek: I have not find out where the openssh dependency in systemd comes from. Libfido itself depends on libudev
<Mic92>
marek: So I would suggest to remove stuff from systemd until you find the offender
<Mic92>
tilpner: That is fair. I think we should organize a meeting. We have not even set a shepherd leader for this one.
__red__ has quit [Quit: WeeChat 2.6]
bridge[evilred] has quit [Remote host closed the connection]
psyanticy has joined #nixos-dev
<andi->
gchristensen: not sure, it could just be that hydra has different scheduling priorities. I went to bed, slept 8h and it finished <100 jobs on 19.09.. :/ I can't see where it might be stuck.
<marek>
dtz: maybe you have an idea? regarding libfido2, please? ^
<marek>
I wish there was a way to trace the depdendency tree during an eval
<Mic92>
marek: use --show-trace
<Mic92>
it is getent and utillinux and kmod in systemd
<Mic92>
marek: I also think this only fails to evaluate on macOS actually.
<marek>
I'm running linux and it fails too
<marek>
or you mean that it fails on some darwin part?
<Mic92>
marek: No. I mean if you use the evaluation nix script and remove the darwin platform from it, it seem to work.
<marek>
Mic92: ooooh, that's true!
<marek>
hm, but why, it doesn't make much sense
<Mic92>
marek: I think libfidor2 should be built without udev on macOS, it does not make any sense to have it their.
<Mic92>
that might fix evaluation
<marek>
Mic92: ! that might be it! :)
<tilpner>
Mic92: Who would be involved in that meeting, and would it occur before or after the selection of a shepherd leader?
<Mic92>
tilpner: Yes, you would be. I hope we can decide on a shepherd leader before the meeting.
<tilpner>
Mic92: So I'll wait for the next meeting, hope that you decide on a leader, and then bring it up again?
<Mic92>
tilpner: I already sent out an email to set up meeting where we would discuss the RFC specifically. Did you got it?
<tilpner>
(Oh, saw your comment about the email just now)
<tilpner>
Ahh, it was marked as spam
<tilpner>
The wonders of hosting your own email server?
<Mic92>
tilpner: Yes. It should have good reputation actually. I do dmarc, spf, ARC and all kinds of stuff.
<tilpner>
I replied, hoping that my mail provider will not get caught in their spam filter and alert them to the conversation even if your mail was marked as spam for them too
<jtojnar>
hmm, we have too many nix commands without `--option experimental-features nix-command` in the tree
<gchristensen>
can we move them back to nix 1.x-style commands?
<jtojnar>
maybe but we probably need to sprint it
<jtojnar>
or maybe we want to dogfood the command
<jtojnar>
and should enable the experiment at the site of use
<gchristensen>
yeah
claudiii has joined #nixos-dev
<jtojnar>
Hmm, even nixos-rebuild switch fails with "Activation script snippet 'nix' failed (1)" when using nix = nixUnstable overlay
<gchristensen>
yes that is not good :/
<jtojnar>
well people should use nix.package option instead
<gchristensen>
more that I mean the activation script shouldn't ever fail
<jtojnar>
but I think the overlay bricked my nixos-rebuild 😀️
<gchristensen>
o
<niksnut>
using a nix overlay rather than nix.package definitely shouldn't break anything, in fact the nixos.org machines all use that
<jtojnar>
nixos-install will fail even with nix.package option though
<gchristensen>
andi-: it looks like only a few hundred were Runnable until about 5 minutes ago when 1500 became Runnable
<andi->
mhm, but those are all aarch64 while the x86_64-linux jos have been queued for >12h and still no progress. Not sure what they are if not runnable
<gchristensen>
andi-: yeah, so I did a bit of an experiment where I made a lot of the Packet machines not having the feature for big-parallel jobs, and instead brought up a separate pool of machines which would only do 1 big-parallel job at a time
<gchristensen>
so maybe I will undo that :)
<andi->
we do not have that many big-parallel jobs, right? I think that is rather wasteful haveing them do only one job for a few jobs that exist but you probably had a good reason.
<gchristensen>
24 hour long chromium builds
<gchristensen>
they would do other jobs, too, of course, not just big-parallel
<andi->
Oh yeah the chromium problem.. I wonder if we could just use more system libs for them an cut the compilation time down.. But I am not interested in looking into another browser.
<gchristensen>
same
<andi->
Also they seem to be mostly painless in maintaining just painful in getting through hydra.
<gchristensen>
looking at the graph of runnable jobs, I see nixos-test,kvm jobs are backlogged, so I just gave all the packet machines the kvm and nixos-test features
<gchristensen>
but I left the experiment of the big machines which will do 1 big-parallel job at a time
<andi->
ok
<gchristensen>
aaaand looking at the graph you can see hydra immediately started like 300 kvm,nixos-test jobs
<gchristensen>
andi-: my thinking / hope was that a super unloaded machine would be able to blow through the tests, too. anyway, that didn't pan out.
<andi->
I could magine that tests aren't really bottle necked at CPU but the inefficient 9pfs
<gchristensen>
yeah
<andi->
which reminds me to look at virtiofs
<gchristensen>
andi-: I think it would be interesting to mark the early bootstrapping phase of nixpkgs, where almost all the jobs are run sequentially due to the natural fanning out of the graph
<gchristensen>
and mark those as big-parallel
<gchristensen>
or fancier, make the hydra evaluator super smart about finding those choke points on the graph and a way to mark machines as good targets for those builds
<andi->
that is interesting.. maybe we want more build tags. E.g. "bootstrap"
<gchristensen>
yeah
<andi->
But not as a mandatory feature but as preference
<gchristensen>
yeah
<gchristensen>
on a big machine like this we can build stdenv in 20min if it gets --jobs 1 --cores 28
<gchristensen>
it is unfortunately hard for us to add more tags, because other hydras won't be able to build them anymore without people finding out the hard way
<andi->
One thing I noticed is that my 3rd gen Ryzen 12 Core desktop is faster then some 24 core epyc when building Firefox..
<gchristensen>
nice :D
<gchristensen>
Nix has a much better error message now w.r.t. missing features, so that is fine
<andi->
I am not sure if that is all the evolution of the CPU or just that fact that the RAM is running 800MHz faster and it has more cache?
<gchristensen>
hard to know
<gchristensen>
it would be good for hydra to have https://hydra.nixos.org/queue-summary show a queue break-down (and information on available machines) for features too
<gchristensen>
looks like today will be a SQL, Perl, Python2, Python3, C++, and Rust kind of day.
<gchristensen>
disasm: can I impose upon your team for a bit? :)
<gchristensen>
I'd like help testing something in ways maybe I'm not
<gchristensen>
andi-: my experiment did actual damage because I incorrectly used the mandatory features field, basically guaranteeing no work could be done by these specially allocated machines.
<gchristensen>
realizing that now and fixing
<gchristensen>
sorry :(
<gchristensen>
roads, good intentions, ...
<andi->
gchristensen: Yeah, no worries. I have the feeling declarative config for that stuff would go a long way and enable use to (re)view things if we think stuff is off. Sadly that puts more work on your shoulders(?) :/
<gchristensen>
aminechikhaoui: let's chat here in case other people have ideas too
<gchristensen>
aminechikhaoui: running `./dev-shell --arg p '(p: [ (p.callPackage ../../input-output-hk/nixops-packet/release.nix {}) ] )'` doesn't work, because the build for the plugin isn't able to import nixops: ModuleNotFoundError: No module named 'nixops'
<gchristensen>
aminechikhaoui: so then, callPackage is, I think, passing in the nixops from nixpkgs -- which is not the version I want -- causing some further problems
<gchristensen>
File "nix_run_setup", line 3, in <module>
<gchristensen>
import setuptools
<gchristensen>
ModuleNotFoundError: No module named 'setuptools'
<gchristensen>
so I think we need like a "bootstrapping" nixops and then one mushed together with the plugins -- the bootstrapping one passed in to the plugins
thefloweringash has quit [Ping timeout: 260 seconds]
arcnmx has quit [Ping timeout: 252 seconds]
colemickens has quit [Ping timeout: 252 seconds]
layus[m] has quit [Ping timeout: 246 seconds]
mkg20001 has quit [Ping timeout: 246 seconds]
tokudan[m] has quit [Ping timeout: 240 seconds]
Nyanloutre[m] has quit [Ping timeout: 248 seconds]
Ox4A6F has quit [Ping timeout: 248 seconds]
jonge[m] has quit [Ping timeout: 248 seconds]
domenkozar[m] has quit [Ping timeout: 248 seconds]
aditsachde has quit [Ping timeout: 245 seconds]
jtojnar has quit [Ping timeout: 246 seconds]
ma27[m] has quit [Ping timeout: 256 seconds]
alienpirate5 has quit [Ping timeout: 256 seconds]
dtz has quit [Ping timeout: 256 seconds]
vaibhavsagar has quit [Ping timeout: 256 seconds]
zimbatm[m] has quit [Ping timeout: 240 seconds]
worldofpeace has quit [Ping timeout: 272 seconds]
abbradar[m] has quit [Ping timeout: 260 seconds]
roberth has quit [Ping timeout: 260 seconds]
thefloweringash has joined #nixos-dev
timokau[m] has joined #nixos-dev
abbradar[m] has joined #nixos-dev
roberth has joined #nixos-dev
aanderse has joined #nixos-dev
<gchristensen>
aminechikhaoui: good grief you approved it lol
<gchristensen>
sounds good
rnhmjoj has joined #nixos-dev
thefloweringash has quit [Quit: killed]
abbradar[m] has quit [Quit: killed]
roberth has quit [Quit: killed]
aanderse has quit [Quit: killed]
timokau[m] has quit [Quit: killed]
rnhmjoj has quit [Client Quit]
emily has joined #nixos-dev
<emily>
is there any workaround for fish users?
<emily>
hm, is /nix/store/y54wzrnzassarw2p3jv361nqpf1hz5x1-fish_patched-completion-generator.drv failing on nixos-unstable known?
<emily>
ah, I see cole-h has a pr :)
ris has joined #nixos-dev
Ox4A6F has joined #nixos-dev
<cole-h>
^^; That was my fault
<gchristensen>
/!\ Today's office hours is a different format /!\ watch live: https://youtu.be/FKpeI8U8-AE discuss in #nixos-officehours, no group Zoom call today. We're discussing the last few weeks: Hydra's memory problems, the flakes.nix merge to nixpkgs, RFC process, etc. We start in 38 minutes
averell has quit [Quit: .]
averell has joined #nixos-dev
drakonis has joined #nixos-dev
abbradar[m] has joined #nixos-dev
Ericson2314 has joined #nixos-dev
colemickens has joined #nixos-dev
jtojnar has joined #nixos-dev
ma27[m] has joined #nixos-dev
alienpirate5 has joined #nixos-dev
rycee has joined #nixos-dev
arcnmx has joined #nixos-dev
layus[m] has joined #nixos-dev
dtz has joined #nixos-dev
thefloweringash has joined #nixos-dev
aditsachde has joined #nixos-dev
jonge[m] has joined #nixos-dev
Nyanloutre[m] has joined #nixos-dev
bennofs[m] has joined #nixos-dev
rnhmjoj has joined #nixos-dev
worldofpeace has joined #nixos-dev
roberth has joined #nixos-dev
timokau[m] has joined #nixos-dev
zimbatm[m] has joined #nixos-dev
pkolloch[m] has joined #nixos-dev
domenkozar[m] has joined #nixos-dev
vaibhavsagar has joined #nixos-dev
aanderse has joined #nixos-dev
tokudan[m] has joined #nixos-dev
mkg20001 has joined #nixos-dev
drakonis has quit [Ping timeout: 245 seconds]
psyanticy has quit [Quit: Connection closed for inactivity]
drakonis has joined #nixos-dev
bhipple has joined #nixos-dev
bhipple has quit [Ping timeout: 255 seconds]
bhipple has joined #nixos-dev
bhipple has quit [Ping timeout: 260 seconds]
justanotheruser has quit [Ping timeout: 245 seconds]
<gchristensen>
it'dbe nice to get more testing on it, but it is hard :P
<gchristensen>
I do think after it merges there will be incentive and motivation to get the other providers on to py3 (see https://github.com/NixOS/nixops-aws/pull/23, but I can't test that because I have no AWS infra)
<aminechikhaoui>
the plugin structure isn't released yet so maybe it's fine
<aminechikhaoui>
gchristensen yeah I think it should be ready, the only problem I see is we would break 2 "officially" maintained plugins (hetzner/aws) so ideally that wouldn't stay for too long
<aminechikhaoui>
niksnut any thoughts on that in general ?
<samueldr>
speaking of nixops and breakage, #80680 fixes nixos for the libvirtd plugin, that is, before the specific fix which isn't ported to the nixos 19.09 release branch
<samueldr>
I'm not using nixops, and I'm not 100% positive if nixos master has a nixops with the libvirtd plugin fix included
<gchristensen>
nixops master doesn't have any libvirtd code anyway
<samueldr>
anyway, the issue is not specific to nixops
<samueldr>
which is why I said "nixos master" :)
<gchristensen>
aminechikhaoui: I think (a) updating nixops to py3 (b) `black`ening the repo ... would go a ways towards making it a bit easier to contribute to
<gchristensen>
aminechikhaoui: I wish there was a way to thoroughly test PRs, I think this is going to haunt us essentially forever. like https://github.com/NixOS/nixops-aws/pull/1 which is a great PR and I basically depend on it for the py3 PR I submitted
<{^_^}>
nixops-aws#1 (by takeda, 22 weeks ago, open): Rewriting to use boto3
<aminechikhaoui>
yeah it's a bit of a manual process and even the functional tests don't cover everything
<gchristensen>
it would be cool to have a working test system for it
<gchristensen>
oh man one thing I didn't test with my nixops PR is creating a new network :P
<aminechikhaoui>
oh lol me too
<gchristensen>
easy enough
<multun>
what's maintaining nixops btw?
<gchristensen>
I'm not sure I can parse that sentence
<multun>
wow
<multun>
who
<multun>
sorry
<gchristensen>
aminechikhaoui: making a new network wokred :P
<multun>
\o/
<gchristensen>
multun: amine here does a lot of it, but maintainership is a bit low
<multun>
the formatting is pretty insanely inconsistent
<gchristensen>
the general concern is with breaking the `git blame`, but also generally +1 on doing it
<gchristensen>
multun: if you sent a PR based on my python2->3 PR (send it to nixops directly) which also checked in travis, it has a good chance of merging
<aminechikhaoui>
yeah I use git blame a lot and that would make it a bit harder but hopefully it is worth it
<samueldr>
rip the black band-aid with the 2to3 conversion
<gchristensen>
samueldr: hm?
<samueldr>
black is code formatting, right?
<gchristensen>
aminechikhaoui: you've seen github's tool to "look behind" a commit?
<gchristensen>
samueldr: yeah
<samueldr>
that makes it the best moment to reformat all the code, as it's already been mechanically recombobuated by 2to3
<aminechikhaoui>
ha yeah true
<samueldr>
then, as long as each contribution is black'd properly, no future git blame issues
<aminechikhaoui>
I haven't seen the feature, I'll look it up
<gchristensen>
true
<samueldr>
though, hindsight is 20/20, it probably should have been blacked once before 2to3
<samueldr>
that way you can git blame to the python 2 equivalent
<gchristensen>
samueldr: still could
<multun>
it's not like it gets completly impossible to blame
<multun>
and nixops needs a good formatter so hard blames would probably be less frequent after reformatting
<gchristensen>
multun: are you going to send that PR? if yes, and if soon, I have alterantive dire8ctions & I'll wait for you :P
<multun>
what do you mean x3
v0|d has joined #nixos-dev
<gchristensen>
I'm wondering if you're going to PR adding `black`
<multun>
oh yeah
<gchristensen>
like if you've already started working on it
<multun>
I didn't start
<gchristensen>
were you going to do that within the next few minutes, or more like tomorrow/etc?
<multun>
I would rather do the coverage CI first
<gchristensen>
sounds hard :x
<multun>
we'll only know once it's done (:
<gchristensen>
okay, want to hack on that, then? I can get a PR in for `black`ing it
<multun>
let's go !
<gchristensen>
yay!
<multun>
:D
<gchristensen>
aminechikhaoui: does this diff look good to you? (caution: very very very short.)
<gchristensen>
okay I'm making thaht diff slightly longer now
justanotheruser has joined #nixos-dev
ixxie has quit [Ping timeout: 260 seconds]
<gchristensen>
multun: I'm pretty sure I got a +1 to merge https://github.com/NixOS/nixops/pull/1228 from aminechikhaoui in a DM, but waiting a bit to find out. I feel very comfortable merging this without the coverage, since black validates the AST.
<{^_^}>
nixops#1228 (by grahamc, 23 minutes ago, open): travis: check files are formatted correctly with `black`
<multun>
gchristensen: oh yeah sure go on
<gchristensen>
and as noted, I'm ready to update the 2to3 pr, too. I am excited to see where you get with your testing work
<multun>
probably not that far
<multun>
I'm just trying to integrate the coverage to the pull request build steps
<gchristensen>
nice
<gchristensen>
it might not actually work, it might need to be able to deploy to real things
<multun>
probably yeah
<multun>
I have no clue
<gchristensen>
let's find out
<gchristensen>
ooops deleted a real nixops network
<multun>
at least the delete feature works :/
<gchristensen>
:D it wasn't an important network anyway