<domenkozar>
genesis: sorry I was just playing around with it, am I the maintainer?
<domenkozar>
pff I am
<domenkozar>
niksnut++
<{^_^}>
niksnut's karma got increased to 7
<domenkozar>
for patchelf
<domenkozar>
niksnut++
<{^_^}>
niksnut's karma got increased to 8
<domenkozar>
for all the software
<edef>
what's the policy on backporting systemd patches?
<domenkozar>
depends on the patch
<edef>
i just backported some stuff that should make wireguard more pleasant to use with systemd-networkd, and i'm considering writing a patch that makes the wireguard module use that if systemd-networkd is available
<edef>
specifically it's a backport of [RoutingPolicyRule] InvertRule=
<emily>
oh no, don't rope me into trying to fix it
<gchristensen>
looks like a great way to get your feet went and get a quick win
* tilpner
mumbles about faketime hooks
<gchristensen>
aszlig: present?
jtojnar has joined #nixos-dev
<aminechikhaoui>
So I got this idea for our company private hydra which is basically running vulnix on a set of jobs derivations e.g (jobs matching *:*:*-prod-*) using the runcommand hydra plugin and dumping the reports in some S3 bucket.
<aminechikhaoui>
e.g not sure about the performance, afaik vulnix has some sort of cache so it should get faster after the initial runs. Also the notification sender is running in a separate thread so sending notifications shouldn't impact the build time only the notifications might be delayed for email/slack etc
<aminechikhaoui>
But a bit worried that it would be a bad idea :) any comments on it ?
<srhb>
aminechikhaoui: I would probably always batch up the notifications in a queue when using runcommand anyway.
<srhb>
aminechikhaoui: Get it out of Hydra land asap.
<aminechikhaoui>
hm how ? all plugins will run with every buildFinished/Failed etc.. event
jtojnar has quit [Quit: jtojnar]
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
orivej has joined #nixos-dev
<aszlig>
gchristensen: just woke up, but yeah
<gchristensen>
I sorted it, aszlig, let's see how you like it. I'm bisecting the kernel
<aszlig>
gchristensen: yeah, i usually do it in a similar way, although using fetchFromGitHub instead of cleanSource/filterSource
<gchristensen>
oh wow, so not literally using `git bisect`?
<aszlig>
gchristensen: i'm using git bisect, but update the git rev for every step
<gchristensen>
ahh ok
<gchristensen>
cool. thank you for checking it out
<aszlig>
gchristensen: the reason is that the nodes in my build farm are much faster churning through the kernel build
<gchristensen>
ah
<gchristensen>
do you test multiple at a time?
<aszlig>
and uploading the kernel source for every step is a bit meh, if you only have 40 mbit/s upload speed
<gchristensen>
I hear that. I'm doing this build direct on the machines.
<aszlig>
gchristensen: nah, that would be overhead
<gchristensen>
gotch
<gchristensen>
git bisect is incredible. Bisecting: 14457 revisions left to test after this (roughly 14 steps)
<aszlig>
if you have a decent workstation, then yeah, it's way simpler to do
<aszlig>
of course, you could also make sure you have a pretty minimal kernel, that'd bring down build times as well
<aszlig>
but my fastest build node builds a full generic kernel in less than 8 minutes, so that's way better than on my local machine (around 40 minutes) =)
<aszlig>
plus no load on the local machine :-)
<aszlig>
a better way would probably be to re-use the already built objects, but especially for projects like linux, it doesn't help that much, because bisecting large ranges of commits will pretty much always involve a full rebuild
<gchristensen>
yeah
<gchristensen>
simpler to do a full rebuild, and more likely to be honest
<gchristensen>
some overhead could cut the human time significantly
<aszlig>
well, ideally you'd fully automate this with git bisect run
<gchristensen>
yea
<aszlig>
hm... this reminds me...
<gchristensen>
a trouble with that is making sure moddirversion matches what the kernel expects
<aszlig>
something like nix-build --bisect would be good to have
<aszlig>
gchristensen: yeah
<aszlig>
that could be solved using IFD on the kernel's makefile
<gchristensen>
nix-build --bisect would be wildly cool
<aszlig>
like if you run with --bisect you'll get exit code 100 if it's a dependency failure or eval error
<aszlig>
err... 125, not 100
<gchristensen>
yea
<aszlig>
hm, or maybe it makes sense to exit with 125 on dep/eval error regardless of whether you give --bisect...
<aszlig>
niksnut: any opinion on that?
<aszlig>
because that might be generally useful not only for bisect
<clever>
gchristensen: i'm looking into building electron under nix-shell, for reasons (lol), and i'm noticing it doesnt use make, configure, or anything else standard, so where do i even start?
<gchristensen>
oh man
<gchristensen>
do they use ninja or something?
<clever>
something called gn is used to generate the ninja files, i think
<clever>
aszlig: bingo, now i rewind node until thats 60
<sphalerite>
samueldr: yes Ithink it makes sense to exclude zfs from latest-kernel installer images. More than the alternative we're currently on, which is not build the latest-kernel installer image whenever zfs is broken IIUC
<samueldr>
yeah, you seem to understand correctly
<niksnut>
aszlig: opinion on what exactly?
<aszlig>
niksnut: returning with exit status 125 if it's a dependency failure or eval error
drakonis has joined #nixos-dev
<niksnut>
yeah that's fine, we already return different exit codes for permanent vs transient failures
<aszlig>
niksnut: that would make it easier using git bisect run
<aszlig>
ah, okay, that's why i got confused with 100, i knew that it was used somewhere
<aszlig>
niksnut: okay, so maybe we then need an extra --bisect flag, because people out there might be relying on 100/101
drakonis_ has quit [Read error: Connection reset by peer]
<niksnut>
I think it was used by hydra but not anymore
<infinisil>
aszlig: I've been using a script for bisections that does `if nix-shell -A <attribute> --run true; then nix-build -A <attribute>; else exit 125; fi`
<aszlig>
infinisil: that would work as well, but isn't as nice as nix-build --bisect :-)
<samueldr>
from a week's worth of looking, it seems to be an at-least daily occurence now :/
<samueldr>
(plain simple OOMs not the heap thing, which might be gaining in frequency too)
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 258 seconds]
orivej has quit [Ping timeout: 245 seconds]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 250 seconds]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 258 seconds]
ckauhaus has quit [Quit: WeeChat 2.2]
<gchristensen>
it is becoming clear that nixpkgs is too big, and we need to find a solution -- probably for 19.09
<gchristensen>
I would like to see multiple proposals for how to do this, I know one (from profpatsch iirc) involves setting groups of supported packages. another I've heard is having <lang>Packages be their own thing.
<gchristensen>
this is probably a harder problem than other distros, who don't have the same atomic model
<gchristensen>
but we can't keep growing like this -- memory and cpus aren't keeping up :)
<nbp>
If we were to rewrite this one using Nix libraries to poke at the internal, then we could literally have a single evaluation for updating "nixos rebuild" command if the Nix binary is unchanged.
<Profpatsch>
gchristensen: This is just for a full evaluation, right?
<Profpatsch>
What happens with packages that depend on language packages?
<gchristensen>
right, I don't know
<gchristensen>
and the answer to that is too big for IRC
<Profpatsch>
I mean I’d say we need more lazyness.
<Profpatsch>
And better debugging tools, flamegraphs and the like.
<gchristensen>
these are all possible answers
<gchristensen>
but we have a serious problem brewing -- we have for a long time
<simpson>
We shouldn't expect a best-for-us answer to necessarily resemble existing solutions, since AFAICT we are in a different size category than everybody else.
<Profpatsch>
Splitting nixpkgs is gonna introduce a versioning nightmare, and with dubious results (because you’d have to evaluate the language layers anyway, packages depend on them).
<gchristensen>
Profpatsch: could be
<gchristensen>
part of why I think it is important to have a handful of proposals, from different approaches, to consider
<LnL>
splitting it into layers implies pulling apart certain package sets partially I think
<Profpatsch>
A cool thing would be: lazy language packages.
<Profpatsch>
Only packages needed for the resulting binaries are checked into nixpkgs.
<LnL>
if the stdenv depends on pythonPackages.foo that also belongs in the stdenv layer
drakonis_ has joined #nixos-dev
<gchristensen>
LnL: that is true
<Profpatsch>
And the rest is available as layers somewhere.
<LnL>
and if dependencies change, things might have to move a layer up or down
drakonis has quit [Ping timeout: 250 seconds]
init_6 has quit [Ping timeout: 245 seconds]
<gchristensen>
right, layers / "sets" would be difficult and complicated
<gchristensen>
debian etc. of course have the ultimate in laziness ;)
<nbp>
gchristensen: one option is to cache the evaluation at the Nixpkgs level and freeze the evaluation on files sha.
<gchristensen>
oh interesting
<Profpatsch>
Our biggest advantage rn is that everything is in a monorepo.
<Profpatsch>
If we split, we lose.
<nbp>
Thus having a Nix expression cache which is stored outside, and keyed to Nixpkgs files hashes.
MichaelRaskin has joined #nixos-dev
<gchristensen>
true
<gchristensen>
also perhaps there are wins which could be made in Nix itself
<nbp>
Which implies that we are able to track the sources or hash the expression inputs.
<gchristensen>
but something has to give
<Profpatsch>
gchristensen: Do you have a clear overview of which parts of nixpkgs take how much memory?
<gchristensen>
no way
<Profpatsch>
I think that has to be the first step.
<gchristensen>
+1
<gchristensen>
having tools to understand this would be huge
<Profpatsch>
I have no insight in the nix language, but it would be a project that should be paid by some of the companies using nix.
<nbp>
Profpatsch: The top-level attribute set takes ~30 MB, just to enumerate all the names.
<nbp>
and its values
<gchristensen>
Profpatsch: well now let's not commit to that, maybe someone is specifically interested in doing this as a hobby of some sort
<gchristensen>
:P
<Profpatsch>
nbp: You mean in pseudo-python: "".join([k for k in pkgs.keys()]) is a string of size 30MB?
<nbp>
Profpatsch: no, the actual bytes in the Nix program.
<Profpatsch>
The internal representation?
<nbp>
yes
<nbp>
Which is why I think we want a lazy update operator for the top-level of Nixpkgs (or just large attribute set in general.
<LnL>
one thing that should already be possible is graphing the drv size, but not sure how much insight that would give
<nbp>
As any overlay will add an extra level of 30 MB, especially if it is using rec.
<Profpatsch>
Making it lazy wouldn’t mean there will be more intermediate arrtsets in memory?
<samueldr>
intuition, with the hydra-eval-job parts mainly, is that somehow we might be losing through fragmentation (the increase heap size error)
<nbp>
Profpatsch: no, this would imply more sharing between all the level of overlay executions.
<samueldr>
though I do not feel qualified enough to know whether this makes sense or not within nix
<nbp>
Profpatsch: self: super: { super-chain = super; } <--- basically look at the memory consumption of Nix every time you clone this overlay under a new name.
<nbp>
Profpatsch: the fact that Nix is lazy implies that having a single reference to super would force Nix to keep it alive.
<nbp>
Profpatsch: such as: super.bar.override
<Profpatsch>
Maybe what we need is bang patterns for strict arguments.
<Profpatsch>
Like in Haskell
<nbp>
!important
<gchristensen>
hehe
<Profpatsch>
I refuse to understand this CSS joke
<simpson>
It's okay, it's !important.
<gchristensen>
lol.
* simpson
!helping
* nbp
is in favor of external maximal laziness.
* gchristensen
doesn't know what that means
<nbp>
You know maximal laziness?
<gchristensen>
yeh
<nbp>
external implies that it is stored else-where.
<gchristensen>
ah
<nbp>
and guarded by its inputs :)
<gchristensen>
the evaluation cache
<gchristensen>
.nixc
<nbp>
exactly
<gchristensen>
samueldr: some memory parameters on chef (hydra's machine) have been tweaked by niksnut, hopefully that helps
<aszlig>
gchristensen: well, i'm busy with other things, but still there =)
<gchristensen>
aszlig: I fear I've bisected to a large commit, and I don't know what to do from here :/
<gchristensen>
I'd love your advice
<aszlig>
which one?
<gchristensen>
0e9da3fbf7d81f0f913b491c8de1ba7883d4f217 (note I'm not 100% certain it is this one -- I still have one step to go -- but it looks relevant)
<gchristensen>
this is the list it could be: 0e9da3fbf7d81f0f913b491c8de1ba7883d4f217, 875736bb3f3ded168469f6a14df7a938416a99d5
<aszlig>
gchristensen: hm... i doubt this one could be related to zfs
<aszlig>
okay, then maybe wait until you're done bisecting before digging through the code :-D
<gchristensen>
okay :)
<aszlig>
because from a rough look at the zfs source, it at least doesn't use that symbol directly
<aszlig>
if btw. the former is the commit result of the bisect, i'd bisect again against 40e020c129cfc991e8ab4736d2665351ffd1468d..00203ba40d40d7f33857416adfb18adaf0e40123
<aszlig>
(don't forget to check the first and last commit first)
<aszlig>
but scrap my comment about "i doubt this one could be related to zfs"
<aszlig>
it's highly likely actually
drakonis has quit [Quit: WeeChat 2.3]
<gchristensen>
aszlig: 0e9da3fbf7d81f0f913b491c8de1ba7883d4f217 is the first bad commit indeed
<gchristensen>
aszlig: I'll bisect 40e020c129cfc991e8ab4736d2665351ffd1468d..00203ba40d40d7f33857416adfb18adaf0e40123 now
<matthewbauer>
How is the NixOS release going everyone? Are we on track for an official release soon?
<gchristensen>
( sphalerite, samueldr )
<infinisil>
gchristensen: (you still need to update the /topic with the new release managers)
<gchristensen>
too bad computers aren't like muscles -- getting better at something the more they do it
<gchristensen>
because if it was, this 19th kernel compile would be so fast
<samueldr>
shhh
<samueldr>
that's a monkey paw wish and you just wished machine learning up now
<gchristensen>
oh no
<infinisil>
Imagine using ML for kernel builds, you'll never be sure if it's actually a compiler error or just the ML messing up.
<aszlig>
infinisil: except if the compiler itself is the classifier :-D
<gchristensen>
you'll certainly never know if what you get out the other end is what you wanted
<infinisil>
probabilistic type checking
<samueldr>
as for the release, I've not been really paying as much attention as I'd like to the issues/PRs :( I've been opportunistically active on IRC since it's easy and low effort, otherwise watching hydra
<aszlig>
hm... i'm pretty sure somebody already has implemented such a compiler :-D
<samueldr>
and using 19.03 on machines here so at least for my subset of nixpkgs things seem smooth
<samueldr>
work and other misc. things have been taking up my time for the last two months :/ though everything's fine if there is worry :)