ekleog changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html https://r13y.com | 18.09 release managers: vcunat and samueldr | https://logs.nix.samueldr.com/nixos-dev
orivej has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
<domenkozar> genesis: sorry I was just playing around with it, am I the maintainer?
<domenkozar> pff I am
<domenkozar> niksnut++
<{^_^}> niksnut's karma got increased to 7
<domenkozar> for patchelf
<domenkozar> niksnut++
<{^_^}> niksnut's karma got increased to 8
<domenkozar> for all the software
<edef> what's the policy on backporting systemd patches?
<domenkozar> depends on the patch
<edef> i just backported some stuff that should make wireguard more pleasant to use with systemd-networkd, and i'm considering writing a patch that makes the wireguard module use that if systemd-networkd is available
<edef> specifically it's a backport of [RoutingPolicyRule] InvertRule=
<edef> (of systemd.network)
ajs124 has left #nixos-dev [#nixos-dev]
pie___ has joined #nixos-dev
pie__ has quit [Ping timeout: 246 seconds]
<ivan> does anyone want to merge https://github.com/NixOS/nixpkgs/pull/54392 it should be good
<{^_^}> #54392 (by ivan, 9 weeks ago, open): ffmpeg, mpv: enable hardware-accelerated decoding with CUDA
drakonis has quit [Quit: WeeChat 2.3]
harrow has quit [Quit: Leaving]
ivan has quit [Quit: lp0 on fire]
drakonis has joined #nixos-dev
harrow has joined #nixos-dev
ivan has joined #nixos-dev
<genesis> domenkozar : feel free to remove you frim maintainer, at least plz review it :)
harrow has quit [Quit: Leaving]
ivan has quit [Quit: lp0 on fire]
ivan has joined #nixos-dev
harrow has joined #nixos-dev
init_6 has joined #nixos-dev
Zer000 has quit [Ping timeout: 255 seconds]
jtojnar has quit [Remote host closed the connection]
jtojnar has joined #nixos-dev
jtojnar_ has joined #nixos-dev
jtojnar has quit [Read error: Connection reset by peer]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
jtojnar_ is now known as jtojnar
ajs124 has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
phreedom has joined #nixos-dev
phreedom_ has quit [Ping timeout: 256 seconds]
{^_^} has quit [Remote host closed the connection]
{^_^} has joined #nixos-dev
johanot has joined #nixos-dev
<gchristensen> https://r13y.com/ is stable again
<Taneb> I guess the low-hanging reproducibility fruit has been eaten
<ekleog> can I poke someone for review of https://github.com/NixOS/nixpkgs/pull/57559 ? (and bonus if someone knows of a nixos test that actually checks the ISO image)
<{^_^}> #57559 (by Ekleog, 2 weeks ago, open): iso-image: make reproducible by not relying on mcopy's readdir
<gchristensen> whoa
<gchristensen> the installer test passed, so I think it is good-to-go
<gchristensen> let's find out :)
<ekleog> :D
jtojnar has quit [Remote host closed the connection]
orivej has quit [Ping timeout: 246 seconds]
johanot has quit [Quit: WeeChat 2.4]
<gchristensen> it sure does!
<emily> oh no, don't rope me into trying to fix it
<gchristensen> looks like a great way to get your feet went and get a quick win
* tilpner mumbles about faketime hooks
<gchristensen> aszlig: present?
jtojnar has joined #nixos-dev
<aminechikhaoui> So I got this idea for our company private hydra which is basically running vulnix on a set of jobs derivations e.g (jobs matching *:*:*-prod-*) using the runcommand hydra plugin and dumping the reports in some S3 bucket.
<aminechikhaoui> e.g not sure about the performance, afaik vulnix has some sort of cache so it should get faster after the initial runs. Also the notification sender is running in a separate thread so sending notifications shouldn't impact the build time only the notifications might be delayed for email/slack etc
<aminechikhaoui> But a bit worried that it would be a bad idea :) any comments on it ?
<srhb> aminechikhaoui: I would probably always batch up the notifications in a queue when using runcommand anyway.
<srhb> aminechikhaoui: Get it out of Hydra land asap.
<aminechikhaoui> hm how ? all plugins will run with every buildFinished/Failed etc.. event
jtojnar has quit [Quit: jtojnar]
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
orivej has joined #nixos-dev
<aszlig> gchristensen: just woke up, but yeah
<gchristensen> I sorted it, aszlig, let's see how you like it. I'm bisecting the kernel
<gchristensen> aszlig: https://gist.github.com/grahamc/bd2d437a900be5b9923b243148439251 here is my .nix, I have a checkout at ./linux which I checked out and started bisecting on
<aszlig> gchristensen: yeah, i usually do it in a similar way, although using fetchFromGitHub instead of cleanSource/filterSource
<gchristensen> oh wow, so not literally using `git bisect`?
<aszlig> gchristensen: i'm using git bisect, but update the git rev for every step
<gchristensen> ahh ok
<gchristensen> cool. thank you for checking it out
<aszlig> gchristensen: the reason is that the nodes in my build farm are much faster churning through the kernel build
<gchristensen> ah
<gchristensen> do you test multiple at a time?
<aszlig> and uploading the kernel source for every step is a bit meh, if you only have 40 mbit/s upload speed
<gchristensen> I hear that. I'm doing this build direct on the machines.
<aszlig> gchristensen: nah, that would be overhead
<gchristensen> gotch
<gchristensen> git bisect is incredible. Bisecting: 14457 revisions left to test after this (roughly 14 steps)
<aszlig> if you have a decent workstation, then yeah, it's way simpler to do
<aszlig> of course, you could also make sure you have a pretty minimal kernel, that'd bring down build times as well
<aszlig> but my fastest build node builds a full generic kernel in less than 8 minutes, so that's way better than on my local machine (around 40 minutes) =)
<aszlig> plus no load on the local machine :-)
<aszlig> a better way would probably be to re-use the already built objects, but especially for projects like linux, it doesn't help that much, because bisecting large ranges of commits will pretty much always involve a full rebuild
<gchristensen> yeah
<gchristensen> simpler to do a full rebuild, and more likely to be honest
<gchristensen> some overhead could cut the human time significantly
<aszlig> well, ideally you'd fully automate this with git bisect run
<gchristensen> yea
<aszlig> hm... this reminds me...
<gchristensen> a trouble with that is making sure moddirversion matches what the kernel expects
<aszlig> something like nix-build --bisect would be good to have
<aszlig> gchristensen: yeah
<aszlig> that could be solved using IFD on the kernel's makefile
<gchristensen> nix-build --bisect would be wildly cool
<aszlig> like if you run with --bisect you'll get exit code 100 if it's a dependency failure or eval error
<aszlig> err... 125, not 100
<gchristensen> yea
<aszlig> hm, or maybe it makes sense to exit with 125 on dep/eval error regardless of whether you give --bisect...
<aszlig> niksnut: any opinion on that?
<aszlig> because that might be generally useful not only for bisect
<clever> gchristensen: i'm looking into building electron under nix-shell, for reasons (lol), and i'm noticing it doesnt use make, configure, or anything else standard, so where do i even start?
<gchristensen> oh man
<gchristensen> do they use ninja or something?
<clever> something called gn is used to generate the ninja files, i think
<aszlig> clever: GN... yeah :-D
<aszlig> chromium :-D
<clever> is gn packaged yet? lol
* aszlig hates it with a passion...
<aszlig> clever: IIRC i did package it back then for chromium but it was removed again
<aszlig> lemme check...
<clever> aszlig: the root problem i have, is that i need to debug something going wrong in electron
<clever> the problem is in the nodejs side, and i can just replace libnode.so if possible
<aszlig> clever: ah, it's still there
<clever> /nix/store/9qkdgz264yh029ddajfij2s4p0shsrn5-electron-3.0.14/bin/electron: symbol lookup error: /nix/store/9qkdgz264yh029ddajfij2s4p0shsrn5-electron-3.0.14/bin/electron: undefined symbol: _ZN6icu_608ByteSink15GetAppendBufferEiiPciPi
<clever> but building libnode.so myself results in this failure
<aszlig> clever: ah, i misremembered, i packaged gyp, not gn
<clever> fixing _ZN6icu_608ByteSink15GetAppendBufferEiiPciPi would also work
<aszlig> clever: hm, maybe do it the dirty way and just patchelf in icu :-D
<clever> aszlig: icu-small is in the source for node, and has been enabled and built, i believe
<clever> -rw-r--r-- 1 clever users 48K Mar 29 12:09 out/Debug/obj.target/icuucx/deps/icu-small/source/common/bytestream.o
<clever> this is left-over from building node
drakonis_ has joined #nixos-dev
<clever> aszlig: and the above file is rebuild if i modify the input, so it is still part of the current build
<clever> objdump then...
<clever> _ZN6icu_608ByteSink15GetAppendBufferEiiPciPi missing
<clever> _ZN6icu_618ByteSink15GetAppendBufferEiiPciPi present
<clever> aszlig: now i get to deal with name mangling!
drakonis has quit [Ping timeout: 244 seconds]
<clever> ahhh, icu_60::ByteSink::GetAppendBuffer vs icu_61::ByteSink::GetAppendBuffer
<clever> source/common/unicode/uversion.h:# define U_NAMESPACE_BEGIN extern "C++" { namespace U_ICU_NAMESPACE {
<aszlig> clever: hm, that sounds like there are tw different icu versions in the build closure
<aszlig> s/\<tw\>/two/
<clever> aszlig: i'm grabbing a semi-random node revision, and then just forcibly replacing the libnode.so in electron
<clever> so electron expects icu60, but i built an icu61 version
<sphalerite> samueldr: so the problem is that newer kernels keep breaking zfs? Yeah I'd think so
<samueldr> "yeah I'd think so" can you reword as if the context was lost through time?
<clever> [clever@amd-nixos:~/iohk/node]$ grep U_ICU_VERSION_MAJOR_NUM deps/icu-small/source/common/unicode/uvernum.h
<clever> #define U_ICU_VERSION_MAJOR_NUM 61
<clever> aszlig: bingo, now i rewind node until thats 60
<sphalerite> samueldr: yes Ithink it makes sense to exclude zfs from latest-kernel installer images. More than the alternative we're currently on, which is not build the latest-kernel installer image whenever zfs is broken IIUC
<samueldr> yeah, you seem to understand correctly
<niksnut> aszlig: opinion on what exactly?
<aszlig> niksnut: returning with exit status 125 if it's a dependency failure or eval error
drakonis has joined #nixos-dev
<niksnut> yeah that's fine, we already return different exit codes for permanent vs transient failures
<aszlig> niksnut: that would make it easier using git bisect run
<niksnut> return timedOut ? 101 : (permanentFailure ? 100 : 1);
<aszlig> ah, okay, that's why i got confused with 100, i knew that it was used somewhere
<aszlig> niksnut: okay, so maybe we then need an extra --bisect flag, because people out there might be relying on 100/101
drakonis_ has quit [Read error: Connection reset by peer]
<niksnut> I think it was used by hydra but not anymore
<infinisil> aszlig: I've been using a script for bisections that does `if nix-shell -A <attribute> --run true; then nix-build -A <attribute>; else exit 125; fi`
<aszlig> infinisil: that would work as well, but isn't as nice as nix-build --bisect :-)
<infinisil> Yeah
<samueldr> nixos:trunk-combined (likely) OOM'd @ ~11:28 UTC log copy here https://gist.github.com/samueldr/9dab6e7934e20e8ca5196d3ab4aeb2cb
<samueldr> from a week's worth of looking, it seems to be an at-least daily occurence now :/
<samueldr> (plain simple OOMs not the heap thing, which might be gaining in frequency too)
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 258 seconds]
orivej has quit [Ping timeout: 245 seconds]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 250 seconds]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 258 seconds]
ckauhaus has quit [Quit: WeeChat 2.2]
<gchristensen> it is becoming clear that nixpkgs is too big, and we need to find a solution -- probably for 19.09
<gchristensen> I would like to see multiple proposals for how to do this, I know one (from profpatsch iirc) involves setting groups of supported packages. another I've heard is having <lang>Packages be their own thing.
<gchristensen> this is probably a harder problem than other distros, who don't have the same atomic model
<gchristensen> but we can't keep growing like this -- memory and cpus aren't keeping up :)
<nbp> If we were to rewrite this one using Nix libraries to poke at the internal, then we could literally have a single evaluation for updating "nixos rebuild" command if the Nix binary is unchanged.
<Profpatsch> gchristensen: This is just for a full evaluation, right?
<Profpatsch> What happens with packages that depend on language packages?
<gchristensen> right, I don't know
<gchristensen> and the answer to that is too big for IRC
<Profpatsch> I mean I’d say we need more lazyness.
<Profpatsch> And better debugging tools, flamegraphs and the like.
<gchristensen> these are all possible answers
<gchristensen> but we have a serious problem brewing -- we have for a long time
<simpson> We shouldn't expect a best-for-us answer to necessarily resemble existing solutions, since AFAICT we are in a different size category than everybody else.
<Profpatsch> Splitting nixpkgs is gonna introduce a versioning nightmare, and with dubious results (because you’d have to evaluate the language layers anyway, packages depend on them).
<gchristensen> Profpatsch: could be
<gchristensen> part of why I think it is important to have a handful of proposals, from different approaches, to consider
<LnL> splitting it into layers implies pulling apart certain package sets partially I think
<Profpatsch> A cool thing would be: lazy language packages.
<Profpatsch> Only packages needed for the resulting binaries are checked into nixpkgs.
<LnL> if the stdenv depends on pythonPackages.foo that also belongs in the stdenv layer
drakonis_ has joined #nixos-dev
<gchristensen> LnL: that is true
<Profpatsch> And the rest is available as layers somewhere.
<LnL> and if dependencies change, things might have to move a layer up or down
drakonis has quit [Ping timeout: 250 seconds]
init_6 has quit [Ping timeout: 245 seconds]
<gchristensen> right, layers / "sets" would be difficult and complicated
<gchristensen> debian etc. of course have the ultimate in laziness ;)
<nbp> gchristensen: one option is to cache the evaluation at the Nixpkgs level and freeze the evaluation on files sha.
<gchristensen> oh interesting
<Profpatsch> Our biggest advantage rn is that everything is in a monorepo.
<Profpatsch> If we split, we lose.
<nbp> Thus having a Nix expression cache which is stored outside, and keyed to Nixpkgs files hashes.
MichaelRaskin has joined #nixos-dev
<gchristensen> true
<gchristensen> also perhaps there are wins which could be made in Nix itself
<nbp> Which implies that we are able to track the sources or hash the expression inputs.
<gchristensen> but something has to give
<Profpatsch> gchristensen: Do you have a clear overview of which parts of nixpkgs take how much memory?
<gchristensen> no way
<Profpatsch> I think that has to be the first step.
<gchristensen> +1
<gchristensen> having tools to understand this would be huge
<Profpatsch> I have no insight in the nix language, but it would be a project that should be paid by some of the companies using nix.
<nbp> Profpatsch: The top-level attribute set takes ~30 MB, just to enumerate all the names.
<nbp> and its values
<gchristensen> Profpatsch: well now let's not commit to that, maybe someone is specifically interested in doing this as a hobby of some sort
<gchristensen> :P
<Profpatsch> nbp: You mean in pseudo-python: "".join([k for k in pkgs.keys()]) is a string of size 30MB?
<nbp> Profpatsch: no, the actual bytes in the Nix program.
<Profpatsch> The internal representation?
<nbp> yes
<nbp> Which is why I think we want a lazy update operator for the top-level of Nixpkgs (or just large attribute set in general.
<LnL> one thing that should already be possible is graphing the drv size, but not sure how much insight that would give
<nbp> As any overlay will add an extra level of 30 MB, especially if it is using rec.
<Profpatsch> Making it lazy wouldn’t mean there will be more intermediate arrtsets in memory?
<samueldr> intuition, with the hydra-eval-job parts mainly, is that somehow we might be losing through fragmentation (the increase heap size error)
<nbp> Profpatsch: no, this would imply more sharing between all the level of overlay executions.
<samueldr> though I do not feel qualified enough to know whether this makes sense or not within nix
<nbp> Profpatsch: self: super: { super-chain = super; } <--- basically look at the memory consumption of Nix every time you clone this overlay under a new name.
<nbp> Profpatsch: the fact that Nix is lazy implies that having a single reference to super would force Nix to keep it alive.
<nbp> Profpatsch: such as: super.bar.override
<Profpatsch> Maybe what we need is bang patterns for strict arguments.
<Profpatsch> Like in Haskell
<nbp> !important
<gchristensen> hehe
<Profpatsch> I refuse to understand this CSS joke
<simpson> It's okay, it's !important.
<gchristensen> lol.
* simpson !helping
* nbp is in favor of external maximal laziness.
* gchristensen doesn't know what that means
<nbp> You know maximal laziness?
<gchristensen> yeh
<nbp> external implies that it is stored else-where.
<gchristensen> ah
<nbp> and guarded by its inputs :)
<gchristensen> the evaluation cache
<gchristensen> .nixc
<nbp> exactly
<gchristensen> samueldr: some memory parameters on chef (hydra's machine) have been tweaked by niksnut, hopefully that helps
<samueldr> thanks, I'll be keeping an eye
<gchristensen> I know you will :)
orivej has joined #nixos-dev
<MichaelRaskin> So. There is one more user having problems without LOCALE_ARCHIVE set on non-NixOS Linux. This time LibreOffice. https://github.com/NixOS/nixpkgs/issues/57957#issuecomment-478130557
<MichaelRaskin> Do we have a canonical issue about that? Should we make a more prominent mention in the manial?
<MichaelRaskin> (Hm, _any_ mention in the manual)
<MichaelRaskin> Should we give up and use full locale archive?
orivej has quit [Ping timeout: 246 seconds]
<niksnut> that would mean a 120+ MB increase in *every* closure size
<MichaelRaskin> Another option: give up only for packages where 120MiB increase in closure size is not really noticeable…
<MichaelRaskin> But yeah, I have already reread the issue about Nix and locales.
Zer000 has joined #nixos-dev
<infinisil> Why is that so big?
<MichaelRaskin> too many locales
<infinisil> Like, for all kinds of languages?
<MichaelRaskin> Yes
<infinisil> Could we use replaceDependency to only make it include a single language, chose by the user?
<infinisil> (replaceDependency so no rebuild needed for every language)
<MichaelRaskin> Making replaceDependency sed by default by all packages sounds like a radical move
<MichaelRaskin> used by default
<infinisil> Yeah, but I don't see any immediate danger in doing this
<MichaelRaskin> Is there a preexisting canonical issue about LOCALE_ARCHIVE in general?
<MichaelRaskin> Or should I just open one?
<gchristensen> should probably open one :)
<MichaelRaskin> Ah, there is a documentation issue from 2015
<MichaelRaskin> #6878
<{^_^}> https://github.com/NixOS/nixpkgs/issues/6878 (by domenkozar, 4 years ago, open): document locale issues
drakonis has joined #nixos-dev
<gchristensen> aszlig: have you gone off yet?
<aszlig> gchristensen: well, i'm busy with other things, but still there =)
<gchristensen> aszlig: I fear I've bisected to a large commit, and I don't know what to do from here :/
<gchristensen> I'd love your advice
<aszlig> which one?
<gchristensen> 0e9da3fbf7d81f0f913b491c8de1ba7883d4f217 (note I'm not 100% certain it is this one -- I still have one step to go -- but it looks relevant)
<gchristensen> this is the list it could be: 0e9da3fbf7d81f0f913b491c8de1ba7883d4f217, 875736bb3f3ded168469f6a14df7a938416a99d5
<aszlig> gchristensen: hm... i doubt this one could be related to zfs
<gchristensen> heh
<aszlig> the latter sounds more plausible
<gchristensen> https://github.com/zfsonlinux/zfs/issues/8545 is my tracking issue
<{^_^}> zfsonlinux/zfs#8545 (by grahamc, 21 hours ago, open): ZoL 0.7.13 + Linux Kernel 5.0.4: GPL-incompatible module zfs.ko uses GPL-only symbol 'preempt_schedule_notrace'
<aszlig> gchristensen: oh...
<aszlig> gchristensen: is this only on aarch64?
<aszlig> as in: x86 works?
<gchristensen> yeah
<gchristensen> afaik anyway :)
<aszlig> okay, then maybe wait until you're done bisecting before digging through the code :-D
<gchristensen> okay :)
<aszlig> because from a rough look at the zfs source, it at least doesn't use that symbol directly
<aszlig> if btw. the former is the commit result of the bisect, i'd bisect again against 40e020c129cfc991e8ab4736d2665351ffd1468d..00203ba40d40d7f33857416adfb18adaf0e40123
<aszlig> (don't forget to check the first and last commit first)
<aszlig> but scrap my comment about "i doubt this one could be related to zfs"
<aszlig> it's highly likely actually
drakonis has quit [Quit: WeeChat 2.3]
<gchristensen> aszlig: 0e9da3fbf7d81f0f913b491c8de1ba7883d4f217 is the first bad commit indeed
<gchristensen> aszlig: I'll bisect 40e020c129cfc991e8ab4736d2665351ffd1468d..00203ba40d40d7f33857416adfb18adaf0e40123 now
<matthewbauer> How is the NixOS release going everyone? Are we on track for an official release soon?
<gchristensen> ( sphalerite, samueldr )
<infinisil> gchristensen: (you still need to update the /topic with the new release managers)
<gchristensen> heh
gchristensen changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html https://r13y.com | 18.09 release managers: vcunat,samueldr | 19.03 RMs: samueldr,sphalerite| https://logs.nix.samueldr.com/nixos-dev
<aszlig> gchristensen: you're missing a space :-P
<gchristensen> :|
gchristensen changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html https://r13y.com | 18.09 release managers: vcunat,samueldr | 19.03 RMs: samueldr,sphalerite | https://logs.nix.samueldr.com/nixos-dev
<gchristensen> thanks :)
<aszlig> :-D
<gchristensen> too bad computers aren't like muscles -- getting better at something the more they do it
<gchristensen> because if it was, this 19th kernel compile would be so fast
<samueldr> shhh
<samueldr> that's a monkey paw wish and you just wished machine learning up now
<gchristensen> oh no
<infinisil> Imagine using ML for kernel builds, you'll never be sure if it's actually a compiler error or just the ML messing up.
<aszlig> infinisil: except if the compiler itself is the classifier :-D
<gchristensen> you'll certainly never know if what you get out the other end is what you wanted
<infinisil> probabilistic type checking
<samueldr> as for the release, I've not been really paying as much attention as I'd like to the issues/PRs :( I've been opportunistically active on IRC since it's easy and low effort, otherwise watching hydra
<aszlig> hm... i'm pretty sure somebody already has implemented such a compiler :-D
<samueldr> and using 19.03 on machines here so at least for my subset of nixpkgs things seem smooth
<samueldr> work and other misc. things have been taking up my time for the last two months :/ though everything's fine if there is worry :)