apaul1729 has quit [Remote host closed the connection]
drakonis1 has joined #nixos-dev
pxc has quit [Ping timeout: 268 seconds]
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 240 seconds]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 240 seconds]
ajs124 has left #nixos-dev [#nixos-dev]
pxc has joined #nixos-dev
pxc has quit [Ping timeout: 252 seconds]
drakonis1 has quit [Quit: WeeChat 2.3]
worldofpeace has joined #nixos-dev
worldofpeace has quit [Remote host closed the connection]
<sphalerite>
gchristensen++
<{^_^}>
gchristensen's karma got increased to 102
<sphalerite>
which eval time?
orivej has quit [Ping timeout: 252 seconds]
colemickens has quit [Read error: Connection reset by peer]
chrisaw has quit [Read error: Connection reset by peer]
alunduil has quit [Read error: Connection reset by peer]
colemickens has joined #nixos-dev
chrisaw has joined #nixos-dev
Haskellfant has joined #nixos-dev
alunduil has joined #nixos-dev
ajs124 has joined #nixos-dev
cocreature has quit [Read error: Connection reset by peer]
Haskellfant is now known as cocreature
ajs124 has left #nixos-dev [#nixos-dev]
andi- has quit [Ping timeout: 258 seconds]
andi- has joined #nixos-dev
andi- has quit [Excess Flood]
andi- has joined #nixos-dev
andi- has quit [Ping timeout: 250 seconds]
andi- has joined #nixos-dev
orivej has joined #nixos-dev
andi- has quit [Ping timeout: 252 seconds]
andi- has joined #nixos-dev
andi- has quit [Ping timeout: 264 seconds]
peti has joined #nixos-dev
<peti>
niksnut: https://hydra.nixos.org/machines shows a couple of ghc builds that are stuck for up to 24h "receiving outputs". I cancelled them through the web interface, but apparently the actual build processes don't abort. Can you help, maybe?
andi- has joined #nixos-dev
<niksnut>
they're probably waiting for memory
<niksnut>
to copy the output from the remote machine
<peti>
niksnut: Well, if there is a way to rescure those builds, then it would be great. But it's fine to just abort them, too, because the latest versions had to be re-built from scratch anyway, so we probably never need those binaries again.
<niksnut>
I don't think they *can* be aborted in the "receiving outputs" stage (except by restarting hydra-queue-runner)
<niksnut>
this is happening because we had to reduce the NAR buffer size to avoid OOMs in hydra-evaluated btw
<peti>
Hmm. I see.
<peti>
ghc to too freaking big.
<peti>
We could aggressively split it into multiple outputs. I suppose that would help. But it's difficult to pull off.
<niksnut>
I'm thinking we could enable some swap for hydra-queue-runner (but not for hydra-evaluator to avoid trashing)
Jackneill has quit [Remote host closed the connection]
Jackneill has joined #nixos-dev
init_6 has joined #nixos-dev
jtojnar has joined #nixos-dev
ajs124 has joined #nixos-dev
orivej has quit [Ping timeout: 264 seconds]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
fadenb has quit [Remote host closed the connection]
fadenb has joined #nixos-dev
orivej has joined #nixos-dev
ivan has quit [Remote host closed the connection]
ivan has joined #nixos-dev
ajs124 has left #nixos-dev [#nixos-dev]
ajs124 has joined #nixos-dev
init_6 has quit []
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 250 seconds]
drakonis has joined #nixos-dev
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 250 seconds]
drakonis has quit [Ping timeout: 240 seconds]
drakonis has joined #nixos-dev
drakonis1 has quit [Ping timeout: 252 seconds]
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 264 seconds]
asymmetric has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 252 seconds]
<infinisil>
Do we still backport to 18.09?
<Taneb>
Until end of April, I think?
<infinisil>
Alright
<infinisil>
Ah the thing I wanted to backport isn't needed on 18.09 anyways
pxc has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
drakonis has joined #nixos-dev
asymmetric has quit [Ping timeout: 264 seconds]
drakonis_ has quit [Ping timeout: 264 seconds]
drakonis has quit [Ping timeout: 250 seconds]
MichaelRaskin has joined #nixos-dev
<gchristensen>
looks like our lib's lists.unique could really use ... uh ... optimising
ivan has quit [Quit: lp0 on fire]
harrow` has quit [Quit: Leaving]
harrow has joined #nixos-dev
<ekleog>
optimizing it sounds pretty hard, though
<gchristensen>
yeah
<ekleog>
we can likely get it to O(n log n) with dichotomy, I guess
ivan has joined #nixos-dev
* ekleog
trying
<ekleog>
hmm wait actually not in the way I was thinking of
<gchristensen>
the samples happen every time a function is called
<ekleog>
oh so it isn't, that'd have been really really weird :)
<gchristensen>
hrm?
<ekleog>
making a 26% reduction in total time
<gchristensen>
ahh rgiht
<ekleog>
well even if it doesn't do much I guess if it's not actively negative it'd be a good thing to put in, as it makes the flamegraph much more readable :°
<gchristensen>
yea
<gchristensen>
I really need toremember to not do tehse exciting experiments on my teeny tiny laptop
<ekleog>
wait you've got 30G RAM on your “teeny tiny laptop”? :D
<ekleog>
yup, I'm thinking it should lead to fewer copies
<gchristensen>
ok, the first set of evaluations are running
<infinisil>
gchristensen: btw, O(n) is not constant memory :P
<gchristensen>
ahh right
<gchristensen>
it is such a pleasure to be in this community. it is incredible, writing little tools and having brilliant people leap and do something incredible
<infinisil>
I guess we could call it constant-per-element though
<sphalerite>
nrAvoided seems like a good thing to rise
<gchristensen>
not sure what nrAvoided means
<gchristensen>
but yeah :P
<sphalerite>
seems to be the number of thunks created but not evaluated
<gchristensen>
curious
<gchristensen>
okay, starting to make the flame graph for yours, infinisil, so I might be unresponsive here a bit
<infinisil>
Oh, there's an old `uniqList` function in deprecated.nix
<ekleog>
infinisil: looking at the tables I find it fun: your change that was supposed to reduce memory reduced it less than mine, which was supposed to reduce cpu time but reduced it less than yours xD
<averell>
there should be an enumerate there right? or is that elemAt super cheap? otherwise just zip with index maybe.
<infinisil>
ekleog: Yeah xD
<infinisil>
Oh I think I see a big optimization for mine
<averell>
is it enumerating the list? :)
* ekleog
eager to have the numbers for the dichotomy-based one
<ekleog>
averell: I'm not sure what you're talking about?
<gchristensen>
ekleog: is that a nudge to move my testing to a bigger system?
<averell>
i meant instead of concatMap with index and elemAt just go over enumerate the list like (value, i)
<averell>
or are lists random access?
<ekleog>
averell: lists are actually arrays in nix
<infinisil>
Random access is O(1)
<averell>
that explains a lot...
<ekleog>
gchristensen: I've heard we have big epyc-based machines 😇 but anyway it's not really a nudge, though it'd be more than great if ofborg could give out these numbers :)
<gchristensen>
I completely agree
<gchristensen>
the machines are not powerful enough to do the flame graphs
<ekleog>
hmm that'll definitely make things harder :/
<gchristensen>
no big deal
<gchristensen>
I can get bigger ones
<gchristensen>
certainly adds another step, though!
<infinisil>
I wonder, if I have a function `foo = x: y: ...` that gets called with the same x a lot of times, is it better to assign `bar = foo 0` and use that?
<gchristensen>
probably
<gchristensen>
(I don't know, we should test)
<infinisil>
I'd think so too, but then you also get an additional thunk for bar, which could be bad if this is in a loop
<infinisil>
So not sure
<ekleog>
let's change the nix language to be just rust with a library
<gchristensen>
lol
<ekleog>
this way performance would be predictable
<ekleog>
I'm being totally serious here 😇
<ekleog>
more seriously, allowing nix-native code FFI might actually be a non-stupid idea, as the evaluation is not done by the daemon anyway so it's outside the trusted code base
<gchristensen>
it is a very nice property that nix code can't do things
<ekleog>
right, pure-mode, forgot about it
<ekleog>
(I had builtins.exec in mind, with all the drawbacks entailed)
<averell>
you should put that quote on a t-shirt
<ekleog>
hmm maybe nix-wasm FFI?
<infinisil>
Pfff, nobody cares about drawbacks when builtins.exec is the only way to write a RTS for nix
<infinisil>
Considering that it takes ages to load that doesn't look good!
<infinisil>
Oh, it's there, just had to scroll down lol
<ekleog>
actually nix-wasm FFI might actually make sense for these things that are pretty much impossible to do sanely in nix (like that `unique` function that really should be using a hashmap and just iterating forward while collecting in a Vec)
<ekleog>
I wonder if such an FFI could be possible with shlevy's plugin system, for initial experiments
<infinisil>
gchristensen: Yeah so this huge spike there is from unique, coming from the recursive isFirst probably
<infinisil>
Wait does Nix not do tail call elimination
<gchristensen>
infinisil: maybe it does but my instrumentation doesn't handle it properly?
<infinisil>
ekleog: Because you first have the cost for remove, and then also the cost for unique on its result
<gchristensen>
averell: they're not benches, really, it is the exact data from evaluating nixpkgs
<gchristensen>
ie, not at all synthetic
pxc has joined #nixos-dev
<ekleog>
infinisil: I'm assuming it'll free the memory used by remove before allocating the memory for unique
<ekleog>
and the memory returned-by-remove ingested-by-unique is counted in the price of both
<ekleog>
now nix is a GC language, so we have no guarantee of that, but if we start down that route then it's hard to compute memory usage
<infinisil>
I see
<infinisil>
I'm unsure myself regarding whether we can make that assumption
<infinisil>
I guess we could test it though
<infinisil>
Evaluate uniq on 2^1, 2^2, 2^3, ..., 2^10 elements and see how memory goes up
<gchristensen>
even with loads of ram, flamegraph.pl takes ~10min to run
<ekleog>
gchristensen: tbh I'm not sure the flamegraph is actually useful here
<gchristensen>
for this specific case, possibly not
<ekleog>
like, my current hunch is we need your stats table to decide whether things get merged, and the flamegraph to figure out how to improve stufff
<gchristensen>
it sure did a nice job pointing out a problem though
<ekleog>
indeed .)
<infinisil>
gchristensen++ for benchmarks and stuff
<{^_^}>
gchristensen's karma got increased to 103
<ekleog>
I wonder whether we can make a derivation that builds the flamegraph and have hydra build it
<ekleog>
and ideally have it end up on the nixos website
<gchristensen>
I don't think hydra would be very impressed by that
<infinisil>
ekleog: Anyways, about my unique I can at least say it's guaranteed O(n) :P Even though it performed worse in general, maybe this can be optimized still though
<infinisil>
I think at least!
<gchristensen>
anyway, I see no reason why this shouldn't be run on each PR
<ekleog>
this way we'd always have an up-to-date flamegraph to know where to improve on stuff, and stats tables would be provided by ofborg
<ekleog>
doesn't the flamegraph take too much memory for ofborg?
<gchristensen>
it takes too much for the puny machines it uses for evals right now, but it doesn't have to
<gchristensen>
it doens't have to run on those machines*
<ekleog>
(while I'd be hoping ofborg can do the stats table, as it can already eval)
<infinisil>
Oh! Couldn't we add parallelism to Nix?
<infinisil>
Things like mapAttrs, concatMap, stuff like that could all be run in parallel
<infinisil>
Tbh we should just add such functions to builtins. There's no reason to keep implementing all these things in nixpkgs/lib, other than for backwards compatibility
<tilpner>
Yes, most people can probably be persuaded to give up minimalism/small API for performance
<tilpner>
And the old Nix implementations can stay around as fallbacks for a while
<ekleog>
if nix was rust I'd happily do it, but tbh I've got enough weird c++ issues in interpreters to try and minimize my interaction with c++ (ie. keep it to when needed for $work) :/
<ekleog>
gchristensen: hmm nothing looking plain wrong in this graph, I guess the memory increase comes from dichotomy that actually needs to keep like 3N/2 memory instead of just the N for the trivial copying-many-times version
<{^_^}>
nix#2748 (by edolstra, 2 weeks ago, open): [WIP] Make nix/unpack-channel.nix a builtin builder
<infinisil>
tilpner: That makes me think, considering we have a lib/minver.nix, we can probably remove some fallbacks in lib by now
<ekleog>
infinisil: I'm pretty much thinking that parallelism here is not necessarily going to improve things much, as nix wasn't really designed for it
<gchristensen>
yes, at this point I'd go back to v1 of your patch, ekleog, and then look at the "next big thing"
tilpner has quit [Remote host closed the connection]
tilpner has joined #nixos-dev
<sphalerite>
gchristensen: backporting what, sorry? The performance improvements?
<gchristensen>
yea
<sphalerite>
mixed feelings
<sphalerite>
on the one hand, yay less tooling problems. On the other hand, pretty fundamental and far-reaching changes which proooobably shouldn't go into a stable release?
<sphalerite>
Would this make borg and hydra work noticeably better?
<sphalerite>
(also NIXRUUUUUUUST :D)
<sphalerite>
niksnut++
<{^_^}>
niksnut's karma got increased to 9
<gchristensen>
it would cut down evaluation ram by like 730mb, which we would definitely notice