<infinisil>
What I found out: In low memory environments, evaluation time can be much much higher
<tomberek>
I've got high hopes for the intensional store/CAS efforts, but the proposals aren't quite in a place where I can effectively contribute. I still believe those efforts are valuable and will allow very interesting features. What would be the best way to fund someone to focus those rfc's?
<infinisil>
Note how with a 804MB limit it takes 20s, while with a 144MB limit it takes 51s
<cole-h>
What do the "low" and "high" results mean? The minimum amount of memory needed and the maximum amount of memory it can run under?
<infinisil>
cole-h: Bisection low and high. High means with this much memory evaluation can succeed, while it shouldn't succeed with low
<cole-h>
I see now
<infinisil>
pkgs.hello can be evaluated in as little as 2MB in ~20 seconds
<infinisil>
But with 60MB of memory it only takes 0.18 seconds
ris has quit [Ping timeout: 272 seconds]
<infinisil>
Firefox can be evaluated with 2.2MB if you can wait 60 seconds
andi- has quit [Quit: WeeChat 2.6]
<clever>
thats what i suspected, give it too little, and it will spent all of its time collecting garbage
<clever>
the old (and now disabled) hydra restarting hack in evals, didnt limit ram at all, but instead just wiped the heap clean if it used "too much"
<clever>
after the attr causing that overflow, finished its eval
<clever>
grep the nixos-org-configuration repo for that config key
<gchristensen>
hydra.nixos.org is using an eexperimental evaluator now, so maybe check out that repo from ~2 weeks ago and grep that
lovesegfault has quit [Ping timeout: 272 seconds]
<infinisil>
Hm, looking at GC_INITIAL_HEAP_SIZE, I fail to see how this can *fix* memory problems
<infinisil>
Heap should be expanded automatically if necessary, why does the initial size matter?
<gchristensen>
please please please find a better answer :P
* infinisil
has all brain cycles spent for today
<infinisil>
I shall activate chill mode now
<gchristensen>
good luck, I have a hard time with that mode.
<simpson>
infinisil: I haven't looked at Boehm in a long time, but IIRC the heap is only grown when necessary; that is, when a typical GC run isn't enough to shrink the heap past some high-water mark, or doesn't make enough room for some new object.
<infinisil>
I see, though why increasing it sometimes makes evaluations suddenly work is still a mystery
<simpson>
Highlights from https://www.hboehm.info/gc/gcdescr.html "In non-incremental mode, we make a decision about whether to garbage collect whenever an allocation would otherwise have failed with the current heap size. If the total amount of allocation since the last collection is less than the heap size divided by GC_free_space_divisor, we try to expand the heap. Otherwise, we initiate a garbage collection.
<simpson>
This ensures that the amount of garbage collection work per allocated byte remains constant."
<simpson>
"In incremental mode, the heap is always expanded when we encounter insufficient space for an allocation. Garbage collection is triggered whenever we notice that more than GC_heap_size/2 * GC_free_space_divisor bytes of allocation have taken place. After GC_full_freq minor collections a major collection is started."
<simpson>
If the heap starts too small, then Nix might simply not generate enough memory pressure to force the heap to grow. Instead, the GC runs more often, leading to the space-time tradeoff you measured earlier.
<infinisil>
Ah neat
<infinisil>
simpson: Wait, so Nix uses non-incremental mode?
<infinisil>
Ah yeah I think non-incremental is the default
<infinisil>
Though that makes me wonder how incremental mode would do
bgamari has quit [Ping timeout: 240 seconds]
bgamari has joined #nixos-dev
v0|d has joined #nixos-dev
* infinisil
failed to activate chill mode and ended up debugging some haskell package build failure
<Profpatsch>
Also for non-students, there’s always something we can do if there’s somebody who wants to invest the effort.
<Profpatsch>
(The effort to implement something useful for the project/community that is)
<Profpatsch>
infinisil: Hm, so if the pkgs/release.nix only takes 150MB and one minute, does that mean that most of our crazy evaluation times comes from the module system?
__monty__ has joined #nixos-dev
cole-h has quit [Ping timeout: 268 seconds]
johnny131 has quit [Ping timeout: 272 seconds]
<tomberek>
Profpatsch: I meant that i'd like to help, but in the current state i'm not able to, until a bit more progress is made. I'd like to help though.
<Profpatsch>
tomberek: which rfc are you talking about specifically? the one by regnat?
<Profpatsch>
tomberek: what can be done to enable you to help the project?
<tomberek>
I can provide time, but the number of changes proposed to the C++ basline is more than I can do. If it can get a bit further along, i can start iterating use cases, testing, fixing bugs, etc. I think we just need some momentum.
<tomberek>
Profpatsch: or rather the changes needed to get momentum are beyond my abilities, so i'd like to provide my time or some funds in any other helpfule ways, or incentivise others to.
<Profpatsch>
fair
<tomberek>
Seems like there are others in a similar situation, we need a champion to get started (and to make some decisions about design to avoid perpetual discussion)
<gchristensen>
rnhmjoj: niksnut is working on improvements to the evaluator to try and fix it without changing nixpkgs
m15k has quit [Remote host closed the connection]
drakonis1 has joined #nixos-dev
drakonis1 has quit [Ping timeout: 246 seconds]
drakonis1 has joined #nixos-dev
drakonis1 has quit [Ping timeout: 240 seconds]
ixxie has joined #nixos-dev
<aminechikhaoui>
hm in hydra if I remove localhost from /etc/nix/machines would preferLocalBuilds derivations run on the frontend still ?
<aminechikhaoui>
I'm seeing some weird issue in my hydra after updating to a more recent version of hydra, when there is a derivation with preferLocalBuild =1 I see the derivation and its deps built in localhost and all appearing in the logs of the final one
<aminechikhaoui>
niksnut (sorry for the direct pings :D) can you think of something that can cause that
<clever>
aminechikhaoui: without a localhost in machines, preferLocalBuilds gets ran on a remote machine, always
<clever>
localhost x86_64-linux,i686-linux - 4 1 local local
<clever>
aminechikhaoui: but you can add a special "local" feature to the mandatory features list
<clever>
which means it can only run preferLocalBuilds and nothing else
<aminechikhaoui>
nothing else coming from the queue-runner right ?
<aminechikhaoui>
I still want it to respect max-jobs for hydra-provisioner
<clever>
queue-runner will only auto-generate localhost if machines is missing
<aminechikhaoui>
in my case I have "localhost x86_64-linux,i686-linux - 1 1000 local local" in /etc/nix/machines
<aminechikhaoui>
so I should make it local instead I guess
<clever>
a single local, would allow any job to be ran
<clever>
also, both ways just mean preferLocalBuilds can run on localhost, but it can also run on remote hosts too
<aminechikhaoui>
so the problem is, somehow there is a bug where it runs the preferLocalBuilds derivation *and its deps*
<aminechikhaoui>
and all logs are showing up in the preferLocalBuilds derivation log
<aminechikhaoui>
quite messed up :D
<clever>
weird
<aminechikhaoui>
I guess maybe I should remove localhost from nix machines until I'm in the mood to investigate it :-)
page has quit [Ping timeout: 272 seconds]
bhipple has joined #nixos-dev
page has joined #nixos-dev
<aminechikhaoui>
I wish there was a command to ask the nix-deamon which derivations it is currently building
<niksnut>
that might become more feasible if we move from a forking to a multi-threaded daemon
<aminechikhaoui>
ls -l /proc/<nix-daemon-process>/fd seems to get me close enough :)
<aminechikhaoui>
after trying each pid :D
<aminechikhaoui>
niksnut since you're around, any idea what can cause that weird issue above in the discussion :)
<niksnut>
hm no, preferLocalBuilds shouldn't affect dependencies
<das_j>
is there any doc on how hydra bumps the channels when the build is successful?
<clever>
aminechikhaoui: i just peek at the names of the temp dirs in `ls -ltrh /tmp`
<das_j>
aminechikhaoui: You can also use nix-top as root which shows the out path
<{^_^}>
ostreedev/ostree#1592 (by jtojnar, 1 year ago, open): installed tests fail with pseudoterminal active
<worldofpeace>
Jan Tojnar: Yeah, I'm not sure I get why they called that app this.
<gchristensen>
ehhhhh that machine might be. I can't look at it further right yet.
<jtojnar>
worldofpeace well it would make sense Polari (or alternatively Parlare, Parlary, Palare, Palarie, Palari; from Italian parlare, meaning 'to talk')
<infinisil>
gchristensen: Hm thanks, guess I'll just apply the fix and not waste much more time on this
<gchristensen>
infinisil: or just restart it
<gchristensen>
almost every machine is running a very very new nix, but it was unlucky
<infinisil>
There's an upstream fix, I'd rather apply that
<samueldr>
do we actually need the fix? is that nix version still supported? should we?
<samueldr>
though, considering it's an upstream fix it's much less of an issue though if it was added
<gchristensen>
that nix version is almost certainly not supported
<samueldr>
do we have guidelines about nix support once a new version is released?
<infinisil>
There is lib/minver.nix
<infinisil>
I think for the whole placeholder thing we waited at least 2 NixOS releases to not cause too much problems for people updating multiple versions
<samueldr>
here I was thinking raw nix, not related to nixpkgs
<samueldr>
not nix usage in nixpkgs
<infinisil>
"raw nix"?
<samueldr>
just nix
<samueldr>
what's the expectations for "support" about previous releasss