samueldr changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS 20.03 Feature Freeze Feb 10 https://discourse.nixos.org/t/nixos-20-03-feature-freeze/5655 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 19.09 RMs: disasm, sphalerite; 20.03: worldofpeace, disasm | https://logs.nix.samueldr.com/nixos-dev
<jtojnar> I will take a look at it tomorrow
<jtojnar> Though I am not sure about bootstrapping NixOS in non-NixOS qemu
<worldofpeace> I'm not really sure what they meant by this actually
<jtojnar> Maybe just aarch64 qemu image?
<samueldr> sounds like they just want to run nix to run the build?
<samueldr> though if it's native aarch64 it'll cause complications
<worldofpeace> yeah, qemu and aarch64 is a bit out of my depth to assist them to be able to play with it
<infinisil> I got a pretty neat script to test memory usage now, using it looks like https://paste.infinisil.com/E7xltbuQlU.txt
<infinisil> What I found out: In low memory environments, evaluation time can be much much higher
<tomberek> I've got high hopes for the intensional store/CAS efforts, but the proposals aren't quite in a place where I can effectively contribute. I still believe those efforts are valuable and will allow very interesting features. What would be the best way to fund someone to focus those rfc's?
<infinisil> Here's a run for `pkgs/top-level/release.nix -A unstable`: https://paste.infinisil.com/WPYWaBrOU4.txt
<infinisil> Note how with a 804MB limit it takes 20s, while with a 144MB limit it takes 51s
<cole-h> What do the "low" and "high" results mean? The minimum amount of memory needed and the maximum amount of memory it can run under?
<infinisil> cole-h: Bisection low and high. High means with this much memory evaluation can succeed, while it shouldn't succeed with low
<cole-h> I see now
<infinisil> pkgs.hello can be evaluated in as little as 2MB in ~20 seconds
<infinisil> But with 60MB of memory it only takes 0.18 seconds
ris has quit [Ping timeout: 272 seconds]
<infinisil> Firefox can be evaluated with 2.2MB if you can wait 60 seconds
andi- has quit [Quit: WeeChat 2.6]
<clever> thats what i suspected, give it too little, and it will spent all of its time collecting garbage
<clever> the old (and now disabled) hydra restarting hack in evals, didnt limit ram at all, but instead just wiped the heap clean if it used "too much"
<clever> after the attr causing that overflow, finished its eval
andi- has joined #nixos-dev
<infinisil> (updated script: https://paste.infinisil.com/gJaFRbtZuo.sh)
<infinisil> clever: Do you happen to know how nixos' hydra limits memory?
<infinisil> Like, to what amount
<clever> i dont know what the current limits are
<clever> but its probably in the nixos-org-configurations epo
<clever> repo*
<gchristensen> it is only limited by free ram
<gchristensen> it has 41g free, 51g available now
<clever> the iohk hydra has had evals fail, because boehm wasnt willing to use more, and i worked around it by increasing the initial heap size
<infinisil> gchristensen: Does only a single eval run at a time?
<infinisil> Probably not
<gchristensen> yes
<infinisil> Ah nice
<clever> [root@nas:~]# grep max_concurrent_evals /var/lib/hydra/hydra.conf
<clever> max_concurrent_evals = 1
<clever> grep the nixos-org-configuration repo for that config key
<gchristensen> hydra.nixos.org is using an eexperimental evaluator now, so maybe check out that repo from ~2 weeks ago and grep that
lovesegfault has quit [Ping timeout: 272 seconds]
<infinisil> Hm, looking at GC_INITIAL_HEAP_SIZE, I fail to see how this can *fix* memory problems
<infinisil> Heap should be expanded automatically if necessary, why does the initial size matter?
<gchristensen> please please please find a better answer :P
* infinisil has all brain cycles spent for today
<infinisil> I shall activate chill mode now
<gchristensen> good luck, I have a hard time with that mode.
<simpson> infinisil: I haven't looked at Boehm in a long time, but IIRC the heap is only grown when necessary; that is, when a typical GC run isn't enough to shrink the heap past some high-water mark, or doesn't make enough room for some new object.
<infinisil> I see, though why increasing it sometimes makes evaluations suddenly work is still a mystery
<simpson> Highlights from https://www.hboehm.info/gc/gcdescr.html "In non-incremental mode, we make a decision about whether to garbage collect whenever an allocation would otherwise have failed with the current heap size. If the total amount of allocation since the last collection is less than the heap size divided by GC_free_space_divisor, we try to expand the heap. Otherwise, we initiate a garbage collection.
<simpson> This ensures that the amount of garbage collection work per allocated byte remains constant."
<simpson> "In incremental mode, the heap is always expanded when we encounter insufficient space for an allocation. Garbage collection is triggered whenever we notice that more than GC_heap_size/2 * GC_free_space_divisor bytes of allocation have taken place. After GC_full_freq minor collections a major collection is started."
<simpson> If the heap starts too small, then Nix might simply not generate enough memory pressure to force the heap to grow. Instead, the GC runs more often, leading to the space-time tradeoff you measured earlier.
<infinisil> Ah neat
<infinisil> simpson: Wait, so Nix uses non-incremental mode?
<infinisil> Ah yeah I think non-incremental is the default
<infinisil> Though that makes me wonder how incremental mode would do
bgamari has quit [Ping timeout: 240 seconds]
bgamari has joined #nixos-dev
v0|d has joined #nixos-dev
* infinisil failed to activate chill mode and ended up debugging some haskell package build failure
cptchaos83 has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
cptchaos83 has joined #nixos-dev
das_j has quit [Remote host closed the connection]
Scriptkiddi has quit [Remote host closed the connection]
ajs124 has quit [Remote host closed the connection]
ajs124 has joined #nixos-dev
das_j has joined #nixos-dev
Scriptkiddi has joined #nixos-dev
v0|d has quit [Remote host closed the connection]
orivej has quit [Ping timeout: 260 seconds]
drakonis has quit [Ping timeout: 272 seconds]
lovesegfault has joined #nixos-dev
bhipple has quit [Remote host closed the connection]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 272 seconds]
<Profpatsch> tomberek: Tweag is very interested in funding efforts towards the intensional store.
<Profpatsch> Also for non-students, there’s always something we can do if there’s somebody who wants to invest the effort.
<Profpatsch> (The effort to implement something useful for the project/community that is)
<Profpatsch> infinisil: Hm, so if the pkgs/release.nix only takes 150MB and one minute, does that mean that most of our crazy evaluation times comes from the module system?
__monty__ has joined #nixos-dev
cole-h has quit [Ping timeout: 268 seconds]
johnny131 has quit [Ping timeout: 272 seconds]
<tomberek> Profpatsch: I meant that i'd like to help, but in the current state i'm not able to, until a bit more progress is made. I'd like to help though.
<Profpatsch> tomberek: which rfc are you talking about specifically? the one by regnat?
<tomberek> Profpatsch: #17 and #62
<{^_^}> https://github.com/NixOS/nixpkgs/pull/17 (by viric, 7 years ago, merged): Fixing npm and updating nodejs to 0.6.19
<{^_^}> https://github.com/NixOS/nixpkgs/issues/62 (by chaoflow, 7 years ago, closed): check virtualenv behaviour
<Profpatsch> haha, {^_^}, that’s the wrong repo
<Profpatsch> tomberek: what can be done to enable you to help the project?
<tomberek> I can provide time, but the number of changes proposed to the C++ basline is more than I can do. If it can get a bit further along, i can start iterating use cases, testing, fixing bugs, etc. I think we just need some momentum.
<tomberek> Profpatsch: or rather the changes needed to get momentum are beyond my abilities, so i'd like to provide my time or some funds in any other helpfule ways, or incentivise others to.
<Profpatsch> fair
<tomberek> Seems like there are others in a similar situation, we need a champion to get started (and to make some decisions about design to avoid perpetual discussion)
<infinisil> rfcs#17 rfcs#62
<{^_^}> https://github.com/NixOS/rfcs/pull/17 (by wmertens, 2 years ago, open): [RFC 0017] Intensional Store
<{^_^}> https://github.com/NixOS/rfcs/pull/62 (by regnat, 9 weeks ago, open): [RFC 0062] Content-addressed paths
<rnhmjoj> what's the status on hydra failures? i see someone managed to, at least, create the new channels for 20.03
johnny101m has joined #nixos-dev
m15k has joined #nixos-dev
<m15k> Can someone tell me how the GitHub build bot checks which architecture should be build?
cdepillabout has joined #nixos-dev
tomberek has quit [Ping timeout: 240 seconds]
cdepillabout has quit [Quit: Leaving]
orivej has joined #nixos-dev
psyanticy has joined #nixos-dev
<gchristensen> rnhmjoj: niksnut is working on improvements to the evaluator to try and fix it without changing nixpkgs
m15k has quit [Remote host closed the connection]
drakonis1 has joined #nixos-dev
drakonis1 has quit [Ping timeout: 246 seconds]
drakonis1 has joined #nixos-dev
drakonis1 has quit [Ping timeout: 240 seconds]
ixxie has joined #nixos-dev
<aminechikhaoui> hm in hydra if I remove localhost from /etc/nix/machines would preferLocalBuilds derivations run on the frontend still ?
<aminechikhaoui> I'm seeing some weird issue in my hydra after updating to a more recent version of hydra, when there is a derivation with preferLocalBuild =1 I see the derivation and its deps built in localhost and all appearing in the logs of the final one
<aminechikhaoui> niksnut (sorry for the direct pings :D) can you think of something that can cause that
<clever> aminechikhaoui: without a localhost in machines, preferLocalBuilds gets ran on a remote machine, always
<clever> localhost x86_64-linux,i686-linux - 4 1 local local
<clever> aminechikhaoui: but you can add a special "local" feature to the mandatory features list
<clever> which means it can only run preferLocalBuilds and nothing else
<aminechikhaoui> nothing else coming from the queue-runner right ?
<aminechikhaoui> I still want it to respect max-jobs for hydra-provisioner
<clever> queue-runner will only auto-generate localhost if machines is missing
<aminechikhaoui> in my case I have "localhost x86_64-linux,i686-linux - 1 1000 local local" in /etc/nix/machines
<aminechikhaoui> so I should make it local instead I guess
<clever> a single local, would allow any job to be ran
<clever> also, both ways just mean preferLocalBuilds can run on localhost, but it can also run on remote hosts too
<aminechikhaoui> so the problem is, somehow there is a bug where it runs the preferLocalBuilds derivation *and its deps*
<aminechikhaoui> and all logs are showing up in the preferLocalBuilds derivation log
<aminechikhaoui> quite messed up :D
<clever> weird
<aminechikhaoui> I guess maybe I should remove localhost from nix machines until I'm in the mood to investigate it :-)
page has quit [Ping timeout: 272 seconds]
bhipple has joined #nixos-dev
page has joined #nixos-dev
<aminechikhaoui> I wish there was a command to ask the nix-deamon which derivations it is currently building
<niksnut> that might become more feasible if we move from a forking to a multi-threaded daemon
<aminechikhaoui> ls -l /proc/<nix-daemon-process>/fd seems to get me close enough :)
<aminechikhaoui> after trying each pid :D
<aminechikhaoui> niksnut since you're around, any idea what can cause that weird issue above in the discussion :)
<niksnut> hm no, preferLocalBuilds shouldn't affect dependencies
<das_j> is there any doc on how hydra bumps the channels when the build is successful?
<clever> aminechikhaoui: i just peek at the names of the temp dirs in `ls -ltrh /tmp`
<das_j> aminechikhaoui: You can also use nix-top as root which shows the out path
<aminechikhaoui> oh nix-top is nice
<aminechikhaoui> thanks das_j !
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
teto has joined #nixos-dev
Jackneill has quit [Remote host closed the connection]
cole-h has joined #nixos-dev
Synthetica has joined #nixos-dev
<Synthetica> ryantm: What's up with r-ryantm?
claudiii has joined #nixos-dev
tomberek has joined #nixos-dev
orivej has quit [Read error: Connection reset by peer]
orivej has joined #nixos-dev
lassulus has quit [Quit: WeeChat 2.6]
lassulus has joined #nixos-dev
justan0theruser has quit [Ping timeout: 246 seconds]
thefloweringash has quit [Ping timeout: 246 seconds]
pkolloch[m] has quit [Ping timeout: 252 seconds]
worldofpeace has quit [Ping timeout: 252 seconds]
mkg20001 has quit [Ping timeout: 252 seconds]
tokudan[m] has quit [Ping timeout: 252 seconds]
roberth has quit [Ping timeout: 246 seconds]
vaibhavsagar has quit [Ping timeout: 246 seconds]
layus[m] has quit [Ping timeout: 240 seconds]
timokau[m] has quit [Ping timeout: 240 seconds]
alienpirate5 has quit [Ping timeout: 240 seconds]
ma27[m] has quit [Ping timeout: 248 seconds]
Nyanloutre[m] has quit [Ping timeout: 248 seconds]
abbradar[m] has quit [Ping timeout: 245 seconds]
arcnmx has quit [Ping timeout: 256 seconds]
jonge[m] has quit [Ping timeout: 256 seconds]
dtz has quit [Ping timeout: 240 seconds]
colemickens has quit [Ping timeout: 260 seconds]
bennofs[m] has quit [Ping timeout: 260 seconds]
rycee has quit [Ping timeout: 245 seconds]
jtojnar has quit [Ping timeout: 248 seconds]
emily has quit [Ping timeout: 248 seconds]
domenkozar[m] has quit [Ping timeout: 240 seconds]
aanderse has quit [Ping timeout: 245 seconds]
Ericson2314 has quit [Ping timeout: 240 seconds]
rnhmjoj has quit [Ping timeout: 260 seconds]
Ox4A6F has quit [Ping timeout: 240 seconds]
justan0theruser has joined #nixos-dev
justan0theruser has quit [Read error: Connection timed out]
edwtjo has joined #nixos-dev
edwtjo has joined #nixos-dev
edwtjo has quit [Changing host]
justan0theruser has joined #nixos-dev
thefloweringash has joined #nixos-dev
worldofpeace has joined #nixos-dev
vaibhavsagar has joined #nixos-dev
tokudan[m] has joined #nixos-dev
pkolloch[m] has joined #nixos-dev
roberth has joined #nixos-dev
mkg20001 has joined #nixos-dev
arcnmx has joined #nixos-dev
jonge[m] has joined #nixos-dev
Nyanloutre[m] has joined #nixos-dev
ma27[m] has joined #nixos-dev
alienpirate5 has joined #nixos-dev
layus[m] has joined #nixos-dev
timokau[m] has joined #nixos-dev
jtojnar has joined #nixos-dev
abbradar[m] has joined #nixos-dev
domenkozar[m] has joined #nixos-dev
rycee has joined #nixos-dev
bennofs[m] has joined #nixos-dev
aanderse has joined #nixos-dev
rnhmjoj has joined #nixos-dev
ris has joined #nixos-dev
ixxie has quit [Ping timeout: 272 seconds]
ixxie has joined #nixos-dev
kraem has joined #nixos-dev
<gchristensen> w00000t 20.02 has a tested job https://hydra.nixos.org/job/nixos/release-20.03/tested
<LnL> \o/
claudiii has quit [Quit: Connection closed for inactivity]
<ris> damn i should pay more attention - could've got some final bumps in for my packages
<ris> ah well
domenkozar[m] has quit [Quit: killed]
thefloweringash has quit [Quit: killed]
tokudan[m] has quit [Quit: killed]
jonge[m] has quit [Quit: killed]
arcnmx has quit [Quit: killed]
roberth has quit [Quit: killed]
bennofs[m] has quit [Quit: killed]
Nyanloutre[m] has quit [Quit: killed]
alienpirate5 has quit [Quit: killed]
ma27[m] has quit [Quit: killed]
mkg20001 has quit [Quit: killed]
rycee has quit [Quit: killed]
rnhmjoj has quit [Quit: killed]
aanderse has quit [Quit: killed]
layus[m] has quit [Quit: killed]
vaibhavsagar has quit [Quit: killed]
jtojnar has quit [Read error: Connection reset by peer]
abbradar[m] has quit [Read error: Connection reset by peer]
worldofpeace has quit [Quit: killed]
timokau[m] has quit [Quit: killed]
<ris> i'll put a note in my calendar for august 1st
pkolloch[m] has quit [Quit: killed]
claudiii has joined #nixos-dev
psyanticy has quit [Quit: Connection closed for inactivity]
pkolloch[m] has joined #nixos-dev
obadz has quit [Quit: WeeChat 2.7]
obadz has joined #nixos-dev
teto has quit [Quit: WeeChat 2.7]
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
{^_^} has quit [Excess Flood]
{^_^} has joined #nixos-dev
ixxie has quit [Ping timeout: 265 seconds]
vaibhavsagar has joined #nixos-dev
timokau[m] has joined #nixos-dev
Ox4A6F has joined #nixos-dev
colemickens has joined #nixos-dev
mkg20001 has joined #nixos-dev
worldofpeace has joined #nixos-dev
tokudan[m] has joined #nixos-dev
thefloweringash has joined #nixos-dev
rnhmjoj has joined #nixos-dev
emily has joined #nixos-dev
rycee has joined #nixos-dev
dtz has joined #nixos-dev
arcnmx has joined #nixos-dev
abbradar[m] has joined #nixos-dev
domenkozar[m] has joined #nixos-dev
alienpirate5 has joined #nixos-dev
aanderse has joined #nixos-dev
ma27[m] has joined #nixos-dev
layus[m] has joined #nixos-dev
Ericson2314 has joined #nixos-dev
roberth has joined #nixos-dev
jonge[m] has joined #nixos-dev
jtojnar has joined #nixos-dev
bennofs[m] has joined #nixos-dev
Nyanloutre[m] has joined #nixos-dev
drakonis has joined #nixos-dev
<worldofpeace> ✨
<worldofpeace> So everything should be sorted?
<gchristensen> I don't want to jynx it but ........ yes?
<worldofpeace> I swoon 😻️
<gchristensen> and indeed we may even be far better off than before: able to add aarch64 to the standard channel
<samueldr> noting here: the tested jobset already had aarch64-linux in, so this brings noething more that could stop channel upgrades
<samueldr> what it brings is more in-sync channels
<{^_^}> #80379 (by worldofpeace, 51 seconds ago, open): ZERO Hydra Failures 20.03
<samueldr> worldofpeace: typo's could be construed as a misuse of the apostrophe
<samueldr> though its use could be slowly evolving as a "hey look 'ma, here's a plural coming"
<samueldr> though I figure you mean in the ZHF issue :)
__monty__ has quit [Quit: leaving]
<worldofpeace> lol
<samueldr> worldofpeace: weird thing, you link to a nixpkgs fork rather to nixpkgs proper for the utility flag
<worldofpeace> samueldr: can we update the topic with https://discourse.nixos.org/t/nixos-20-03-beta/5935 ?
<worldofpeace> hmm, will fix that
<samueldr> gchristensen can help you/us with that
<samueldr> if you're still too matrix'd for it, I can
worldofpeace_ has joined #nixos-dev
<worldofpeace_> samueldr: hehe, on real IRC.
<worldofpeace_> gchristensen: hey, 👋️
drakonis has quit [Ping timeout: 240 seconds]
<gchristensen> worldofpeace_: you got it :)
<gchristensen> oops
<samueldr> lol
<gchristensen> okay now you got it, worldofpeace_ x)
worldofpeace_ changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS 20.03 BETA Announced https://discourse.nixos.org/t/nixos-20-03-beta/5935 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 19.09 RMs: disasm, sphalerite; 20.03: worldofpeace, disasm | https://logs.nix.samueldr.com/nixos-dev
<gchristensen> worldofpeace_++
<{^_^}> worldofpeace_'s karma got increased to 2
<worldofpeace_> alt worldofpeace is out 👋️
worldofpeace_ has quit [Quit: worldofpeace_]
<flokli> worldofpeace++
<{^_^}> worldofpeace's karma got increased to 66
* gchristensen wonders off for some nosh
<worldofpeace> Thanks gchristensen flokli ❤️
<gchristensen> s/o/a/ # definitely need that nosh
<worldofpeace> I've actually never heard of the word nosh ever, thanks merriam webster
<jtojnar> https://en.wiktionary.org/wiki/nosh has also meaning from Polari
claudiii has quit [Quit: Connection closed for inactivity]
<infinisil> Hm, currently debugging https://hydra.nixos.org/build/113074265, which I can't reproduce locally
<{^_^}> crystal-lang/crystal#8609 (by lugia-kun, 8 weeks ago, closed): Spec fails in Colorize if stdout is not TTY
<infinisil> And I can apply the patch to fix it, but what worries me is that there's an impurity regarding the tty somehow
<infinisil> Like, does Nix locally expose a tty to the builder which won't be present on hydra?
<samueldr> I seem to recall something about ttys changing in nix
<worldofpeace> Jan Tojnar: 🤣
<worldofpeace> Jan Tojnar: But I'm certain he didn't mean this
<infinisil> samueldr: Hm there's https://github.com/NixOS/nix/pull/2878
<{^_^}> nix#2878 (by edolstra, 39 weeks ago, merged): Run builds in a pseudo-terminal
<jtojnar> worldofpeace I wonder if the name of GNOME’s IRC client is related
<infinisil> Maybe hydra is using an older version and I'm using a newer one?
<gchristensen> unlikely
<gchristensen> t4a ... let me see.
<jtojnar> infinisil sounds similar to https://github.com/ostreedev/ostree/issues/1592
<{^_^}> ostreedev/ostree#1592 (by jtojnar, 1 year ago, open): installed tests fail with pseudoterminal active
<worldofpeace> Jan Tojnar: Yeah, I'm not sure I get why they called that app this.
<gchristensen> ehhhhh that machine might be. I can't look at it further right yet.
<jtojnar> worldofpeace well it would make sense Polari (or alternatively Parlare, Parlary, Palare, Palarie, Palari; from Italian parlare, meaning 'to talk')
<infinisil> gchristensen: Hm thanks, guess I'll just apply the fix and not waste much more time on this
<gchristensen> infinisil: or just restart it
<gchristensen> almost every machine is running a very very new nix, but it was unlucky
<infinisil> There's an upstream fix, I'd rather apply that
<samueldr> do we actually need the fix? is that nix version still supported? should we?
<infinisil> Looks like it was indeed an unlucky machine: https://hydra.nixos.org/job/nixos/release-20.03/nixpkgs.crystal.x86_64-linux
<infinisil> Well, might
<samueldr> though, considering it's an upstream fix it's much less of an issue though if it was added
<gchristensen> that nix version is almost certainly not supported
<samueldr> do we have guidelines about nix support once a new version is released?
<infinisil> There is lib/minver.nix
<infinisil> I think for the whole placeholder thing we waited at least 2 NixOS releases to not cause too much problems for people updating multiple versions
<samueldr> here I was thinking raw nix, not related to nixpkgs
<samueldr> not nix usage in nixpkgs
<infinisil> "raw nix"?
<samueldr> just nix
<samueldr> what's the expectations for "support" about previous releasss