samueldr changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS 20.03 Feature Freeze Feb 10 https://discourse.nixos.org/t/nixos-20-03-feature-freeze/5655 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 19.09 RMs: disasm, sphalerite; 20.03: worldofpeace, disasm | https://logs.nix.samueldr.com/nixos-dev
drakonis has quit [Quit: WeeChat 2.6]
bhipple has quit [Remote host closed the connection]
bhipple has joined #nixos-dev
tomberek has joined #nixos-dev
justanotheruser has quit [Ping timeout: 260 seconds]
cptchaos83 has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
cptchaos83 has joined #nixos-dev
ris has quit [Ping timeout: 248 seconds]
justanotheruser has joined #nixos-dev
drakonis has joined #nixos-dev
bhipple has quit [Ping timeout: 260 seconds]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 272 seconds]
bhipple has joined #nixos-dev
cptchaos83_ has joined #nixos-dev
cptchaos83 has quit [Ping timeout: 268 seconds]
<samueldr> bhipple: around? about the nix review tool
drakonis has quit [Quit: WeeChat 2.6]
bhipple has quit [Ping timeout: 268 seconds]
bhipple has joined #nixos-dev
noonien has quit [Quit: Connection closed for inactivity]
lovesegfault has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.7]
bhipple has quit [Ping timeout: 268 seconds]
ixxie has joined #nixos-dev
bhipple has joined #nixos-dev
phreedom has quit [Remote host closed the connection]
phreedom has joined #nixos-dev
cole-h has quit [Ping timeout: 265 seconds]
bhipple has quit [Ping timeout: 240 seconds]
bhipple has joined #nixos-dev
bhipple has quit [Ping timeout: 265 seconds]
bhipple has joined #nixos-dev
tomberek has quit [Remote host closed the connection]
__monty__ has joined #nixos-dev
ris has joined #nixos-dev
jared-w has quit [Ping timeout: 245 seconds]
jared-w has joined #nixos-dev
bhipple has quit [Remote host closed the connection]
orivej has joined #nixos-dev
phreedom has quit [Ping timeout: 240 seconds]
phreedom has joined #nixos-dev
<ris> hmmm - am i right in thinking that switching out the stdenv for a python package (thru buildPythonPackage) involves creating a whole separate python toolchain?
__monty__ has quit [Quit: leaving]
<ris> because stdenv gets injected to buildPythonPackage very early, from `python.stdenv`
<ris> the only way i can see of having a buildPythonPackage with a different stdenv is feeding a stdenv-overridden python to `mk-python-derivation.nix`
<ris> which of course will build a whole separate pythonPackages
justan0theruser has quit [Ping timeout: 245 seconds]
orivej has quit [Ping timeout: 268 seconds]
claudiii has joined #nixos-dev
bennofs has quit [Ping timeout: 246 seconds]
averell has quit [Quit: .]
bhipple has joined #nixos-dev
averell has joined #nixos-dev
orivej has joined #nixos-dev
bennofs has joined #nixos-dev
bennofs has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
bennofs has joined #nixos-dev
johnny101m2 has joined #nixos-dev
johnny131 has joined #nixos-dev
johnny101m has quit [Ping timeout: 260 seconds]
johnny101m2 has quit [Ping timeout: 272 seconds]
drakonis has joined #nixos-dev
bennofs has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
bennofs has joined #nixos-dev
cole-h has joined #nixos-dev
__monty__ has joined #nixos-dev
<infinisil> ris: Idea: Let's have a separate repository (or folder) for bootstrap tools. Those expressions should be self-contained (only depending on perhaps nixpkgs/lib)
<ris> not sure i'd know where to start
<infinisil> Then this solves above problem, and it also means we don't have to rebuild everything for every core change
<infinisil> s/core change/change involving deep dependencies like python
<infinisil> I guess once Flakes are a thing this could be very nice
justanotheruser has joined #nixos-dev
bhipple has quit [Ping timeout: 272 seconds]
bhipple has joined #nixos-dev
tomberek has joined #nixos-dev
* ris strokes beard
<ris> you might be right
<simpson> ris: What's wrong with building an entirely fresh Python interpreter? What did you want to change in your stdenv?
<ris> just wanted to build a specific package with clang, that's all
<simpson> Like, yeah, one has to plan one's evening around how many Pythons one is going to build, but that's not too big of a deal in the long run.
<ris> it's fine, i just did a CC= hack in the end
<simpson> Hm, that works? That's surprising; normally I'd expect that Python extension modules need to all be built with roughly the same compiler and libc.
<ris> yeah it's just a slightly absurd restriction that you can't have a python module compiled with gcc with one that needs clang for some reason
<ris> because they both insist on existing in separate python universes
<simpson> I presume it's because of API and ABI fragility in CPython's C API.
<ris> C API is solid
<emily> I mean, clang and gcc are pretty solidly ABI-compatible
<emily> even for C++ I think
<ris> you can interchange C code between clang and gcc freely
<emily> those languages actually have stable platform-defined ABIs
<ris> darwin is another story of course but lets not go there
<emily> CPython's ABI changing based on clang vs. gcc would be kind of scary if anything
<ris> the "ideal world" of having two good compilers is being able to pick and choose the best for each task
<simpson> It's bad enough that (1) we consider it luck if it works, rather than a plan, and (2) if you need to call C code, use cffi; if you need to go faster, use PyPy.
<simpson> (If you need a fast binary made mostly of C, consider a subprocess-oriented design!)
<ris> cffi is less than ideal for some scenarios
<aminechikhaoui> hm what's the right way to set services.hydra-dev.dbi when it's running in the same server w/ no password ?
<emily> i think that's a slightly biased perspective, people deploy native cpython extensions pretty widely/successfully
<emily> the entire data science ecosystem kinda depends on that
<aminechikhaoui> "dbi:Pg:dbname=hydra;host=localhost;user=hydra" doesn't seem to work out of the box
<ris> cffi is really about calling code that you're going to write fully in C-land or possibly already exists and is python-naive... if you're looking to write code that heavily uses/interacts with python objects, it doesn't give you a lot of guidance
<aminechikhaoui> niksnut around by any chance ? :) (see question above)
<ris> (if anyone's interested what i'm working on, https://github.com/risicle/cpytraceafl - i've just got it working for mixed python/c code)
<ris> (still tidying that up for release though)
ixxie has quit [Ping timeout: 272 seconds]
justanotheruser has quit [Ping timeout: 246 seconds]
<niksnut> aminechikhaoui: I think it's just "dbi:Pg:dbname=hydra;user=hydra;" (which is the default)
<aminechikhaoui> yeah but I think I need to add host=/tmp
<aminechikhaoui> let me try that
<aminechikhaoui> after checking the issue is that it's looking for the wrong postgres socket dir
<aminechikhaoui> doing some long overdue update the LB hydra :D
<aminechikhaoui> niksnut yay host=/tmp worked
<aminechikhaoui> which I guess I need to revert when switching to 19.09
<niksnut> hm right
justanotheruser has joined #nixos-dev
CRTified has quit [Quit: Gateway shutdown]
CRTified has joined #nixos-dev
noonien has joined #nixos-dev
<infinisil> Hm, I wonder if nixpkgs evaluation could be made to use less memory by using builtins.seq in specific places
<infinisil> Or deepSeql
<gchristensen> basically the existence of overrides means almost nothing can be garbage collected ever
<gchristensen> one thing which would help is changing «attr».«system» to «system».«attr» since that does allow for entire archs worth of sets to be GC'd
<infinisil> gchristensen: You mean from like `nix-build nixos/release.nix -A manualHTML.x86_64-linux`?
<gchristensen> yea
<infinisil> Hm not getting that, why would `x86_64-linux.manualHTML` allow for more gc?
<infinisil> Other systems aren't evaluated as far as I can see in any case
<samueldr> hydra?
<gchristensen> hydra
<samueldr> instead of having all systems at the leaves, they'd be at the root of the trunk
<samueldr> while not forbidden, I don't think we have a platform referrencing another platform in nixpkgs
<infinisil> Ahh I see now
<samueldr> (though keep your eyes peeled as it could seep through from good intentions, https://nixos.wiki/wiki/Cross_Compiling#Lazy_cross-compiling )
<gchristensen> eee
<gchristensen> shut it down lol
<gchristensen> somebody get me a server with 2T of RAM and 48 cores, hydra needs an upgrade
clever has quit [Ping timeout: 260 seconds]
<infinisil> Idea: Change the implementation of callPackage such that the resulting .override doesn't reference any existing thunks, but it instead reimports the file it was called with
<infinisil> So .override is like actually calling `callPackage` again
clever has joined #nixos-dev
clever has joined #nixos-dev
clever has quit [Changing host]
<Profpatsch> gchristensen: Problem with that is that nobody else is ever going to be able to evaluate anything when hydra assumes 128G of RAM just to do basic tasks …
<Profpatsch> This kills outside contributions.
<gchristensen> nobody has to evaluate like hydra does
<gchristensen> but I agree it isn't good
<Profpatsch> Similar to how nobody ever contributes to Chrome because it’s impossible to work with as a non-googler
<gchristensen> it was a joke
<Profpatsch> You get a caste of people with access to the necessary hardware, and the rest loses interest (or even if they have it, they can’t do anything grass-roots anymore)
<gchristensen> okay
<Profpatsch> Just an observation, not that Nix(OS) is there (yet?)
<gchristensen> it is
<gchristensen> you can't use it on the smallest AWS machines
<Profpatsch> that’s true.
<infinisil> How does Nix's gc work? Does it deallocate stuff immediately once they aren't referenced anymore, ARC style?
<gchristensen> it uses Boehm which is a heap scanning
<gchristensen> which actually causes unrelated problems -- there is a high probability that random points in the heap look like pointers to data which aren't
<clever> GHC has a whole wiki page on how to say what is a pointer and what isnt
<clever> nix doesnt use such logic, and just checks the contents of the entire object for any addr
<gchristensen> yeah well its boehm :P
<infinisil> So it's a tracing GC that runs continuously in a separate thread?
<gchristensen> look up boehm P
<gchristensen> look up boehm*
<infinisil> I am but it's kind of hard to read what it actually does and when..
<clever> ping?
<gchristensen> pong
<clever> my router is crapping itself again
<gchristensen> messy
<clever> Feb 16 21:05:17 router kernel: e1000 0000:04:02.1 enp4s2f1: Detected Tx Unit Hang
<clever> --- 192.168.2.1 ping statistics ---
<clever> 723 packets transmitted, 362 received, 49.9308% packet loss, time 739126ms
<infinisil> I want to figure out how I can test memory usage reasonably
<clever> infinisil: specifically, look at the layout field on the wiki i linked, that is what nix needs to remove false gc roots
<infinisil> Like, do I just run `NIX_SHOW_STATS=1 nix-instantiate -A hello` then look at heapSize?
<clever> infinisil: show-stats can also dump the stats to a file as json
<infinisil> Yeah, but I need to know that I'm looking for the right thing
<clever> 1743 GC_get_heap_usage_safe(&heapSize, 0, 0, 0, &totalBytes);
<clever> 1797 gc.attr("heapSize", heapSize);
<clever> 1798 gc.attr("totalBytes", totalBytes);
<clever> then cross-reference that function in the Boehm docs
<clever> infinisil: line 688-ish, heapSize includes bytes used and bytes unusable due to fragmentation, but excludes bytes that are "free" i believe
<clever> totalBytes, is not a sum of what exists, but a counter, that increases every time you allocate and never goes down
<infinisil> I guess what I'd really need is a "how much memory is the minimum amount necessary for it to have enough"
<clever> heapSize wont give a true answer to that, since it can go down when GC's free stuff
claudiii has quit [Quit: Connection closed for inactivity]
<infinisil> Yeah
<infinisil> So I guess I'll write some small wrapper around some memory limiting thing (https://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process)
<clever> and totalBytes would be how much you would need, to never gc
<infinisil> That bisects some evaluation to find the minimum required
<clever> i think ulimit and the time binary, can give you highwater marks
<infinisil> I like this answer the most: systemd-run --scope -p MemoryLimit=500M <command>
<clever> [clever@amd-nixos:~/apps/nix/src/libexpr]$ /run/current-system/sw/bin/time -v ls
<clever> Maximum resident set size (kbytes): 2528
<clever> that is the highwater mark
<infinisil> clever: Hm, `sudo systemd-run --scope -p MemoryLimit=10M nix-instantiate -A hello` succeeds
<infinisil> But time -v shows Maximum resident set size (kbytes): 65124
<infinisil> I guess it's an upper bound though
<infinisil> I think I'll use the systemd-run thing for bisection
<clever> i think the gc is more aggressive, when its low on ram
<infinisil> Makes sense
<clever> if it has room to breathe, it will let the garbage pile up more, before a gc
<niksnut> fyi there is a precise garbage collector for nix: https://github.com/NixOS/nix/commits/precise-gc
<clever> and then use less cpu overall
<samueldr> what is the state of precise-gc?
<samueldr> any drawbacks? issues?
<gchristensen> niksnut: I'm supposing you've seen the recent eval errors on hydra?
<niksnut> samueldr: it works, it's reasonably fast until it needs to do collection (since the collector is unoptimized and single-core), and it uses less memory
<niksnut> but most of our memory problems are not caused by boehm
<clever> niksnut: i think if the aggregate job was fixed, the restarting would actually be able to do its job better
<samueldr> I was only curious about the status, I didn't really ascribe any issue to the other GC, I don't _know_ that is has issues, so I won't pretend to :)
<samueldr> wondering also if it's expected to see it replace boehm or if it's really only an experiment
__monty__ has quit [Quit: leaving]
andi- has quit [Ping timeout: 240 seconds]
andi- has joined #nixos-dev
cptchaos83_ has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
cptchaos83 has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.7]
drakonis has joined #nixos-dev
<samueldr> restarting?
<samueldr> let me rephrase: should I restart or is there something that can be looked at?
<aminechikhaoui> samueldr there was a kernel OOM of a nix process around the same time, might be related
<samueldr> likely yes
<samueldr> thanks for checking
<samueldr> I'll requeue
<aminechikhaoui> there is also a coredump of the process (well many) but doesn't seem very useful
<worldofpeace> Jan Tojnar: do you think you could respond to https://github.com/flashrom/flashrom/issues/125#issuecomment-586633361 ?
<worldofpeace> (the dev is super super willing to get it fixed for us, it appears)