gchristensen changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | 18.09 release managers: vcunat and samueldr | https://logs.nix.samueldr.com/nixos-dev
drakonis has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
<gchristensen> builds that don't bit-by-bit reproduce, are they idempotent?
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
lezed1 has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
lassulus_ has joined #nixos-dev
lassulus has quit [Ping timeout: 250 seconds]
lassulus_ is now known as lassulus
pie___ has joined #nixos-dev
pie__ has quit [Ping timeout: 245 seconds]
lezed1 has quit [Quit: Connection closed for inactivity]
<simpson> Does anybody else happen to care about pypyPackages? pypyPackages.mock has been unhappy for a while due to a failing test, and I'm wondering whether `doCheck = false;` is appropriate on such a central library.
<simpson> The irony of tests failing on a testing library *is* pretty delicious.
<infinisil> simpson: why are they failing?
<simpson> > pypy
<{^_^}> "<derivation /nix/store/caycq9bgpr44pv3xfnzzy41d6xpp36n3-pypy-6.0.0.drv>"
<simpson> > pypy.isPyPy
<{^_^}> false
<simpson> infinisil: I will explain more in my commit, likely, but the short version is that the test is known broken and that upstream has it patched out on their master branch.
<simpson> Given that this package tracks a 3-to-2 stdlib backport, it is *very* unlikely that we'll get upstream to release a new version just for this. So I'll leave a comment explaining that we shouldn't do the checks on PyPy for this version.
phreedom has quit [Ping timeout: 256 seconds]
phreedom has joined #nixos-dev
<infinisil> That sounds reasonable
<simpson> infinisil: ##55182
<simpson> #55182
<{^_^}> https://github.com/NixOS/nixpkgs/pull/55182 (by MostAwesomeDude, 1 minute ago, open): Unbreak many PyPy packages
__Sander__ has joined #nixos-dev
hedning has joined #nixos-dev
domenkozar has joined #nixos-dev
init_6 has joined #nixos-dev
init_6 has quit []
orivej has joined #nixos-dev
shlevy has joined #nixos-dev
<shlevy> clever: gchristensen reached out about #2665... How do you expect to reuse the value object exactly? Once created Values are pure from the perspective of the language (thunk forcing is unobservable as long as it's not bottom), but you'd need to force the thunk in a different context for different calls to the fn right?
<{^_^}> https://github.com/NixOS/nixpkgs/issues/2665 (by third3ye, 4 years ago, closed): Creating assets and a toolchain for building the NixOS wiki via the github repo
<clever> shlevy: one sec
<clever> { args1 }: { args2 }: let expr = 5*5; in body
<clever> { args1 }: let expr = 5*5; { args2 }: body
<shlevy> Sure
<clever> shlevy: in the 1st case, when the ExptLet is being evaled, it will maybeThunk() the 5*5, creating a thunk for that ExprApp
<clever> and if you run the 2nd lambda 200 times, that creates 200 thunks
<clever> which each have to re-compute 5*5
<shlevy> Yes
<Synthetica> clever: There is a check for that in nix-linter
<clever> in the 2nd example, it can reuse the result, when calling the 2nd lambda repeatedly
<shlevy> But unless we're doing static analysis for references to arg2 (which is feasible but not trivial) we can't reuse it
<shlevy> I get the difference. I'm just not clear on what you're proposing we do
<clever> add a std::map to Expr, that will map from Env* to Value*
<shlevy> By pointer equality?
<clever> so, when maybeThunk is creating a thunk for a given Env, it can possibly return an old Value for the same thunk
<clever> yeah, pointer equality seems simplest for a first try
<shlevy> How do you want to identify "equivalent" Env*s?
<clever> oh, hmmm, i can see an issue there now
<clever> the Env* will include the args2 key
<shlevy> I'd expect the Env to be different at each call
<shlevy> Right
<clever> and will be different for each eval
<shlevy> And even if you called it with the exact same value it'd be different
<shlevy> qua pointer
<clever> yeah, it would be a new (but identical) Env
<clever> you would need to analyze what the Expr is accessing, and what parts of the Env it cares about
<shlevy> builtins.memoise (on an experimental branch) could cover the same-args case, static analysis could identify opportunities for lifting
<clever> Synthetica: where is that linter?
<shlevy> But if nix-linter is already doing that analysis then let's see if there's even a big win possible first :) FWIW I already manually do this transformation when I write expressions, I'm not sure how many others do though
<clever> shlevy: oh, but let lifting wouldnt actually have found this, in its original form
<clever> originally, it was in the form of
<clever> configureFlags = [ ... ] ++ lib.optionals (expr) map (arg: "--hsc2hs-option=" + arg) ["--cross-compile" "--via-asm"];
<clever> i initially put it into a let block one step up, but it didnt solve it, moving it another scope further up (and fixing the import of this file to do the same), is what shaved 2gig off the heap usage
<clever> Synthetica: could the linter have found this?
<shlevy> clever: If [ ... ] is constant, it should be able to find it yeah
<shlevy> You can lift that to the very top of the file before you even define the top-level function
<shlevy> Then Nix's import caching will cover you
<clever> the constant part, was the `map (arg: "--hsc2hs-option=" + arg) ["--cross-compile" "--via-asm"]
<clever> that map only has to be ran once
<shlevy> Oh, right
<clever> but, NIX_COUNT_CALLS reveals the fn for map was ran 20,000 times
<shlevy> Yeah you'd need subexpression lifting there
<shlevy> Which I'd expect to be too expensive to run every eval
<shlevy> Much better for a linter
<clever> i had to lift it several steps up the scope chain, and repeat the same in another file, to even benefit from it
<clever> default.nix was doing the same thing, args1: arg2: let foo = import ./comp-builder.nix arg1; ...
<shlevy> clever: If you lifted it above even thetop level { stdenv, ... } I'd expect you to be able to contain it to one file
<clever> so even when comp-builder.nix was 100% fixed, a new Value was being made for every call to the outer lambda
<shlevy> You'd be fine then because import *does* cache values by filename (as long as you're not using scopedImport)
<clever> ah, yeah, in this exact case, the map is 100% pure, and relies only on the global scope, so it could have been pushed even higher up
<shlevy> So the top-level let would be reused
<clever> import caches the Expr, not the Value
<shlevy> Ah really?
<clever> i believe
<clever> let me double-check...
<shlevy> That's annoying
<shlevy> You're always going to eval it right after the import...
<shlevy> Even if that's just as a maybeThunk
<clever> i think the problem, is that Value's dont exist within the Expr tree
<shlevy> They don't need to
<clever> Expr * e = state.parseExprFromFile(resolveExprPath(realPath), staticEnv);
<shlevy> Right
<shlevy> Annoying
<shlevy> There's no reason we couldn't cache that at the value level
<shlevy> Not sure how much we'd gain, depends on how many exprs could have a top-level self-contained expr
<clever> you cease to gain anything, once you need something in lib for that const-expr
<shlevy> Sure
<shlevy> I'd expect flakes to be a good time to fix this
<clever> hmmm, where is that import cache again...
<clever> i dont think its inside parseExprFromFile
<shlevy> It's in parceExprFromFile I think
<shlevy> Or traceable from there for sure
<clever> doesnt look like it
<clever> oh, i see the problem
<clever> parseExprFromFile is for the scopedImport case
<shlevy> Ah right
<clever> state.evalFile is the cache-able case
<shlevy> evalFile does cache the values
<shlevy> I hope
<shlevy> Yep
<shlevy> So we're good there
<clever> so the top-level Value import returns, is in that cache, and when the root node is an ExprLet...
<clever> this eval is only ran once, at import time (non-lazily)
<clever> that will generate a new Env for the let body, and populate it with every key=value in the let block, as maybe-thunks
<clever> and then eval the body in that context
<clever> if it was a: let foo = 5*5; in { args }: ..., that body is an ExprLambda
<clever> which just shoves itself into a Value, along with the Env the let block just made
* gchristensen watches
<clever> everything past that point is an Expr, and wont get much sharing
<shlevy> Right
<shlevy> but you can lift the let foo even higher
<shlevy> And if you *couldn't*, then you wouldn't beable to have this optimization in any case
<clever> shlevy: i think nix-shell and/or nix-build isnt using the cache, when it first imports the root file?
<clever> though thats of less importance, you would potentially have infinite recursion if you ever re-import it
<shlevy> Yeah I mean that's at most 2
<shlevy> imports
<clever> i could also see it slightly delaying the infinite recursion tests
<clever> though that heavily depends on the Expr->Value conversions again
<shlevy> Yeah it definitelywould
<shlevy> Blackholing is by pointer
<clever> shlevy: i also found something slightly weird in another area...
<clever> shlevy: why does one branch use lambda.body->eval, and the other fun.lambda.fun->body->eval ?
<clever> ExprLambda & lambda(*fun.lambda.fun);
<clever> when lambda is a reference to fun.lambda.fun, and so they should be identical?
<shlevy> Yeah looks like lambda was added later as a convenience
<shlevy> Patch would be nice
<shlevy> It *might* somehow impact tail-recursion but I don't think it should
<clever> ah, i can see how it might
<clever> lambda is on the stack, and has to still be on the stack, to de-reference it
<shlevy> It's a reference so it shouldn't impact lifetimes
<shlevy> Once you've de-referenced there's no destructor or anything to worry about
<clever> but, for tail-recursion to work, it would be clearing the stack up before the call, and then doing jmp rather then call
<shlevy> Right
<clever> but it still has to remove that reference from the local stack
<shlevy> But you've already dereferenced by then
<shlevy> The key is that you can (and indeed must) dereference beforethe call
<shlevy> And once you have you don't need it around
<clever> depends on if the compiler can save it into a register while clearing the stack for tail-call or not
<shlevy> It has to save it into a register :P
<shlevy> It's the call target
<shlevy> or the jump target
<clever> but, will the compiler be smart enough to do know it can tail-recurse, and save it to a reg before cleaning the stack? or will it not re-arrange things, and then fail to tail-recurse?
<clever> would need to try it, and then disassemble
<shlevy> clever: It's definitely going to separate the deref from the call and do tail-call analysis after
<shlevy> There's no way gcc and clang devs are leaving this $100 bill on the ground :P
<clever> lol
hedning has quit [Quit: hedning]
<clever> the only way i can see nix optimizing this let thing at runtime, is if an Expr propagates all ExprVar's up the tree
<shlevy> Yeah
<clever> i think ExprVar is the only one that deals with the Env an Expr is ran within?
<shlevy> And constant subexpression lifting isn't always even what you want
<shlevy> The only direct one
<shlevy> But anything containing ExprVars would have the same issue if not also lifted
<clever> let me check the latest profile run...
<clever> "/home/clever/iohk/cardano-nsis/haskell.nix/builder/comp-builder.nix:41:24 null ran 1214672",
<clever> 41 makePairs = map (p: rec { key="${val}"; val=p.components.library; });
<clever> the fn passed to map
<clever> first thing i would change here, is key = val; the "${val}" seems a bit pointless?
<clever> and the key is only used by genericClosure
<clever> wait, does it even need a val? could val just be renamed to key?
<clever> ok, something funky is going on, it later does val.config....
<clever> so how is "${val}" valid?
<shlevy> If it has an outPath or __toString
<clever> yeah, confirming which it is...
<clever> its deep within the options framework...
<clever> simpler to just throw in a trace now...
<clever> trace: /nix/store/pxrzg78cknrgl6nivjm2bx6kmhyi1snp-unix-2.7.2.2-lib-unix
<clever> not sure how, but it evals to this
<clever> which doesnt exist
<clever> but the trace has revealed that it appears to be re-computing this whole thing, many times
<clever> shlevy: yep, val is a derivation with an outPath
<clever> and genericClosure cant compare sets for the key
<clever> so "${val}" is right
<clever> ah, thats also why its not in my store, thats the $out of a derivation it has yet to build
<shlevy> Or val.outPath ;)
<clever> i was more thinking about collapsing val and key into just key
<clever> since they are pointing to the same thing
<clever> but key has to be a string, and val has to be a set
__Sander__ has quit [Ping timeout: 240 seconds]
__Sander__ has joined #nixos-dev
<clever> shlevy: another problem i discovered recently, is getting backtraces in non-error condition
<clever> s
hedning has joined #nixos-dev
hedning has quit [Read error: Connection reset by peer]
hedning has joined #nixos-dev
__Sander__ has quit [Ping timeout: 246 seconds]
__Sander__ has joined #nixos-dev
hedning_ has joined #nixos-dev
hedning_ has quit [Remote host closed the connection]
hedning_ has joined #nixos-dev
hedning has quit [Ping timeout: 250 seconds]
hedning_ is now known as hedning
edef has left #nixos-dev [#nixos-dev]
hedning has quit [Client Quit]
__Sander__ has quit [Quit: Konversation terminated!]
tilpner has quit [Ping timeout: 268 seconds]
tilpner has joined #nixos-dev
WilliButz has quit [Read error: Connection reset by peer]
orivej has quit [Ping timeout: 244 seconds]
tilpner has quit [Quit: WeeChat 2.3]
WilliButz has joined #nixos-dev
tilpner has joined #nixos-dev
drakonis has joined #nixos-dev
tilpner has quit [Quit: WeeChat 2.3]
tilpner has joined #nixos-dev
orivej has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-dev
{^_^} has quit [Remote host closed the connection]
{^_^} has joined #nixos-dev
garbas has joined #nixos-dev
garbas has left #nixos-dev [#nixos-dev]