infinisil changed the topic of #nix-lang to: Channel for discussing Nix as a language - - Logs:
<qyliss> are you concerned that it means that a function that gets passed an attrset doesn't know if accessing one of its names will be lazy?
<infinisil> Nah that's no problem, the implementation will be able to handle that
<qyliss> no I mean like, is that not going to be confusing to the Nix programmer?
<infinisil> Ah
<infinisil> Hmm I can't see how this would be confusing
<infinisil> It's the declaration site of `<expr> // <expr>` that needs to decide whether it wants it to be lazy or not
<qyliss> I'm just concerned about adding more edge cases to Nix
<qyliss> there are already so many
<infinisil> And the choice of laziness is probably then made with a reason, namely that e.g. the left side might not be evaluated
<infinisil> Tbh the evaluated is pretty clean
<infinisil> evaluator*
<qyliss> I think puck would disagree
<infinisil> Oh yeah I guess there are some naughty features
<infinisil> Oh and bugs..
<qyliss> some of which are unfixable!
<qyliss> so making the language even more complicated and edge-casey for a 5% performance gain is scary to me
<qyliss> like, with this change there'll be two different types of attribute set names
<infinisil> Oh there's no 5% performance gain, it's a loss of 5%
<infinisil> If it were enabled by default at least
<qyliss> yeah, that's what I meant
<qyliss> much of a muchness
<qyliss> and so there will almost certainly be edge cases resulting from the different types of attribute names
<infinisil> But I'm 100% certain that this feature is sound and pure
<qyliss> I'm sure Eelco was when he wrote lots of the rest of it :P
<infinisil> And in fact, the implementation is so clean now that it's easy to also add a lazy `builtins.lazyElemAt`
<qyliss> I think this is great work, btw!
<qyliss> I just think a small performance hit is not worth making the language even more complicated than it already is
<infinisil> Oh well the performance hit doesn't exist if it's just opt-in with builtins.lazyAttrsUpdate
<qyliss> that's my point
<infinisil> Huh
<infinisil> Not sure I get your point
<qyliss> making it opt in makes the language more complicated to avoid a small performance hit
<qyliss> I am doubtful that will be worth it in the long term
<infinisil> Oh, so I'm implementing this because I need it
<qyliss> yes
<qyliss> I think it's a good feature
<qyliss> I think it should just be how all attrsets work
<infinisil> Ah so you're arguing for making it the default?
<qyliss> Yes
<infinisil> Got it
<infinisil> Yeah tbh I'd love to make it the default
<qyliss> Sorry, could I have been clearer?
<infinisil> No worries, maybe yeah :)
<qyliss> if you can identify how I'd love to know :)
<qyliss> sometimes I don't communicate as well as I think I do
<qyliss> actually reading back I think I see it :)
<infinisil> Hehe alright
<infinisil> But yeah, I just don't know how the 5% performance penalty could be removed
<qyliss> I'd happily take a 5% performance hit for a simpler language (and presumably a more smaller and therefore more maintainable codebase)
<infinisil> And actually I believe this is just an inherent property of this feature
<infinisil> Making this opt-in with a builtin only adds about as much code as to define a builtin, not much else
<qyliss> there don't have to be two different attrset lookup paths?
<infinisil> Not entirely sure what that means, but the answer is probably "no" :)
<infinisil> I could explain how it works in some detail if you're interested
<qyliss> please do!
<infinisil> qyliss: Okay so the basics of it is that previously there was the Expr::eval function, which evaluated an *expression* into weak head normal form
<infinisil> And this function was called at most once on every expression (in a given environment)
<infinisil> That's because as soon as it's evaluated, you don't have an expression anymore but a *value*
<infinisil> So previously `throw "" // { x = 0; }` would evaluate the whole thing into a single value, of type "attribute set" (tAttrs)
<infinisil> But with lazy attribute names it works a bit differently: You can evaluate an expression multiple times, each time getting a different attribute from the result
<infinisil> Without evaluating the whole thing
<qyliss> that's interesting
<infinisil> So the latest implementation uses an *EvaluationStrategy* for this, which determines how values get evaluated, with support for early exit
<qyliss> is each attribute name evaluated only once?
kalbasit has joined #nix-lang
<infinisil> Yeah, every variable in Nix is a thunk, and once that's evaluated, it's stored for the future
<infinisil> attribute values are thunks too
<infinisil> So now in the latest version there's two evaluation strategies: A `ForceEvalStrategy` and an `AttrEvalStrategy`
<infinisil> These are objects which are passed under the common superclass of `EvalStrategy` into the `eval` function (which is renamed to `evalWithStrategy`)
<infinisil> Currently such an evaluation strategy only has one method: `handleAttrs`, which gets called when the result of the evaluation is an attribute set
<infinisil> In addition to setting object fields, these strategies can also return whether they're "done"
<infinisil> Aka it can terminate early
<infinisil> So e.g. the ForceEvalStrategy always returns false, because it can never terminate early
<infinisil> Whereas the AttrEvalStrategy terminates early if it can find the attribute it's looking for
<qyliss> How does looking up an attribute name work?
<infinisil> The above code evaluates the right side of an // with the strategy, and if it's not done yet, evaluates the left side
<infinisil> Which calls Attr * attr = e->evalAttr(state, env, vTmp, name); in line 1170, which does the actual lookup of a name
<qyliss> attr lookup is logarithmic, right? so that must require evaluating multiple names?
<infinisil> And the definition of evalAttr is just a simple wrapper around `e->evalWithStrategy`:
<infinisil> qyliss: It's logarithmic yeah, but for `let attrs = { a = null; b = null; } // { c = null; d = null; }; in attrs.d`, it only needs to do the logarithmic lookup in the { c, d} set
<infinisil> If you do `attrs.a`, it first does the algorithmic lookup on the right side, then on the left, then it notices that both sides are fully evaluated already, so it combines them into `{ a = ; b = ; c = ; d = ; }` for faster lookup in the future
<infinisil> Oh also what's really neat: We can introduce arbitrary other evaluation strategies
<infinisil> Another one would be lazy list indexing
<infinisil> And another cool one might be type lookup without actually evaluating the expressions
<puck> i'd be interested in some improvements, but it's a lil annoying that this needs weird tricks to make it work?
<infinisil> puck: The current implementation is by far the least weird of all my other implementation attempts, sooo yeah :P
<infinisil> I think it's actually really beautiful
<puck> having this be a special case on the way that // is parsed is annoying
<infinisil> It's not parsed differently though
<puck> well, it's evaluated differently
<infinisil> Yea
<infinisil> But as said before, I want it to make opt-in instead, so you have to use `builtins.lazyAttrsUpdate` to get that behavior
<puck> my test implementation of this (before i realised nix doesn't even implement this optimization) was basically just an item in enum Value { PartialAttrset(HashMap<String, Value>) }
<infinisil> Hmm interesting
<puck> but it also uses a way different architecture for the evaluator, which doesn't have infinite recursion issues
<infinisil> There are infinite recursion issues?
<puck> well, like. some expressions, when evaluated, exhaust the entire stack
<puck> i recently found my code for a prototypal nix evaluator, which has optimizations such as ({ a = 5; b = 7; }).b -> 7
<infinisil> Ah well then it's not infinite recursion, just a stack overflow :P
<infinisil> puck: So you figured out how to make tail call optimization maybe/
<infinisil> ?
<puck> i had some different ways of fixing this
<infinisil> Can I look at the code somewher?
<puck> one iteration, every single value was a rust Future, which waited on the values it depends on
<infinisil> Oh rust
<puck> and another, more trivially, was just "build a stack of items to evaluate, pop the stack, etc"
<puck> of course
<puck> i'm not going to try writing an evaluator in a language as prone to errors as C++ :p
<infinisil> So one potential problem with your approach is that it can't do infinite recursion detection anymore
<puck> it can
<puck> you just check the stack of items
<infinisil> Hmm..
<puck> and you could still do blackholing, etc
<infinisil> Yeah I guess that works, never mind :)
<infinisil> So is the code online anywhere?
<puck> no, it's all one big mess and very unfinished
<infinisil> I see
<puck> it's just barely capable of doing arithmetic and simple lambdas :p
<puck> it also fixes some fun nix issues in general tho, i had fun writing the latest parser
<puck> > let a = rec { }; a.b = c; c = 5; a.c = 7; in a
<{^_^}> { b = 7; c = 7; }
<puck> or even
<puck> > let a = rec { }; a = { b = c; c = 7; }; c = 5; in a
<{^_^}> { b = 7; c = 7; }
<infinisil> Whack
<gchristensen> why do you do this to me
<puck> infinisil: hrmm, in your patchset, does ({ a = throw "beep"; } // { b = throw "boop"; }).b work as expected?
<puck> iii have a feeling this could probably get me a cool primitive i have been looking for for a while
<infinisil> puck: I'd expect that to boop, which it does yeah
<gchristensen> oh no
<infinisil> But it does that today already
<infinisil> > ({ a = throw "beep"; } // { b = throw "boop"; }).b
<{^_^}> boop
<puck> yeah. that's not quite the form i was looking for
<infinisil> > (throw "beep" // { b = throw "boop"; }).b
<{^_^}> beep
<infinisil> It even boops for this
<puck> no, that's a beep
<infinisil> Well, that's the thing about lazy attribute names though, it doesn't attempt to evaluate left sides if there's a matching one on the right
<puck> infinisil: does this evaluate in your patchset? let a = {"${builtins.toJSON a}" = 1; }; b = {"${builtins.toJSON c}" = 2; }; c = a // b; in c
<infinisil> Let's see..
<infinisil> > let a = {"${builtins.toJSON a}" = 1; }; b = {"${builtins.toJSON c}" = 2; }; c = a // b; in c
<{^_^}> infinite recursion encountered, at undefined position
<infinisil> First here
<infinisil> Nope
<puck> hrm
<infinisil> Same inf rec
<infinisil> Well, it shows a position at least
<puck> hrm, fun
<infinisil> I wouldn't expect that to work though, why would it?
<puck> i didn't have too much hope for this working, but i suspect that i can use this to quantum-lock an attrset with the lazy patch
<infinisil> "quantum-lock"??
<infinisil> Well if you're looking for bugs to abuse, my lazy attribute name patch isn't for you :)
<puck> hold on, i'm doing a local build now :p
<puck> this'll take a few minutes
<infinisil> hehe
<puck> i originally had assumed that i could do a certain trick on nix, but it required more laziness, the lazy name evaluation might just be enough :p
<infinisil> I see :o
<puck> aand the build failed
<infinisil> Did you `nix-build`?
<puck> yes
<puck> test failed
<infinisil> Oh yeah it does that
__monty__ has quit [Quit: leaving]
<puck> doesn't look like your fault, just why would you have tests if they're broken
<infinisil> I did doCheck and doInstallCheck = false in flake.nix
<infinisil> Hm nah I think that might be the PR's fault
<infinisil> Yeah
<infinisil> It's still WIP okay!
<gchristensen> I wish nix had more smol tests
<infinisil> +1 to that
<puck> oh, i see what's up, this doesn't work quite as i thought
<puck> it still evaluates the full right-hand side of the //
<infinisil> It evaluates it as far as necessary
<puck> not afaict
<infinisil> Ah, I guess you mean that all attribute names of a { ... } are evaluated at once
<puck> yes
<infinisil> Yeah that's not done in my PR
<puck> that's what i thought this added :p
<infinisil> Yee nope, and I think that would probably be an anti-feature
<puck> eh, it /should/ be equivalent. of course it isn't due to nix but
<puck> (there's a few simple impurities that aren't fixable)
<infinisil> puck: Such as?
<puck> this allows creating objects that, when called multiple times with the same input, output different values
<infinisil> What's the underlying impurity?
<puck> builtins.toFile and builtins.pathExists
<infinisil> Hmm..
<puck> this works decently in normal nix, but if ran in pure mode really shines, as it actually abuses the "allowed paths" mechanism
<puck> i used this to build multiplayer rock-paper-scissors that can be played by two people on the same machine, against a nix store
<infinisil> Noice
<puck> and this allows for storing information inbetween nix evaluations
<gchristensen> could we delete enough features to fix it?
<infinisil> I'm thinking we need to move builtins.toFile "nearer" to the build phase
<infinisil> So that evaluation itself can't influence the store
<gchristensen> sounds good
<gchristensen> I wish I could `nix-build` an expression which is just a path
<infinisil> Oh yeah
<gchristensen> I feel that really spoils some of the transparency
<infinisil> paths all around are bit odd
kalbasit has quit [Ping timeout: 256 seconds]
tilpner_ has joined #nix-lang
ris has quit [Ping timeout: 240 seconds]
tilpner has quit [Ping timeout: 246 seconds]
tilpner_ is now known as tilpner
<infinisil> qyliss: Oooo I just made an optimization that might be pretty good
<infinisil> For lazy attribute names
<infinisil> Oh jeez it's already 3am
<qyliss> :DDD
<qyliss> I'm making 3am pasta
<qyliss> which is a problem because I have a busy day tomorrow
<infinisil> Oh nice, I just had some pasta too like an hour ago
<infinisil> qyliss: It be saturday though!
<qyliss> every day is the same to me :p
<infinisil> I see..
<infinisil> Well darn, it seems that this made it *slower*
<infinisil> ...?!
<infinisil> Another try..
<qyliss> oh nooo
ekleog has quit [Ping timeout: 244 seconds]
ekleog_ has joined #nix-lang
MichaelRaskin has quit [Ping timeout: 240 seconds]
clever_ has joined #nix-lang
clever has quit [Ping timeout: 240 seconds]
MichaelRaskin has joined #nix-lang
tilpner_ has joined #nix-lang
tilpner has quit [Ping timeout: 260 seconds]
tilpner_ is now known as tilpner
<MichaelRaskin> I thought the solution was to block pathExists on store?
<MichaelRaskin> Just moving toFile closer to build sounds like it won't help between-evaluation communication, no?
clever_ has joined #nix-lang
clever_ has quit [Changing host]
clever_ is now known as clever
clever_ has joined #nix-lang
clever_ has joined #nix-lang
clever_ has quit [Changing host]
clever has quit [Disconnected by services]
clever_ is now known as clever
__monty__ has joined #nix-lang
cirno-999 has quit [Ping timeout: 272 seconds]
cirno-999 has joined #nix-lang
ris has joined #nix-lang
kalbasit has joined #nix-lang
kalbasit has quit [Ping timeout: 240 seconds]
nf has quit [Quit: Fairfarren.]
nf has joined #nix-lang
siraben has quit [Ping timeout: 244 seconds]
jtojnar has quit [Ping timeout: 244 seconds]
jeanete70 has quit [K-Lined]
colemickens has quit [Ping timeout: 260 seconds]
l33[m] has quit [Ping timeout: 246 seconds]
JJJollyjim has quit [Ping timeout: 244 seconds]
l33[m] has joined #nix-lang
jtojnar has joined #nix-lang
siraben has joined #nix-lang
evanjs has quit [Quit: ZNC 1.8.2 -]
evanjs has joined #nix-lang
colemickens has joined #nix-lang
rmdashrf has quit [Quit: ZNC 1.8.2+deb1~bpo10+1 -]
evanjs has quit [Read error: Connection reset by peer]
evanjs has joined #nix-lang
__monty__ has quit [Quit: leaving]
evanjs has quit [Quit: ZNC 1.8.2 -]
evanjs has joined #nix-lang