infinisil changed the topic of #nix-lang to: Channel for discussing Nix as a language - https://nixos.org/nix/manual/#chap-writing-nix-expressions - Logs: https://logs.nix.samueldr.com/nix-lang/
rajivr has joined #nix-lang
mpickering has quit [Ping timeout: 268 seconds]
mpickering has joined #nix-lang
__monty__ has joined #nix-lang
ddellacosta has joined #nix-lang
ddellacosta has quit [Ping timeout: 246 seconds]
Profpatsch has joined #nix-lang
<Profpatsch> okay, you got me
<infinisil> o/
<infinisil> > :v mergeAttrsWith
<{^_^}> mergeAttrsWith = op: a: b: a // b // intersectAttrsWith op a b
<infinisil> > :v intersectAttrsWith
<{^_^}> intersectAttrsWith = op: a: b: lib.mapAttrs (n: bv: op a.${n} bv) (builtins.intersectAttrs a b)
<infinisil> This is what I wrote recently
<infinisil> Those take an `op :: Value -> Value -> Value` which is called on collisions
<infinisil> Could put a `throw` in there
<Profpatsch> what does intersectAttrs do?
<Profpatsch> Just an intersect?
<infinisil> What you'd expect, intersects attribute sets, preferring the right side on collisions
<Profpatsch> mergeAttrsWith looks like it’s doing a bit more work than necessary
<infinisil> Hmm
<infinisil> I'm not sure if it can be improved
<Profpatsch> infinisil: do attrsets force WHNF on values?
<infinisil> Nope
<infinisil> > builtins.seq { x = throw ""; } null
<{^_^}> null
<Profpatsch> > builtins.seq { "${throw ""}" = 32; } null
<Profpatsch> > builtins.seq { "${throw "foo"}" = 32; } null
<{^_^}> foo
<infinisil> Hehe, guess it doesn't show empty errors
kalbasit has joined #nix-lang
<Profpatsch> how about
<Profpatsch> > mergeAttrsUnique = left: right:
<{^_^}> error: syntax error, unexpected ';', at (string):234:33
<Profpatsch> let intersect = lib.intersectLists (lib.attrNames left) (lib.attrNames right); in
<Profpatsch> assert
<Profpatsch> lib.assertMsg (intersect == [])
<Profpatsch> (lib.concatStringsSep "\n" [
<Profpatsch> "mergeAttrsUnique: keys in attrset overlapping:"
<Profpatsch> "left: ${lib.generators.toPretty {} (lib.getAttrs intersect left)}"
<Profpatsch> "right: ${lib.generators.toPretty {} (lib.getAttrs intersect right)}"
<Profpatsch> ]);
<Profpatsch> left // right;
<Profpatsch> lol
<Profpatsch> > mergeAttrsUnique = left: right: let intersect = lib.intersectLists (lib.attrNames left) (lib.attrNames right); in assert lib.assertMsg (intersect == []) (lib.concatStringsSep "\n" ["mergeAttrsUnique: keys in attrset overlapping:" "left: ${lib.generators.toPretty {} (lib.getAttrs intersect left)}" "right: ${lib.generators.toPretty {} (lib.getAttrs intersect right)}"]); left // right;
<{^_^}> error: syntax error, unexpected ';', at (string):234:387
<Profpatsch> > assert true; "foo"
<{^_^}> "foo"
<Profpatsch> > x = assert true; "foo"
<{^_^}> x defined
<Profpatsch> > mergeAttrsUnique = left: right: let intersect = lib.intersectLists (lib.attrNames left) (lib.attrNames right); in assert lib.assertMsg (intersect == []) (lib.concatStringsSep "\n" ["mergeAttrsUnique: keys in attrset overlapping:" "left: ${lib.generators.toPretty {} (lib.getAttrs intersect left)}" "right: ${lib.generators.toPretty {} (lib.getAttrs intersect right)}"]); left // right
<{^_^}> mergeAttrsUnique defined
<Profpatsch> mergeAttrsUnique { foo = "bar"; } { foo = 42; }
<Profpatsch> >mergeAttrsUnique { foo = "bar"; } { foo = 42; }
<Profpatsch> > mergeAttrsUnique { foo = "bar"; } { foo = 42; }
<{^_^}> assertion (((lib).assertMsg (intersect == [ ])) (((lib).concatStringsSep "\n") [ ("mergeAttrsUnique: keys in attrset overlapping:") (("left: " + (((lib).generators.toPretty { }) (((lib).getAttrs...
<Profpatsch> doesn’t print the trace
<Profpatsch> > mergeAttrsUnique { foo = "bar"; } { bar = 42; }
<{^_^}> { bar = 42; foo = "bar"; }
<infinisil> Profpatsch: Seems like a subset of mergeAttrsWith behavior
<Profpatsch> infinisil: yeah, once that lands it could be refactored
<infinisil> And with O(n^2) runtime
<infinisil> (the lib.intersectLists)
<Profpatsch> O(nm)
<infinisil> Yea
<Profpatsch> I think it would make sense to annotate all runtimes and check whether there’s any low-hanging fruit
<Profpatsch> Esp the attrsets stuff
<Profpatsch> free real estate
<infinisil> Lol
<infinisil> All the tail's in attrsets.nix would be nice to avoid
<Profpatsch> some of the stuff in deprecated.nix is horrible lol
<Profpatsch> uhhh, tail is O(n)?
<infinisil> Yeah
<Profpatsch> ouch!
<Profpatsch> infinisil: I said in another channel that a lazy tuple might be a good idea
<Profpatsch> builtins.tuple { _1 = …; _2 = … }
<infinisil> That's just a list :)
<Profpatsch> which translates to a space-efficient repr in newer nixes
<Profpatsch> Yeah, and then we could build actually lazy cons-lists
<Profpatsch> which are somewhat nice in their space-usage as long as you don’t do … stuff
<infinisil> We can already
<infinisil> One change to Nix I want to make soon is allow .<number> for accessing list elements, which will make this nicer
<Profpatsch> well, it’s an optimization for { car = …; cdr = … } which I’d guess uses some space to save the labels
<infinisil> > [0 1].0
<{^_^}> attempt to call something which is not a function but a list, at (string):459:1
<Profpatsch> infinisil: but that evaluates the cells to WHNF
<infinisil> Nah
<infinisil> > builtins.seq [ (throw "nope") ] null
<{^_^}> null
<Profpatsch> It doesn’t?
<infinisil> Nope, lists are lazy in elements
<Profpatsch> > builtins.length [ (throw "nope") ] null
<{^_^}> attempt to call something which is not a function but an integer, at (string):459:1
<Profpatsch> > builtins.length [ (throw "nope") ]
<{^_^}> 1
<Profpatsch> puck: you lied to me!!! :D
<infinisil> :o
<Profpatsch> we have stuff like recursiveUpdateUntil which is used in only two places in nixpkgs
<Profpatsch> lol, it actually has a release note for 18.09 because it was buggy
<infinisil> > fastnixarray = import <nixbotlib/fastnixarray>
<{^_^}> fastnixarray defined
<infinisil> > fastnixarray
<{^_^}> { count = 4; emptyArray = <CODE>; foo = <CODE>; get = <CODE>; lib = <CODE>; measure = <CODE>; seqListElems = <CODE>; set = <CODE>; test = <CODE>; toArray = <CODE>; toList = <CODE>; }
<infinisil> > fastnixarray.toArray (lib.range 0 100)
<{^_^}> { capacity = <CODE>; contents = [ [ [ [ 0 1 2 3 ] [ 4 5 6 7 ] [ 8 9 10 11 ] [ 12 13 14 15 ] ] [ [ 16 17 18 19 ] [ 20 21 22 23 ] [ 24 25 26 27 ] [ 28 29 30 31 ] ] [ [ 32 33 34 35 ] [ 36 37 38 39 ] [ 40...
<infinisil> > fastnixarray.toArray (lib.range 0 10)
<{^_^}> { capacity = <CODE>; contents = [ [ 0 1 2 3 ] [ 4 5 6 7 ] [ 8 9 10 null ] [ ] ]; empty = <CODE>; }
<infinisil> Using lists for this as well :)
<infinisil> > set 1000 "foo" (fastnixarray.toArray (lib.range 0 10))
<{^_^}> undefined variable 'set' at (string):460:1
<infinisil> > fastnixarray.set 1000 "foo" (fastnixarray.toArray (lib.range 0 10))
<{^_^}> { capacity = <CODE>; contents = [ [ <CODE> ] [ ] [ ] [ [ ] [ ] [ ] [ [ ] [ ] [ [ ] [ ] [ "foo" null null null ] [ ] ] [ ] ] ] ]; empty = <CODE>; }
<infinisil> > :p fastnixarray.set 1000 "foo" (fastnixarray.toArray (lib.range 0 10))
<{^_^}> { capacity = 1024; contents = [ [ [ [ [ 0 1 2 3 ] [ 4 5 6 7 ] [ 8 9 10 null ] [ ] ] ] ] [ ] [ ] [ [ ] [ ] [ ] [ [ ] [ ] [ [ ] [ ] [ "foo" null null null ] [ ] ] [ ] ] ] ]; empty = null; }
<infinisil> > :p fastnixarray.set 10000 "foo" (fastnixarray.toArray (lib.range 0 10))
<{^_^}> { capacity = 16384; contents = [ [ [ [ [ [ [ 0 1 2 3 ] [ 4 5 6 7 ] [ 8 9 10 null ] [ ] ] ] ] ] ] [ ] [ [ ] [ [ ] [ ] [ ] [ [ [ ] [ [ "foo" null null null ] [ ] [ ] [ ] ] [ ] [ ] ] [ ] [ ] [ ] ] ] [ ] ...
kalbasit_ has joined #nix-lang
rajivr has quit [Quit: Connection closed for inactivity]
kalbasit has quit [Ping timeout: 256 seconds]
kalbasit_ has quit [Ping timeout: 256 seconds]
kalbasit has joined #nix-lang
kalbasit has quit [Remote host closed the connection]
kalbasit has joined #nix-lang
<puck> <Profpatsch> puck: you lied to me!!! :D <- you still can't make a list that's infinitely long :p
ddellacosta has joined #nix-lang
<infinisil> Yet
<Profpatsch> puck: sure, just use car/cons cells
<Profpatsch> Or does that hit the recursion depth?
<Profpatsch> infinisil: I’m just noticing, (listToAttrs (map ()) is everywhere
<Profpatsch> Now I wonder how much we could save if we made that into a builtin instead of listToAttrs
<infinisil> Well listToAttrs is a builtin and so is map, so I don't expect a lot
<infinisil> I guess one less list allocation though
<Profpatsch> mapAttrs['], genAttrs and zipAttrsWithNames use it just in attrsets.nix
<Profpatsch> I guess ideally there would be some kind of rewriting system that does these automatically
<infinisil> Hm yeah
<Profpatsch> Same with the listToAttrs … attrsToList
<Profpatsch> stuff
<infinisil> Hm wait
<infinisil> Ah
<Profpatsch> well, maybe more in a sense of rewriting rules.
<infinisil> builtins.sort builtins.lessThan is optimized
<infinisil> But not builtins.sort (a: b: a < b) I think
<infinisil> TIL about builtins.lessThan
<Profpatsch> Maybe hnix can do AST-based rewrites
<Profpatsch> But I guess I’m faster with helm-edit :P
<infinisil> Rewrite it how though?
<infinisil> Need a new builtin for the listToAttrs map thing
<Profpatsch> I mean rewrite as in go through nixpkgs and rewrite all occurences
<infinisil> I think this internal rewriting might be nicer because of this
<infinisil> Yeah but to what
<Profpatsch> the lib function
<Profpatsch> I wonder how fast the lib lookup for `builtins ? foo` is
* infinisil doesn't get it, you want to rewrite `listToAttrs (map ..)` with which lib function?
<Profpatsch> because that is something we could easily remove statically the first time the evaluator passes it
<infinisil> Except builtins can change :P
<infinisil> let builtins = ...; in ...
<Profpatsch> infinisil: I mean rewrite textually first in nixpkgs, to the lib definition, then use `builtins.mapListToAttrs or <definition>` once the builtin exists
<infinisil> Ah got it
<Profpatsch> And then the nix evaluator can have a static shortcut that rewrites these `builtins.foo or foo-def` terms to `builtins.foo` if the builtin exists
<Profpatsch> So that it doesn’t have to go through the branch everytime the function is called
<Profpatsch> e.g. for concatMap = builtins.concatMap or (f: list: concatLists (map f list));
<Profpatsch> which might be quite substantial? Maybe not
<Profpatsch> But feels like an easy win
<infinisil> Profpatsch: Hmm, what if there's some function optimizer in general
<infinisil> Which can detect these kinds of optimizations automatically
<Profpatsch> that’s a JIT, no?
<Profpatsch> But that requires a complete rewrite of the evaluator
<infinisil> Hmm well it wouldn't compile it to machine code presumably
<infinisil> Or would it
<Profpatsch> I think JITs usually work on a bytecode representation firts
<Profpatsch> but I guess why not, libjit exists
<Profpatsch> Like how Emacs is gonna be 3 times faster from the next release
<Profpatsch> Although I feel like nixpkgs has bigger problems than what a 3 times speedup can fix …
<infinisil> Hehe yeah
<Profpatsch> There was this flamegraph setup that gchristensen had
<Profpatsch> Ah, nix_count_calls
<Profpatsch> omg, that’s not printed to stdout
<Profpatsch> infinisil: So, -A hello gives me fold' 3510 calls, foldr 2048 calls
<Profpatsch> mapListToAttrs (after adding it) 2049 calls :)
<Profpatsch> I wonder if we could push this stuff to graphana somehow
<MichaelRaskin> I wonder if that would lead to discovery of a trick which reduces the number of calls but uses something builtin but still expensive-ish so the overall performance still goes down
<Profpatsch> infinisil: TIL json merge-patch https://tools.ietf.org/html/rfc7396
<Profpatsch> We could implement it in nix
__monty__ has quit [Quit: leaving]
<infinisil> ,profiling
<{^_^}> Use NIX_COUNT_CALLS=1 and/or NIX_SHOW_STATS=1 to profile Nix evaluation
<infinisil> ,profiling = Use NIX_COUNT_CALLS=1 and/or NIX_SHOW_STATS=1 and/or --trace-function-calls to profile Nix evaluation
<{^_^}> profiling redefined, was defined as Use NIX_COUNT_CALLS=1 and/or NIX_SHOW_STATS=1 to profile Nix evaluation
<infinisil> Profpatsch: ^
<Profpatsch> yeah, you have to use both if you actually want any COUNT_CALLS output to show