worldofpeace changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | NixOS 20.09 Nightingale ✨ | | | 20.09 RMs: worldofpeace, jonringer |
<ekleog> hmmmm yeah I'm wondering if we gain much compared to having the config-generating script (I'm assuming all these things are going to be 99% used for generating config files) just calling the .command things itself
<infinisil> Because this server part would then have to run as root, at which point there's no reason to have any context restrictions
<ekleog> oooh that's why
<ekleog> that actually somehow scares me
<ekleog> because then it means that unless we do proper access control, all programs can try asking for dynamic config values
<ekleog> and if we do proper access control we're close to square one
<infinisil> ekleog: How about /run/nixos-dynamic/per-user/*/values.sock, which is only accessible to the * user
<infinisil> Then all the module needs is to run as that user
<infinisil> Hmm, and the server side would probably also run as that user
<infinisil> Or so
* infinisil waves hands around
<ekleog> ^^'
<infinisil> I believe there's some nice solution there
<ekleog> in this case I think the module could also directly run the commands themselves
supersandro2000 has quit [Disconnected by services]
supersandro2000 has joined #nixos-dev
<ekleog> the only solution I see would be to 1. have the module implementation declare which values it's going to access under which user, and 2. have the module system socket-handling daemon run as root and do access control before giving values out
<infinisil> Oh, if we have an abstraction like that though, we could only expose a filesystem to the module
<ekleog> but proper implementation of 2 somehow makes me scared
<infinisil> The module only needs to implement static and dynamic types, where dynamic is just `cat` ing a specific file
<infinisil> But the file is actually like a socket, fifo, or fuse
<infinisil> So it doesn't need to run the command on its own
<infinisil> ekleog: Ohh, and then we can run each dynamic value provider in its own tightly restricted (by default) systemd service, with DynamicUser
<ekleog> from the module implementer point of view, I'm not sure replacing `\$(${theOption.command})` by `\$(cat ${theOption.file})` is that big a gain
<infinisil> So the permissions of the part that provides the value can be independent from the part that consumes the value
<infinisil> I think it's a big gain from a security point of view, because of ^^
<ekleog> as soon as the dynamic value provider does not run as root, it possibly cannot access the files
<ekleog> IMO `RunExecStartPreAsRoot` or I don't remember which is the name of the systemd thing about that is enough for all practical purposes and reduces risk of wrong implementation
<infinisil> I guess it might need a bit more user declaration
<infinisil> E.g. `mkFile { path = "/some/path"; serviceConfig.User = "some-user-that-can-access-that-file"; }`
<ekleog> that sounds doable, but I'm not sure what we gain compared to requiring the end-user to just set the file to the right permissions
<infinisil> Hmm
<infinisil> Oh but yeah, we don't need to run this as a special user
<infinisil> `root` is fine, because `root` is also the permission you need to nixos-rebuild and apply the configuration that chooses that path
<infinisil> And by separating the providers permissions from the consumers, we don't need to give root to the consumer
<infinisil> So the module just cat's `/run/nixos-dynamic/per-user/*/<value>` at runtime, which is configured to be only accessible to that module
<infinisil> While the underlying producer cat's the configured file as root
<infinisil> Also something I just thought of: The module only needs to implement a way to get the value dynamically, because once it can do that, it implicitly supports a static value too
<ekleog> Oooh, like the module implementation first runs `touch /blah/theOption; chmod u=rw,go= /blah/theOption; chown user:group /blah/theOption; ${theOption.command} >> /blah/theOption` as root, and then in a second step reads it to generate the actual configuration file?
<ekleog> That'd allow having `command` contain `sudo -u foobar`, which sounds good indeed :)
<infinisil> Yeah, pretty much!
<infinisil> Oh and here's a very good example of why this is needed:
<{^_^}> #92711 (by LEXUGE, 26 weeks ago, open): minecraft-server: add declarative setting for ops.json
<ekleog> though it only moves little of the config-file-generating process out of root, so… I'm kind of wondering whether the whole config file should not just be run as root, thus avoiding this boilerplate?
<ekleog> (and then the config file be chowned as some user and then ran as that user)
<infinisil> Currently the minecraft module has a `declarative = true` option, which only makes a certain set of options declarative, `ops.json` isn't one of them. Now introducing a declarative `ops.json` kind of needs another `declarativeOps` option
<infinisil> ekleog: Oh that sounds good too
<infinisil> `my-service --config /run/nixos-dynamic/per-user/my-service/settings.json` as ExecStart
<siraben> Profpatsch: wow, I see you've already been busy in #108979
<{^_^}> (by Profpatsch, 2 hours ago, open): doc: stdenv.lib -> lib
<siraben> #108978, rather
<{^_^}> (by Profpatsch, 3 hours ago, open): treewide: with stdenv.lib; in meta -> with lib;
<infinisil> Oh, that's what Profpatsch used hnix for :o
<ekleog> Sounds like a great idea overall, I'm not going to say I'm gonna write an RFC for it because realistically I'll never get to actually do it, but if there's one I'd love to support it :)
<siraben> yay haskell
<infinisil> Profpatsch: Lol you coward " echo "replacing stdenv.lib meta in $file" >&2 sed -e '/${metaString}/ s/stdenv.lib/lib/' \ -i "$file"
<infinisil> You recently talked about hnix so I was expecting that here xD
<infinisil> ekleog: Awesome, thanks for the discussion, I'll keep this chat in mind in case I write one :)
<ekleog> Thank you for the idea in the first place! :D
<ekleog> (and well, for the great discussion too :p)
<infinisil> :D
<siraben> lol how does one review a 5000+ file PR
<supersandro2000> you don't
<ekleog> 1. look at the command used to generate it, 2. land to staging, 3. ???, 4. profit
<cole-h> Last time this happened, we split it into chunks I think.
<ekleog> (not sure about “to staging” actually, as this specific PR is supposed to not change any derivation)
<{^_^}> #83909 (by OmnipotentEntity, 40 weeks ago, closed): treewide: Per RFC45, remove all unquoted URLs
<infinisil> cole-h: I'm still a bit salty that I wrote a tool explicitly for this use case, with a test suite and all, but it wasn't even used for the mass replacement -.-
<cole-h> infinisil: I remember that hehe
<infinisil> >> And my macro was roughly: /^\s*[\w.]\+\s*=\s*\w\+:\/\/<Enter>f=wi"<Esc>/\s*;\s*\(\#.*\)*<Enter>i"<Esc>
<{^_^}> firing: RootPartitionLowInodes:
copumpkin has joined #nixos-dev
copumpkin has quit [Changing host]
copumpkin has joined #nixos-dev
supersandro2000 has quit [Quit: The Lounge -]
supersandro2000 has joined #nixos-dev
mkaito has quit [Quit: WeeChat 3.0]
rajivr has joined #nixos-dev
jonringer has joined #nixos-dev
jonringer has quit [Remote host closed the connection]
jonringer has joined #nixos-dev
srk has quit [Ping timeout: 240 seconds]
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
Taneb has quit [Quit: I seem to have stopped.]
Taneb has joined #nixos-dev
kalbasit has joined #nixos-dev
<elvishjerricco> Hm. Adding udev rules to make something work ended up making everything worse... Yay.
<{^_^}> firing: RootPartitionLowInodes:
orivej has quit [Ping timeout: 260 seconds]
<qyliss> cole-h: I just wrote the two sentences in the manual that kicked this whole thing off. It's Profpatsch and siraben who're really running with it.
<cole-h> qyliss: Without you, they wouldn't have started running with it. However, Profpatsch++ and siraben++, of course.
<{^_^}> Profpatsch's karma got increased to 18
<{^_^}> siraben's karma got increased to 0b101
<siraben> :D
jonringer has quit [Ping timeout: 264 seconds]
<qyliss> ngl I'm extremely proud of this comment:
<cole-h> qyliss++ Wow, that's great.
<{^_^}> qyliss's karma got increased to 113
<cole-h> Also puck++
<{^_^}> puck's karma got increased to 41, it's a crit!
<qyliss> the funny thing is puck doesn't even know she helped with this yet
<qyliss> she just posted that regex on IRC a few days ago
<cole-h> haha
<qyliss> and when I needed it right now I just got out my logs
saschagrunert has joined #nixos-dev
<Profpatsch> qyliss: wow, of course somebody merges a treewide change, changing lines right besides mine in a few dozen files over night …
<qyliss> Profpatsch: you can just regenerate yours though, right?
<Profpatsch> qyliss: partially
<qyliss> ah
<Profpatsch> well, have to bite the bullet
<Profpatsch> There’s probably a git command for semi-automating these, but I don’t know it
<qyliss> semi-automating what exactly
<qyliss> maybe I can save you some time :)
<Profpatsch> <<<<<<< HEAD
<Profpatsch> meta = with stdenv.lib; {
<Profpatsch> =======
<Profpatsch> enableParallelBuilding = true;
<Profpatsch> meta = with lib; {
<Profpatsch> >>>>>>> 86ea9f38c89... treewide: with stdenv.lib; in meta -> with lib;
<Profpatsch> this changes happened a few dozen times
<Profpatsch> of course the headers are also modified because they are usually on one line
<Profpatsch> Can’t you say something like “if the >>> side contains X, replace the <<< wtih regex R”
<qyliss> I don't see a way to :(
<qyliss> there's no merge strategy option for this
<qyliss> you could global find an replace that exact string though (all lines) if it's literally that exact conflict every time
<Profpatsch> nearly done :)
<Profpatsch> Okay, if I have to do this again, I will need to at least automate the header part …
<Profpatsch> lol, it took 10 minutes but it felt like an eternity. just programmer things
<cole-h> Profpatsch: looks like you amended into a staging-next merge?
tom39291 has quit [Remote host closed the connection]
tom39291 has joined #nixos-dev
<cole-h> s@?@@
<Profpatsch> cole-h: uh, interesting
<Profpatsch> I thought I had rebase --continued
<Profpatsch> crap
<cole-h> :D
<Profpatsch> do I have to do this again :(
<cole-h> You could probably just soft-reset HEAD^ and then redo the actual commit
<Profpatsch> I will beat it into shape with macros
<qyliss> I'm gonna try to make a graph of stdenv.lib occurrences over time
<Profpatsch> qyliss: so, it looks like ofborg will not add any labels until it verifies that everything evaluates, which takes hours … I wonder if that’s worth it.
<Profpatsch> Maybe it’s possible to do the same evaluation locally.
<Profpatsch> My approach was just to run a separate eval for everything in pkgs
<cole-h> (There's a section in the readme that shows you how ofborg computes the outpaths, which will at least show eval errors)
<Profpatsch> cole-h: doesn’t show me how it evaluates
<Profpatsch> The example uses -A manual
<cole-h> Further down
<Profpatsch> ugh nix-env
<Profpatsch> I don’t think I have enough RAM for this
<qyliss> Last time I tried to do that it used too much memory on my machine
<qyliss> yeah
<cole-h> I'll run it for you, then :P
<Profpatsch> If you could, that would be sweet
<Profpatsch> Basically I just need to find any breakage that still exists
<qyliss> As long as somebody has checked it's rebuild-0
<qyliss> and evaluates
<Profpatsch> qyliss: I’d be fine with some leaf drvs not evaluating, since this is easy to fix
<cole-h> Profpatsch: pkgs/tools/misc/ethminer/default.nix
<cole-h> undefined lib
<cole-h> because it uses `clangStdenv` in the top-level
<cole-h> (rather, the top-level deps, not necessarily top-level/all......)
<Profpatsch> It feels like while the scope has grown, our tool has not.
<cole-h> Profpatsch: Actually, I'll just send you a diff after I get through all these lol
<Profpatsch> cole-h: yeah, there’s probably a bunch
<Profpatsch> Like, it’s still impossible to evaluate all of nixpkgs without crude workarounds
<Profpatsch> And heaps of RAM
<cole-h> FWIW, I was able to run the outpaths tool with "only" 16GiB earlier last year
<cole-h> (I've since upgraded to 32GiB)
<Profpatsch> I mean some people will probably say the solution is flakes, but that’s like saying the solution to being hit by lightning is carrying around a metal pole all the time
<qyliss> lol
<cole-h> :P
<Profpatsch> cole-h: yeah, 16 is what I have, but part of it is used. :)
<cole-h> That said, it's definitely happy to eat whatever ram is available -- `free -h` showed 29Gi at one point
<Profpatsch> I guess if you start by listing exhaustively you can split it up into multiple evals
<Profpatsch> cole-h: ah, yes, boehm I think
<Profpatsch> It will be a lot faster with more RAM
<cole-h> fancy
<Profpatsch> maybe that’s bs :)
<cole-h> Maybe, but it's fancy bs ;P
<Profpatsch> But at least in theory you can replace values with thunks again and recompute when you need them again
<Profpatsch> And have something like a LRU cache
<Profpatsch> idk what nix does
kalbasit has quit [Ping timeout: 240 seconds]
teto has joined #nixos-dev
<cole-h> I wonder if --keep-going has any effect on this script. Would be nice to not have to fix these one-by-one...
<cole-h> Doesn't :(
sgrunert has joined #nixos-dev
<qyliss> graph slowly being generated here:
saschagrunert has quit [Ping timeout: 246 seconds]
<qyliss> cc siraben
<Profpatsch> cole-h: oh, you mean you have to restart every time it hits an evaluation problem?
<qyliss> (I think it was you who was saying it would be cool?)
<cole-h> Yarp
<Profpatsch> qyliss: This graph is just following the number of commits per time :)
<cole-h> Hehe
<Profpatsch> I think you have to divide by commits
<Profpatsch> cole-h: ah, hm, yeah. Don’t feel like you have to spend an undue amount of time on this
<Profpatsch> I thought it might actually skip and record broken expressions
<qyliss> Profpatsch: nah, mostly what I'm interested in is watching it peak and then decrease when it catches up to now
<Profpatsch> qyliss: Ah :)
<cole-h> I'll give it another 5 minutes and send you the patch with what I have so far. Then you can (try to) continue from there :P
<qyliss> but that's going to take a long time because I am not generating this data at all efficiently
<Profpatsch> Currently nixpkgs uses throw on the toplevel, so you can’t even mapAttrs over over it
<cole-h> Efficiency? pfft
<Profpatsch> cole-h: cool
<cole-h> oh
<cole-h> huh
<cole-h> finished
<cole-h> One more run for sanity check
<Profpatsch> Yeah, my replacement script yesterday booted a python interpreter for each item, so it only went through three replacements per second :)
<Profpatsch> So even for 6k files it took like 10–15 minutes
<Profpatsch> I could have used writeRust instead of writePython, that would have sped it up by an order of magnitude at least
<Profpatsch> python3 is really slow to boot up when you import them modules you basically aways need, os, sys and json
<Profpatsch> Like 200+ms
<Profpatsch> cole-h: amazing, thanks
<Profpatsch> Ah, good to see that my randomwalk caught most already
<cole-h> FWIW, each run (on my system) was ~2m tops. Including the successful ones.
<Profpatsch> the long tail is the hard thing
<cole-h> Profpatsch: btw, was just saying there were overlapping changes, not that there were any conflicts per-se (or at least, none git couldn't solve itself). It's cleaner being able to read the diff when it's unique :P
<Profpatsch> using git am and adding to my list of inscrutable & unhelpful git messages I’ve never seen before
<Profpatsch> cole-h: oh, but I didn’t know about the other PR
<cole-h> that's a diff, so you'll have to `git apply`
<cole-h> :P
<Profpatsch> wtf
<Profpatsch> I’m so happy I can usually use magit
<cole-h> git am expects a mailbox-formatted-patch basically
<Profpatsch> okay, i guess then let’s wait for the ofborg tags and merge
<Profpatsch> provided there’s no new conflicts by then.
<cole-h> SGTM. Have fun with that :P
<Profpatsch> qyliss: After that change I only see 6k stdenv.lib calls remaining in nixpkgs
<qyliss> wow
<qyliss> that's gonna look good on the graph
<qyliss> I hope you're not done by the time the graph catches up lol
<Profpatsch> I think part of them can be done with simple sed
<cole-h> qyliss: ofborg's been chugging today, so since we're waiting on that... I think you'll have time
<qyliss> oh good
<cole-h> (chugging as a result of 2 evaluators total)
<Profpatsch> strange though, I still see a lot of meta = with stdenv.lib; calls
<Profpatsch> I wonder if I only caught a subset
garbas_ has quit [Quit: WeeChat 2.9]
<Profpatsch> eh, in that case we adjust and do the same thing again
<cole-h> Profpatsch: staging/-next just merged
garbas has joined #nixos-dev
<cole-h> ~1h ago
<Profpatsch> Ah, maybe it’s that
<cole-h> 2h ago
<Profpatsch> I feel like my next task is bringing nix-tree-sitter up to snuff
<Profpatsch> So that we can do AST-replacements like that better in the future
<Profpatsch> hnix is okay, but it doesn’t provide source spans for anonymous nodes
orivej has joined #nixos-dev
<cole-h> Profpatsch++ Sounds like a fun project.
<{^_^}> Profpatsch's karma got increased to 19
<Profpatsch> tree-sitter is the best thing to happen to code in the last decade, convince me otherwise
<Profpatsch> maybe on par with LSP
<cole-h> Anyways, that's all for me tonight. o/
<Profpatsch> cole-h: oh right, timezones. :)
<Profpatsch> Sleep well
<cole-h> Hehe
srk has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
cole-h has quit [Ping timeout: 240 seconds]
srk has quit [Remote host closed the connection]
srk has joined #nixos-dev
garbas has quit [Quit: WeeChat 2.9]
<Profpatsch> peculiar!
<qyliss> sup?
<Profpatsch> ofBorg took 30 minutes to start, and in the meantime there was a merge conflict
<Profpatsch> So it immediately stopped with a merge error
<Profpatsch> I am … uncertain how to proceed
<Profpatsch> I mean I can try again and hope for a better result.
<qyliss> can we split this into smaller PRs?
<qyliss> doesn't even matter how it's split
<Profpatsch> Is there any admin with superpower to ping abaut the ofborg queue?
<Profpatsch> qyliss: I don’t think it matters if ofborg takes half an hour to start
<qyliss> you'd be less likely to get conflicts
<Profpatsch> I mean yes, we reduce the chance.
<Profpatsch> But that sounds like a workaround
<qyliss> sounds like a very pragmatic workaround to me
<qyliss> OfBorg admins are gchristensen and cole-h AIUI
<Profpatsch> otoh, we cole-h already did a full evaluation an hour ago
<Profpatsch> So we are only waiting for a green check mark from authority here
<qyliss> oh
<qyliss> I think skipping OfBorg is probably fine in that case
<qyliss> if we've run the same checks locally there's not much reason to wait
<qyliss> Profpatsch: if you fix the conflicts I'll run the checks on a machine with 128GiB RAM
<qyliss> and then I think we should just merge after that, without waiting for OfBorg
<qyliss> (assuming the checks all pass)
<Profpatsch> qyliss: rebased
<qyliss> okay let me figure out how to do this
<qyliss> no need to link me
<Profpatsch> qyliss: wait I’ll link you the command
<qyliss> I said no need :)
<qyliss> I know where to find it
<qyliss> 72Gi free atm
<qyliss> HEAD is now at 1a25769a6c9 treewide: with stdenv.lib; in meta -> with lib;
<qyliss> running
<qyliss> we _can_ skip OfBorg, right?
<qyliss> the check doesn't block merge?
<qyliss> for the record I still think smaller PRs would have been the way to go but I think this is fine
<qyliss> Profpatsch: okay, I'm not sure how I tell that was successful
<qyliss> it exited 0
<qyliss> and wrote out-paths
<qyliss> So I think that means it worked?
<Profpatsch> qyliss: I think that’s it … you could try removing a lib from one package header and see what happens
<Profpatsch> yeah, the ofborg check doesn’t block merge
<Profpatsch> It’s too flaky for that
<Profpatsch> qyliss: do you know an easy way to split up a PR into smaller ones? Maybe scripted?
<qyliss> okay let's try that
<qyliss> (removing a lib)
<qyliss> Profpatsch: I do not know of a way
<Profpatsch> I guess combining the hub tool and stashing could be scripted
<qyliss> I don't think it's something you can reasonably do in a generic way
<qyliss> Because in most PRs you're going to have dependencies between files
<Profpatsch> yeah
<qyliss> in this PR there are, I assume, almost none of those
<qyliss> while I'm doing this can you run the check I posted earlier?
<qyliss> the +lib -stdenv one
<Profpatsch> I mean, this is the same strategy as splitting a monorepo into multiple repos because the tools are too bad
<Profpatsch> It works, but it is a workaround that produces lots of work
<qyliss> yeah, but this doesn't have permanent bad effects
<qyliss> except a few more commits than necessary
<qyliss> Profpatsch: okay yeah when I manually broke it eval failed
<Profpatsch> check is +lib -stdenv
<qyliss> so that means it passed before
<qyliss> okay I think we're good
<Profpatsch> few
<Profpatsch> phew
<Profpatsch> let us not speak of this again :P
<qyliss> hope this doesn't come back to bite us
<Profpatsch> I think we did anything we could to make sure it didn’t break anything
thibm has joined #nixos-dev
<qyliss> We should keep an eye on things for the next few hours
<Profpatsch> +1
<qyliss> hmm, shame you went for lib, stdenv over the more common stdenv, lib
<qyliss> I should have noticed that befroe
<Profpatsch> qyliss: we can do an import resorting sometime in the future if you want :P
<qyliss> I guess it'll be easy once you have tree-sitter
<Profpatsch> should be easy enough. While we’re at it, we could also automatically break them onto multiple lines
<Profpatsch> Less merge conflicts all the time
<qyliss> -1
<qyliss> makes it very hard to read
tilpner has quit [Remote host closed the connection]
<Profpatsch> I usually write inherits
<Profpatsch> inherit
<Profpatsch> foo
<Profpatsch> bar
<Profpatsch> ;
<Profpatsch> So that
tilpner has joined #nixos-dev
<Profpatsch> they don’t lead to merge conflicts
<Profpatsch> Also it makes commenting out line-based so much easier
<qyliss> I think readability is more important than mergeability
<Profpatsch> but yeah, once we have a tree sitter parser these changes should be mostly trivial
tilpner has quit [Remote host closed the connection]
tilpner has joined #nixos-dev
tilpner has quit [Remote host closed the connection]
tilpner has joined #nixos-dev
<qyliss> really kicking myself for not spotting that tbh
<Profpatsch> qyliss: haha, the graph is starting to fall steeply
<Profpatsch> qyliss: spotting what?
<qyliss> I did a fancy script-based review but didn't look enough at the actual diff
<qyliss> lib, stdenv
<qyliss> Profpatsch: where did your 6k number come from?
<qyliss> I just had to check my graph code because I didn't see a drop of anything like that
<qyliss> but it seems correct
<qyliss> git --git-dir ~/code/nixlib.git grep '[sS]tdenv.*\.lib' nixpkgs/master | wc -l
<Profpatsch> 6k files?
<qyliss> 37708
<qyliss> oh files
<Profpatsch> yeah
<Profpatsch> rg -l 'stdenv.lib' | wc -l
<Profpatsch> I think many occurences are fixed by changing generator scripts
<siraben> qyliss: I love how Nix makes it trivial to share reproducible scripts
thibm has quit [Ping timeout: 264 seconds]
thibm has joined #nixos-dev
Graypup_ has quit [Quit: ZNC 1.6.1 -]
Graypup_ has joined #nixos-dev
dhess has joined #nixos-dev
<dhess> Hi. I have a multi-output derivation, let's call it "foo" with `outputs = ["out", "lib", "installedTests"]``. I have a Flake whose `packages` attribute assigns nice aliases to these: `foo-lib = foo.lib`, `foo-installed-tests = foo.installedTests`.`
<dhess> I'm using flake-compat. When I `nix-build -A` I get a single result pointing to the Nix store for foo-lib
<dhess> When I `nix build` I get 3 results, one for each output in derivation `foo`
<dhess> And if I haven't already built, say, `foo-installed-tests` then the symlinks to the `foo-installed-tests` and `foo` are invalid, because they've never been downloaded/built on this machine. (e.g., if `foo-lib` hits a binary cache)
<qyliss> that sounds like a bug
<dhess> This is with "nix (Nix) 2.4pre20201201_5a6ddb3" BTW
<qyliss> the last bit especially
<dhess> qyliss: right, `nix build` should only output just the one result, right?
<qyliss> idk, but it definitely shouldn't output invalid links
<rnhmjoj> can anyone confirm that staging is not evaluating?
<qyliss> rnhmjoj: error: undefined variable 'kdeFrameworks' at /home/qyliss/src/nixpkgs-lib/pkgs/top-level/all-packages.nix:19919:14
<ehmry> has anyone ever tried building a BSD with the nixos modules? not something i would try, just curious
<ehmry> I've got a feeling that porting to a BSD init system would be less messy than navigating systemd workarounds on linux
<rnhmjoj> qyliss: thank you
__monty__ has joined #nixos-dev
thibm has quit [Ping timeout: 272 seconds]
thibm has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
__monty__ has quit [Ping timeout: 264 seconds]
__monty__ has joined #nixos-dev
<{^_^}> resolved: RootPartitionLowInodes:
<gchristensen> andi-: ^ see, scenic route :P
<lukegb> Hahaha
<lukegb> Nice!
<lukegb> ehmry: you could probably use the module system but the modules themselves are unlikely to be very useful? Most of them just create systemd services, after all
<{^_^}> grahamc/packet-nix-builder#12 (by grahamc, 1 minute ago, open): Provision with Terraform, get secrets from Vault from within the repository
<andi-> gchristensen: nice!
<andi-> gchristensen: do not really have time these days.. Will see if I can spent an hour or so looking at stuff tonight
jonringer has joined #nixos-dev
<gchristensen> andi-: you definitely don't need to review :)
thibm has quit [Ping timeout: 246 seconds]
thibm has joined #nixos-dev
dstzd has quit [Quit: ZNC -]
dstzd has joined #nixos-dev
<domenkozar[m]> btw didn't someone say using libvirtio instead of 9p would speed up tests?
unrooted has joined #nixos-dev
<unrooted> hey, I'm coming here to ask about suckless nixOS
<unrooted> all I could found is the mailing list from around 4 years ago time
<unrooted> I even asked about it on nix and nixOS Matrix channels like a few days ago, but I got a recommendation to mail one guy who works on making nixOS 'suck less' and not 'suckless'
<adisbladis> unrooted: Afaik nothing ever materialised from that
<unrooted> hm, ouch, do you know any devs who worked on that? I'd love to give myself a shot in a spare time, maybe using notOS:
<tilpner> domenkozar[m]: I've only heard that about virtio-fs
<infinisil> domenkozar[m]: Yeah libvirt is the virtualisation api, virtio-9p is a standard for sharing guest filesystems with the host, and virtio-fs is a faster standard for the same thing
<gchristensen> unrooted: I don't know of any serious in-roads on that at al
<unrooted> ouch, I get it
<unrooted> and anyone know anyone who tried with the notOS?
unrooted has quit [Ping timeout: 272 seconds]
dstzd has quit [Quit: ZNC -]
dstzd has joined #nixos-dev
mikroskeem has joined #nixos-dev
<arianvp> I wish Nix could pattern match on nested things..
<arianvp> pkgs@{ stdenv.mkDerivation, ...}: brings mkDerivation in scope
<arianvp> is this a thing in some shape or form; or not?
<Ericson2314> the secret is hnix itself almost generalizes to other languages :D
<Ericson2314> oops replying to something many pages above
<domenkozar[m]> btw
<{^_^}> nix#4442 (by domenkozar, 10 minutes ago, open): allowSubstitutes/preferLocalBuild are an anti-feature
<domenkozar[m]> I'd really like to make a PR for that in nixpkgs
<domenkozar[m]> any objections?
<symphorien[m]> Remote builders on a slow network are really painful without preferLocalBuild
<symphorien[m]> But I won't miss allowSubstitutes if it goes away
<symphorien[m]> It tends to silently break nixos-install
<domenkozar[m]> the problem is that those two flags make Nix extremely hard to reason about
<domenkozar[m]> everything works this way, except for a few flags that you can't possibly see make everything do another thing
thibm has quit [Ping timeout: 256 seconds]
<symphorien[m]> Well the problem they try to solve still exists, and removing those flags does not remove the problems
<domenkozar[m]> yet it does introduce another 10 problems
<domenkozar[m]> so I'd rather have one that at least is predictable
<symphorien[m]> Well you understand my point, and I understand yours
<niksnut> domenkozar[m]: all those flags exist for a reason
thibm has joined #nixos-dev
<niksnut> one person's anti-feature is another person's essential feature
<niksnut> btw I don't see why they make Nix hard to reason about, since remote building and substitution doesn't change the result of a build
<domenkozar[m]> because you pass flags like --builders to Nix
<domenkozar[m]> and it doesn't respect it
<domenkozar[m]> how does the user know what's going on?
<domenkozar[m]> they think some config is overriding a flag
<domenkozar[m]> note that I've seen that happen before and today, a developer lost a few hours
<symphorien[m]> Well the right fix is to make it respect the flag
<symphorien[m]> Not remove the functionality
<domenkozar[m]> that's the same thing :D
<domenkozar[m]> niksnut: the problem is that this feature is used extensively in nixpkgs so there's no way to not use it
<symphorien[m]> <domenkozar[m] "that's the same thing :D"> Huh?
<niksnut> --builders doesn't mean "build on this machine", it means "this machine is available for building"
<domenkozar[m]> how about --max-jobs 0?
<domenkozar[m]> how would that work then?
<domenkozar[m]> that would then need another set of flags on cli to specify what to do
<gchristensen> I'm not keen on removing useful features to avoid a bug on another useful feature
<domenkozar[m]> since there's no sane default
<domenkozar[m]> gchristensen: can you be more specific?
<adisbladis> Hm, I was recently thinking we should use preferLocalBuild more
<gchristensen> I commented on the ticket :)
<adisbladis> Especially with potentially slow binary caches like ipfs
<domenkozar[m]> might as well turn off binary caches then :D
<domenkozar[m]> if we end up flipping switches manually that's just insane
<adisbladis> domenkozar[m]: Don't be hyperbolic
<niksnut> domenkozar[m]: with --max-jobs 0, it's still supposed to build preferLocalBuild derivations
<domenkozar[m]> I have no idea what that means
<niksnut> /* Make sure that we are allowed to start a build. If this
<niksnut> maxBuildJobs is 0. */
<niksnut> derivation prefers to be done locally, do it even if
<niksnut> of course, this could be broken...
<gchristensen> I do consider that to be a bug
<domenkozar[m]> it does that but it makes no sense whatsoever given what the user told Nix to do
<niksnut> ideally though, preferLocalBuild should build remotely if it can't build locally, since it's "prefer", not "require" local build
<{^_^}> nix#3810 (by grahamc, 25 weeks ago, open): preferLocalBuild should not override max-jobs=0
<domenkozar[m]> if we use these flags even more, there'll be an endless fight what are good defaults and there's no single answer
<gchristensen> you're right there is no good answer
<domenkozar[m]> on top of the fact that it's a complete shock to a Nix newbie
<niksnut> ehm
<niksnut> I think most newbies would barely notice
<domenkozar[m]> so would you rather have software behave slowly in some corner cases or be so hard to use that users drop it?
<qyliss> are Nix newbies generally using remote builders?
<domenkozar[m]> yes
<domenkozar[m]> I'm not making this up...
<niksnut> I mean, why would they notice?
<niksnut> except that things go faster doing them locally
<qyliss> i ask purely because i've still not managed to figure out how to make them work after two years :P
<gchristensen> about 2yr ago I wrote up a short personal thing of a project I'd like to work on: Remote Building: Flexible Workflows ... probably niche, but it might be nice to have some fancy workflow support. For example, evaluate on machine A, build on machine cluster B, and don't copy results back to me, but instead copy them to machine C. Or even copy the results to machines C,D,E, and F.
<domenkozar[m]> well the more knobs you add the more consultants you'll need but at some point people will see that's not feasible
<domenkozar[m]> something to think about long term
<gchristensen> that seems pretty reductionist
<domenkozar[m]> well you can put the labels on it, but it's the feedback I got, not something I made up
<gchristensen> imo the surprising thing here is 2 features which indepndently work in a predictable way come together and work in an unpredictable way
<domenkozar[m]> or more like, 2 features that completely change how Nix works by design
<gchristensen> ???
<gchristensen> I'm going to disengage now
<domenkozar[m]> if it needs special handling all over the scheduler it's probably a bad idea
mkaito has joined #nixos-dev
<domenkozar[m]> I don't think it makes sense to debate anyway, because all experienced users on this channel won't have a problem with it, since the problem exists only for users that don't know about these two flags
<domenkozar[m]> and the way to discover them is through lots of lost hours
<gchristensen> are you really trying to say you're the only person who is experienced with Nix who also has contact with and understanding of new Nix users?
<domenkozar[m]> I have never said that
<domenkozar[m]> but I do run a service with 2000 users that gives me more exposure that most developers usually have
<gchristensen> I should have qualified it to "only person who is here, who is also experienced"* since that seems to be what you said here: "because all experienced users on this channel won't have a problem with it"
<domenkozar[m]> so I can confidently say that I do have a much better insight than anyone else
<domenkozar[m]> and I can't say I'm the only one
<domenkozar[m]> gchristensen: so I hope that clarifies my first statement
<gchristensen> thahnks
<domenkozar[m]> I do get a support ticket about once a week
<domenkozar[m]> that someone is confused why the hell is cachix not substituting
<domenkozar[m]> I even have a template answer
<domenkozar[m]> and the response is usually: oh wow
<domenkozar[m]> maybe it's every two weeks, I'll start logging it
<niksnut> btw I'm not really against removing preferLocalBuild, it was mostly added for hydra but no longer uses it
<domenkozar[m]> I'll start asking all users that hit this write to that issue, so that my word doesn't have to be trusted blindly
<gchristensen> I believe you, domenkozar[m], I don't agree that the solution to this is removing some of the few policy features that are important to users across the spectrum.
<domenkozar[m]> I see that preferLocalBuild can save minutes, but allowSubstitutes is really marginal
<qyliss> having both is definitely confusing
<domenkozar[m]> is that statement a true fact?
<gchristensen> I agree that allowSubstitutes is super marginal
<lukegb> I think it is easier to argue that allowSubstitutes is marginal
<lukegb> I'd be happy enough to see it not used in nixpkgs
<domenkozar[m]> you can't even not save time uploading the closure
AlwaysLivid has joined #nixos-dev
<domenkozar[m]> so it's really just about substituting
<domenkozar[m]> ok great :)
<abathur> is it feasible to address this, and perhaps others, with a ~debug flag that reports the build/substitution plan and the "why" for each decision?
<qyliss> that sounds like a real solution
<domenkozar[m]> not sure how feasible that is without a substantial rewrite
<lukegb> preferLocalBuild still makes tons of sense to me for the "write this file to disk" trivial builds
<domenkozar[m]> since afaik there's no separation of planning/execution
<lukegb> I'm not sure how useful that would be in practice either
<abathur> nod, I realize it may not be
<adisbladis> Just an observation: preferLocalBuild is used for two different things. To avoid copying large sources to a remote and to avoid quering a binary cache for small builds.
<domenkozar[m]> hope I'm wrong :)
<abathur> just thinking the blue-sky thoughts the naive can think...
<qyliss> lots of things with large sources are small builds
<qyliss> so it's a heavily overlapping use case I suspect
<lukegb> does preferLocalBuilds imply !allowSubstitutes
<adisbladis> qyliss: For sure. I thought it's worth a mention that it's not just a knob for a single use case.
<qyliss> oh yeah sure
<niksnut> adisbladis: the second is allowSubstitutes
<niksnut> preferLocalBuild still allows substitution
<niksnut> though originally it didn't
<gchristensen> my hope is that describes a predictable and less magic / invasive implementation of preferLocalBuilds
<{^_^}> nix#3810 (by grahamc, 25 weeks ago, open): preferLocalBuild should not override max-jobs=0
<adisbladis> niksnut: Oh, I thought it still worked the same way
<domenkozar[m]> I'd just like to note that you can observe even experienced Nix devs don't know how these two flags behave
<domenkozar[m]> and that is not to say someone it nor or is experienced, but that it's complex
<domenkozar[m]> someone is*
<domenkozar[m]> and even I have to look it up for answers like lukegb asked
<domenkozar[m]> so gchristensen would your proposal remove preferLocalBuild eventually?
<domenkozar[m]> if feels to me that if both live at the same time that makes it even harder to reason about
<gchristensen> whether we keep it or not I dunno, syntactic sugar can be useful
<gchristensen> no opinion
pmy has quit [Ping timeout: 256 seconds]
pmy has joined #nixos-dev
<domenkozar[m]> having local-preferred feature sounds a misnomer to me
<domenkozar[m]> maybe something like quick-build is closer to the problem it's solving
<qyliss> doesn't that make it even less clear what it does?
<domenkozar[m]> maybe? :)
<gchristensen> domenkozar[m]: I don't understand your comment here:
<domenkozar[m]> zupo added a comment to explain
<gchristensen> ah, may I suggest rewording slightly to say "one reason" instead of "the reason"?
<domenkozar[m]> done
thibm has quit [Ping timeout: 246 seconds]
<gchristensen> cool, that parses for me now
thibm has joined #nixos-dev
<gchristensen> I don't love having to set the local system in nix.conf to a uuid
<qyliss> oh no lol
<lukegb> I can't help but feel that running the Nix evaluation on the tiny VPSes isn't really a great usecase here either though and they should be copying the system closure onto the VPSes and not running builds of any sort there...
<domenkozar[m]> lukegb: that is the plan
<domenkozar[m]> lukegb: however buliding remotely would solve 80% of the problems
<domenkozar[m]> but there's no way to do that with Nix now
AlwaysLivid has quit [Remote host closed the connection]
AlwaysLivid has joined #nixos-dev
kalbasit has joined #nixos-dev
kalbasit_ has joined #nixos-dev
kalbasit_ has quit [Ping timeout: 256 seconds]
<infinisil> domenkozar[m]: Arguably they should probably move to a nixops-style deployment, where the nix evaluation runs on a different host
<lukegb> I think that's what he meant by "that is the plan"
<domenkozar[m]> there are two separate problems: 1) too much HDD usage 2) takes a lot of time and hogs CPU
<domenkozar[m]> 1) could be solved with remote builders, but NixOS is mostly about preferLocalBuild
<domenkozar[m]> 2) is planned by moving evaluation outside the machines but 1) can be solved now
costrouc has quit [Quit: costrouc]
AlwaysLivid has quit [Remote host closed the connection]
AlwaysLivid has joined #nixos-dev
kalbasit_ has joined #nixos-dev
teto has quit [Ping timeout: 260 seconds]
dtz has joined #nixos-dev
thibm has quit [Ping timeout: 260 seconds]
thibm has joined #nixos-dev
rajivr has quit [Quit: Connection closed for inactivity]
sgrunert has quit [Remote host closed the connection]
teto has joined #nixos-dev
tokudan has quit [Remote host closed the connection]
bennofs has joined #nixos-dev
<bennofs> I made a derivation for the python bindings shipped with clang. Where would be the best place to have this inside nixpkgs? Probably it should go into python-packages.nix, so it can work for all python versions?
tokudan has joined #nixos-dev
cole-h has joined #nixos-dev
kalbasit_ has quit [Ping timeout: 272 seconds]
<domenkozar[m]> adisbladis: I hope you were joking about IPFS+preferLocalBuild
<yorick> can someone review ?
<{^_^}> #107394 (by yorickvP, 2 weeks ago, open): beam-packages: move wxSupport arg up to package set, add beam_nox
<gchristensen> andi-: are you actually going to review that PR, or is it aspirational? no worries if it is just aspirational, but I'll just merge it if so
<andi-> gchristensen: I actually just finished work and starting to go through my things :)
<gchristensen> oh cool :)
<andi-> having a job is generally a bad idea. Distracts from the important stuff :D
<gchristensen> haha
<andi-> I'm going to ignore the commit messages. (c'mon x3)
<gchristensen> "make it work in CI", baby
<cole-h> "c'mon x3" and "c'mon 2x"
<cole-h> That irks me, Graham. Why did you do this.
<cole-h> yorick: A bit curious why you're prefixing e.g. `beam.interpreters` with `pkgs.`. Does that really make a difference?
<yorick> cole-h: oh! I thought it was rec
<yorick> then yes
<cole-h> "yes" meaning yes it makes a difference and is necessary?
AlwaysLivid has quit [Ping timeout: 240 seconds]
mikroskeem has quit [Quit: WeeChat 3.0]
<yorick> cole-h: yes meaning it's unneccesary
<yorick> I'll fix it
<qyliss> I swear that used to be a rec
<qyliss> did it used to be a rec?
<qyliss> hmm, not since I started using Nix
<qyliss> I wonder why I thought that
<yorick> qyliss: I swear aswell
<yorick> anyways, fixed
<qyliss> because it made overlays weird, right?
teto has quit [Ping timeout: 260 seconds]
<abathur> anyone know why something built with buildPythonApplication would leak its propagatedBuildInputs/PYTHONPATH into build environments that use it as a buildInput?
<symphorien[m]> isn't it the definition of propagatedBuildInputs ?
<abathur> perhaps, but the definition of buildPythonApplication is also that it's not supposed to leak build inputs, and AFAIK propagated is the correct place for python runtime module deps
<abathur> well, let me find what the doc says, I'm not phrasing precisely
<abathur> "Python applications can be installed in your profile, and will be wrapped to find their exact library dependencies, without impacting other applications or polluting your user environment."
<symphorien[m]> installed in your profile and put in buildInputs is a different thing
<symphorien[m]> but I agree that this behavior is unwanted
<abathur> ok, yeah, that was on my list
<symphorien[m]> I thought there was another attribute like `pythonPath` but I can't find it in the docs
<abathur> there is, though it's marked as deprecated in the code so I've avoided it
<symphorien[m]> ah...
<abathur> resholve's python deps currently leak into the environment of anything that uses resholvePackage; I've found a workaround for resholvePackage (WIP in, but I'm trying to figure out if there's a correct/canonical way to just fix the expression for resholve
<{^_^}> abathur/resholve#20 (by abathur, 2 hours ago, open): try to fix leaky pythonpath
<abathur> which is basically, yeah, to avoid putting it in buildInputs
thibm has quit [Quit: WeeChat 2.9]
<qyliss> You should ask jonringer or FRidh
<abathur> I suppose; I've been wondering if may eventually fix the underlying issue anyways, in which case a stopgap that only fixes resholvePackage is probably fine
<{^_^}> #102613 (by FRidh, 9 weeks ago, open): WIP: Python: wrap and patch using `requiredPythonModules` instead of `propagatedBuildInputs`
xwvvvvwx has quit [Quit: ZNC 1.8.2 -]
xwvvvvwx has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
teto has joined #nixos-dev
orivej has joined #nixos-dev
puckipedia has joined #nixos-dev
puck has quit [Ping timeout: 265 seconds]
puckipedia is now known as puck
orivej has quit [Ping timeout: 256 seconds]