worldofpeace changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | NixOS 20.09 Nightingale ✨ https://discourse.nixos.org/t/nixos-20-09-release/9668 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 20.09 RMs: worldofpeace, jonringer | https://logs.nix.samueldr.com/nixos-dev
das_j has quit [Quit: Bridge terminating on SIGTERM]
Scriptkiddi has quit [Quit: Bridge terminating on SIGTERM]
ajs124 has quit [Quit: Bridge terminating on SIGTERM]
ajs124 has joined #nixos-dev
das_j has joined #nixos-dev
Scriptkiddi has joined #nixos-dev
supersandro2000 has quit [Disconnected by services]
supersandro20000 has joined #nixos-dev
srk has joined #nixos-dev
jonringer_ has quit [Remote host closed the connection]
justanotheruser has joined #nixos-dev
abathur has quit [Quit: abathur]
supersandro20000 has quit [Quit: The Lounge - https://thelounge.chat]
supersandro2000 has joined #nixos-dev
abathur has joined #nixos-dev
jonringer has joined #nixos-dev
rajivr has joined #nixos-dev
<gchristensen> I'm trying to "remotely" build to a local machine using redirected stores... any ideas on how to make this work? https://gist.github.com/grahamc/89f9df54ae6e6298731f2912b357d971
<gchristensen> note the second paste is a script named `ssh` in PATH, which is the key to making this "work"
<gchristensen> I've futzed with it a bit more and can't seem to persuade it to work without spelunking into the code and perhaps a debugger. if anyone has ideas, I'd love to put this to work for testing Hydra's remote builder behavior :)
<gchristensen> but I'm off to bed, g'night 'all
orivej has quit [Ping timeout: 272 seconds]
justanotheruser has quit [Ping timeout: 264 seconds]
tv has quit [Ping timeout: 246 seconds]
tv has joined #nixos-dev
justanotheruser has joined #nixos-dev
BaughnLogBot has quit [Ping timeout: 272 seconds]
BaughnLogBot has joined #nixos-dev
aszlig_ is now known as aszlig
jonringer has quit [Remote host closed the connection]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 272 seconds]
janneke_ is now known as janneke
<regnat[m]1> gchristensen: Actually remote building is now systematically used on hydra https://github.com/NixOS/hydra/pull/874, with Nix having some logic to bypass ssh (a bit like what your gist does) if the remote is `localhost`
<{^_^}> hydra#874 (by regnat, 2 days ago, merged): Remove the `sendDerivation` logic from the builder
jonringer has joined #nixos-dev
jonringer has quit [Remote host closed the connection]
jonringer has joined #nixos-dev
<siraben> How do I make a new package set with an overlay? Say I wanted to make `pkgs.dockerImages.X` buildable, where `X` is a name of a package in Nixpkgs
<siraben> I looked into now pkgsMusl and other sets do it, using a function called nixpkgsFun (with a comment saying it's hacky apparently)
<siraben> s/now/how/
<siraben> looks like authored by Ericson2314
jonringer has quit [Ping timeout: 264 seconds]
page_ is now known as page
JJJollyjim has joined #nixos-dev
JJJollyjim is now known as Guest67960
Guest67960 has quit [Quit: authenticating]
Guest67960 has joined #nixos-dev
Guest67960 has quit [Client Quit]
Guest67960 has joined #nixos-dev
rnhmjoj has quit [Changing host]
rnhmjoj has joined #nixos-dev
rnhmjoj has quit [Changing host]
rnhmjoj has joined #nixos-dev
cole-h has quit [Ping timeout: 272 seconds]
sphalerite_ is now known as sphalerite
terrorjack has quit [Quit: The Lounge - https://thelounge.chat]
terrorjack has joined #nixos-dev
Guest67960 has quit [Quit: authenticating]
Guest67960 has joined #nixos-dev
Guest67960 is now known as JJJollyjim
JJJollyjim has quit [Client Quit]
JJJollyjim has joined #nixos-dev
JJJollyjim is now known as Guest26650
Guest26650 has quit [Client Quit]
Guest26650 has joined #nixos-dev
Guest26650 has quit [Client Quit]
Guest26650 has joined #nixos-dev
Guest26650 has quit [Client Quit]
__monty__ has joined #nixos-dev
ScottHDev5 has joined #nixos-dev
orivej has joined #nixos-dev
mkaito has joined #nixos-dev
risson_ is now known as risson
mkaito has quit [Quit: WeeChat 3.0]
orivej has quit [Ping timeout: 276 seconds]
<gchristensen> Regnat[m]1: yeah I saw that but I want to specifically test something where two stores are involved
<regnat[m]1> gchristensen: Ah it's not enough then, indeed 😕 .
<regnat[m]1> Maybe you could use a chroot store as the second store. Not sure hydra would like that though
<regnat[m]1> (I mean directly, not going through ssh at all)
mkaito has joined #nixos-dev
mkaito has joined #nixos-dev
mkaito has quit [Changing host]
<gchristensen> Regnat[m]1: that is an interesting idea
<clever> this might be of use
<clever> its a bash script to compile a new build of nix, that is rooted on /home/$X/nix/store
<gchristensen> I *think* I tried those once
<clever> but, it also throws a rootfs thing in, so it lands in /home/clever/rootfs/home/$X/nix/store
<clever> and it expects you to copy the whole thing to /home/$X/nix/store before it works
<clever> the end goal, is that you can build a nix that will work from the "wrong" dir, without having root on either machine
<clever> `ssh://root@target?remote-store=local?root=/mnt` may also be of use
<gchristensen> hmm
<clever> that connects to a remote machine (root@target) via ssh, but then tells the remote nix to open `local?root=/mnt` instead
<gchristensen> error: build of '/home/grahamc/projects/github.com/NixOS/hydra/meh/local/realstore/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' on 'ssh://bogus?store=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store&state=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/var/nix&log=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/var/log/nix' failed: error:
<gchristensen> dependency '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/x74z7ass7s9lgxgd35v03w171y17zzhf-builder.sh' of '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' does not exist, and substitution is disabled
<gchristensen> error: opening lock file '/home/grahamc/projects/github.com/NixOS/hydra/meh/local/nix/var/nix/current-load/ssh:__bogus?remote-store=local&store=_home_grahamc_projects_github.com_NixOS_hydra_meh_remote_nix_store&state=_home_grahamc_projects_github.com_NixOS_hydra_meh_remote_nix_var_nix&log=_home_grahamc_projects_github.com_NixOS_hydra_meh_remote_nix_var_log_nix-0': File name too long :D
<clever> lol
<gchristensen> error: build of '/home/grahamc/projects/github.com/NixOS/hydra/meh/local/realstore/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' on 'ssh://bogus?remote-store=local&store=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store&state=/home/grahamc/projects/github.com/NixOS/hydra/meh/rxvn&log=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/var/log/nix' failed:
<gchristensen> error: dependency '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/x74z7ass7s9lgxgd35v03w171y17zzhf-builder.sh' of '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' does not exist, and substitution is disabled
<Ericson2314> siraben: i've always beenn grumpy about the `pkgsBlah`
<Ericson2314> I think matthewbaueer did it
avn has joined #nixos-dev
<Ericson2314> niksnut: https://github.com/NixOS/nix/pull/4570 is pushed and passing CI
<{^_^}> nix#4570 (by Ericson2314, 3 days ago, open): Split {,local-}derivation-goal.{cc,hh}
<Ericson2314> (well, pushed -> newly rebased as discussed)
bennofs_ has joined #nixos-dev
srk has quit [Remote host closed the connection]
srk has joined #nixos-dev
jonringer has joined #nixos-dev
cole-h has joined #nixos-dev
jared-w has quit [Ping timeout: 272 seconds]
jared-w has joined #nixos-dev
orivej has joined #nixos-dev
justanotheruser has joined #nixos-dev
justan0theruser has quit [Ping timeout: 272 seconds]
rajivr has quit [Quit: Connection closed for inactivity]
<niksnut> Ericson2314: thanks, merged!
<Ericson2314> niksnut: thanks!
<Ericson2314> I found a deleeted test to restore
<Ericson2314> I'll make a PR for that
<Ericson2314> niksnut: https://github.com/NixOS/nix/pull/4580 here it is
<{^_^}> nix#4580 (by Ericson2314, 12 seconds ago, open): Restore now-working build-remote-content-addressed-fixed test
sterni has quit [Ping timeout: 260 seconds]
sterni has joined #nixos-dev
<niksnut> Ericson2314: hm, it's failing
<Ericson2314> niksnut: huh weird, maybe it was commented out for a reason?
<Ericson2314> I'll test harder locally
copumpkin has quit [Quit: Hmmm]
copumpkin has joined #nixos-dev
<NinjaTrappeur> rmcgibbo[m]: what's the point of https://github.com/NixOS/nixpkgs/pull/114489#issuecomment-786799135 ?
<NinjaTrappeur> ofborg is already building the package and running the passthrough tests.
averell has quit [Remote host closed the connection]
<cole-h> probably more useful for when packages are dependencies for others and not leaves
<rnhmjoj> i don't think ofborg tests the reverse dependencies of the packages touched by a PR
<NinjaTrappeur> right
<NinjaTrappeur> there's no reverse dependency here
<NinjaTrappeur> it's generating a spam email
<cole-h> it could though.... ;)
averell has joined #nixos-dev
averell has quit [Remote host closed the connection]
<NinjaTrappeur> why can't this post a PR status?
<cole-h> requires a token to the repo
<NinjaTrappeur> For a good reason.
<cole-h> pretty sure we don't want to be handing those out willy-nilly
averell has joined #nixos-dev
<NinjaTrappeur> this is not the first time I have to mute a nixpkgs review spambot
<rnhmjoj> yeah, you never know: i learned by using nixpkgs-review that some obscure haskell package (from the generated set) was a reverse dependency and broken when i bumped a python library
<NinjaTrappeur> this has to stop...
<adisbladis> I agree with NinjaTrappeur (and I've argued the same before)
<adisbladis> Github notifications are already spammy enough as-is
<symphorien[m]> The suggestions (when there are some) are often useful, to the defense of the bot
<NinjaTrappeur> right
<qyliss> I do like the suggestions, although I'd prefer that be a status check
<adisbladis> I don't think anyone is arguing against the functionality itself
<NinjaTrappeur> So, let's open a chat and let's host this bot with a NixPkgs token.
<NinjaTrappeur> I have no issue with the check.
<qyliss> ideally it would be
<qyliss> just part of OfBorg
* gchristensen puts up the cole signal
<cole-h> 👀
<cole-h> One moment..
<adisbladis> It's also pretty annoying when reviewing
<adisbladis> If a PR has comments I assume someone has started a review process
<qyliss> omg yes cole-h save us from the botpocalypse
<cole-h> Yes, I agree. I drafted an issue for "internal review" but am just now getting around to posting it :)
<sterni> supersandro2000: regarding #114290, i think it'd better probably to merge the initial commit now and then do the fixups making them all get evaluated in separate PRs
<{^_^}> https://github.com/NixOS/nixpkgs/pull/114290 (by sternenseemann, 1 day ago, open): treewide: add missing lib inputs
<NinjaTrappeur> > You must be a human to create an Account. Accounts registered by "bots" or other automated methods are not permitted. We do permit machine accounts:
<{^_^}> error: syntax error, unexpected ':', expecting ')', at (string):488:148
<{^_^}> ofborg#551 (by cole-h, 13 seconds ago, open): Build dependencies of packages
<qyliss> NinjaTrappeur: there's an official way to do bots with the API
<NinjaTrappeur> yes
<NinjaTrappeur> that's my point
<qyliss> oh, right
<sterni> supersandro2000: the subsequent changes should all be reviewed by different people probably like the maintainers for chicken or magnetophonDSP etc.
<qyliss> cole-h: do you have an issue for the lint checks as well?
<cole-h> ofborg is in a good place to be a self-hosted stand in for nixpkgs-review since it already has a list of affected attrs
<cole-h> qyliss: I added that as a sort of addendum at the bottom of that issue, since they're somewhat related
<cole-h> but I should probably make it a separate issue
<cole-h> (I'm pretty sure the lints used in r\mcgibbo[m]'s bot are from nixpkgs-hammering)
<samueldr> and depending on the implementation, could reduce the "surface" an equivalent API token actually has access too
<samueldr> (if you were to provide a way to puppeteer ofborg through an API)
<rmcgibbo[m]> NinjaTrappeur: The reason is because my build finished before ofborg.
<rmcgibbo[m]> Had ofborg finished the build before, it would not have been posted.
<{^_^}> ofborg#552 (by cole-h, 8 seconds ago, open): Investigate integration of nixpkgs-hammering
<gchristensen> I'd like to also +1 it being part of ofborg itself for purposes of community continuity
<rmcgibbo[m]> I'd like to turn the r-rmcgibbo bot into something that can post its results using the github Checks API rather than comments.
<NinjaTrappeur> rmcgibbo[m]: Congrats for winning the build race. Please try to get a token to send this as a status or upstream that to ofborg.
<qyliss> gchristensen: I wonder what could be done to make it easier to get OfBorg to do stuff like this in the first place
<gchristensen> however, I could also see the argument of https://twitter.com/SysQuotes/status/1364605332736073732
<cole-h> qyliss: Maybe one step would be to have ofborg be "pluggable"
<qyliss> yeah
<gchristensen> we could use github actions and host our own github actions runners
<qyliss> if you have a script interface so people could write checks in the language of their choice that would be helpful maybe
<rmcgibbo[m]> If someone makes an interface into ofborg that allows me to plug into it, I am 100% on that.
<cole-h> qyliss: Seems like a self-hosted GitHub Actions runner would be an ideal choice for that
<rmcgibbo[m]> Sounds amazing. I'm happy to send my reports to some kind of webhook or whatever.
<{^_^}> nix-community/infra#5 (by zimbatm, 1 year ago, open): WIP: github-actions
<gchristensen> ah
<adisbladis> It's pretty painful to build from source :/
<gchristensen> do we need to?
<rmcgibbo[m]> Integration with the github Checks API is obviously highly desired -- this isn't the first time it's come up -- it's just technically complicated and not the kind of thing I can unilaterally make happen today.
<gchristensen> moving to github actions means we could make a blanket rule of no bots without an RFC without stalling development, that might be the most equal thing to do
<gchristensen> I guess it is clear enough that I have no special attachment to ofborg, and any position I have on bots is more related to the effort to fit in to workflows in an unobtrusive way
<gchristensen> and to be incredibly careful with the X's
<NinjaTrappeur> rmcgibbo[m]: thanks for taking my feedback into account, that's greatly appreciated.
<NinjaTrappeur> I however feel like the token requirement to create a bug is a feature, not a bug. This is the kind of stuff that should get some level of community approval before rolling in. Especially when it impersonates a "real user" flow ending up generating emails/notifications.
<NinjaTrappeur> *to create a bot, not a bug ofc ><
<rmcgibbo[m]> If it's the consensus of the community that the r-rmcgibbo bot is providing negative value over the alternative of simply turning it off... if you just want me to turn it off -- then I'm fine with that. Obviously I'm biased, but it seems to me that it's a net plus even if there are some rough edges that could be improved.
<rmcgibbo[m]> Integration with the checks API (and being incredibly careful with the red Xs) is my end goal, and I think everyone agrees that it's better than either the current status quo with r-rmcgibbo or the pre r-rmcgibbo status quo.
<cole-h> gchristensen: So then (assuming we do get self-hosted GHA running), I guess the "integration of a bot" would be by sending a PR to some repo which will get added to the list of running checks on the next redeploy of whatever machine that is?
<gchristensen> it'd be a PR to nixpkgs to change its .github/workflow YAMLs
<cole-h> Oh, that's how we'd do it
<gchristensen> arguably that is a better workflow anyway, since it keeps the checks easy to see and possibly run locally
<cole-h> Yeah, right, OK
<cole-h> Brain is currently orienting
<gchristensen> rmcgibbo[m]: I don't think it is providing negative value
<rmcgibbo[m]> I've definitely _tried_ to get some level of community approval. But you, know, it's tricky. I've posted on discourse, I've hung out here in the irc rooms, I've talked extensively to Sandro who was running very similar checks before, I've talked to gchristensen a bit. But yeah it's nothing formal, no RFC.
<gchristensen> I think it is really hard to integrate well, and we're seeing chafing from that
<rmcgibbo[m]> (And I've also put in many tens of hours into improving nixpkgs-hammering, which keeps getting smarter and smarter.)
<NinjaTrappeur> +1 gchristensen
<NinjaTrappeur> rmcgibbo[m]: I don't have any gripe with your bot. My issue is the notification system it's currently using.
<gchristensen> Sandro's bot created a lot of chafing too :)
<qyliss> rmcgibbo[m]: fwiw I'm personally happy enough being on your special list that notifies me only when there's actually a problem
<qyliss> I wonder if that should be the default (unless you changed it already)
<rmcgibbo[m]> Re self-hosted GHA: is it possible for us to configure it in such a way that the GHA could ping me with a webhook, and I could respond once I've finished the build by pinging the GHA on a web hook? Or would something like my bot have to run on the same server that the GHA runner is running on.
<adisbladis> I think it's important to understand where these frustrations are coming from
<adisbladis> Even ofborg used to post comments
<gchristensen> my opinions here are more in the realm of: useful tools are great, but having them run by assorted people out of the goodness of their heart is less good, I'd rather have them run by a collective or the foundation or something to that effect
<cole-h> =1
<cole-h> ...+1
<adisbladis> cole-h: I mentally parse that as 0.001 ;)
<cole-h> hehe
<gchristensen> "very smol agree"
justan0theruser has joined #nixos-dev
<sterni> is there a good and intended way to have lib.types.attsOf which must have certain attributes?
justanotheruser has quit [Ping timeout: 260 seconds]
<rmcgibbo[m]> Agreed. Infrastructure-wise though, I think there are some advantages of having things a bit distributed. If the GHA runner machine has to run everything, it might be more challenging to do both x86_64 and aarch64, for example. Or it might be more complicated to handle failure modes like disk-full. r-rmcgibbo runs build in pretty ephemeral VMs that are spun up on demand, which seems like a good thing that we might not
<rmcgibbo[m]> want to give up when switching its method-of-posting-results to the checks API
<gchristensen> I think you want a submodule sterni
<adisbladis> sterni: I think you're looking for
<adisbladis> lib.types.submodule
<gchristensen> rmcgibbo[m]: yeah, if they were "just" GHA's I'd be inclined to manage them like the build farm's builders: ephemeral machines that get wiped daily
<rmcgibbo[m]> So, I think the big question then is... how do we get from "here" to "there"? It seems like we all agree that having more or less running nixpkgs-review + nixpkgs-hammering on every PR (or even every build of every PR, when there are force pushes) and posting the results using the green check mark API is where we want to get to, right?
<rmcgibbo[m]> How do we get there/
<rmcgibbo[m]> sorry for the typos.
<gchristensen> I'm a professional at typos and people don't seem to mind
<gchristensen> it sounds like cole-h is making noise about getting these things running in ofborg, I'd be interested in his input
<cole-h> Well, running nixpkgs-review (building changed attrs) is fairly trivial (at a glance -- haven't actually attempted it yet)
<gchristensen> I suspect some people would be unhappy with github actions due to vendor lock-in (though I'm not sure how much that holds up when the interesting part of the checks are not so locked in, the "lock in" is how to get the feedback posted) but not sure
<rmcgibbo[m]> (I don't quite understand the interplay between the Checks API and Github Actions, but that's on me.)
<qyliss> i'm one of the people you might expect to be unhappy about that, and I don't think it matters
<samueldr> if it's the OSS runtime for github actions that is being used as an interface, self-hosted, to me it doesn't sound too much like vendor lock-in... more like artificial interface limitations
<cole-h> vendor lock-in is a concern, but also is the same with ofborg. The only difference (IME) is that ofborg is written in Rust and deployed to "our" machines, while GHA runs on GitHub's machines
<gchristensen> qyliss: you're on the list :P
<gchristensen> samueldr: is also on that list
<qyliss> samueldr says it better than I could
<gchristensen> anyone want to get it packaged? :P
<cole-h> So my opinion is: I'd like to build dependent packages in ofborg since we already generate what's changed. But for nixpkgs-hammering integration: I don't care either way.
<adisbladis> gchristensen: I'd argue that a well done github actions flow has less lock-in than we have now
* gchristensen wanders off to go on a walk
<cole-h> In fact, it might be easier to add nixpkgs-hammering to a workflow so that it can be updated "easier"
<adisbladis> As long as all the github actions specifics are as thin as possible it's a clear improvement
<adisbladis> It should be easier to move if most of the logic is scripts in nixpkgs that are not directly concerned with what CI is running them
<rmcgibbo[m]> It's seems like some people are talking about GHA running on github's machines and some people are talking about GHA running on self-hosted machines?
<adisbladis> rmcgibbo[m]: I think we're all mostly talking about self-hosted
<gchristensen> yeah, we can't really use GH's machines for many of the important tasks since they have maybe even just a fraction of the ram we need
<rmcgibbo[m]> Okay. Sorry I guess I misunderstood what cole-h was saying.
<sterni> gchristensen: oh of course
<rmcgibbo[m]> Alright, well let me know if there's anything I can do to help. As you might imagine I have a bit of experience with the nuances of nixpkgs-review now. I have pretty accurate data for the build time associated with most of the attributes in nixpkgs currently, so I know how to predict the time requirements for builds and to filter the set of attrs to be built for large PRs, etc etc.
<rmcgibbo[m]> I'd love to get out of the business of operating the bot.
mkaito has quit [Quit: WeeChat 3.0]
justanotheruser has joined #nixos-dev
justan0theruser has quit [Ping timeout: 272 seconds]
cole-h has quit [Ping timeout: 265 seconds]
tom39291 has joined #nixos-dev
tom39291 has quit [Client Quit]
tom39291 has joined #nixos-dev
tom39291 has quit [Client Quit]
tom39291 has joined #nixos-dev
__monty__ has quit [Quit: leaving]
<jtojnar> rmcgibbo: looking at https://docs.github.com/en/rest/reference/checks#create-a-check-run, the comment issue could be switched to that with minimal changes
<jtojnar> main issue is that the check will probably disappear on push
<jtojnar> (from the checks box, it will probably be accessible through the commit, unless force pushed)