<gchristensen>
note the second paste is a script named `ssh` in PATH, which is the key to making this "work"
<gchristensen>
I've futzed with it a bit more and can't seem to persuade it to work without spelunking into the code and perhaps a debugger. if anyone has ideas, I'd love to put this to work for testing Hydra's remote builder behavior :)
<gchristensen>
but I'm off to bed, g'night 'all
orivej has quit [Ping timeout: 272 seconds]
justanotheruser has quit [Ping timeout: 264 seconds]
tv has quit [Ping timeout: 246 seconds]
tv has joined #nixos-dev
justanotheruser has joined #nixos-dev
BaughnLogBot has quit [Ping timeout: 272 seconds]
BaughnLogBot has joined #nixos-dev
aszlig_ is now known as aszlig
jonringer has quit [Remote host closed the connection]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 272 seconds]
janneke_ is now known as janneke
<regnat[m]1>
gchristensen: Actually remote building is now systematically used on hydra https://github.com/NixOS/hydra/pull/874, with Nix having some logic to bypass ssh (a bit like what your gist does) if the remote is `localhost`
<{^_^}>
hydra#874 (by regnat, 2 days ago, merged): Remove the `sendDerivation` logic from the builder
jonringer has joined #nixos-dev
jonringer has quit [Remote host closed the connection]
jonringer has joined #nixos-dev
<siraben>
How do I make a new package set with an overlay? Say I wanted to make `pkgs.dockerImages.X` buildable, where `X` is a name of a package in Nixpkgs
<siraben>
I looked into now pkgsMusl and other sets do it, using a function called nixpkgsFun (with a comment saying it's hacky apparently)
<clever>
the end goal, is that you can build a nix that will work from the "wrong" dir, without having root on either machine
<clever>
`ssh://root@target?remote-store=local?root=/mnt` may also be of use
<gchristensen>
hmm
<clever>
that connects to a remote machine (root@target) via ssh, but then tells the remote nix to open `local?root=/mnt` instead
<gchristensen>
error: build of '/home/grahamc/projects/github.com/NixOS/hydra/meh/local/realstore/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' on 'ssh://bogus?store=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store&state=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/var/nix&log=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/var/log/nix' failed: error:
<gchristensen>
dependency '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/x74z7ass7s9lgxgd35v03w171y17zzhf-builder.sh' of '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' does not exist, and substitution is disabled
<gchristensen>
error: opening lock file '/home/grahamc/projects/github.com/NixOS/hydra/meh/local/nix/var/nix/current-load/ssh:__bogus?remote-store=local&store=_home_grahamc_projects_github.com_NixOS_hydra_meh_remote_nix_store&state=_home_grahamc_projects_github.com_NixOS_hydra_meh_remote_nix_var_nix&log=_home_grahamc_projects_github.com_NixOS_hydra_meh_remote_nix_var_log_nix-0': File name too long :D
<clever>
lol
<gchristensen>
error: build of '/home/grahamc/projects/github.com/NixOS/hydra/meh/local/realstore/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' on 'ssh://bogus?remote-store=local&store=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store&state=/home/grahamc/projects/github.com/NixOS/hydra/meh/rxvn&log=/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/var/log/nix' failed:
<gchristensen>
error: dependency '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/x74z7ass7s9lgxgd35v03w171y17zzhf-builder.sh' of '/home/grahamc/projects/github.com/NixOS/hydra/meh/remote/nix/store/m8wyzn96afyw8sr3lcj32n3lp9w2wa0s-my-build-product.drv' does not exist, and substitution is disabled
<Ericson2314>
siraben: i've always beenn grumpy about the `pkgsBlah`
<NinjaTrappeur>
ofborg is already building the package and running the passthrough tests.
averell has quit [Remote host closed the connection]
<cole-h>
probably more useful for when packages are dependencies for others and not leaves
<rnhmjoj>
i don't think ofborg tests the reverse dependencies of the packages touched by a PR
<NinjaTrappeur>
right
<NinjaTrappeur>
there's no reverse dependency here
<NinjaTrappeur>
it's generating a spam email
<cole-h>
it could though.... ;)
averell has joined #nixos-dev
averell has quit [Remote host closed the connection]
<NinjaTrappeur>
why can't this post a PR status?
<cole-h>
requires a token to the repo
<NinjaTrappeur>
For a good reason.
<cole-h>
pretty sure we don't want to be handing those out willy-nilly
averell has joined #nixos-dev
<NinjaTrappeur>
this is not the first time I have to mute a nixpkgs review spambot
<rnhmjoj>
yeah, you never know: i learned by using nixpkgs-review that some obscure haskell package (from the generated set) was a reverse dependency and broken when i bumped a python library
<NinjaTrappeur>
this has to stop...
<adisbladis>
I agree with NinjaTrappeur (and I've argued the same before)
<adisbladis>
Github notifications are already spammy enough as-is
<symphorien[m]>
The suggestions (when there are some) are often useful, to the defense of the bot
<NinjaTrappeur>
right
<qyliss>
I do like the suggestions, although I'd prefer that be a status check
<adisbladis>
I don't think anyone is arguing against the functionality itself
<NinjaTrappeur>
So, let's open a chat and let's host this bot with a NixPkgs token.
<NinjaTrappeur>
I have no issue with the check.
<qyliss>
ideally it would be
<qyliss>
just part of OfBorg
* gchristensen
puts up the cole signal
<cole-h>
👀
<cole-h>
One moment..
<adisbladis>
It's also pretty annoying when reviewing
<adisbladis>
If a PR has comments I assume someone has started a review process
<qyliss>
omg yes cole-h save us from the botpocalypse
<cole-h>
Yes, I agree. I drafted an issue for "internal review" but am just now getting around to posting it :)
<sterni>
supersandro2000: regarding #114290, i think it'd better probably to merge the initial commit now and then do the fixups making them all get evaluated in separate PRs
<NinjaTrappeur>
> You must be a human to create an Account. Accounts registered by "bots" or other automated methods are not permitted. We do permit machine accounts:
<{^_^}>
error: syntax error, unexpected ':', expecting ')', at (string):488:148
<qyliss>
NinjaTrappeur: there's an official way to do bots with the API
<NinjaTrappeur>
yes
<NinjaTrappeur>
that's my point
<qyliss>
oh, right
<sterni>
supersandro2000: the subsequent changes should all be reviewed by different people probably like the maintainers for chicken or magnetophonDSP etc.
<qyliss>
cole-h: do you have an issue for the lint checks as well?
<cole-h>
ofborg is in a good place to be a self-hosted stand in for nixpkgs-review since it already has a list of affected attrs
<cole-h>
qyliss: I added that as a sort of addendum at the bottom of that issue, since they're somewhat related
<cole-h>
but I should probably make it a separate issue
<cole-h>
(I'm pretty sure the lints used in r\mcgibbo[m]'s bot are from nixpkgs-hammering)
<samueldr>
and depending on the implementation, could reduce the "surface" an equivalent API token actually has access too
<samueldr>
(if you were to provide a way to puppeteer ofborg through an API)
<rmcgibbo[m]>
NinjaTrappeur: The reason is because my build finished before ofborg.
<rmcgibbo[m]>
Integration with the github Checks API is obviously highly desired -- this isn't the first time it's come up -- it's just technically complicated and not the kind of thing I can unilaterally make happen today.
<gchristensen>
moving to github actions means we could make a blanket rule of no bots without an RFC without stalling development, that might be the most equal thing to do
<gchristensen>
I guess it is clear enough that I have no special attachment to ofborg, and any position I have on bots is more related to the effort to fit in to workflows in an unobtrusive way
<gchristensen>
and to be incredibly careful with the X's
<NinjaTrappeur>
rmcgibbo[m]: thanks for taking my feedback into account, that's greatly appreciated.
<NinjaTrappeur>
I however feel like the token requirement to create a bug is a feature, not a bug. This is the kind of stuff that should get some level of community approval before rolling in. Especially when it impersonates a "real user" flow ending up generating emails/notifications.
<NinjaTrappeur>
*to create a bot, not a bug ofc ><
<rmcgibbo[m]>
If it's the consensus of the community that the r-rmcgibbo bot is providing negative value over the alternative of simply turning it off... if you just want me to turn it off -- then I'm fine with that. Obviously I'm biased, but it seems to me that it's a net plus even if there are some rough edges that could be improved.
<rmcgibbo[m]>
Integration with the checks API (and being incredibly careful with the red Xs) is my end goal, and I think everyone agrees that it's better than either the current status quo with r-rmcgibbo or the pre r-rmcgibbo status quo.
<cole-h>
gchristensen: So then (assuming we do get self-hosted GHA running), I guess the "integration of a bot" would be by sending a PR to some repo which will get added to the list of running checks on the next redeploy of whatever machine that is?
<gchristensen>
it'd be a PR to nixpkgs to change its .github/workflow YAMLs
<cole-h>
Oh, that's how we'd do it
<gchristensen>
arguably that is a better workflow anyway, since it keeps the checks easy to see and possibly run locally
<cole-h>
Yeah, right, OK
<cole-h>
Brain is currently orienting
<gchristensen>
rmcgibbo[m]: I don't think it is providing negative value
<rmcgibbo[m]>
I've definitely _tried_ to get some level of community approval. But you, know, it's tricky. I've posted on discourse, I've hung out here in the irc rooms, I've talked extensively to Sandro who was running very similar checks before, I've talked to gchristensen a bit. But yeah it's nothing formal, no RFC.
<gchristensen>
I think it is really hard to integrate well, and we're seeing chafing from that
<rmcgibbo[m]>
(And I've also put in many tens of hours into improving nixpkgs-hammering, which keeps getting smarter and smarter.)
<NinjaTrappeur>
+1 gchristensen
<NinjaTrappeur>
rmcgibbo[m]: I don't have any gripe with your bot. My issue is the notification system it's currently using.
<gchristensen>
Sandro's bot created a lot of chafing too :)
<qyliss>
rmcgibbo[m]: fwiw I'm personally happy enough being on your special list that notifies me only when there's actually a problem
<qyliss>
I wonder if that should be the default (unless you changed it already)
<rmcgibbo[m]>
Re self-hosted GHA: is it possible for us to configure it in such a way that the GHA could ping me with a webhook, and I could respond once I've finished the build by pinging the GHA on a web hook? Or would something like my bot have to run on the same server that the GHA runner is running on.
<adisbladis>
I think it's important to understand where these frustrations are coming from
<adisbladis>
Even ofborg used to post comments
<gchristensen>
my opinions here are more in the realm of: useful tools are great, but having them run by assorted people out of the goodness of their heart is less good, I'd rather have them run by a collective or the foundation or something to that effect
<cole-h>
=1
<cole-h>
...+1
<adisbladis>
cole-h: I mentally parse that as 0.001 ;)
<cole-h>
hehe
<gchristensen>
"very smol agree"
justan0theruser has joined #nixos-dev
<sterni>
is there a good and intended way to have lib.types.attsOf which must have certain attributes?
justanotheruser has quit [Ping timeout: 260 seconds]
<rmcgibbo[m]>
Agreed. Infrastructure-wise though, I think there are some advantages of having things a bit distributed. If the GHA runner machine has to run everything, it might be more challenging to do both x86_64 and aarch64, for example. Or it might be more complicated to handle failure modes like disk-full. r-rmcgibbo runs build in pretty ephemeral VMs that are spun up on demand, which seems like a good thing that we might not
<rmcgibbo[m]>
want to give up when switching its method-of-posting-results to the checks API
<gchristensen>
I think you want a submodule sterni
<adisbladis>
sterni: I think you're looking for
<adisbladis>
lib.types.submodule
<gchristensen>
rmcgibbo[m]: yeah, if they were "just" GHA's I'd be inclined to manage them like the build farm's builders: ephemeral machines that get wiped daily
<rmcgibbo[m]>
So, I think the big question then is... how do we get from "here" to "there"? It seems like we all agree that having more or less running nixpkgs-review + nixpkgs-hammering on every PR (or even every build of every PR, when there are force pushes) and posting the results using the green check mark API is where we want to get to, right?
<rmcgibbo[m]>
How do we get there/
<rmcgibbo[m]>
sorry for the typos.
<gchristensen>
I'm a professional at typos and people don't seem to mind
<gchristensen>
it sounds like cole-h is making noise about getting these things running in ofborg, I'd be interested in his input
<cole-h>
Well, running nixpkgs-review (building changed attrs) is fairly trivial (at a glance -- haven't actually attempted it yet)
<gchristensen>
I suspect some people would be unhappy with github actions due to vendor lock-in (though I'm not sure how much that holds up when the interesting part of the checks are not so locked in, the "lock in" is how to get the feedback posted) but not sure
<rmcgibbo[m]>
(I don't quite understand the interplay between the Checks API and Github Actions, but that's on me.)
<qyliss>
i'm one of the people you might expect to be unhappy about that, and I don't think it matters
<samueldr>
if it's the OSS runtime for github actions that is being used as an interface, self-hosted, to me it doesn't sound too much like vendor lock-in... more like artificial interface limitations
<cole-h>
vendor lock-in is a concern, but also is the same with ofborg. The only difference (IME) is that ofborg is written in Rust and deployed to "our" machines, while GHA runs on GitHub's machines
<gchristensen>
qyliss: you're on the list :P
<gchristensen>
samueldr: is also on that list
<qyliss>
samueldr says it better than I could
<gchristensen>
anyone want to get it packaged? :P
<cole-h>
So my opinion is: I'd like to build dependent packages in ofborg since we already generate what's changed. But for nixpkgs-hammering integration: I don't care either way.
<adisbladis>
gchristensen: I'd argue that a well done github actions flow has less lock-in than we have now
* gchristensen
wanders off to go on a walk
<cole-h>
In fact, it might be easier to add nixpkgs-hammering to a workflow so that it can be updated "easier"
<adisbladis>
As long as all the github actions specifics are as thin as possible it's a clear improvement
<adisbladis>
It should be easier to move if most of the logic is scripts in nixpkgs that are not directly concerned with what CI is running them
<rmcgibbo[m]>
It's seems like some people are talking about GHA running on github's machines and some people are talking about GHA running on self-hosted machines?
<adisbladis>
rmcgibbo[m]: I think we're all mostly talking about self-hosted
<gchristensen>
yeah, we can't really use GH's machines for many of the important tasks since they have maybe even just a fraction of the ram we need
<rmcgibbo[m]>
Okay. Sorry I guess I misunderstood what cole-h was saying.
<sterni>
gchristensen: oh of course
<rmcgibbo[m]>
Alright, well let me know if there's anything I can do to help. As you might imagine I have a bit of experience with the nuances of nixpkgs-review now. I have pretty accurate data for the build time associated with most of the attributes in nixpkgs currently, so I know how to predict the time requirements for builds and to filter the set of attrs to be built for large PRs, etc etc.
<rmcgibbo[m]>
I'd love to get out of the business of operating the bot.
mkaito has quit [Quit: WeeChat 3.0]
justanotheruser has joined #nixos-dev
justan0theruser has quit [Ping timeout: 272 seconds]