samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs: https://logs.nix.samueldr.com/nixos-infra/
cole-h has quit [Quit: Goodbye]
tilpner_ has joined #nixos-infra
tilpner has quit [Ping timeout: 260 seconds]
tilpner_ is now known as tilpner
LnL- has joined #nixos-infra
LnL- has joined #nixos-infra
LnL- has quit [Changing host]
cole-h has joined #nixos-infra
niksnut has joined #nixos-infra
cole-h has quit [Quit: Goodbye]
Mic92 has joined #nixos-infra
domenkozar[m] has joined #nixos-infra
<Mic92> Right now we don't have a macos builder in our ci for quite a while. That's why we want to use github actions. Domen offered to add a cachix key into github, which we can use to build & cache packages.
<Mic92> Also the macos builder was restricted to trusted users, which github action would not be.
<Mic92> What is your opinions on that.
<Mic92> which allows to build the nixpkgs manual, which tends to break every once in a while because it is not built.
<domenkozar[m]> I think moving to github actions would allow also to have way more automation and having that transparent and easy to change
<niksnut> BTW we do have macOS builders in our CI and they're working: https://hydra.nixos.org/machines
<Mic92> I am talking about ofborg here. https://monitoring.nix.ci/d/000000002/ofborg?orgId=1&refresh=10s we cannot use hydra to review pull request.
<Mic92> And for security it is probably the best to keep hydra seperate from pull requests.
<gchristensen> maybe we could buy a mac, or find budget for a rented one, or convert a hydra one to ofborg for the time being.
<gchristensen> (or move away from ofborg to github actions)
<Mic92> That is an alternative
<Mic92> But I really would keep ofborg / github actions seperate from hydra. There could be always security vulnerabilities in the builder, which should be kept away from the build building the official binary cache.
<Mic92> gchristensen: but would this allow also contributors to build packages?
<domenkozar[m]> note there's also https://github.com/NixOS/nix/issues/4047
<gchristensen> not really, macos has poor isolation. maybe it could work if macos was used as a remote builder, instead of running ofborg directly on a mac
<gchristensen> then, at least, no secrets would be in the mac
<gchristensen> this could work with a virtualised macos -- like how hydra.n.o runs them (and how I ran my mac before)
<domenkozar[m]> why not use microsofts free infra in that case? :)
<niksnut> speaking of which, are we likely to run into CPU limits or whatever?
<domenkozar[m]> nope
<niksnut> there are no limits?
<niksnut> so say a chromium build won't get killed?
<domenkozar[m]> not for OSS
<niksnut> oh great
<domenkozar[m]> oh per build?
<domenkozar[m]> per build it's 12h IIRC
<gchristensen> whoa
<Mic92> well. this limit is higher than ofborg's 60min limit
<gchristensen> can we create _a lot_ of builds? :) (like, hundreds of thousands a day ...)
<niksnut> then we should just move the entire nixpkgs channel build to github
<niksnut> provided we can manage the cache.nixos.org signing key securely
<Mic92> Maybe we should ask github about that first
<domenkozar[m]> Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.
<domenkozar[m]> Each workflow run is limited to 72 hours. If a workflow run reaches this limit, the workflow run is cancelled.
<domenkozar[m]> I mean not using this infra and building it ourselves is quite irrational :D
<Mic92> To manage the signing key security I would create a repo seperate from nixpkgs
<domenkozar[m]> gchristensen: we probably could, but there's also concurrency limits
<domenkozar[m]> to you're limited by concurrency + workflow limit
<domenkozar[m]> 20 concurrent jobs + 72h workflow limit
<domenkozar[m]> which is quite a lot :D
<Mic92> yes.
<domenkozar[m]> (macos has 5 concurrent jobs)
<gchristensen> please don't move actual builds to github actions :o
<niksnut> 'You can run an unlimited number of jobs'
<niksnut> I wonder what would happen if we actually create a workflow with 80K jobs...
<domenkozar[m]> probably nothing
<domenkozar[m]> they'd weight we're a very popular repo
<domenkozar[m]> they run on their own infra, it's basically free for them
<niksnut> can jobs be dynamically populated or do they have to be statically defined in .github/workflows?
<niksnut> it would be annoying to have to define a job for every attribute in pkgs
<gchristensen> oh no
<gchristensen> this can't be real, lol
<domenkozar[m]> you probably could generate the yml file
<domenkozar[m]> and trigger another build
<domenkozar[m]> so eval would get attributes
<domenkozar[m]> and then push a commit that would generate jobs to build each one
<domenkozar[m]> you might need some way to check for dependencies though
<domenkozar[m]> I'd start with a simple job though
<domenkozar[m]> ohhhhh
<domenkozar[m]> The Actions expression language has a fromJSON(value) method that can take a stingified JSON object and bind it to a property.
<domenkozar[m]> Combining this with job.outputs you can build a workflow that has a fully dynamic matrix.
<domenkozar[m]> so you could basically make a remote builder on github actions :D
<domenkozar[m]> eval -> generate jobs
<domenkozar[m]> wait for dependencies to land in binary cache, build
<Mic92> domenkozar[m]: I would not be surprised if we hit other limits with this.
<domenkozar[m]> it would be quite expensive if you had to pay for minutes though
<gchristensen> we're not talking about actually building all of nixpkgs using actions, right?
<domenkozar[m]> Mic92: well yeah, you can have 20 concurrent jobs
<domenkozar[m]> gchristensen: no, I meant this only for PRs
<Mic92> that sounds fine
<domenkozar[m]> anyway, probably best to have one job per platform for now
<domenkozar[m]> 6h is plenty to build stuff
<zimbatm> my understanding is that the 20 concurrency limit applies over the whole repo
<zimbatm> meaning that one PR could easily block everybody else
<Mic92> zimbatm: realistically 20 builds is more than what ofborg currently offers
<Mic92> I think we should limit the maximal build so.
<Mic92> fromJSON() in github action is really cool. I will use for NUR at some point
<domenkozar[m]> so what's the conclusion? :D
<zimbatm> I don't see why it can't be tried at minimum
cole-h has joined #nixos-infra
niksnut has quit [Ping timeout: 256 seconds]
<cole-h> Has anybody looked into https://github.com/NixOS/nixpkgs/issues/98281