<Profpatsch>
(this is why most RFCs don’t lead to any actions I think)
<Profpatsch>
I feel that usually I can’t get much more done than one or two things in an 8 hour workday, and maybe review one or two small PRs
<colemickens>
danderson: I'd be sort of curious to hear about the process of trying to sell NixOS for production. Seems like an interesting challenge to think about- strategy, technical arguments, maybe some (reverse) psychology. Especially if it winds up being successful and it works out or blows up, etc.
<danderson>
colemickens: I don't have much to tell you there. I complained about how our ubuntu machines are already drifting out of spec and we only have 4 of them
<danderson>
and the CEO said "I don't think classic config management will ever work, I think something like nixos is the way to go"
<danderson>
so... I said hell yes, and I'm planning the migration :P
<danderson>
but the basic argument is that we're a security company, and a small company. We want an auditable OS, and the ability to know exactly what's running at all times.
<colemickens>
Interesting. That specifically is more important to you than reproducability/reliability?
<danderson>
NixOS is the only thing that fits the bill for change management. With a little tooling, I can exactly review the closure diff on all machines.
<colemickens>
Also, your job/product sound more fun each time I hear about it.
<danderson>
reliability - we're deliberately engineering simple systems. Little daemons run by systemd, nothing complex.
<danderson>
so, we really just need a robust linux userspace to run on, with a bit of monitoring.
<colemickens>
Yeah, that's very much how I feel about NixOS. Thinking of maintaining servers with Puppet, I just really don't like to even think about playbooks, etc.
<genesis>
i get some systemd issue if you are in :)
<genesis>
we lack of service dependancies.
<danderson>
reproducibility - I have more faith in nixos being reproducible than ubuntu, tbh
<danderson>
if I wanted to, I could disable the cache and rebuild our OS from scratch. Can't do that with ubuntu :)
<genesis>
you could have suprise anyway
<colemickens>
Yeah, that's more what I meant with reproducability/reliability since reproducability is sort of overload (or I misuse it). The fact that I can always clone my repo and `nix-build` and get /the exact same build/ (even if some bits are different), and see in one place exactly what versions are running, is just invaluable.
<colemickens>
To me, worth the time I spend maintaining/contributing to nixpkgs, rebasing and all.
<danderson>
right. And I think that's a better security posture than the alternative, which is "well, if we use Ubuntu we can tell Fortune 500s that we use Ubuntu, and they'll check the security box"
<danderson>
it'll take longer to explain how things work with a NixOS-based prod, but I think it's the right thing to do.
<genesis>
i'd gentoo to prod so i guess it's better :D
<danderson>
also, part of our product plan is supporting on-premises deployment. NixOS is amazing for that.
<danderson>
"here's an update. Didn't work? Hit the big red rollback button, and it's fixed"
<Dandellion>
do we track how big our binary seed is in nixpkgs? (for a super basic thing like the installation ISO, obviously excluding anything non-free)
<colemickens>
also something I've thought about for some sort of utopic, perkeep-in-the-cloud-and-at-home sort of product.
<danderson>
Anyway. Sorry for offtopic. Point is, I currently use NixOS on my personal computers, but Real Soon Now I'm going to run it in real prod
<danderson>
so I have a sudden increased interest in making sure nixpkgs doesn't die from the weight of PRs :)
<genesis>
1800, not so much
<Profpatsch>
danderson: If you make sure the packages you are using for prod get updated, there’s already a lot gained
bhipple has joined #nixos-dev
<genesis>
i tried to close a bunch, but i've no power, it seems people like ot alive :)
orivej has quit [Ping timeout: 240 seconds]
<genesis>
<danderson> Right now what I can do is walk around open PRs, going "sure, that looks fine. But I can't leave review comments, or merge, so what's the point?"
<genesis>
i think we get a point here anyway, people fear too much to do something
<danderson>
are there usage stats for nix derivations anywhere?
<danderson>
Working through the security bug list, one of them is easiest to fix by deleting a package closure that seems to be abandoned (>2y out of date, CVEs, etc. - upstream has updated since then).
<danderson>
But I'd like to know if this is something 2000 people are using without realizing it's super broken :)
<qyliss>
danderson: no
<qyliss>
Mostly, nobody is using abandoned software
<qyliss>
And if they are they can propose a revert.
<danderson>
even if upstream is still active? nixpkgs hasn't updated, but if someone does the work, it could be upgraded
<danderson>
marked vulnerable in nixpkgs in November last year, so it hasn't been _very_ long yet.
drakonis has quit [Quit: WeeChat 2.7.1]
<qyliss>
if it's abandoned upstream I'd be inclined not to bother waiting
<colemickens>
Does nix-review check commit messages? That would be really nice, even if it only did it in cases where there's only one commit to check since I think it's expensive to do the diff.
<qyliss>
We should really have some guidance around when it's okay to use binaries in packages, I think
<qyliss>
Downloading and patching a binary built by somebody else who knows how, when source is available, feels to me like defeating the purpose of Nix...
Jackneill has joined #nixos-dev
FRidh has joined #nixos-dev
<qyliss>
At the very least there should be some clear marker, like a mandatory -bin suffix
<qyliss>
Or so goes my current thinking
<colemickens>
I am surprised when I find out a package isn't built from source and lacks a -bin suffix, for sure.
<andi->
qyliss: I feel the same. What is your thought on those pre-minified JS bundles that we sometimes fetch?
<lovesegfault>
I second colemickens, bin pkgs without a -bin suffix always surprise me
__monty__ has joined #nixos-dev
orivej has joined #nixos-dev
<Profpatsch>
andi-: I guessi it depends on whether they are executed as nodejs or in the browser
<niksnut>
once flakes have landed, it will become more feasible to move binary packages out of nixpkgs
<timokau[m]>
I'm still not very happy about that prospect, I think splitting up the monorepo will introduce more problems than its worth. Creating a multi-tier system within nixpkgs that people can opt-in and opt-out of would be better in my opinion.
<Profpatsch>
we can already mark broken and nonfree packages, nothing speaks against adding a tag for binary packages
<Profpatsch>
no need to ever split this out of nixpkgs
<adisbladis>
I'm a bit torn on this issue.. On the one hand it's awesome to be able to grep through nixpkgs for pretty much anything you want to find out more about. OTOH splitting things up a bit can ease maintenance of large ecosystems like https://github.com/nix-community/emacs-overlay which is currently periodically copied into nixpkgs.
<niksnut>
Profpatsch: I disagree. For example, I maintain a blender-bin flake providing blender binaries. That's not the kind of thing that should be in Nixpkgs (being intended as a source distribution).
<niksnut>
also, not being in nixpkgs means it doesn't have to go through the nixpkgs PR bottleneck
<FRidh>
reducing the main repo should allow us to focus on core packages instead of causing us to avoid working on those because of the potential work it generates fixing reverse dependencies.
<FRidh>
at the same time...a flake is not really that different from a container or any other lock file, which means the flake maintainer becomes responsible for updating it and its dependencies.
<flokli>
I think a lot of the PRs in nixpkgs are a) stalling because there's no clear consensus about how to proceed, so they just stall indefinitely b) either awaiting a follow-up review from the reviewer or awaiting feedback from the reporter
<FRidh>
but that's solvable with CD again
<flokli>
Some more clear guidelines for a), and more automation for b) could help keep the count down
<flokli>
and allow more throughput for the remaining PRs, because it's not that much
<niksnut>
FRidh: CD?
<FRidh>
continuous delivery
<niksnut>
flokli: a) is not solvable in general. The best solution is to have a clearly defined scope, which means saying no more often.
<FRidh>
or, actually, its more trying to automatically update and release
<flokli>
so often I click at a issue/PRs, and was like, ah, I already read that, it's still waiting feedback or an upstream fix
<niksnut>
flokli: the great thing about putting things in a separate repo is that you don't *need* consensus
<flokli>
well, if it's a NixOS-wide thing, it need it
<niksnut>
sure, there are core packages/modules that should not be moved out
<flokli>
fine for having multiple different approaches to a small flavour of doing things in a minimal subset
<flokli>
but one can't easily shift things like cross-compilation or the vm test infrastructure out
<flokli>
same goes for a subset of the python packageset - it's just required by too many packages of the base system.
<niksnut>
not cross-compilation, but vm testing absolutely could be
<niksnut>
for example the perl vm testing infrastructure could be moved into a separate flake for users that still need it
<flokli>
niksnut: the perl one maybe
<timokau[m]>
We could still achieve the scope reduction by having support tiers in nixpkgs, with the lowest tier saying "PR reviews only need to check that they are non-malicious, breakage on dependency updates is not checked".
<flokli>
but the tests are linked from packages
<timokau[m]>
That way we don't need external repos, we don't need to keep saying no. We can gradually move a package up the tiers if it seems well-maintained and there is some evidence of usage.
<gchristensen>
having low quality packages in nixpkgs costs people's time, effort, money, and reputation
<timokau[m]>
Point is that I see very little advantage of splitting up the repo, but quite some disadvantages
<flokli>
I think packages should have a certain quality level to get into (and stay in) nixpkgs. Back in the past, we haven't been that strict on that, but with the grown size, we have to.
<niksnut>
gchristensen: right, and it bloats the repo
<flokli>
I don't think randomly now "externalizing" packages helps
<niksnut>
and you still have nixpkgs as a bottleneck
<flokli>
if a package is badly maintained, and nobody cares, drop it. somebody will pick up, either maintains it externally, or reintroduces it with better quality
<niksnut>
imagine a world where (say) every rust crate had to exist inside a single monorepo
<gchristensen>
the only one which sticks out as definitely viable, to me, is the haskell package set -- which is already maintained in that style.
<gchristensen>
I don't feel we *need* to break up nixpkgs. but we don't have a good way for people to *not* put a package in nixpkgs, and still distribute it. that feels like the big change, to me.
<timokau[m]>
niksnut: That is not really comparable. Distros are all about integration, which is why a monorepo is good in this case
<michaelpj>
the metadata about rust crates *does* exist inside a single monolithic source of truth: crates.io
<FRidh>
gchristensen: due to dependencies towards the haskell package set, the small nixpkgs would have to choose to include the generic builder and some packages in nixpkgs
<FRidh>
same goes for every other language in the end
<niksnut>
rust crates also need to be integrated (maybe more tightly than typical packages)
<gchristensen>
FRidh: nixpkgs can still depend on the haskell set
<timokau[m]>
Yes I agree its nice to have the option to put a package outside of nixpkgs, but I do not think that we should split up the existing set or deny new (well-defined) packages because of it
<niksnut>
michaelpj: but you don't actually need to use it, since you can also point to your repos from cargo.toml
<FRidh>
gchristensen: right, in that case we have effectively the same content just spread over multiple repo's
<gchristensen>
but it is maintained in that style already. bulk, atomic updates merged on a weekly basis. the difference is each week our repo grows by a couple mb, instead of 5kb
<FRidh>
except the loads of individual packages
<gchristensen>
hm?
<michaelpj>
I think we'd rapidly find ourselves with a whole new set of challenging integration problems, just this time at the flake level
<niksnut>
yes, it does shift the responsibility onto the user
<gchristensen>
I don't feel like we need to break up nixpkgs. I feel nixpkgs changes the game and equation about whether people should/need-to upstream something to nixpkgs
<FRidh>
yes it would become easier for them to choose to distribute expressions themselves, which is now not as easy
<timokau[m]>
+1, but I still think we should not split up the existing set and also not increase the hurdles for adoption too much. If a package is not very popular yet but has a highly-motivated maintainer, it is suitable for adoption in my opinion.
<gchristensen>
what is that curses program for exploring disk usage?
<timokau[m]>
ncdu
<michaelpj>
gchristensen: ncdu?
<gchristensen>
thanks, yeah
<gchristensen>
./objects/pack
<gchristensen>
36G
claudiii has joined #nixos-dev
<adisbladis>
gchristensen: Not curses, but filelight is my favourite tool to explore disk usage
<adisbladis>
For this task a gui makes a lot of sense
<gchristensen>
cool
bhipple has joined #nixos-dev
<FRidh>
pff, I don't really like the glibc bump, changing python or python.buildEnv now requires a stdenv rebuild :(
<gchristensen>
ow.
<gchristensen>
niksnut: btw, have you followed this ca-references breakage?
<niksnut>
I saw an issue about it I think
ixxie has quit [Ping timeout: 265 seconds]
<gchristensen>
nix-copy-closure on .drv's doesn't work anymore, is I think the long and short of it
bhipple has quit [Ping timeout: 260 seconds]
<LnL>
FRidh: that's why I duplicated the (minimal) python expression in python/boot.nix before
<LnL>
glibc and llvm only need it to run a build script, so none of the things like setup-hooks are needed at that point
<jtojnar>
but the bind : int option, forces you to use port-only bind without arguments
<infinisil>
jtojnar: It can still be overridden if necessary
<ash_braker>
jtojnar: How about bind: string instead?
<jtojnar>
infinisil not easily afaict
<infinisil>
This is the cool thing about these settings options, as you can just do `services.smartdns.settings.bind = ":53 -no-rule -group example"`
<ash_braker>
infinisil: overridden could be also possible but I think string could be better because we then don't need to remind users to set in `extraConfig` instead
<infinisil>
ash_braker: (With what I commented there is no extraConfig anymore)
<jtojnar>
infinisil even when it has type = int?
<ash_braker>
infinisil: (fine, then I mean `settings`
<infinisil>
jtojnar: The type of `services.smartdns.bindPort` doesn't influence what values are allowed in services.smartdns.settings.bind
<jtojnar>
infinisil but then we would get two bind directives in the config, would not we?
<ash_braker>
(then I think string could be better to proceed)
<infinisil>
jtojnar: Since it will be specified as `services.smartdns.settings.bind = mkDefault ":${toString cfg.bindPort}"`, this setting will be overridden by another one without a priority
<jtojnar>
ooh
<infinisil>
ash_braker: Whatever you think is better works, string or an int
<jtojnar>
that sounds nice, thanks for explaining
<infinisil>
I personally would think an int is better, because it's very easy to understand and one doesn't need to document the string format
<infinisil>
And people can still override it if necessary (but using settings)
<jtojnar>
do not we have types.port even?
<jtojnar>
that would be great for bindPort option
<infinisil>
Oh yeah we do
<ash_braker>
infinisil: thanks for explanation.
<infinisil>
:)
<ash_braker>
I decided to remove `servers` option because it would be obstructive for me to put `server` into `setting` with mkDefault
<genesis>
Mic92 : i don't see hw you have time to close something i explicitly ask to not close, and no time for #82266
<ash_braker>
infinisil: So there would be only `cfg.bind` and `cfg.settings`. Will it be a little unnecessary to keep it outside there?
<ash_braker>
If we cut `cfg.bind` off, then there would be only one universal `cfg.settings`
<infinisil>
The disadvantage of this is that no options is cfg.settings are type checked or documented
<infinisil>
But if you don't see much value behind other options then that's fine by me
<Mic92>
genesis: well closing a PR is easier than having to review and test something to be merged.
<ash_braker>
infinisil: Anyway, then I keep bind there. I think documentation should not be the issue as long as I give some distinctive and illustrative examples.
<arcnmx>
is that why the channels haven't been updating? o:
<gchristensen>
arcnmx: which?
<ash_braker>
A quick ask: can I put multiple examples?
<arcnmx>
gchristensen: all afaict
orivej has quit [Ping timeout: 240 seconds]
<gchristensen>
hm, not sure
<gchristensen>
are they not updating?
__monty__ has quit [Quit: leaving]
<infinisil>
ash_braker: Unfortunately not
<infinisil>
As of now anyways, I should look into changing it so that's possible
<ash_braker>
hmmm, then I might give a big comprehensive example instead
<arcnmx>
gchristensen: I haven't noticed any movement in a few days and https://status.nixos.org/ seems to agree, though hydra appears to have been building the whole time?
<arcnmx>
the test links show successful builds at least
<jtojnar>
ash_braker you are encouraged to add module doc
<ash_braker>
jtojnar: Thanks for your suggestion. I will first finish the code
<samueldr>
(maybe it's eval from derivation that we need to think about?)
<adisbladis>
Let's just keep the acronym and change the meaning to Input From Derivation
<infinisil>
samueldr: Oh that sounds better
<samueldr>
adisbladis: { stdenv, hello }: stdenv.mkDerivation { buildInputs = [ hello ]; } that's an input, from derivation?
<adisbladis>
samueldr: Darn it
<samueldr>
[15:05:08] <samueldr> though that's not right, since derivations can be input, don't mind me
<samueldr>
:D
<samueldr>
literally my train of thought
<infinisil>
,IFD
<{^_^}>
import-from-derivation (IFD) is when you evaluate nix from a derivation result, for example `import (pkgs.writeText "n" "1 + 1")` will evaluate to 2. This is sometimes problematic because it requires evaluating some, building some, and then evaluating the build result. It has been described as "such a nice footgun."
<samueldr>
>> when you evaluate nix from a derivation result
<samueldr>
EFD!
<gchristensen>
Input-from- is good I like that
<samueldr>
no, it can't work! forget I ever said it!
* infinisil
likes eval-from-derivation
<infinisil>
How about "multi-stage evaluation"
<infinisil>
Because instead of a single stage "eval/instantiate -> build" it's "eval/instantiate -> build -> eval/instantiate -> build"
Jackneill has quit [Remote host closed the connection]
<infinisil>
Or "derivation-dependent evaluation"
<infinisil>
"eval sandwich" :D
<infinisil>
"build-eval-build sandwich"
<infinisil>
Wait it should be "eval-build-eval sandwich" (because the last build step might not be needed)
<cole-h>
How about just sandwich
<cole-h>
Make everybody hungry
<infinisil>
Perfect
<infinisil>
I think I like "derivation-dependent evaluation" the most though
<infinisil>
Makes it very clear what it is
<drakonis>
so, will impure derivations get merged?
<drakonis>
based on the latest flakes meeting
<drakonis>
it'd serve quite well as means for a nix CI
orivej has joined #nixos-dev
<niksnut>
maybe as an experimental feature
psyanticy has quit [Quit: Connection closed for inactivity]
FRidh has quit [Quit: Konversation terminated!]
<aminechikhaoui>
what's impure derivations ?
<drakonis>
a derivation that yields different results every time it is executed
<drakonis>
in constract with pure derivations, which will always yield the same output
<drakonis>
its particularly useful for fetching the latest versions of software made available in repositories
<drakonis>
building off the development repository rather than tarballs
<aminechikhaoui>
ah so a combination of fixed-output derivations without the hashes
<drakonis>
i can see it being useful for fetching nightly rust binaries
<aminechikhaoui>
ah good I was just about to ask about interaction with binary caches
<drakonis>
there's a lot that can be done with this
<adisbladis>
Wow, that's terriawesome
<drakonis>
feature request: optionally output the hash obtained from fetching the input file to a nix file
<drakonis>
for pinning purposes
<samueldr>
since FOD can depend on an impure derivation, don't you already have that?
<drakonis>
FOD?
<samueldr>
fixed-output derivation
<drakonis>
no, this is for moving targets
<drakonis>
as input
<samueldr>
[16:18:21] <drakonis> feature request: optionally output the hash obtained from fetching the input file to a nix file
<samueldr>
you can do that by making an FOD depend on the impure derivation, then pin the tofu'd hash
<samueldr>
unless I'm missing something
<drakonis>
its accessory to using impure derivations, if i build something with impure mode, i'd like to have the file used to generate the impure derivation to be stored along with it, but with a pinned hash
<drakonis>
for that particular attempt
<drakonis>
if it is necessary to rebuild that specific derivation again
<drakonis>
there is a content addressed hash from what i read
ixxie has quit [Ping timeout: 256 seconds]
rajivr___ has quit [Quit: Connection closed for inactivity]
<drakonis>
i'm not sure if i'm being clear here
<samueldr>
reading the motivation for tarball/unpacked I think that what you need is already present, though I may be missing something from what you want
<drakonis>
what i want is to have a impure derivation also generate a FOD mirror of the nix file used to generate the derivation
<drakonis>
optionally
<drakonis>
for the purposes of pinning things that can be pinned.
<drakonis>
i think i used FOD wrong here
<samueldr>
the impure derivation may not have enough information for it to be useful, unless you trace all system calls to figure out what git commit ID was used by `git clone $repo` and then that's going to be hell
<samueldr>
or in the example of date > $out, nothing can help pin it
<drakonis>
oh no i just want the input hash, i dont want the rest of that
<drakonis>
but then, i suppose that's not very useful for the majority of cases
<samueldr>
I think you'll have to figure out a strategy to keep the relevant information for your use cases yourself
<drakonis>
ah, well then.
<samueldr>
e.g. if it's the git commit ID, git rev-parse > $out/.git-commit
<samueldr>
if it's a "release/latest/source.tar.gz" kind of URL that redirects, maybe curl can be coaxed in outputting the URL it redirects to
<samueldr>
I may still be missing what you want/need though
<drakonis>
actually that'd be what i want but its not a viable thing to do
<samueldr>
the main issue I can see from what I understand of your needs, is that there's just no way to know what the impure derivation will do
<drakonis>
i see.
<samueldr>
though a generic "fetchImpureGitBranch" could be made with well-known paths to figure things out
<colemickens>
It can handle git/mercurial sources from github and non-github sources.
<colemickens>
Spits out a metadata.nix file that can be used to pull in pinned nixpkgs, or follow a branch for a package's metadata rev/sha256.
<samueldr>
yeah, I think the main difference is that now this can be a nix derivation
<drakonis>
and it'd make so many things easier
* colemickens
reads a bit more scrollback and gets more of the picture. ignore me!
<drakonis>
how viable would it be to add packages that always fetch the latest tag?
<samueldr>
colemickens: no worries, an apt comment, that this is an incremental change that may help some use cases, but nothing ground-breaking :)
<samueldr>
drakonis: problematic considering any build with that impure system is not cached, unless you cache it with an FOD, and there you won't follow the latest build unless you invalidate the sha... unless I'm missing something in that change
<drakonis>
actually they are cached with it
<drakonis>
content addressed
<drakonis>
hm
<drakonis>
oops i read it wrong.
<drakonis>
okay fair enough.
<samueldr>
they are saved in a content-addressed location, but the result is not cached
<samueldr>
yeah
<drakonis>
i would propose to have a mechanism to convert them onto FODs
<drakonis>
that's what we were talking about earlier
<drakonis>
rather, what i was waffling about.
<drakonis>
the first paragraph regarding does mention the ability to implement a mechanism for reusing impure builds.
<drakonis>
i feel i should stop.
<drakonis>
i'm getting ahead of myself, as it should be first made available before these concerns can be addressed in practice
<drakonis>
ah, i think i have confused pure derivations with FOD
bhipple has joined #nixos-dev
Jackneill has joined #nixos-dev
abathur has quit [Ping timeout: 256 seconds]
abathur has joined #nixos-dev
Jackneill has quit [Remote host closed the connection]
bhipple has quit [Read error: Connection reset by peer]
teto has quit [Quit: WeeChat 2.7.1]
abathur has quit [Ping timeout: 258 seconds]
bhipple has joined #nixos-dev
<aminechikhaoui>
hm nix-build -K doesn't work with distributed builds ?