<domenkozar[m]>
Consider releasing 19.03, there's only kernel 5.0 which you won't include, then 5.1 comes out but that one is also left out because it will be EOL before NixOS 19.09
<domenkozar[m]>
so who are you preventing to shoot themselves in the foot vs helping with bleeding edge kernel is a though one :)
<andi->
I think the fine line here is that we always have (had?) a latest/testing kernel but not old/no longer maintained releases. Newer kernels might be requires for hardware support.
<andi->
*required
<domenkozar[m]>
but I agree that including kernels that do go EOL during our release cycle without any notification support from our side is tricky
<ekleog>
Well, IMO stable is stable, and if you need new hardware support you just take the kernel from unstable. Nowwe might benefit from better tooling around mixing things from multiple nixpkgs versions
<domenkozar[m]>
goes back to philosophy of decisions behind NixOS, things like freedom to pick software vs security
<ekleog>
OTOH, I do understand that there is sometimes a need for bleeding-edge stuff, but that's just our failing to make mix-and-match easy
<Taneb>
(point of data: to get my laptop to work I needed a kernel >= 4.20)
<ekleog>
domenkozar[m]: we could have both: people on stable with a kernel from unstable would totally make sense to me
<Taneb>
(I'm currently using 5.0 from nixpkgs 19.03)
<domenkozar[m]>
> if you need new hardware support you just take the kernel from unstable
<domenkozar[m]>
I'm not sure though, if I needed that on the server for some reason, I'd be totally unhappy, but as you mentioned there are workarounds
<{^_^}>
error: syntax error, unexpected ')', expecting THEN, at (string):256:1
<ekleog>
domenkozar[m]:well, the kernel *is* unstable anyway, might as well make it explicit :)
<andi->
Isn't that mostly up to those people that are doing the kernel work in NixOS? If they can support newer kernels on stable releases that a good thing. Not something we've to make (more) painful for users. If those people decide to no longer do that for whatever reason it is fine..
<ekleog>
having unstable kernels on our stable series is luring people into a false sense of stability
<domenkozar[m]>
unstable still breaks from time to time and being forced to use it due to missing stable kernel is a bit inconvenient
<domenkozar[m]>
so it's a trade off :)
<ekleog>
andi-:the thing is currently we do have the latest kernel, but it just doesn't respect any kind of stability, and 5.0 got dropped from 19.03 just before -- so people cannot actually support newer kernels on stable releases
<timokau[m]>
Couldn't we just have one `linux-latest` or even `linux-unstable` attribute which is the only unstable one we maintain on stable releases?
<ekleog>
domenkozar[m]:does unstable's 5.0 kernel break more often than 19.03's 5.0 kernel? I'd be surprised
<timokau[m]>
The instability should be obvious then, and EOL versions can be replaced by the current newest kernel when they go EOL
<ekleog>
timokau[m]:it would at least make it explicit that we don't give any stability guarantees around it
<ekleog>
thouhg I really believe we should make it trivial to use a kernel from unstable with userspace from 19.03 (or the opposite), which would solve the issue by itself -- your solution is a nice stop-gap, though :)
<andi->
ekleog: those people that were on 5.0 can switch to 5.1... Maybe we just have to make it more explicit that those "intermediate" kernel releases might disappear in the nixos release as soon as upstream stops supporting them or it becomes unfeasible for us..
<domenkozar[m]>
I dislike `unstable` indirection, usually I want 5.0 or X.X due to some kernel bugs
<domenkozar[m]>
so then it's just a matter of time until someone bumps that to 5.2 and boom
<andi->
timokau[m]: there is linux_testing and linux_latest IIRC.. and I am infavor of just using those as kernels for people to go to.. No guarantees that the version number is stable or not..
<ekleog>
domenkozar[m]:well, that's exactly the issue with having 5.0 in stable: we're guaranteeing that we'll support the system for 6 months, and in the middle of said 6 months we droip the 5.0 (because w can not reasonably be expected to handle backports ourselves)
<qyliss>
Agree. There shouldn’t be attributes in stable that we don’t know we’ll be able to support for the lifetime of the release.
<qyliss>
linux_latest is fine. linux_5_0 is not. linux_4_19 is, because it’s an LTS.
<domenkozar[m]>
ekleog: I'm not entirely sure, I think our stance is "we support what upstream supports at given time"
<ekleog>
qyliss:even linux_latest is edge-y to me: the same nixos configuration, automatically upgraded by some script, may just upgrade the kernel and break
<domenkozar[m]>
I'd only dare to say "we're guaranteeing we'll support" with commercial support
<domenkozar[m]>
i.e. redhat
<ekleog>
domenkozar[m]:it definitely is our stance for unstable. I don't see how we could call our releases stable with that definition :)
<domenkozar[m]>
(hope someone does that for NixOS)
<ekleog>
right, the “guarantee” is with 0%SLA, but you get the idea
<{^_^}>
#63735 (by Ekleog, 1 minute ago, open): manual: remind to drop kernels that will get EOL'd
<domenkozar[m]>
guarantees without consequences are zero guarantes in my eyes :)
<domenkozar[m]>
and probably in eyes of business in general
<domenkozar[m]>
but anyway I'm nitpicking sorry.
<qyliss>
ekleog: I think that’s up to the user. It’s pretty clear you’re opting out of stability in your kernel at that point, but you’re at least not going to be left with an insecure kernel.
<domenkozar[m]>
someone should do commercial NixOS support
<domenkozar[m]>
problem solved :P
<ekleog>
qyliss:let me suggest the un-suggest-able: what if we hadan “unstable” attribute in our release that'd point to the latest nixos-unstable
<gchristensen>
no thanks
<qyliss>
impurely?
<ekleog>
domenkozar[m]:I've seriously thought about that at some point. Just find me enough people ready to pay for it and I'm willing to try and backport all the things :p
<gchristensen>
typically commercial distro support is more "take me back as closely as possible to 1995 as you can get me"
<domenkozar[m]>
doesn't need to be typical
<domenkozar[m]>
always room for inovation
<ekleog>
it's un-suggest-able, and I don't really want that. The overall idea being, if we had that, then all would be good from an end-user's perspective (assuming it's pure and we just bump a hash every day in the release branch or something)
<domenkozar[m]>
ekleog: it will certainly be a long term effort
<domenkozar[m]>
the question is how to start small :)
<domenkozar[m]>
there's 0 commercial support
<domenkozar[m]>
so the bar is low :)
<samueldr>
(x-post from #nixos-aarch64) we currently don't build the aarch64 package set in nixos-19.03; I think simply adding an aarch64 jobset for nixos like previous should be enough, even without adding a channel that tracks it; tests are still ran for 19.03, so that would be more about covering the binary cache
<domenkozar[m]>
if I'm a company that needs such support I'd rather pay 1 person to work on it rather than have no choice
<domenkozar[m]>
my 2c, bbl
<ekleog>
and now, if we take that same un-suggest-able idea, and turn it into “we can easily match channels” (eg. having the release + unstable channels by default and the two package sets imported by default), then all is good, isn't it?
<gchristensen>
typically commercial distro support is more "take me back as closely as possible to 1995 as you can get me"
<gchristensen>
oops
<qyliss>
ekleog: I’m hesitant to endorse any idea that uses channels, although honestly I think that makes sense given the current channel model.
<ekleog>
domenkozar[m]: isn't it what companies that hirepeople from mayflower or similar pay for?
<ekleog>
qyliss:well, I don't really see non-power-users using something else than channels :)
<domenkozar[m]>
ekleog: yeah but it's still more like consulting than a service/product
<qyliss>
ekleog: not right now, but flakes may change this
<qyliss>
(as skeptical as I am)
<ekleog>
flakes will be the whole new world, when we'll understand what they are and it's still true the following day :p
<ekleog>
domenkozar[m]: the thing is, as far as I understand, the idea would be 1. take a given nixpkgs, 2. freeze all versions in time, do nothing but backports, debian-like, 3. iterate for many years
<gchristensen>
commercial support for NixOS is quite complicated, as a company choosing to do that would likely be imposing a lot of rules around stable and what gets backported when, which may not be compatible with the community's way of doing it
<ekleog>
gchristensen: stable rules would likely be debian-stability-like rules, which would be completely incompatible with the current community effort
<gchristensen>
exactly
<qyliss>
gchristensen: I expect they would have their own tree, and hopefully upstream things that made sense given the community’s rules
<qyliss>
(norms is probably a better word)
<gchristensen>
right
<domenkozar[m]>
yeah it would be a fork of stable
<domenkozar[m]>
that seams reasonable?
<ekleog>
well, to be precise (I've thought just a bit about it), it'd be possible to reuse the community effort to know which things may need fixing and easily identifying the places that are vulnerable
<ekleog>
like “release bumped package XXX to version VVV, let's see the changelog and add the relevant patches”
<gchristensen>
perhaps though a better route would be having them work with the community to define more carefully what the stable release means, and not fork
<ekleog>
gchristensen:well, the first issue is already the one of release duration -- a commercial release that lasts only 6 months isn't going to attract anyone I think
<qyliss>
especially since there’s no time to upgrade
<qyliss>
If you don’t upgrade immediately on release you’re sort of out of luck, with our current model
<andi->
It will probably be very difficult to come to a common ground.. e.g. we commonly bump some packages (sqlite, nss, nspr, …) every 4-5 months even on stable branches because things depend on them. Not sure if that would still fit on their "stable" description and would probably impose a lot of extra work to volunteers if we have to obey to some of thsoe rules :/
<gchristensen>
the duration a release is maintained is not fixed in stone to be 6mo
<qyliss>
No, but in practice people stop backporting to older releases
<gchristensen>
andi-: true, and definitely an important consideration
<domenkozar[m]>
I'd start by building on top of stable, extending the EOL
<gchristensen>
qyliss: in this model of course the company would likely do a lot of that work
<qyliss>
That’s true
<domenkozar[m]>
usually company would innovate in it's own bucket, then propose changes upstream
<gchristensen>
yeah
<gchristensen>
well so I think there isn't an appetite for commercial support sufficiently large to fund how much work would need to be done
<domenkozar[m]>
gchristensen: based on a feel or some research you did?
<gchristensen>
I've scoped it out a bit :)
<domenkozar[m]>
I'm asking as I'm curious
<samueldr>
I guess that the only reason that no release has been LTS yet, is that there's not been any outside help in helping us (in all ways) maintain a branch for that long
<gchristensen>
it is very expensive to maintain a stable release
<domenkozar[m]>
gchristensen: could start very lean, by just offering consulting work on this area
<domenkozar[m]>
slowly building infrastructure and understanding the case better
<gchristensen>
and not just expensive now, but expensive forever, and it would be very harmful to start a stable release as a business and then fail a couple years in, as that is not stable. it would tarnish it. so, something like having 5 yrs of funding ready to invest
<gchristensen>
yeah, the consulting stuff we already do at Tweag
<domenkozar[m]>
but noone knows about that
<gchristensen>
oh :)
<domenkozar[m]>
or is there a marketing for it?
<gchristensen>
we're working on that part
<domenkozar[m]>
marketing site*
<gchristensen>
also since Tweag has hired Eelco, I think it makes it much trickier to do something like "a stable NixOS", as we must be very careful to not give impressions like Tweag is stealing Eelco, or harming Nix, etc.
<domenkozar[m]>
> be very harmful to start a stable release as a business and then fail a couple years in, as that is not stable. it would tarnish it. so, something like having 5 yrs of funding ready to invest
<{^_^}>
error: syntax error, unexpected THEN, expecting ')', at (string):255:61
<andi->
(just leaving this here: and then there is the whole (social?) problem with people that do it in the spare time vs those being paid for it… It is a though problem alltogether. ;))
<domenkozar[m]>
ohh I so much disagree :D but IRC is the worst form
<gchristensen>
andi-: VERY tough :P
<domenkozar[m]>
andi-: we already have that for a long time
<domenkozar[m]>
so I guess having more people paid makes things better?
<andi->
well we certainly need the project sturctures around that to avoid things moving in the direction of a few companies with money..
<gchristensen>
it is definitely a goal of mine to get as many Nix people in to jobs which pay them to do Nix stuff
<gchristensen>
andi-++
<{^_^}>
andi-'s karma got increased to 13
<andi->
I am already disliking how flakes doesn't have an RFC before implementation ;)
<qyliss>
andi++
<qyliss>
andi-++
<{^_^}>
andi-'s karma got increased to 14
<ekleog>
andi-:++
<domenkozar[m]>
could be Tweag spin off ;)
<ekleog>
andi-++
<{^_^}>
andi-'s karma got increased to 15
<qyliss>
A commercial supporter would probably need to do something about time to channel update too
<andi->
(But I am also not the biggest fan of the RFC process as is and I can't express things in a timely fashion..)
<qyliss>
It shouldn’t take a couple of days for a critical security fix to become available because hydra is flakey
<domenkozar[m]>
andi-: also one of the reasons I didn't join Tweag :)
<gchristensen>
that is tricky too, because it was in a fork but that was weird ("is this a takeover?") and at the moment it is largely PoC work -- which I think can be helpful to have prepared a PoC for an RFC (working code making it easier to provide feedback on)
<gchristensen>
qyliss: you should see how long it takes Debian and RedHat ...!
<gchristensen>
domenkozar[m]: why is thath?
<andi->
or SuSe..
<qyliss>
gchristensen: I make beating them a point of personal pride :P
<andi->
qyliss: we are good on that most of the time..
<samueldr>
I *think* that for stable, the minimum time is Σ(tests) + new stuff to build; but it's not queued in front of already existing jobs
<gchristensen>
^
<ekleog>
qyliss: presumably a commercial supporter would have their own hydra anyway, and only adding patches should make less test failures (ideally none as things are supposed to be _stable_)
<domenkozar[m]>
gchristensen: I'd like that we can build ecosystem where people are paid and centralizing that seems wrong
<domenkozar[m]>
because it limits experimentation
<gchristensen>
+1
<domenkozar[m]>
I prefer failure over bullet proofing things
<qyliss>
domenkozar[m]++
<{^_^}>
domenkozar[m]'s karma got increased to 2
<gchristensen>
seems good
<domenkozar[m]>
otherwise I wouldn't start cachix/hercules
<domenkozar[m]>
I guess that's the TLDR version for IRC :D
<qyliss>
It would be interesting to know roughly how many people are paid to work on nix things, how much of their time they spend on it, where money comes from, etc
<qyliss>
I know of Mayflower, Target, Tweag and NLnet
<gchristensen>
domenkozar[m]: thank you for doing that :) I feel it is critical to have many different companies doing Nix things, and not few hot spots
orivej has quit [Ping timeout: 272 seconds]
<ekleog>
qyliss: https://standard.ai/ also does nix stuff (I'm currently long-term-contracting for them)
<qyliss>
Oh nice
<ekleog>
(though my work is half-way between rust and nix, and until now has too rarely been upstream-able stuff)
<domenkozar[m]>
so in my eyes, if someone start commercial support for NixOS and fails that's ok, there's lots of lessons learned and someone else can take over. failure happens over many different reasons, but the market stays
<domenkozar[m]>
there's a difference between firing people before their 5 year cliff and being human while failing
<qyliss>
Oh and edef does nix things about mutable.io
<gchristensen>
edef++
<{^_^}>
edef's karma got increased to 2
<andi->
commercial support != LTS versions.. I think commercial support is largely accepts just doing LTS releases is a bit more complicated
<ekleog>
something that's possible too is to guarantee 1 year really-stable release, and have that extend-able depending on the amount of nfunding received
<gchristensen>
that is not exactly a thing a company would pay for
<andi->
but that requires proper ahead of time planning e.g lining up with kernel release schedules, browers, …
<ekleog>
well, it allows the company to know it can still pay more to get more years if things fail
<ekleog>
like, bootstrap of such a system is always going to be experimental anyway
<gchristensen>
andi-: that is a great point
<ekleog>
andi-:I think for browsers even debian doesn't try to just backport issues, does it?
<andi->
gchristensen: what makes you say that? In the past I've always been told $company is using $Distri LTS because of the overhead of having to upgrade every year. Sure that might be less painful with Nix/NixOS but they probably wouldn't believe it right away. :/
aszlig has quit [Quit: Kerneling down for reboot NOW.]
<gchristensen>
andi-: the thing not interested in paying for is a 1 year LTS which "might" go longer. typically if they're paying for an LTS, they want an LTS for the whole duration for sure
<andi->
ahh, thats your point. Agreed.
aszlig has joined #nixos-dev
<ekleog>
gchristensen:reword this in “this is a subscription-based LTS where the sum price paid by all subscribers is $X”
<ekleog>
$X/yr *
<ekleog>
and once the thing is stable it can give a fixed price
<ekleog>
plus it has nice network effects where companies will encourage other companies to join
<gchristensen>
yeah, I'm not sure that would work
<gchristensen>
but maybe? :P
<andi->
I'd rather see people being employed to do that kind of work full-time but that is probably a few orders of magnitude too high for our current users.. And if enough of those people exist they can see if it makes sense to run an LTS release..
<ekleog>
dunno either
<andi->
it is not like without an LTS release we do not have enough work to do..
<ekleog>
first question being “what would $X be?”
<gchristensen>
the first question is what does LTS mean. the second question what is $X
<ekleog>
well I assume LTS is somewhere between 5 and 10 years
<ekleog>
long enough that on average the patch backport is painful but hopefully aligned to debian|rhel releases
cjpbirkbeck has joined #nixos-dev
<andi->
10y is a loong time.. I'd rather see users moving to a "faster" cycle of upgrades. Maybe providing better upgrade tests/compatiblity would be worth spending money on?
<gchristensen>
;+1
<ekleog>
debian is 3 years full support + 2 years LTS, or so it seems
<andi->
I do see reasons to run systems for 10y+ (spacecrafts, maybe some science labs like LHC?) but usually most "online" systems shouldn't be untouched for 10y.
<ekleog>
RHEL appears to be ~4 years, plus ~4 years “extended”
<ekleog>
andi-:go tell that to the people who actually buy LTS support
<andi->
but isn't LTS always an excuse to not touch your systems. To run them in the way you'be been running them for the last 20y?
<andi->
ekleog: I tell them whenever I talk to people about it.. It is complete nonsense to freez things for 10y.. Maybe if you are on the project for 8.5y and do not want to touch it ever again..
<ekleog>
so yeah basically start a LTS at the same time a RHEL version gets out, and piggy-back on it
<gchristensen>
sounds expensive
<ekleog>
?
Guanin has joined #nixos-dev
<gchristensen>
we'd then have slower security updates than RHEL, and unless at the RHEL stable time we made sure all our packages matched theirs, we'd maybe not really be advantaged at all
<ekleog>
right, I assumed it'd be based on using the same package versions as RHEL's
<andi->
What's the scope of packages you consider for LTS? Every, ruby, python, haskell, rust,.. Package in nixpkgs? If so that is really expensive. I'd expect that someone running LTS also invests in security/Stability research of the releases and not just piggy backing.
<ekleog>
andi-: my assumptions of what people want from LTS, potentially ill-placed, are “stability and security”. There's no really better way than piggybacking on rhel/debian for the time being afaik -- thouhg that doesn't preclude also monitoring the news
alpounet has joined #nixos-dev
<samueldr>
remember that some distro's LTS are not all-encompassing; [citation needed] IIRC ubuntu's LTS do not cover desktop and graphical apps at all
<timokau[m]>
qyliss: Since you mentioned you don't like channels: What are you doing instead?