<ekleog>
(hopefully 1.severity:blocker is the correct tag for “let's not release while this isn't fixed”)
<samueldr>
right, but as far as NixOS is concerned, has the location of things changed? or the module changed settings in an incompatible way, not considering schema upgrades from the software
<ekleog>
the nixos config stays exactly the same, you can nixos-rebuild --upgrade but then if you nixos-rebuild --rollback nextcloud can't start any longer
<samueldr>
room, correct me if I'm incorrect, but I'm pretty sure this isn't a case of stateVersion :/
<ekleog>
… wait that's what I understood as the whole meaning of stateVersion?
<samueldr>
when the nixos internals change with relation to the module
<samueldr>
e.g. location of /var/ data being moved...
<gchristensen>
hrm we had a similar stateversion pin around postgres
<samueldr>
there's the case of postgreSQL default changing, e.g. superuser from root to postgres
<samueldr>
but yeah, I was about to say: there's precedent in postgreSQL pinning a version
<samueldr>
but they are concurrently maintained version
<samueldr>
I'm kinda confident in assuming that rollback to previous releases of many "stateful" software will fail within NixOS... and might be somewhat out of scope, but would be great if we had a better story about that
<gchristensen>
yeah, I'm not sure what to do here
<gchristensen>
I feel backing up data prior to major releases should be a thing people do
<samueldr>
having to keep all software versions for everything stateful for rollback purposes behind stateVersion is not manageable
<gchristensen>
+1
<ekleog>
hmm my understanding of stateVersion was “changing this might need manual intervention, be it for the change of for the rollback of the change”, which made sense for me, but it's true that we might be underutilizing this
<samueldr>
and furthermore, gating the nexcloud version behind stateVersion would preclude keeping a previous stateVersion due to *another* stateful thing needing it
<ekleog>
well, it would make sense to me for software that aren't forward-compatible, but then I'm always erring on the side of “we should just keep all versions of all packages in nixpkgs so long as it's not becoming a burden in download times & co”, so… :°
<samueldr>
e.g. on a 17.09 stateVersion, I upgrade to 19.03 and need/want nextcloud 15; I shouldn't need to change stateVersion to 19.03; especially since here it means the big changes in the postgreSQL "nixos schema" things (var file locations, superusers)
<gchristensen>
yeah, that isn't a thing we do, for sure
<gchristensen>
stateVersion is generally pretty nasty
<samueldr>
stateVersion is really about "oopsie nixos is incompatible with older nixos releases" and not about the internals of the software
<gchristensen>
yea
<ekleog>
well, I react this way because I know I tried to upgrade one of my machines like a week ago and failed to find a correct opensmtpd config after its config file overhaul, and had to rollback until I find further time… and I'm happy to have attempted that upgrade on not-my-nextcloud-server because otherwise I'd have been blocked in a state where either opensmtpd or nextcloud was broken
<ekleog>
(or I should have restored from a backup, which is time-consuming)
<gchristensen>
I think a better route is to ensure the release notes indicate that rolling back is not possible due to the nextcloud data thing, and to encourage backups to users of that service
<samueldr>
though the notice is really "anything stateful might not rollback right"
<samueldr>
which also is valid for user software
<gchristensen>
hehe
<gchristensen>
while true, we have more specific knowledge here :P
<samueldr>
I'm thinking that only noticing nextcloud may end up leading the user to think "I'm not using nextcloud, no need to backup my $server stateful data"
<samueldr>
"so I'm safe to rollback"
<gchristensen>
cool
<gchristensen>
good point
<ekleog>
well, as we're fixing the world I'd have hoped we would have some way of avoiding that and preserving the “nixos-rebuild --rollback works unless one does an action that explicitly breaks it”, but I guess we're not there yet, then
<gchristensen>
Please take backups before each major NixOS version upgrade, as some services may have incompatible updates which prevent rollbacks. In particular, the 18.09 to 19.03 upgrade includes nextcloud..."
<ekleog>
(closed the abovementioned issue as it was not a case of stateVersion)
vcunat has joined #nixos-dev
<vcunat>
gchristensen: about staging... do you prefer to discuss it publicly in this channel?
<gchristensen>
vcunat: sure!
<gchristensen>
btw I emailed vcunat asking about staging workflow and what the status is, since it feels many people don't understand how to operate it
<vcunat>
OK, I've been too busy lately around ietf.org meeting and other work-related stuff, so I wasn't doing anything bigger around nixos.org.
<gchristensen>
ietf work sounds awesome :)
<vcunat>
Apparently @FRidh also wasn't working on staging stuff, and noone else either :-)
<gchristensen>
:D
<vcunat>
I hear the workflow is unclear, so we'll have to improve the docs, but first I suppose we'll do an iteration or two to make master up to date.
<gchristensen>
okay, that sounds good
<gchristensen>
I am curious what sort of active participation there is, and which parts are more "automatic"
<vcunat>
The next step blocking us is to stabilize what we have in the staging-next branch.
<vcunat>
and I didn't have enough time for triaging/fixing them.
<vcunat>
Oh, and 19.03 hasn't happened yet, either, has it?
<gchristensen>
hopefully very soon
<vcunat>
I'm confident I'll have at least one day during this weekend for NixOS stuff, but the question is what needs doing the most :-)
<gchristensen>
I think you're clear of obligations on 19.03 :P
<vcunat>
I see staging-next regressions on x86_64-linux are just a few now. The other two platforms have thousands each, though most might be transient.
<vcunat>
I'm not aware of any real NixOS obligations for me :-)
<gchristensen>
me either :)
<gchristensen>
I am inclined to retry that llvm build and see what happens
<gchristensen>
but that feels crappy
<vcunat>
Human time is more precious than machine time.
<gchristensen>
+1
* gchristensen
presses the big "restart all failed" button
<gchristensen>
we might want to mark llvm as big-parallel
<andi->
I'd like to help stabilizing the things but so far the states of the branches have been a bit opaque to me. I have a very own pet peeve: the openssl-1.1 branch looks like it could go into staging soon. Haven't really found anything that is caused by it vs "normale" staging breakage..
<vcunat>
well, release-19.03 with its ZHF is clear option to "stabilize"
<vcunat>
and currently the most lag is in the staging-next branch which needs stabilization before getting merged to master
<vcunat>
I see we got a large security rebuild today to all branches (gettext #58997).
<vcunat>
I saw people complaining the same way on the corresponding RFC.
<vcunat>
So at some point we'd better do someting about that, too.
<vcunat>
The point was that most changes would be merged to staging, just as it used to be.
* andi-
feels very sorry for having merged that
<gchristensen>
no worries
<gchristensen>
andi-: it isn't a big mistake, and if it all comes down to it, undoing it is a revert and a revert-revert away :)
<vcunat>
I thought it's normal for CVE fixes to get fast-forward to staging-next, so they're not blocked by more work on stabilization due to more intrusive changes.
* gchristensen
writes that down
<andi->
vcunat: that "security" issue is mostly just an edge case on some cli tooling
<andi->
so I intended to let it go the "slower" route
<vcunat>
Oh, OK. So I'll "move" that merge to staging.
<andi->
thanks!
<gchristensen>
vcunat: I guess in summary, I would prefer you spend time on getting staging/staging-next moving8 a bit
<gchristensen>
vcunat: if I can help you by doing something, or providing a big machine for you to work with, I can do that
<vcunat>
OK. I have the community machine for aarch64 and my cheap 16-threaded x86 desktop, so that will most likely be fine, as usual.
<gchristensen>
cool
<samueldr>
so, staging-next is "stabilize-to-merge-in-master" and staging is "break-stuff-and-fix-please" right?
<andi->
that sounds about right from my previous knowledge and the now (again) swapped branch names.
<vcunat>
Well, ideally even staging should take too risky changes directly.
<vcunat>
But we had the problem that merging new mass-rebuild stuff was disturbing stabilization efforts.
<vcunat>
gchristensen: BTW, no news about the bigmac machine? Hydra's been certainly bottle-necked on darwin in the past weeks, and this machine would make a big difference.
<gchristensen>
bigmac went to the big datacenter in the sky
<gchristensen>
I can try and bring it back to life
<gchristensen>
but it would be (again, just like before) temporary
<vcunat>
My core2duo mac-mini has been dead for months, but that's such negligible impact...
<samueldr>
can we get someone like eelco working for apple? :)
<samueldr>
(on nix stuff)
<vcunat>
Apple would pay someone to do nix stuff?
<gchristensen>
wishful thinking
<gchristensen>
vcunat: I will try to bring it back to life this weekend
<vcunat>
A few years ago it actually seemed quite hard to get paid for nix stuff (by anyone).
<gchristensen>
indeed ...
<vcunat>
So far my darwin approach was that there were thousands unfinished builds, but I didn't want to block other platforms so I merged (staging-next -> master) regardless.
<vcunat>
I can keep doing that.
<gchristensen>
sure
<vcunat>
(I've never used Apple stuff; it's not important platform *to myself*.)
<vcunat>
So choose as you prefer :-)
<gchristensen>
well
<vcunat>
There's never enough time to fix everything, not even half of the things.
<gchristensen>
we're in a hard spot on both ends here, hard that we haven't done staging stuff in a long time -- hard that darwin is hard to support
<gchristensen>
exactly
<gchristensen>
so I would rather make forward movement on staging and then play catch-up on darwin
<vcunat>
It could be approched the way that we/you mainly fix darwin on the stable nixos-xx.yy branches.
<vcunat>
If the rest is too fast moving.
<gchristensen>
sounds like a scary every 6mo to see if master works with darwin :P
<gchristensen>
but yeah
<vcunat>
I'm trying the llvm_7 build on aarch64.nixos.community right now.
<vcunat>
$ df -h /tmp
<vcunat>
Filesystem Size Used Avail Use% Mounted on
<vcunat>
tmpfs 63G 63G 511M 100% /
<gchristensen>
oops
<vcunat>
gchristensen: ^^ it seems I don't have permissions to clear it
<vcunat>
nix-build-zfs-kernel-* are large, but there must be something much bigger that I can't see
<gchristensen>
I had many repos cloned in my ~
<vcunat>
but 63G?
<gchristensen>
yeah that doesn't make sense
<samueldr>
ah
<vcunat>
Oh, it's / and not ust /tmp
<samueldr>
each ofborg builder
<samueldr>
they have their own nixpkgs checkout
<samueldr>
and when users keep stuff in their home, like a nixpkgs checkout, it compounds
<gchristensen>
vcunat: do you have enough space now, or shall I do more?
<gchristensen>
I can also reboot and it'll get you fresh disk space
<vcunat>
Seems sufficient for a llvm build.
<gchristensen>
ok
<samueldr>
though the nix store is kept on the hard drive, right?
<samueldr>
(but /tmp I know)
<gchristensen>
indeed it is!
<gchristensen>
that one is erased on each boot, too
averell has quit [Quit: .]
<vcunat>
OK, success.
<vcunat>
I didn't notice it had been running on Hydra in the meantime, but this was much faster... probably because there was almost nothing else running on this machine.
averell has joined #nixos-dev
vcunat has quit [Quit: Leaving.]
<samueldr>
:/ I can't shake off the cpu features reproducibility thing from my mind, wondering how Debian handles it, though can't find notes about it on the reproducible builds website
<gchristensen>
samueldr: maybe ask in #Reproducible-builds on oftc?
<samueldr>
the friction of joining another IRC network :| (I should)
<gchristensen>
what is your irc client?
<samueldr>
complicated, but not an issue
<gchristensen>
...hrm
<gchristensen>
...
<samueldr>
znc <- quassel
<gchristensen>
oh, znc
orivej has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
<gchristensen>
disabling DHCPheh
<gchristensen>
heh
<gchristensen>
disabling DHCP in all the NixOS tests which wait for multi-user.target saves 30s!
<gchristensen>
nix-build ./test.nix 1.00s user 0.18s system 9% cpu 12.131 total
<gchristensen>
vs. nix-build ./test.nix 0.94s user 0.15s system 2% cpu 41.649 total
<{^_^}>
#50930 (by cprussin, 19 weeks ago, open): Add option to make dhcpcd non-blocking
<gchristensen>
multi-user depends on networking, networking by default (as long as dhcpcd is enabled) depends on dhcpcd
obadz has quit [Quit: WeeChat 2.4]
drakonis has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
justanotheruser has joined #nixos-dev
drakonis has joined #nixos-dev
qyliss^work_ has joined #nixos-dev
harrow` has joined #nixos-dev
ekleog_ has joined #nixos-dev
drakonis_ has quit [Ping timeout: 264 seconds]
orivej_ has joined #nixos-dev
jtojnar_ has joined #nixos-dev
niksnut_ has joined #nixos-dev
tv has quit [Ping timeout: 255 seconds]
jtojnar has quit [Ping timeout: 255 seconds]
ekleog has quit [Ping timeout: 255 seconds]
niksnut has quit [Ping timeout: 255 seconds]
qyliss^work has quit [Ping timeout: 255 seconds]
qyliss^work_ is now known as qyliss^work
tdeo has quit [Ping timeout: 246 seconds]
harrow has quit [Ping timeout: 246 seconds]
disasm has quit [Ping timeout: 246 seconds]
aszlig has quit [Ping timeout: 246 seconds]
copumpkin has quit [Ping timeout: 246 seconds]
orivej has quit [Ping timeout: 246 seconds]
tv has joined #nixos-dev
copumpkin has joined #nixos-dev
jtojnar_ is now known as jtojnar
kgz has quit [Ping timeout: 246 seconds]
kgz has joined #nixos-dev
orivej_ has quit [Ping timeout: 245 seconds]
<ekleog_>
matthewbauer: seeing you gave an r+ without merging on https://github.com/NixOS/nixpkgs/pull/58504 ; is there something blocking merging it or was it just to leave other people the time to react?