<cole-h>
Gwern's website is quite honestly a work of art
<pie_>
man cloning big mercurial repos is ridiculous
<pie_>
or at least the jvm from openjdk
drakonis has quit [Quit: WeeChat 2.8]
waleee-cl has quit [Quit: Connection closed for inactivity]
endformationage has quit [Quit: WeeChat 2.6]
cole-h has quit [Quit: Goodbye]
evanjs has quit [Read error: Connection reset by peer]
evanjs has joined #nixos-chat
liff has quit [Ping timeout: 260 seconds]
liff has joined #nixos-chat
monsieurp has joined #nixos-chat
<sphalerite>
elvishjerricco: do you have the nixos gnupg module enabled?
<elvishjerricco>
sphalerite: Yep
<sphalerite>
Is the agent running through the systemd user service?
<eyJhb>
As much as I like DoAs, I hate there are so many small ports, and none really big ones.
<eyJhb>
Especially since it is hard to see how committed they are to keeping it up-to-date, and secure
<sphalerite>
dead on arrival?
<eyJhb>
`Dead on arrival (DOA), also dead in the field and brought in dead (BID), indicates that a patient was found to be already clinically dead upon the arrival of professional medical assistance, often in the form of first responders such as emergency medical technicians, paramedics, or police.` ?
<elvishjerricco>
sphalerite: Yea I think so
<sphalerite>
eyJhb: yeah, what do you mean by DoAs?
<eyJhb>
Yeah, but what happens if lose my file locations?
<eyJhb>
No problem, the locations are just metadata! Your files are still there, sitting in π - they're never going away, are th
<eyJhb>
^^ Tecnically true
<eyJhb>
Have anyone considered changing names, so their lastname does not always come first?
<eyJhb>
Currently getting screwed over my lastname for my exam dates because of that
<FireFly>
heh.. novel reason for changing name :p
<eyJhb>
Someone should do a study on that tbh. :p
<eyJhb>
I have a exam on the 8th of June, and another on the 10th. Which is doable if they hadn't screwed things up in the start of the semester. But some have exams on the 12th+. I want those two extra days damn it! :(
<eyJhb>
Just need a lastname with Å. Ågesen
<FireFly>
Zorro is also permissible, if you're okay with not being at the very end of the alphabet
<FireFly>
:p
<FireFly>
hm, now wondering what danish restrictions on last names are
<srhb>
FireFly: There's a whitelist of surnames
<FireFly>
aw
<srhb>
FireFly: Every surname that more than 2000 (living) people bear gets on the list.
<srhb>
If it's not on, there's a procedure for requesting it, but if it's on, it's automatic and just costs money.
<FireFly>
I think it's essentially the same in Sweden
<eyJhb>
Isn't it less than 2000 srhb ?
<srhb>
eyJhb: I don't think so, no. :)
<srhb>
You're probably thinking of first names
<srhb>
One person bearing a first name whitelists it.
energizer has joined #nixos-chat
<energizer>
ninjin: hit me
<ninjin>
energizer: Sure, let me see if I can dig up a good presentation or do you prefer raw code?
<srhb>
I'm also listening in. My days of scientific computing is gone, but I'm interested and know at least one astrophysicist who was into it.
<ninjin>
I was an early “Deep Learning” adopter for Natural Language Processing, so beware of my biases.
<energizer>
i'm happy to look at whatever you'd like to show
<ninjin>
They are good competition though, even if I seem to remember Lattner leaving which can hardly be a good thing.
<Taneb>
eyJhb: UK
<Taneb>
I think the only rule is it's not inciting a crime
<Taneb>
And you can do it for free with a single witness
<eyJhb>
Offtopic from the offtopic. Does ZFS still require 1 GB ram for every 1 TB disc space? (without dedup). srhb adisbladis
<eyJhb>
Taneb: that sounds nice tbh.. But getting new drivers license will still cost etc.?
<Taneb>
Yeah
<Taneb>
But I think just as much as renewing them? I haven't changes my name myself
<srhb>
eyJhb: Don't know, sorry. It's hungry for sure, but you can tweak the cache size.
<adisbladis>
eyJhb: Absolutely not :)
<adisbladis>
I've heard that number thrown around for dedup
<adisbladis>
But not for regular use
<adisbladis>
Contrary to popular belief ZFS isn't that memory hungry if you tune it
<eyJhb>
adisbladis: I see 5GB for dedup :p
<adisbladis>
Let's forget about dedup and pretend it's not a thing ^_^
<eyJhb>
Considering btrfs, but... That isn't very friendly
<etu>
eyJhb: Don't use dedup
<eyJhb>
And I hate that ZFS will not be part of the kervel
<etu>
I wouldn't use btrfs for temporary files
<eyJhb>
kernel**
<adisbladis>
Anyway, the 1G per 1T storage advice is from old Sun docs
<adisbladis>
I think they were overly conservative
<eyJhb>
Is there any current?
<adisbladis>
Nah, it's tunable
<adisbladis>
I've been running ZFS on as little as 512M RAM
<eyJhb>
So... 16 TB, what would you recommend?
<adisbladis>
Is this purely a storage server?
<adisbladis>
Or a desktop?
<adisbladis>
If the former, I'd just leave the defaults
<adisbladis>
If the latter, I'd leave the defaults for now and see if I need to tune
<adisbladis>
Pretty much
monsieurp has joined #nixos-chat
<eyJhb>
But, no recommended ram size for a desktop?
<eyJhb>
Wonder if it will ever be part of the kernel. And searching for ZFS and seeing proprietary is not nice :(
<adisbladis>
It's not proprietary
<adisbladis>
It's just not GPL compatible
<eyJhb>
What is the license?
<JJJollyjim>
asisbladis: unless you're Canonical
<JJJollyjim>
CDDL
<eyJhb>
Werid license
<adisbladis>
The one that was explicitly created to be GPL-incompatible :P
<adisbladis>
Omg, the `with` keyword in Nix has to die -.-
<FireFly>
agree
<etu>
eyJhb: It's actually free software
<adisbladis>
I just spent 15 minutes staring at the wrong function -.-
<etu>
eyJhb: It's foss in the same way gpl is
<etu>
eyJhb: But different
<eyJhb>
etu: the thing that is the most annoying is just, that it can't be part of the mainline kernel, which would be nice
<etu>
eyJhb: https://2.5admins.com/2-5-admins-03/ -- contains a fun breakdown of some differences between ZFS and Btrfs at the middle/end of the episode
<talyz>
eyJhb: btrfs is nice, but not as featureful as zfs
<talyz>
eyJhb: I use it on all my systems currently, though :)
<JJJollyjim>
offline deduplication is the feature of btrfs that stops me investigating zfs
<JJJollyjim>
i can't spare the memory for the zfs approach
<eyJhb>
etu, adisbladis: DEDUP ^^^
<eyJhb>
talyz: considering btrfs tbh. but need to investigate a little more
<eyJhb>
atm. I am happy with ZFS, but haven't used it much. But I guess anything is better than ext4
<etu>
eyJhb: I don't need dedup. /nix can be dedupped by nix and then I don't tend to duplicate a lot of big files
__monty__ has joined #nixos-chat
<adisbladis>
btrfs is fine but a bit meh
<adisbladis>
I'm still salty from my data corruption re send/recv
<adisbladis>
But that was close to a decade ago now
<MichaelRaskin>
A decade ago I think I managed to fully break multiple BtrFS partitions
<talyz>
I've used send/recv without issues many times
<MichaelRaskin>
But that was a decade ago
<MichaelRaskin>
I did not even do anything interesting with them
<adisbladis>
talyz: Me too, after that kernel bug was fixed ;)
<MichaelRaskin>
Currently I use BtrFS for store and ext4 for /home
<MichaelRaskin>
I like that BtrFS does not commit Rampant Layering Violation™ unless I ask it to (and I do not)
<sphalerite>
I quite like the Rampant Layering Violation™. What are its disadvantages?
<__monty__>
Which layers are we talking about? I'm assuming not OSI?
<MichaelRaskin>
Fault isolation. Switching details of one layer without changing the entire logic across the stack.
<talyz>
adisbladis: right :)
<JJJollyjim>
osi makes about as much sense applied to filesystems as it does applied to the modern web :P
ninjin has quit [Remote host closed the connection]
ninjin has joined #nixos-chat
<eyJhb>
Damn it, I hate working with certs and self signed stuff and how to distribute it
<eyJhb>
HA! Found another way
* etu
have heard that btrfs still can be buggy with send/recieve
<etu>
If the transfer dies
<etu>
It still can think that it was complete on the recieving end
<etu>
And look complete until you start walking the filesystem
ninjin has quit [Quit: WeeChat 2.7.1]
<bqv>
I've used btrfs for a good many years, its just so *easy*
<bqv>
Best of all worlds
monsieurp has quit [Remote host closed the connection]
<eyJhb>
bqv: ever used ZFS before?
<bqv>
No, tbh
<bqv>
But I've been around people who use it a lot, it sounds like a real fuff even for trivial things
<samueldr>
though I'm not too fond to see they implemented what looks like their own syntax
monsieurp has quit [Ping timeout: 256 seconds]
<bqv>
sphalerite: i mean, every time i've considered setting up zfs, i've been hit by the immediate roadblock of "ok you gotta set up a pool, tune several variables, and architect a filesystem hierarchy". with btrfs, i just mkfs and then i can instantly mount it. if i really care, i can make subvolumes, but it's not forced
<bqv>
granted i don't like zfs on principle so i've not tried very hard, but i get the impression it's more involved than a simple btrfs volume
monsieurp has joined #nixos-chat
<sphalerite>
bqv: ah right. Well you don't have to do all those things, you can also just `zpool create pool /dev/sdX` and then you'll have a zfs filesystem mounted on /pool.
<sphalerite>
but if you don't like zfs on principle I guess I don't need to try and convince you otherwise :)
<bqv>
heh, fair
<sphalerite>
It's just that at the point where you're the target audience for zfs, you _do_ want to "architect a filesystem hierarchy" to be able to make full use of it, but that's no different with btrfs
drakonis has joined #nixos-chat
drakonis_ has quit [Read error: Connection reset by peer]
<sphalerite>
(I'd probably add `-o ashift=12 -O compression=on -O mountpoint=legacy -O acltype=posixacl` to that command, so I do see where you're coming from!)
<bqv>
:p
<__monty__>
Btrfs being part of mainline gives me peace of mind. If ever I need to rescue data from a live system I can just pop in whatever live usb I have around. Don't have to go through contortions to get a live usb supporting zfs first.
<gchristensen>
why would you need to rescue anything from a system with zfs :)
<gchristensen>
(or btrfs)
<JJJollyjim>
sometimes i let zfs down :(
<gchristensen>
backups were sucky and slow in the sad old days. we don't need that anymore. with modern filesystems it is completely feasible to back up your system every minute if you wanted. I back up 5 minutes because it is still much faster than any single step of trying to restore something from the disk
<eyJhb>
gchristensen: every 5 minutes?
<gchristensen>
yeah
<eyJhb>
What did you use for it again?
<bqv>
damn...
<gchristensen>
I use znapzend
<immae>
just out of curiosity: you backup every 5 minutes, but can you: send this "backup" remotely while still using your system? remove old versions of the backup to avoid keeping too much?
<eyJhb>
Also when you are on-the-go?
<immae>
(I don’t know much about zfs)
<eyJhb>
ldlework: my scrollbar won't go away in URXVT atm.
<eyJhb>
Annyoing. What was you WM?
<adisbladis>
immae: Yes you can
<adisbladis>
You take a snapshot and send it over the network somewhere
<bqv>
i once had a comprehensive snapshot system on my desktop, but too many times i left it to it's own device(s) for weeks and then ended up with no disk space left
<bqv>
which sounds like something that should be quite avoidable automatically, but snapper on arch was always such a PITA
<immae>
adisbladis: ok. And about removing old backups? (I suppose so, otherwise it would be quite a mess)
<adisbladis>
Since a snapshot zfs is CoW and a snapshot is an immutable pointer there is nothing stopping you from using your system while sending the backup
<immae>
ok
<adisbladis>
immae: Occasionally you want to send a full stream
<immae>
(I’m not sure what you mean by "full stream")
<adisbladis>
Idk, maybe there is a better way to prune old incremental snapshots
<eyJhb>
adisbladis: I got libreoffice questions, do you use it?
<gchristensen>
immae: yes
<adisbladis>
eyJhb: Luckily no
<adisbladis>
Only to read other peoples non plain-text formats
<adisbladis>
I don't use it for writing
<gchristensen>
immae: yeah you can prune continuously of course :)
<immae>
good :)
<gchristensen>
pruning snapshots doesn't make it less of a full system
<immae>
I’m just trying to grasp what zfs is able to do :)
<adisbladis>
World peace
<adisbladis>
Feed the needy
<sphalerite>
I keep 5-minutely snapshots for 15 minutes, 15-minutely snapshots for 4 hours, hourly snapshots for 4 days, daily snapshots for a week, and weekly snapshots for a month
<sphalerite>
on my laptop that is
<immae>
sphalerite: yes that’s exactly the kind of scenario I had in mind :p
<sphalerite>
the latest ones always get synced to my backup server, which keeps daily snapshots for 2 months and monthly snapshots for 5 years
<eyJhb>
adisbladis: just wondered for recovering files. I use it to do my books
neeasade has joined #nixos-chat
<__monty__>
For those of us who don't have their digital life in order, the peace of mind is nice : p
<sphalerite>
(but every new snapshot gets synced, that way the state on the backup server isn't from the beginning of the day at the end of the day where I lose my laptop
<immae>
Last question: say I use zfs with snapshot and all, and slowly in time I reduce the number of snapshots until I get the diskcomplletely full. Does this "full" correspond to the same size as if I only filled the disk from 0 to 100% without any snapshot? (in other words: does the fact that I use snapshots, even temporarily, make me lose space forever?)
<sphalerite>
immae: you can do that kind of stuff with borg too (and get deduplication that isn't as expensive as with zfs), but it's slower since it needs to scan all the files, whereas zfs knows what's changed
<sphalerite>
immae: a snapshot only starts costing significant space once the files referenced by it are removed/overwritten
<gchristensen>
immae: if you create a snapshot, that snapshot takes up as much space as the difference from that snapshot to your current hd's contents
<gchristensen>
immae: once you delete it, its space is released
<immae>
ok good
<__monty__>
immae: Well, a snapshot contains (or rather points to) blocks of files that have since changed. So they do take space but very little in addition and...
<gchristensen>
give it a try, you don't need to use real disks
<immae>
Seems like it’s well designed :)
<gchristensen>
you can just make files on your disk and create a zfs pool with the files as backing data stores
<sphalerite>
I feel like half the time #nixos-chat is talking about how great zfs is x)
<immae>
ah right I should try to play with it
<adisbladis>
sphalerite: If you ever thing it's too quited drop a "zfs sucks!!" or something like that =)
<__monty__>
And if you consider zfs is too much after all, check out btrfs : >
<gchristensen>
can y'all think of various user utilities which talk to the kernel, where the utility and kernel must be a "matched pair" or suffer protocol incompatibilities?
<emily>
zfs tools
<sphalerite>
gchristensen: ^ and bcc
<adisbladis>
Speaking of file systems, how is bcachefs coming along
<sphalerite>
gchristensen: probably also a bunch of proprietary drivers, like nvidia and virtualbox stuff
<sphalerite>
or does an X graphics driver not count as a user utility?
<adisbladis>
sphalerite: My gut tells me that's not the case, why would the ABI that user space interacts with change?
<adisbladis>
I'd guess you need a matched pair of nvidia-driver & nvidia-util
<adisbladis>
But that the kernel version shouldn't matter?
<adisbladis>
After all even the utility is proprietary, no?
<sphalerite>
I thought the kernel module counts as part of the kernel
<adisbladis>
We might mean the same thing =)
<sphalerite>
adisbladis: so I mean that the nvidia userspace (X) driver 323.abc will complain if the nvidia kernel-space driver 342.xyz is loaded
<sphalerite>
and possibly fail to run
<sphalerite>
are we on the same page there?
<adisbladis>
Yes
<adisbladis>
What I mean is that whether it's kernel version 5.4 or 5.6 shouldn't matter
monsieurp has quit [Remote host closed the connection]
<sphalerite>
right
<emily>
I mean, you still need to match the kernel module binary, and the kernel module binary changes depending on what kernel version you're running
<emily>
but with a module the "kernel version" is effectively the module version in this ABI-compatibility sense I'd say
<emily>
for the statement "userspace tool version and kernel version must match"
<emily>
the point is you have to match whatever's actually listening on the other side
<adisbladis>
Haha this discussion is not very helpful
<bqv>
Lmao
<eyJhb>
I need someone to tell me something pretty. If I have have urxvt in my configuration.nix, and I now have to use it in a config file again, would I then do pkgs.urxvt, or just omit it and say it is in my path?
<eyJhb>
Ups, #nixos
<gchristensen>
I'd refer absolutely
<eyJhb>
gchristensen: but then it might be two different versions? eg. I have `services.urxvtd.enable` so I want to use the urxvtc binary. Would you then still use pkgs..../bin/urxvtc?
<gchristensen>
It's Complicated
<eyJhb>
gchristensen: how would you go about it? I need ideas, I also have this `"/run/wrappers/bin/physlock` which I guess I need because it is wrapped with security.wrappers :(
<emily>
eyJhb: you could refer to services.urxvtd.package if it exists
<emily>
that's the best way to express your intent here, "a client to communicate with the configured server"
<emily>
if it doesn't exist and just hardcodes pkgs.urxvt then there's no harm in doing that yourself too, right?
waleee-cl has joined #nixos-chat
<eyJhb>
emily: but e.g. physlock does not allow this, and hardcoding it will give me a path without the security wrapper
<eyJhb>
That gets a little more tricky
<emily>
sure. for wrappers you need to hardcode the wrappers path. that's how wrappers work
<emily>
that's the "public interface" that security.wrappers defines
<eyJhb>
When you say that emily , is there then a "good" way, or what might be the way to do that in this case?
<emily>
it's the correct way to refer to a wrapped program, you can think of /run/wrappers as part of the exported API of security.wrappers
<emily>
it's an unfortunate global singleton-y thing but nixos is full of those
<emily>
maybe there could be a ${security.wrappers.something.executable} or something you could use, but there isn't afaik and it wouldn't really buy you much
<eyJhb>
True, would just be nice. Then I will keep it as-is for now
<emily>
the only way to make it "nice" would be to have fully modular security wrappers that act like independent packages or whatever, and I think nobody likes caps enough to want to do that
<emily>
"ideally" physlock would have a physlockd that runs with the necessary privileges and gets talked to by an unprivileged client rather than relying on setuid, or similar
<emily>
(hard for some things like sudo though)
<eyJhb>
That would be ideal yes
<eyJhb>
You used physlock, right?
<emily>
yeah
<emily>
just the default configuration, I don't have idle locking right now though I'd like to set it up
<emily>
tried to figure out if it could be nicely integrated with gnome and came to the obvious conclusion of "no, they removed support for anything but their own thing years ago"
<eyJhb>
But they got the added enter feature in the gnome lock! \s
<eyJhb>
I would have expected you to run a tiling window manager
<emily>
I really like the scrolling tiling model and the workspace stack
<emily>
it's the most touchpad-friendly tiling WM I've ever used too, which I realize is probably not in most tiling WM users' priority lists, but I like it for laptop use
<emily>
it has a shell.nix so you know it's good software :p
<emily>
previously I used sway but it's super buggy...
drakonis_ has joined #nixos-chat
drakonis has quit [Ping timeout: 272 seconds]
<MichaelRaskin>
emily: I do not have setuid sudo/doas and run most thing via a privileged daemon. I think the most annoying part with that is the need to do something about things not owning the terminal or terminal not having access to send signals to processes inside.
<MichaelRaskin>
Otherwise it's fine
drakonis has joined #nixos-chat
<emily>
mhm. there is s6-sudo{d,c}
drakonis_ has quit [Ping timeout: 244 seconds]
<MichaelRaskin>
Oh, never knew
<eyJhb>
emily: newer heard of that before, looks very nice for anything touch! :D
<eyJhb>
gchristensen: how do you manage your channels with a tmpfs /? Or is /root persistent?
<gchristensen>
eyJhb: on bootup, by default, nixos subscribes the root user to the channel used to create the system being booted
<gchristensen>
so, that is how
cole-h has joined #nixos-chat
<gchristensen>
"When a fail-safe system fails, it fails by failing to fail-safe."
<__monty__>
Aka it fails to save-face.
<eyJhb>
gchristensen: but that only adds the base channel, nothing more. e.g. unstable, home-manager etc. will not be added
<gchristensen>
fair
<gchristensen>
I don't have any other channels
<eyJhb>
Is there a way to manually do this?
<eyJhb>
Is it just symlinks?
<eyJhb>
Christ, so all the channels are in the /root/.nix-defexpr dir, buth .nix-channels is just the nixos channel
<eyJhb>
Lovely!
drakonis1 has joined #nixos-chat
burkelibbey_ has joined #nixos-chat
liszt has quit [Ping timeout: 246 seconds]
emilazy has quit [Ping timeout: 272 seconds]
betawaffle has quit [Ping timeout: 246 seconds]
betawaffle has joined #nixos-chat
emilazy has joined #nixos-chat
betawaffle has quit [Max SendQ exceeded]
wildtrees has joined #nixos-chat
wildtrees has quit [Max SendQ exceeded]
wildtrees has joined #nixos-chat
peel has quit [Ping timeout: 252 seconds]
emilazy has quit [Ping timeout: 272 seconds]
liszt has joined #nixos-chat
liszt has quit [Max SendQ exceeded]
peel has joined #nixos-chat
betawaffle has joined #nixos-chat
<ashkitten>
what order of magnitude of text string size can postgres store before it starts to degrade performance?
liszt has joined #nixos-chat
emilazy has joined #nixos-chat
<ashkitten>
ah, it stores those separately
<gchristensen>
depending on quantity, be careful
<gchristensen>
storage on a database server is operationally expensive
<__monty__>
Hang on, isn't that what databases are all about?
<gchristensen>
storing bytes of data on a disk isn't what databases are all about
<gchristensen>
much cheaper to, say, store the hash of the data and write the actual data to a content-addressed system
<gchristensen>
or store the data under a uuid in s3
<gchristensen>
it just depends on how many orders of magnitude we're actually talking here
<ashkitten>
eh, dw about it
<gchristensen>
once you're talking orders of magnitude I get spooked :P
<ashkitten>
i was only worried about performance issues not capacity
<ashkitten>
lol
<ashkitten>
it's a mastodon instance, i removed the character limit
<gchristensen>
oh probabl yfine
<ashkitten>
also like, i aint running a professional service here
<bqv>
heh, you know that set of errors you get in shells when you delete the directory the shell is currently in? with xonsh, that's now a set of python errors :D
<gchristensen>
yeah 2xprobably
<bqv>
FileNotFoundError
<gchristensen>
ashkitten: back at tumblr we'd joke how the "tags" table was eventually just going to never be written to anymore as every possible tag was going to exist already
<ashkitten>
wow nerd
<gchristensen>
:o
<ldlework>
deny it
<ldlework>
:P
* gchristensen
conspicuously does not
<ashkitten>
i was on tumblr briefly and it was all just memes about being queer and depressed, which as a depressed queer appealed to me at the time
* __monty__
makes mental note gchristensen single-handedly ran tumblr
<cole-h>
lol
<ldlework>
Does Eelco Dolstra still work on NixOS?
<bqv>
8-ball says yes
<cole-h>
Depends on how you mean "work on NixOS".
<gchristensen>
he's a bit hands-off, but definitely still involved and pays close attention
<cole-h>
Most likely because he's more focused on Nix now that we seem to have a pretty good group of people running the show (or at least moving it along) for NixOS
<ldlework>
I guess I just meant Nix in general
<gchristensen>
__monty__: man, keeping the databases up, alone, was an 8 person team
<gchristensen>
ldlework: undoubtedly yes
<ldlework>
They don't even remark that they invented Nix on their resume!
<gchristensen>
he's very humble!
<ldlework>
My resume would start "The Guy Who Created NixOS"
<gchristensen>
oh well armijn invented nixos and it almost certainly is. armijn is very vain (if he were here, this joke would land well with him.)
<ldlework>
I suppose it makes sense not all the parts were created at once.
<samueldr>
there's a nixcon talk from 2020's edition about that
<samueldr>
oops
<__monty__>
gchristensen: No, I'm sure, was mostly trying to prompt you for more information playfully ; )
<gchristensen>
I still can't believe it was neither taken or reserved
<ashkitten>
ah yeah i recognize some names of popular accounts you've reblogged
<__monty__>
gchristensen: Such an audible swallow.
<cole-h>
gchristensen: It's always fun when you're early to the party and get a sweet name.
<gchristensen>
cole-h: I got this account in, like, 2016!
<adisbladis>
cole-h: Likely Darmstadt
<cole-h>
Wait what
<adisbladis>
As that was the plan for 2020
<gchristensen>
actually, 2017!
<ashkitten>
on the fediverse i can have any username i want!
<cole-h>
gchristensen: Are you sure you didn't just mess with the db and make it available? :D
<gchristensen>
hah
<cole-h>
Oh, Germany, cool!
<__monty__>
Then monitor support db for any complaints relating to the username.
<gchristensen>
__monty__: in that talk I give some numbers of scale and write speed iirc, so that might have the details you're wondering :)
<adisbladis>
ldlework: Re Eelco, maybe you've seen niksnut on irc?
<adisbladis>
That's him
<emily>
gchristensen: wait, are you telling me that if you write half an essay in a tumblr tag it eagerly creates a row in one huge shared monolithic table? amazing
<gchristensen>
lol yeah
<gchristensen>
it is like the 1,000,000 monkeys thing
<ashkitten>
i have in fact seen several essays in tumblr tags
<emily>
tumblr really is exactly what happens when your hacked-up-in-a-few-weeks PHP blog platform ends up serving millions of people huh
<adisbladis>
I have in fact seen many monkeys
<adisbladis>
One of my previous employers did devops before gchristensen's time there :)
<adisbladis>
For tumblr I mean
<gchristensen>
emily: the tech behind it is, frankly, the easy part. there *is* design behind the community interactions
<ashkitten>
i have a lot of complaints about tumblr but its infrastructure never caused me any problems
<gchristensen>
a number of the behaviors are designed to be "negative feedback" systems instead of "positive feedback" systems
waleee-cl has quit [Quit: Connection closed for inactivity]
<gchristensen>
in comparison, twitter replies / quote retweets are designed to be a positive feedback cycle, especially if the content is bad
<gchristensen>
on tumblr, typically, a bad post just gets dumped on but doesn't get shared around
<gchristensen>
as aggressively*
<ashkitten>
gchristensen: you mean where people can't see replies to posts without looking real deep so when misinformation gets spread nobody can correct it?
<gchristensen>
heh
<gchristensen>
you mean the new twitter thing?
<ashkitten>
that was also a big problem
<ashkitten>
i don't know what new twitter thing, since i don't use twitter
<MichaelRaskin>
… I am not sure it is actually less of a problem on Twitter …
<ashkitten>
MichaelRaskin: it used to be, when i used twitter at least
<gchristensen>
twitter has a new feature where you can mark tweets as nobody being able to reply
<ashkitten>
oh
<gchristensen>
so the *only* way to register your displeasure is to quote-retweet
<cole-h>
Except the mentioned person/people, I think
<cole-h>
(Or that part is optional, probably)
<gchristensen>
-> positive feedback cycle
<bqv>
i really wish the fediverse would succeed real damn hard and displace twitter/tumblr/youtube/etc, since the latter are horrifying, but the fediverse is basically self-perpetuating drama given life
<MichaelRaskin>
ashkitten: trying to read some people writing interesting things on Twiiter, leads me to think there are multiple weakly interacting bubbles. Many of them do not get much disinformation in hte first place
<ashkitten>
i remember now, someone posted about the official @twitter account dunking an their entire userbase by using an unreleased feature to prevent people from replying to their post asking if there's a better app
<gchristensen>
bqv: I'm still hoping for a fediverse thing I can host myself at my own domain so I can control my identity, which doesn't consume $alloftheresources
<gchristensen>
ashkitten: apps, which btw, they actively shut down by shutting off the APIs
<bqv>
gchristensen: try pleroma, if you're alright with being accused of being a nazi
<bqv>
but afaik it's incredibly light compared to mastodon, and i have half a nix expression for it
<gchristensen>
well that is a non starter lol
<ashkitten>
the pleroma users are nazis thing is still going?
<ashkitten>
i thought we had matured past that
<bqv>
i don't know, i don't keep up with fediverse drama
<__monty__>
I believe federation leads to centralization.
<bqv>
i know kaniini still won't go back, i talked to them about it recently
<ashkitten>
some of the pleroma devs tick me off real bad which is why i stopped bothering with pleroma
<eyJhb>
gchristensen: any idea where it gets the URL from at boot, to populate .nix-channels?
<gchristensen>
this is another problem though: picking something (software, homeservers) has implications I don't care about. I don't have to care about that with twitter
<ashkitten>
gchristensen: i can point you to some good ones if you like
<adisbladis>
I think fediverse is not good enough tbh
<bqv>
howso?
<ashkitten>
adisbladis: i agree, and i'm trying to push people to make it better
<bqv>
you can't fault it's success, despite being a literal conflict magnet
<bqv>
or maybe, due to that..
<__monty__>
I think the shortcomings are inherent. You really need a distributed solution, rather than just decentralized.
<adisbladis>
bqv: Just my personal opinion, but I think we need full-on decentralization.
<bqv>
oh i totally agree, but that's just unrealistic
<ashkitten>
i agree that decentralization is a goal to work towards
<ashkitten>
maybe not, but i don't think it's because of the technology
<__monty__>
Scuttlebutt's also not all that decentralized with its "pubs."
<bqv>
:p
<MichaelRaskin>
Well, experiment shows that you give _developers_ multiple decentralised options, and they manage to, somehow, …
<adisbladis>
At least to me federated services suffers much of the same problems as centralized platforms
<ashkitten>
__monty__: scuttlebutt pubs are optional, they just relay posts and stuff
<gchristensen>
adisbladis: where does email fall in to the decentralized vs. federated vs. centralized in your mind?
<bqv>
at least with federation you have the option of owning your node
<adisbladis>
gchristensen: Definitely more towards federated
<__monty__>
ashkitten: Yeah but they're not very avoidable is my impression. It's like the whole convenience v. security tragedy.
<adisbladis>
Let's say I sign up to Mastodon, it's not exactly trivial for me to just up & move my social identity elsewhere
<gchristensen>
adisbladis: exactly why I haven't done it
<bqv>
you mean, removing your data?
<bqv>
or specifically moving it to another server?
<bqv>
or to another platform?
<ashkitten>
maybe it will make some people feel better about this whole thing, but someone recently did a poll asking about people's relationships with their instance admins, and the results look really encouraging to me https://vulpine.club/@starkatt/104213750173751227
<adisbladis>
I mean let's say my mastodon host goes down
<__monty__>
bqv: I don't care much about the node. It's my data that matters to me.
<gchristensen>
and why I do have a twitter account. picking a home server on mastodon *means* something, and that meaning can change over time. being like, on twitter, means nothing but you're on twitter
<adisbladis>
Now I need to create a new identity
<bqv>
adisbladis: that's why i mentioned the option of owning your node, so you don't have that risk, and you still control your data
<ldlework>
Get rid of your cloud overlords by picking new cloud overlords!
<bqv>
but on the flipside tech noobs can still sign up with a dedicated instance
<adisbladis>
ldlework: Yeah, kind of...
<bqv>
i.e. i own my own single-user mastodon instance
<adisbladis>
bqv: But then I have infrastructure to maintain...
<adisbladis>
I don't want that either
<gchristensen>
and it isn't simple either
<ldlework>
Single-user mastadon instances shoul be a human right.
<bqv>
ok well you can't have you cake and eat it too :p
<adisbladis>
I have enough shit to take care of without bothering with that
<bqv>
and honestly i haven't touched my mastodon instance for months
<bqv>
it's still fine
<ldlework>
that said, infrastructure is easier than it ever has been
<bqv>
(lie, i did update it recently, but that was trivial)
<ashkitten>
matrix is working on solving all these problems, it's just not a social media platform
<adisbladis>
I don't really trust myself to maintain security over time
<ldlework>
I gave up on security.
<adisbladis>
Given that it's not something I'm likely to care about for long durations of time
<gchristensen>
I really want a fediverse compatible identity server running entirely statically, but afaik it can't just be a static document
<ldlework>
If someone wants to hack ldlework.com and download the html to the site, so be it.
<ashkitten>
gchristensen: technically it can, but you also need a mechanism to deliver posts
<MichaelRaskin>
nix-build, obviously
<adisbladis>
I don't even bother enough to maintain a website :P
<adisbladis>
I just point my now only domain to my github profile
<ashkitten>
for what it's worth i think adisbladis's concerns are well-founded. i've been on the fediverse since late 2016 and i've seen things like the mass exodus from witches.town and all the aftermath of that. the fediverse doesn't have nomadic identity (yet?) and it can be an issue
<bqv>
guess i missed the refugee witches... i was around for the tumblr influx though
<ashkitten>
and what about when a.weirder.earth lost their database?
<bqv>
didn't even know that one was a thing
<ashkitten>
(they're just weirder.earth now because of federation issues afterward)
<bqv>
my experiences with the fediverse have been extremely punctuated because every time i convince myself to try and get into it, i quickly lose the will when i see half the content on there and the non-stop drama
<ashkitten>
the fediverse has issues, you can't just ignore that and tell people to run their own instance or something
<adisbladis>
Perhaps unpopular opinion: Registering identity on a blockchain is not so bad
<ashkitten>
for what it's worth i think an issue the fediverse doesn't have is recentralization
<adisbladis>
Identity is a problem of distributed consensus
<MichaelRaskin>
Blockchains also come and go
<__monty__>
ashkitten: How so?
<adisbladis>
MichaelRaskin: Sure, it's not a panacea
<ldlework>
I'm really glad to have my website.
<MichaelRaskin>
And _identity_ is not a problem of consensus at all in the last twenty years
<MichaelRaskin>
Nicknames, probably
<bqv>
well frankly, the only reason i support the fediverse is because it is already succeeding. i don't really agree with all the technical routes, nor the culture, but it vaguely works and is popular and has a chance of becoming ubiquitous
<MichaelRaskin>
Indentity itself — no
<bqv>
every other option is dead in the water
<adisbladis>
MichaelRaskin: I mean identifiers
<pie_>
doesnt twitter also have that <bqv> my experiences with the fediverse have been extremely punctuated because every time i convince myself to try and get into it, i quickly lose the will when i see half the content on there and the non-stop drama
<ldlework>
When I die, I hope they burn my website to a gold record and shoot it out into space.
<ashkitten>
__monty__: since about 2017 there have been way more small instances popping up. of course, mastodon.social isn't going anywhere, and it's even still growing somewhat, but barely anyone i know is on mastodon.social and the fediverse would survive without it
<bqv>
pie_: yes except i wouldn't convince myself to use twitter, there's no benefit to using twitter
<adisbladis>
And blockchains are a suitable place to tie identifiers to cryptographic keys
<bqv>
(and there's significant detriment to using twitter)
drakonis1 has quit [Quit: WeeChat 2.8]
<MichaelRaskin>
adisbladis: or you can embrace the necessity for nickname locality
<ldlework>
Yeah, how is identity not tied to private keys? Seems like the obvious thing.
<bqv>
amuses me how matrix is losing to mastodon r.e. centralization. imagine matrix without matrix.org, it wouldn't even function on a technical level let alone otherwise
<ldlework>
Doesn't matter what network or whatever. Only I can produce messages with my key.
<ldlework>
Anywhere.
<adisbladis>
MichaelRaskin: Then we're back at square 1
<MichaelRaskin>
bqv: why do I need to imagine?
<MichaelRaskin>
I mean, I have seen Matrix with matrix.org down
<MichaelRaskin>
For weeks
<adisbladis>
I cannot take my account from let's say chaos.social to my own hosted instance
<bqv>
sounds about right
<MichaelRaskin>
Wasn't that bad
<adisbladis>
And keep my identity
<bqv>
adisbladis: actually, migration was implemented
<adisbladis>
ldlework: How do you route a message to a public key?
<MichaelRaskin>
adisbladis: you need to be able to reach out to your followers and prove it is still you. Preferably automatically
<adisbladis>
ldlework: How do you roll your keys if the pubkey is your identity?
<MichaelRaskin>
Or update the DHT entry of current-route-to-this-key
<ldlework>
adisbladis: you route a message to systems where the key has been registered.
<qyliss>
fediverse instances should let you point your own domain at them
<qyliss>
that'd solve a lot of this
<ldlework>
"roll your keys" ?
<MichaelRaskin>
qyliss++
<{^_^}>
qyliss's karma got increased to 63
<gchristensen>
qyliss: truth
<bqv>
fair
<qyliss>
although I think that wouldn't actually just work at a protocol level
<adisbladis>
As the hiphopers say: "Word"
drakonis_ has joined #nixos-chat
<ashkitten>
qyliss: it's complicated
<qyliss>
you'd need to swap out instance-based message signing for DANE or something I guess
<qyliss>
but damn that'd be cool
<adisbladis>
ldlework: In the case of key compromise you want to generate new keys
<adisbladis>
Without invalidating your identity
<MichaelRaskin>
qyliss: well, you can say «I accept the instance signature because it is the target of SRV»
<qyliss>
that's true
<adisbladis>
These are all solvable problems, but no one in the fediverse space has tackled them.
<ldlework>
adisbladis: Same way you registered your identity with the service in the first place.
<__monty__>
ashkitten: I could see things (new features) like better videochat latency with your friends driving recentralization.
<ldlework>
nevermind
<ashkitten>
really what i'm trying to push people to do is create a generic activitypub server that just speaks activitypub S2S and C2S protocols and you bring your own user agent (like mastodon or pleroma)
<adisbladis>
Fair enough
<ashkitten>
__monty__: what video chat
<adisbladis>
Human friendly identifiers aside, public key based social networking would be sweet
drakonis has quit [Ping timeout: 260 seconds]
<__monty__>
ldlework: How do you safely distribute someone's private key? People want to be able to easily log in on their laptop and their phone and maybe their friend's phone.
<ldlework>
Why would you distribute someone else's private key?
<__monty__>
ashkitten: *(new features)*
<ldlework>
Why would you even have someone else's private key?
<bqv>
adisbladis: i mean i think ssb fulfils most of what you're after, but it's incredibly inconvenient and sparsely populated
<ashkitten>
__monty__: i'm confused though, because video chat solutions generally pick one server to relay communication or just establish a direct link
<adisbladis>
bqv: Hm, I keep hearing about ssb but haven't looked into it. I should now though.
<ldlework>
adisbladis: I haven't been paying attention to federation stuff, but I honestly always thought they worked this way.
<MichaelRaskin>
__monty__: distributing between devices within USB range of each other is a problem with many annoying solutions, but well, most of people use at least one
<__monty__>
ldlework: No, how would you allow someone to post from a different device? It's not as if key management is a solved problem.
<ldlework>
Interesting to learn they are basically just open-source twitter.
<adisbladis>
ldlework: It's basically email all over again.
<ldlework>
__monty__: you always send messages from the device itself
<__monty__>
ashkitten: Well that's the driving factor behind the recentralization.
<ldlework>
the services don't have your private key
<ashkitten>
__monty__: what
<ashkitten>
__monty__: that doesn't make any sense
<bqv>
ldlework: fyi, that's really not a solution, and is one of the things i hate about ssb
<__monty__>
ldlework: Because so many people are ok with accessing their social media from one device only?
<ldlework>
bqv: what is "that"
drakonis1 has joined #nixos-chat
<bqv>
"always send messages from the device"
<ldlework>
__monty__: you can copy a text file to other devices..
drakonis_ has quit [Read error: Connection reset by peer]
<ldlework>
Why is it not a solution?
<bqv>
because the problem is "how do you let someone post from a different device", and that explicitly doesn't solve that
<ldlework>
If your private key was literally your social media identity
<ldlework>
people would get used to moving it around and protecting it
<ldlework>
how does it not?
<ldlework>
move the private key to new device
<ldlework>
post from new device
<MichaelRaskin>
If you want indeed to post from non-owned devices, then the hierarchy cold-stored key->own-device temporary key->trusted server short-lived no-delegation key is needed
<__monty__>
ldlework: You've never been out somewhere and went "agh, wish I had my ssh key here?"
<ldlework>
what am i missing lol
<adisbladis>
I've designed something like this before: You generate an initial key, the fingerprint of this key is you "root" identity, with the root identity you then sign one or more sub-identities
<ldlework>
__monty__: that's fine you haven't moved your identity to that device
<ldlework>
seems like it still solves the identity problem
<adisbladis>
You can freely add more sub-identities (devices)
<ldlework>
adisbladis: you still need your original key
<ldlework>
you can't impromptu tweet from a phone you just bought
<bqv>
i can't remember what it is about ssb that means you really shouldn't copy identities/keys between devices
<ldlework>
i don't see this as a "oh well we shouldn't go down this path" problem
<adisbladis>
ldlework: No, but one sub-identity could add another one
<ldlework>
bqv: we'll assume it doesn't exist for now then :)
<ldlework>
adisbladis: you still need some originating identity
<bqv>
it does, or people would be doing that
<adisbladis>
ldlework: Only for a short while
<ldlework>
adisbladis: you're missing the point
<bqv>
adisbladis: i like that solution btw
<ldlework>
original key, sub key, etc
<__monty__>
ldlework: What do you mean it's "fine"? People resort to easily memorized passwords for convenience do you really think they'll accept "well you should've transferred your key in advance?"
<ldlework>
you still need SOME KEY on the new device
<ldlework>
to create a new one
<adisbladis>
After that you need a messaging protocol to propagate changes to your ACL
<ldlework>
this is what __monty__ is complaining about
<adisbladis>
bqv: This system has 1+ million users by now ;)
drakonis_ has joined #nixos-chat
<ldlework>
__monty__: we're talking about people who care to engage with a system where your identity works this way
* bqv
raises three eyebrows
<adisbladis>
And they use Nix!
<ldlework>
not grandma
<ldlework>
Though literally everyone in my immediatey family except perhaps grandparents use LastPass and don't memorize passwords.
<ldlework>
Of course it took a family-wide intervention to make it happen..
<adisbladis>
ldlework: We rolled this out to a tonne of non-techy users
<ldlework>
adisbladis: you can also be less serious about securing sub-identitites right
<ldlework>
like you could hand out sub-identities to service who could then manage them
<ldlework>
and you can revoke them at anytime with a higher-identity
<adisbladis>
Yes, exactly
<ldlework>
adisbladis: social media systems that work like this already exist and you rolled them out with Nix?
<adisbladis>
This wasn't social media
<MichaelRaskin>
As I said, some subidentities are allowed to create sub-sub-identities, some are not
<__monty__>
ashkitten: Re the video stuff. I'd imagine someone like google or facebook coming in with a "federated" cloud instance and the catch'll be that fancy features like group video chat will be just outright better if your conversation participants are on the same cloud instance.
<adisbladis>
This was a PKI for enterprise platform
drakonis1 has quit [Ping timeout: 272 seconds]
<adisbladis>
s/was/is/
<bqv>
though, i reckon some security-conscious people would say something like "this just introduces a single point of failure" compared to a system with one unrelated key per device
<ashkitten>
__monty__: shrug, i don't really know or care about that though. i have better ways to contact my friends and the fediverse has historically been extremely resistant to being touched by capitalism
<__monty__>
When it comes to password managers for non tech-inclined acquaintances/family do any services stand out?
<bqv>
ashkitten: still waiting for that twitter decentralisation project eh?
<ashkitten>
__monty__: really though, at this point you're just proposing hypotheticals
<ashkitten>
bqv: what? i don't care about twitter
<bqv>
ashkitten: ..ok you clearly haven't heard, one sec
<ashkitten>
bqv: i've heard
<ashkitten>
i just don't care
<bqv>
oh
<bqv>
heh, fair enough
<ashkitten>
i'll block twitter dot com on my instance if they try to federate with us
<ashkitten>
but i don't think they will
<bqv>
i think it's realistic that twitter could pose a threat
<bqv>
that's the only avenue to recentralization i see
<ashkitten>
the thing i care about is that twitter users don't have compatible culture and if they try to federate there's gonna be a flood of un-cw'd posts and shit from them and we're too small for them to see us complaining
<ashkitten>
but i don't think this is very productive to talk about
<__monty__>
Un-cw'd?
<ashkitten>
content warnings
<bqv>
mastodon has an elaborate system of spoiler-like content warnings
<ashkitten>
i wouldn't call it elaborate, more half-assed
<bqv>
i call it a headache, tbh
<ashkitten>
it's a problem when people don't use them though
<ashkitten>
but i'm not gonna get into social media politics any more
<ashkitten>
nobody ping me i'm exhausted from this discussion
<adisbladis>
ashkitten: ping
ashkitten has left #nixos-chat ["WeeChat 2.8"]
<bqv>
arsehole :')
<adisbladis>
I had to...
<cole-h>
LMaooo
<adisbladis>
Temptations are my only weakness
<cole-h>
adisbladis: May I tempt you into fixing my `nixops rollback` issue, then? ;^)
<adisbladis>
Hm, not too tempting right now
<cole-h>
Hahaha
<MichaelRaskin>
I am somehow reminded of «Look who thinks he’s nobody»
<joepie91>
adisbladis: don't do that...
<adisbladis>
joepie91: What exactly?
<cole-h>
Don't give in to temptation!
<joepie91>
adisbladis: pinging someone after they've asked not to
<emily>
more like "don't deliberately violate people's boundaries in response to them explicitly stating them"
<adisbladis>
Sorry. It was in bad taste.
<joepie91>
or more generally- yeah, what emily said
<cole-h>
adisbladis: An "Adam" already gave in to temptation a long time ago, and look where that got us... ;P
<adisbladis>
I shouldn't have done that.
<adisbladis>
cole-h: Re your nixops issue did you report it?
<cole-h>
No, I was mostly wondering if it had been seen before. Will send a ticket soon.
monsieurp has joined #nixos-chat
waleee-cl has joined #nixos-chat
<gchristensen>
cole-h: btw that phone number field should be gone now. it was from when skillsmatter was doing in-person stuff
<cole-h>
Yay! Thank you!
<gchristensen>
yeah, freaked me out too
<cole-h>
RIP whoever has Jenny's number with the area code you told me to use :P
<MichaelRaskin>
I mean, «me» me would want to hear which version of in-store /etc you settled upon
<samueldr>
I was answering the more vague "not even nixos implementation details" question
<samueldr>
gchristensen: poking around an immutable system to debug issues
<gchristensen>
cool
<samueldr>
gchristensen: how immutable is immutable
<cole-h>
How you handle deployment, what kind of problems you might face by going this route, how to resolve those problems
<gchristensen>
"ish"
<samueldr>
gchristensen: (that was not a question, but something to talk about in the talk)
<gchristensen>
yeah
<eyJhb>
Why isn't there a NixOS modules for channels? Wait, should I use flakes?
<MichaelRaskin>
Lifetime distribution in the infrastructure might be telling, and so should be presented at some point
<samueldr>
eyJhb: because channels are stateful
<gchristensen>
I spent a while today combing through old backups (I didn't have many :() looking for pictures of an old "server" I had in my closet with a box fan pointing in to the side. I really wanted to put it in as slide 0
<gchristensen>
samueldr: incredibly useful answers, thank you
<samueldr>
you can set nixPath to something specific in your configuration to *replace* channels though
<samueldr>
gchristensen: never too late to dramatized reconstruction
<MichaelRaskin>
You probably have things that can be made fully immutable and recycled daily, and a DB server
<samueldr>
well, once on stage, it may be too late
<MichaelRaskin>
Well, if the stage has a matching fan!
<gchristensen>
samueldr: my basement "rack" might satisfy
<samueldr>
it ain't no rack if it ain't a lack rack
<cole-h>
gchristensen: Complete with blood?
<cole-h>
:P
<gchristensen>
oh it isn't even a lack rack lol
<gchristensen>
it is a shelf with a fire hazard on top
<samueldr>
if it's a shelf, it's still better than my solution
<samueldr>
hm
<samueldr>
fire hazard might be worse
<MichaelRaskin>
_or_ a more realistic evaluation
<samueldr>
I use some piece of furniture that's not deep enough, and has a door on the bottom part which can't close fully
<eyJhb>
samueldr: wait, I am overthinking this. I can just specify a git, shove it into my store and point to that. The only con is I need to get the hash for each and every single upgrade
<samueldr>
eyJhb: yeah
<samueldr>
I kinda do this
<samueldr>
but in a cheaty mutable-ish way
<samueldr>
(I upgrade more frequently than what the repo seems to imply)
<immae>
gchristensen: now that I read your links about zfs I’m quite sad because it would solve many problems I have but I see no proper (and safe) way to convert my current setup (which use ext4) to it...
<cole-h>
There is a proper and safe way to convert your current setup: re-install :D
<immae>
well
<immae>
that would require a huge downtime :p
<eyJhb>
ldlework: then I need to change platform again :(
<ldlework>
eyJhb: for the last time!
<pie_>
immae: if you have the disk space you could set it up in a vm and copy things over
<eyJhb>
ldlework: have you used it for anything? :D
<ldlework>
eyJhb: I have been building a system just like Styx for a few years, for building ldlework.com
<immae>
pie_: I’m not sure I understand your suggestion
<ldlework>
But the problem is that it is in Python, so it is totally up to the user to carefully order the processing of data
<ldlework>
But in Nix, it takes care of this ordering for you
<ldlework>
eyJhb: So I am abandoning my own system Blot, and taking over maintainership of Styx
<pie_>
immae: im not sure about your problem either tbh
<ldlework>
As soon as I do the work to get a 0.8 release out, I will rebuild my site on Styx
<pie_>
immae: you dont neecesssarily need downtime
<eyJhb>
Sweeet
<eyJhb>
ALso...
<ldlework>
eyJhb: I have read every line of code in Styx, many several times :P
<eyJhb>
ROBODUELS?!
<ldlework>
so hopefully that counts enough
<ldlework>
eyJhb: Yeah, you can battle real robots over the internet for money
<ldlework>
it's pretty legit
<immae>
pie_: to reinstall from scratch (as cole-h suggested) I would. I’m trying to think of a way to migrate my current setup (two disks in RAID1, single partition with whole system) to a setup using zfs
<pie_>
wait lol ldlework made that?
<ldlework>
made what?
<pie_>
the ueling robots site
<ldlework>
oh no, i built their backend infrastructure
<pie_>
aha
<eyJhb>
Sounds cool with everything, keep me posted about styx! :D
<eyJhb>
And the battlebots.
<ldlework>
eyJhb: come hang out in #nixos-styx :)
<ldlework>
or don't
<ldlework>
yeah the bots is super fun
<ldlework>
they run it every week and i'm pretty sure it's still free?
<eyJhb>
Joined :p
<eyJhb>
Everything Nix soon!
<ldlework>
Everything "build-y" anyway!
<eyJhb>
And I still don't understand the language! 10/10 idea :p
<ldlework>
I'm warming up to it.
<ldlework>
It's mostly a library problem for me at this point.
<manveru>
i think i'll just make a github action for that
<energizer>
when would you have access to nix-lib without pkgs?
<infinisil>
energizer: Many projects only need pkgs.lib
<infinisil>
And pulling in a nixpkgs for those is pretty slow and wasteful
kgz has joined #nixos-chat
<gchristensen>
"This project is under development. I personally use it to manage several user configurations but it may fail catastrophically for you. So beware!"
<gchristensen>
scary
<__monty__>
gchristensen: Isn't the whole disclaimer thing pretty typically american?
<manveru>
sounds like home-manager :)
<emily>
infinisil: what projects build without anything from stdenv, exactly?
<manveru>
anything written in pure nix
<infinisil>
Hm, my https://github.com/Infinisil/nixus does rely mainly on the module system, but I notice that it also uses pkgs in some places
<manveru>
i even write my own derivations from scratch when i want something minimal :P
<infinisil>
Can't think of one that doesn't need pkgs actually
<Valodim>
infinisil: is the background video on niteo supposed to be that choppy? looks pretty odd
<Valodim>
(assuming you're involved in that)
<infinisil>
Valodim: Huh on the website? I'm just a small contractor for them, got nothing to do with that
<emily>
manveru: ok, but it's basically the subset of Nix without derivations, right?
<emily>
I feel like there's only so much pure utility code you can write around that
<emily>
I do think it'd be nice if nixpkgs.lib was a separate flake (preferably in the same repo), just kinda skeptical that any "leaf" project can get away without a nixpkgs dependency
<manveru>
it's not easy, yeah
<manveru>
but i think that's also because the nixpkgs lib is pretty anemic
<infinisil>
adisbladis: Oh also, we could have separate issues/prs for nixpkgs-boot. If in addition we also split language builders into e.g. nixpkgs-builders, nixpkgs itself would pretty much only contain package definitions
<infinisil>
Oh and I guess NixOS
<infinisil>
But then again, this might come with a lot of problems too, monorepo or not, yadayada
<infinisil>
I'd be interested to see how well multiple repos work
<infinisil>
I believe it might just work much better, especially since the current nixpkgs is getting rather bloated
<MichaelRaskin>
I remember that at some point NixOS and Nixpkgs were split as two repos
<infinisil>
Yeah, also thought of that
<MichaelRaskin>
(half-accidentally)
<MichaelRaskin>
I remember they were remerged and I support that re-merging
<MichaelRaskin>
So for splitting into many real repos (and not just formal ones because GitHub is bad at issue tracking) I would be on the side asking for some convincing arguments why it will be better now
leah2 has joined #nixos-chat
<MichaelRaskin>
staging is as much about evaluating impact on the rest of Nixpkgs as it is about changing stdenv per se
<MichaelRaskin>
(This is not an argument against in-repo policy about allowed references, possibly with multiple evaluation-cache units inside the single repo)
<MichaelRaskin>
(Also, having an auto-exported from Nixpkgs smaller repo with just the lib could be indeed convenient)
<pie_>
id put independent components on separate branches and subtree merge them
<pie_>
that way you get both your lockstep and independence
<pie_>
basically channels
<pie_>
nixpkgs-boot, lib, nixpkgs, nixos -> subtree merge into into channel branches
<MichaelRaskin>
I am afraid that cross-cutting PRs risk becoming even more complicated
<pie_>
yeah i guess
* pie_
makes a mental note of that for his workflow
<hyperfekt>
git transactions
<pie_>
heh what :p
<joepie91>
no energy for extended discussion atm, but one observation I've made within the JS ecosystem is that if splitting things across many repos produces a cross-cutting PR concern, that is almost always a sign that you split things up wrong or nonsensically (eg. tightly coupled code that's in separate repos but not actually separately usable, and lacking generic interfaces/abstractions)
* pie_
attempts to not incide joepie91 to spend more energy x)
<joepie91>
which makes sense, cross-cutting PRs can only really be a thing if a change in one thing requires a change in another that would not be independently useful, which means tight coupling
<pie_>
well, but in the case where something is a direct dependency its going to necessitate ...
<pie_>
yeah so like, dependencies
<pie_>
with two subtrees youll have to fork them both and merge them both at once
<joepie91>
no, dependencies can be (and should be) loosely coupled
<pie_>
if you make a breaking api change?
* pie_
tries to come up with an example but idk
<infinisil>
forward/backward compatibility
<pie_>
yeah i guess you have to have some deprecation process
<infinisil>
Or deprecation yeah
<pie_>
i mean lets take a concrete argument with a change in lib or in -boot
<pie_>
well what i really meant to say is a change in stdenv
<joepie91>
pie_: you can make breaking API changes in a loosely-coupled thing... the key point is that you are treating your own dependent thing as a third-party consumer
<joepie91>
which means API changes need to be documented and generic
<pie_>
ok
<joepie91>
new major version, changelog, etc.
<joepie91>
"design it like a library" to put it bluntly
<infinisil>
We could definitely use more of that for nixpkgs imo
<pie_>
MichaelRaskin: with the above, so just for the sake of argument, what if we were making one of the fancy nice stdenv changes we wanted like , idk, structuredattrs
<pie_>
(which i havent actually looked at so idk anything about it)
<pie_>
though i guess thats not great because sturcturedattrs doesnt break anything
<joepie91>
in those cases cross-cutting PRs are not necessary; because if the dependency is designed as a loosely-coupled library, then whatever interface the dependent needs would need to be generically useful, and thus can be argued on its own merits and separately documented, without needing to coordinate with the dependent
<joepie91>
you still have a dependent PR, but it is no longer cross-cutting
* pie_
scratches head
<pie_>
you mean theres no reverse dependency?
<joepie91>
correct
<joepie91>
say that you are developing bar and foolib
<joepie91>
bar depends on foolib
<MichaelRaskin>
Well, unlike JS ecosystem we inherit the blast radius from upstream
<joepie91>
bar needs to do something with foolib that foolib currently does not support in its API
<MichaelRaskin>
And so either we radically change the version discrepancy inside Nixpkgs — or we have cross-cutting concerns
<pie_>
i mean maybe its making a mountain out of a molehill^Wwhat is already a mountain; but if we make a breaking change to stdenv theres a bidiretional dependency there
<joepie91>
if loosely coupled, the steps are now: 1) submit a generically-useful design change to foolib's API, that is not specifically designed for bar, but that fills in the missing functionality that bar can make use of; 2) once that gets merged, update bar to use the new API
<joepie91>
step 1 is totally independent of whether step 2 happens
<joepie91>
it's a generic change so even if 2 gets lost into the void you've still improved foolib
<pie_>
joepie91: yeah i get that i think
<joepie91>
and therefore the dependency is unidirectional; bar depends on foolib to improve its API, but foolib does not depend on bar to make use of it
<pie_>
joepie91: the improvements to the lib stand on their own
<pie_>
but like in the interim, do you make an stdenv2 or what
<joepie91>
MichaelRaskin: I'm making no particular argument for the nixpkgs structure here (yet), just addressing the "cross-cutting PRs" thing and where it comes from :P
<joepie91>
pie_: you block the change in bar until foolib changes its API, and if it's urgent you fork foolib; like if foolib were a third-party library
<joepie91>
that's really the crux here; the easiest way to design loosely-coupled things is to pretend that consumer-you and developer-you are two different people
<joepie91>
as if it were a third party using your lib, and you using a third-party lib
<pie_>
joepie91: we are in perfect (?) agreement insofar as the changed of the library dont break downstream; but once they break a large downstream you have to fork till you can merge downstream?
<MichaelRaskin>
Well, Nixpkgs has inversion to basic trade-offs at every inherent step, then we go ahead and adopt policies that do not mitigate these different trade-offs but underline them (and fit them!)
<joepie91>
(this also has a lot more subtle benefits like making forks easier, by essentially putting the core maintainers and third-party users on an equal playing field)
<MichaelRaskin>
Nixpkgs has incredible build time _per line of Nixpkgs code_
<pie_>
MichaelRaskin: but you can fix individual packages incrementally
<pie_>
MichaelRaskin: also i dont understand what you were saying about inheriting upstream and version discrepancies
<MichaelRaskin>
Nixpkgs policies go towards minimising simultaneous multi-version use as much as possible
<MichaelRaskin>
We do not use how fontconfig and glibc interact
<joepie91>
pie_: not sure what you mean with the "fork till you can merge downstream"
<MichaelRaskin>
We do not use what optimisations gcc10 includes
<MichaelRaskin>
Oops
<MichaelRaskin>
We do not choose
<MichaelRaskin>
We use the interactions that are chosen upstream
<pie_>
joepie91: sorry, that was overcompresed;im just restating, i think if you make a breaking upstream change for a large downstream, you have to fork both till the downstream can merge
<joepie91>
pie_: I still have no idea what you mean with "the downstream can merge". merge what?
<joepie91>
and who is the upstream vs downstream here?
<pie_>
joepie91: downstream has to fork to support the new upstream, they cant merge their fork until theyve finished integrating the upstream
<pie_>
upstream is stdenv downstream is nixpkgs
<joepie91>
cole-h: "Beating source bit-rot: sources can be stored and shared via IPFS so we don’t lose the data needed to reproduce builds." -- yeaaaaahhh uhhhh, that's not gonna help :P
<joepie91>
IPFS doesn't "store" anything, it's basically just more granular bittorrent...
<pie_>
yeah like ever heard of dead torrents lol
<joepie91>
pie_: can you restate that with bar vs. foolib please
<joepie91>
because I don't follow how what you're saying relates to my explanation
<pie_>
k hold on
<infinisil>
cole-h: Nice!
<cole-h>
I'm not related to it at all, but I saw the CAS RFC get bumped by that post and thought that'll be real cool
<pie_>
joepie91: the generically useful change to foolib breaks bar and everybody else; it takes bar a long time to migrate to the new implementation. durig that time there is a fork of foolib and of bar because foolib doesnt want a somefunction2. so until (effectively) everyone moves to the implementation that is somefunction2, you cant merge foolib or bar.
<infinisil>
Yeah that sounds really exciting
<pie_>
joepie91: unless youre saying it would be fine to have an stdenv.mkDerivation2 or somethig
<pie_>
and then you hve to rename mkDerivation2 to mkDerivation and rename all usages of mkDerivation2 to mkDerivation once everything is done.
<pie_>
joepie91: no mention of "dead torrents" in your comment? :p
<pie_>
ill just reply
<cole-h>
joepie91: Well, doesn't Hydra already store the sources needed to build from? Maybe Hydra could then be augmented to be a "seeder" for the IPFS stuff -- I don't see Hydra dying anytime soon, at least
<joepie91>
cole-h: my problem is with when people start expecting IPFS to magically persist stuff
<energizer>
isn't that the point of a filesystem?
<cole-h>
joepie91: I see. So not that IPFS can't persist stuff, but that it requires some thought put into how it should mostly-persist stuff?
<joepie91>
"oh now we don't need the Hydra storage anymore"
<joepie91>
this is a super common misconception
<joepie91>
like, sure, you could set up Hydra to be a seeder for your data chunks, but then how is IPFS really adding any availability or resistance to bitrot there
<joepie91>
it's really Hydra doing the storage there, like it was before, adding IPFS didn't change that
<joepie91>
cole-h: a more useful way to think about it is "IPFS distributes, it does not store"
<pie_>
something something cdn?
<joepie91>
energizer: yes, this is one reason I am more than a little dissatisfied with the IPFS marketing...
<pie_>
but on other peoples computers
<joepie91>
anyway
<infinisil>
joepie91: What we could potentially get rid of is the CDN for the caches
wildtrees_ has joined #nixos-chat
<joepie91>
infinisil: yes but that is my point, that is distribution, not storage
<infinisil>
Yeah, just wanted to mention that
<energizer>
that would be storing the caches on various users' home computers?
<joepie91>
pie_: regarding your example case, I don't understand why there would be a fork of bar *and* foolib
<infinisil>
energizer: Anybody using Nix could then contribute to redistribution
<infinisil>
Well anybody that runs the IPFS integration thing
<joepie91>
I think (opt-in) P2P build delivery is an interesting idea, FWIW
<joepie91>
I'm just very worried that too many people are going to be misled by the marketing of IPFS specifically
<joepie91>
and expect it to magically keep data around
<joepie91>
because that is more or less what happens every time someone tries to add IPFS to a thing
<energizer>
infinisil: the idea being i can download from my nextdoor neighbor faster than from the cdn across town?
<infinisil>
energizer: Yup, that's pretty much the idea behind IPFS
wildtrees has quit [Ping timeout: 256 seconds]
<joepie91>
(that isn't necessarily true btw, but that's a different discussion :P)
<infinisil>
Yeah
<energizer>
is that a realistic proposition? presumably the major cdns do a huge amount of optimization relative to my neighbor who's probably on a crappy connection
<infinisil>
energizer: In an ideal world, everybody runs IPFS, in which case *all* your neighbors, in the whole town, can improve your speeds
<joepie91>
not just that but your connection to your neighbour also goes to an exchange point first- where, presumably, the CDN has a server sitting - and back
<joepie91>
you don't connect directly to your neighbour
<joepie91>
at least not in most network setups
<joepie91>
so all that does is add an extra roundtrip
<bqv>
Hey I remember when I was trying to suggest exactly this in #nixos and emily wasn't buying it :p
<joepie91>
pie_: to refer back to your sample case for a moment, normally you would release a new major version of foolib, possibly maintain the previous major version of foolib for a while, and each consumer - including when you are that consumer yourself - uses whichever version of foolib they are compatible with
<bqv>
joepie91: fwiw one luxury of nix is that a cache miss (due to dead torrent syndrome) isn't usually fatal, so its still a plus
<pie_>
joepie91: isnt that a fork?
<joepie91>
bqv: the claim in the thread is "prevent source tarball linkrot", more or less
wildtrees__ has joined #nixos-chat
<joepie91>
ie. the thing where that is fatal :)
<pie_>
joepie91: i dont really see how thats any different than build output linkrot tbf...
<joepie91>
pie_: isn't what a fork?
<pie_>
joepie91: "normally you would release a new major version of foolib, possibly maintain the previous major version of foolib for a while"
<joepie91>
then no, that is not a fork
<joepie91>
it is the same project by the same maintainers
<emily>
I think joepie91 is right: IPFS might be interesting for distribution, not so much for storage. did it stop being way too slow?
<joepie91>
just multiple version branches maintained, like Debian has multiple maintained releases at the same time
<joepie91>
Debian 7 is not a fork of Debian 6 either
<infinisil>
emily: The recent IPFS update said to improve speeds a lot
<emily>
2x faster adding, ok. my impression was that Hydra was writing much more than 2x what IPFS could take to S3 in the past?
<emily>
so there still might be some gap to go.
<pie_>
something something timing attack lol <joepie91> you don't connect directly to your neighbour <joepie91> at least not in most network setups
<pie_>
privacy or decentralization pick one xD
<joepie91>
anyway enough discussion for tonight
<pie_>
o/
<joepie91>
back to my validation library project
burkelibbey_ has quit [Quit: Connection closed for inactivity]
wildtrees has joined #nixos-chat
wildtrees_ has quit [Ping timeout: 258 seconds]
wildtrees_ has joined #nixos-chat
wildtrees__ has quit [Ping timeout: 258 seconds]
wildtrees has quit [Ping timeout: 256 seconds]
<pie_>
joepie91: fwiw werent you just saying that we should behave as if the various api proveders and users were separat entities? ;p but im just nitpicking
<joepie91>
pie_: ?
<pie_>
joepie91: nevermind nevermind have fun with your library c:
<joepie91>
pie_: no really, how do you mean?
<pie_>
entitcant remember your phrasing, gotta find it in scroll..
<pie_>
<joepie91> as if it were a third party using your lib, and you using a third-party lib
<pie_>
<joepie91> that's really the crux here; the easiest way to design loosely-coupled things is to pretend that consumer-you and developer-you are two different people
<pie_>
so i kind of put words in your mouth there
<joepie91>
pie_: well yes, but how does what I said / am doing conflict with that :P
<pie_>
<joepie91> then no, that is not a fork
<pie_>
<joepie91> it is the same project by the same maintainers
<joepie91>
sure? this doesn't conflict
<joepie91>
v1 and v2 of a thing are the same project by the same maintainers, not a fork
<joepie91>
this holds true for third-party libs also
<pie_>
MichaelRaskin: ok so maybe -boot would not really work but lib?
<MichaelRaskin>
lib, as unimpacted by upstream, possibly — although not really clear what it would yield compared with just providing automatic snapshots in a separate repo
<colemickens>
would zfs on rpi4 (4gb) be... a bad idea?
<MichaelRaskin>
Note that changes to lib by mission do not stand on their own — they must show that they are not making a worse tradeoff than before for Nixpkgs
<joepie91>
pie_: incidentally, have a look at that library to see me applying the principle I explained :P though most of the docs are still missing
<MichaelRaskin>
As long as there is a claim that Nix is not a general-purpose programming language, the preference system of lib is defined in terms of Nixpkgs
<pie_>
joepie91: i did peek at i tthe other day but im bad at internalizing things
<joepie91>
pie_: (each validator is a separate package, and 'official' packages have no special privileges over third-party packages, they have to deal with exactly the same external-facing API)
<pie_>
MichaelRaskin: well, stuff like it not being such a big deal to reimport nixpkgs (or lib rather) when it gives you infrec issues in modules
<pie_>
though that feels like more of a workaround than anything
<pie_>
yeah the no-batteries-included is rather annoying of lib sometimes
<pie_>
decentralization is good but you you want some form of centralization for collaboration
<pie_>
or you end up with lisp problems
<pie_>
<joepie91> it is the same project by the same maintainers
<MichaelRaskin>
Note that for Common Lisp code the Lisp curse is _way_ less burden than for a «package the OSS world» project
<MichaelRaskin>
openssl changes are necessary _and_ break downstream horribly.
<cole-h>
colemickens: I don't think so? I mean, IIRC gchristensen said he has a Pi running ZFS.
<colemickens>
oh I didn't see that, that's a boost in confidence about doing so then
<joepie91>
pie_: in Validatem, I intend to address the "no batteries included" problem by maintaining a centralized list of categorized validator libs, first-party and third-party
<joepie91>
IMO that's a model that many more projects could benefit from
<joepie91>
because ultimately it is a documentation/discovery problem, and bundling stuff is a bad solution for a documentation/discovery problem :P
<elvishjerricco>
colemickens: Only drawback is that you have to use USB, which is notoriously unreliable for storage. ZFS people like to point this out, but I dunno if it's worse on ZFS or if they're just more conscious of it because of ZFS's great reliability track record
wildtrees has joined #nixos-chat
<elvishjerricco>
And if you're using multiple disks, doing it all on one bus or hub can be rather limiting in performance
<colemickens>
nothing too complicated, just one drive, I figure it's just an rpi. Which part is unreliable though? linux usb stack, cheap adapters with bad firmware, just more prone to getting bumped/disconnected?
<elvishjerricco>
USB drives are likely to claim that a write is committed when it actually hasn't been
<colemickens>
or is it just generally that zfs+usb has popped up on your radar enough to note it?
<colemickens>
aha
<elvishjerricco>
something that sata/sas basically NEVER do
<elvishjerricco>
So ZFS goes ahead and writes an uberblock pointing to an incomplete write that it thinks is complete, and bam you've got a corrupt disk if you powered off at the wrong time
<colemickens>
It's a SATA drive in a cheap USB adapter, so I wonder what characteristics mine will have. It's an effectively stateless machine anyway, so low risk.
wildtrees_ has quit [Ping timeout: 256 seconds]
<colemickens>
oh, I see the point
<elvishjerricco>
SATA behind USB makes no difference. Basically all external drives are like that
<elvishjerricco>
I'm not sure why it's so common for USB drives to do this
<elvishjerricco>
I wonder if the protocol just sucks
<colemickens>
Is there a workload one can run to stress test a suspected drive?
<colemickens>
Hm
<elvishjerricco>
colemickens: Just keep writing stuff, and powering off the drive suddenly, then plugging it in and scrubbing it
<elvishjerricco>
It's important that the drive powers off, if it has its own power supply
<elvishjerricco>
ZFS scrub will always detect these kinds of issues
<colemickens>
okay, thanks for the tips
<elvishjerricco>
colemickens: I'd probably just run a big fio and unpower the drive in the middle, or send/recv a big dataset from an existing machine
wildtrees_ has joined #nixos-chat
<clever>
elvishjerricco: something ive wanted to do, for hardening testing, is to basically run linux in a VM, and record a log of every block write that happens (maybe a fuse block dev would work too)
<elvishjerricco>
clever: What for?
<clever>
elvishjerricco: and then, i can replay that IO log, and try to mount the fs at every point in the log
<elvishjerricco>
Oh, that's a really cool idea
<clever>
what would happen if it got interrupted after the N-th write?
<pie_>
huh neat
<clever>
but it sounds like your saying usb is worse, the 5th write can get lost, while the 6th write went to disk
<elvishjerricco>
clever: That's what I've heard
<elvishjerricco>
No experience with that myself
wildtrees has quit [Ping timeout: 240 seconds]
<MichaelRaskin>
I think it might be actually the drive batching writes
wildtrees__ has joined #nixos-chat
<MichaelRaskin>
If there is no sufficient SATA-level barrier injection or if the drive just lies
<elvishjerricco>
That's certainly plausible
<clever>
as for the other side of things
<MichaelRaskin>
(so that nearby writes happen more efficiently)
<clever>
zfs treats each write() syscall as atomic
<elvishjerricco>
On properly functioning block devices, I believe it is
<clever>
and i think all write()'s to a single dataset, will be garenteed to occur in the order they where made, up to the point of failure
<clever>
either the entire operation worked, or none of it worked, and nothing from the future leaks thru
<elvishjerricco>
Really? I've not heard that before
<clever>
ive asked in #zfsonlinux before
<clever>
if you dont sync things, you can loose some data, but it will always be atomic to the write() syscalls
<clever>
and i think it will remain in-order, even between files?
<gchristensen>
elvishjerricco: I use it on mmc
<elvishjerricco>
Related, a single write() is not guaranteed to be atomic in the event of something like power failure, even with O_SYNC. It is able to write half of the data and then crash, leaving the file half written.
wildtrees_ has quit [Ping timeout: 258 seconds]
<clever>
elvishjerricco: i think zfs will just rollback the entire action, and none of it will have happened
<elvishjerricco>
gchristensen: rpi has mmc?
<gchristensen>
or whatever that thing is
<clever>
elvishjerricco: i know a decent amount about the rpi
<elvishjerricco>
clever: Nope. ZFS works in terms of records, even for sync writes. It'll happily write only some of the records
<gchristensen>
yeah, mmc
<gchristensen>
elvishjerricco: when does the block pointer get written, in such a way that a partial write was possible?
<elvishjerricco>
gchristensen: With async writes, your write will be batched into txgs, and if it spans multiple, it can be interrupted. With sync writes, it writes one record at a time to the ZIL
<elvishjerricco>
And the ZIL is more like a journal than a CoW fs
<gchristensen>
ah
<elvishjerricco>
Though that gets murky...
<gchristensen>
what is it that gets you all the good knowledge?
<clever>
if you are using a seperate log device, and there are now power failures, that journal will be 100% write-only
<clever>
zfs writes to the journal first, to be sure it can repeat the actions later
<elvishjerricco>
clever: Yes, but power failures are the failure mode I'm concerned with here :P
<clever>
then at some point in the future (turning it async), it writes a real copy, to the CoW storage
<elvishjerricco>
gchristensen: Lots of free time and reading and playing around :P
<clever>
if there is a power failure, it can replay that journal upon import
<clever>
to recover any records that didnt get written
<elvishjerricco>
clever: Yes, but if it's only in the ZIL, and you get a power failure, your ZIL may have only part of the write
<elvishjerricco>
e.g. only a few of the 128k records in a 1M write
<clever>
elvishjerricco: i think the write() syscall wont return, until the disk confirms a flush to the ZIL
<elvishjerricco>
clever: Yes, but it will still be written
<elvishjerricco>
Your application won't get returned to, but the intermediate state is now the active state
<elvishjerricco>
Decidedly nonatomic
<clever>
if the disk is doing its job, the data should be safe once its flushed
<clever>
let me see if i can find the irc logs...
<elvishjerricco>
clever: Say you write 1M in a sync syscall. ZFS writes a single 128k record at a time to the ZIL. Every record becomes active as soon as it's written. You get a power failure after the third record. You boot back up. Your file now only has 3 records of your 1M write syscall
<elvishjerricco>
So your file appears incomplete
<elvishjerricco>
It's neither the state before the write began, nor the state you wanted after it would have finished
<clever>
elvishjerricco: that doesnt agee with what i last heard in #zfsonlinux
<gchristensen>
would the read succeed, elvishjerricco?
<elvishjerricco>
gchristensen: The block was still written CoW style, sorta, but in a journal style data structure. So if the record was written to the ZIL, it's good
<elvishjerricco>
But I'm pretty sure I tested this by creating a large random file, loading it into memory, and writing the whole buffer with O_SYNC to another file, and killing the VM mid-write.
<elvishjerricco>
On reboot, the file was partially written, and scrub was OK
<elvishjerricco>
Killed it after only a second or so to make sure it was spanning multiple TXGs
<elvishjerricco>
make sure it was NOT spanning multiple TXGs
<gchristensen>
might be worth opening a bug report?
<elvishjerricco>
gchristensen: I don't think it's a bug. I think ZFS just doesn't guarantee fully atomic write(). Like, it definitely can't if you DO span multiple TXGs, so why guarantee it for a single TXG?
<gchristensen>
well, it says things like "In the event of a shorn write (a system crash or power loss in the middle of writing a file), the entire original contents of the file are still available and the incomplete write is discarded."
<gchristensen>
and "Every write in ZFS is an atomic transaction, because ZFS makes use of barriers to complete transactions. This prevents reordering of writes that might cause inconsistencies due to incomplete writes."
<elvishjerricco>
If your write takes 30 seconds to complete, and you have one TXG every 5s, and TXGs are the unit of atomicity in ZFS, how could that write possibly be atomic?
<clever>
elvishjerricco: i would expect it to create new blocks for the partial write, but not point the file to those new blocks
<gchristensen>
so it is just that everything you're saying here goes counter to what they say about it, which is why I think it would be worth a bug report. either to update docs, clarify the situation, or get a reason why it isn't the way it is
<elvishjerricco>
So I don't think "write" in those claims means "a write() syscall" I think it means "every write in a txg"
<clever>
elvishjerricco: and only after the entire thing has been written to disk, would it update the file to point to those blocks
<gchristensen>
in other words, I'm not inclined to believe you until it is discussed in a bug report, because the documentation says it is part of their design goal to do otherwise
<elvishjerricco>
Lemme see if I can reproduce my test...
wildtrees__ has quit [Quit: Leaving]
<gchristensen>
it isn't that I'm unwilling to believe you
<gchristensen>
just that if what is say is true, it should probably be in a bug report