<infinisil>
Ah then maybe not, was just a semi-random guess :)
<infinisil>
(because I posted this just recently)
<gchristensen>
ah
<infinisil>
gchristensen: What does it do?
<infinisil>
(a 2-node version of the pdf there is rather simple, and I have a bash version of it implemented too)
<gchristensen>
automates a snapshot and a zfs | send, negotaiating the incremental send and no opinion about if the sending side is on the other side of SSH, or on the local side of SSH
<infinisil>
Ahh, neat :D
<infinisil>
Why regret though?
<infinisil>
bash?
<gchristensen>
that ...
<gchristensen>
but also the sending end looks a lot like: `ssh jump -- ssh bastion -- nixops ssh target -- zfs send ...` and the receiving side is also an SSH jump away
<gchristensen>
so the snapshot data is flowing through far far too many systems lol
<pie_[bnc]>
ive got the nix build "why is the build doing this thing" blues again
<gchristensen>
also it doesn't do any clean-up
<infinisil>
Hehe
<gchristensen>
infinisil: do you know zpool checkpoint?
<infinisil>
Seen it, never used it though
<gchristensen>
pretty neat
<elvishjerricco>
infinisil: That write lock idea is neat
<elvishjerricco>
You don't technically need to use multiple snapshots. Bookmarks would work just as well for that use case
<infinisil>
elvishjerricco: Yeah, I'd use bookmarks where possible
<infinisil>
Yeah I like that idea, would love to implement it with a potential zfs library :D
<gchristensen>
I don't understand why bookmarks are a thing
<elvishjerricco>
gchristensen: What do you mean? They're fairly useful
<infinisil>
I still don't quite fully get how bookmarks work, but I believe the main idea is that when you want to send the diff from A to B that you don't need the data from snapshot A for that
<infinisil>
(correct me if that's wrong)
<gchristensen>
how can you diff A->B without knowing what A was?
<elvishjerricco>
Yea bookmarks are literally just the transaction number of what used to be/is a snapshot
<elvishjerricco>
gchristensen: You just send all blocks with a newer birth transaction
<gchristensen>
they're not all guaranteed to exist, though, right?
<elvishjerricco>
gchristensen: How could data in a snapshot you want to send not exist?
<gchristensen>
it isn't a snapshot
<elvishjerricco>
You can't send a bookmark, but you use it as an incremental source for sending a snapshot
<elvishjerricco>
So all blocks in the snapshot newer than the bookmark are sent
<gchristensen>
right, and how do you know what A was if you don't have A anymore?
<gchristensen>
ohh
<elvishjerricco>
gchristensen: Birth transactions are monotonically increasing, and every block has a birth transaction. If a block's birth transaction is newer than the bookmark, it's a block that has been written since the bookmark
<infinisil>
It's neat how zfs allows for this kind of thing
<elvishjerricco>
Very
<elvishjerricco>
The term bookmark is very fitting. Writing new data is like writing a new page in a book, and a bookmark is literally a bookmark in that book. Sending the diff is copying all pages after the bookmark
<gchristensen>
so it walks the tree and sends nodes newer than the ID? neat
<elvishjerricco>
Yea
<gchristensen>
that isn't a good term, no
<elvishjerricco>
Meh, I like it :P
* gchristensen
is explaining why it isn't a good term
<gchristensen>
because if my bookmark is at txg 1, and snapshot Foo (which I want to send) is at txg 1,000,000,001 and since txg 1 I have written 100TB to my 1GB disk, I have torn out the first billion pages of data I supposedly wanted to keep
<gchristensen>
this is where I got hung up on. how are you "bookmark"ing data without any promise that it'll be available for sending later
<elvishjerricco>
Hm that's a fair point
<elvishjerricco>
Maybe something closer to "timestamp" would have been better?
<gchristensen>
not sure actually :)
<gchristensen>
fwiw though, I've read a lot about bookmarks and it still didn't make sense until you wrote it so plainly
<infinisil>
Same ^^
<infinisil>
elvishjerricco++
<{^_^}>
elvishjerricco's karma got increased to 14
<elvishjerricco>
gchristensen: Yea they're not greatly explained anywhere for some reason
<elvishjerricco>
I only picked up on it because I'd seen some talks about ZFS internals :P
<gchristensen>
in fact, they're horribly, incorrectly explained in the man pages, based on what you said
<gchristensen>
"A bookmark is like a snapshot, a read-only copy of a file system or volume."
<elvishjerricco>
Oof
<gchristensen>
and then "Bookmarks can be created extremely quickly, compared to snapshots, and they consume no additional space within the pool." and them I'm thinking okay so snapshots apparently take up enough metadata to warrant a new lighter weight version of the snapshot???
<elvishjerricco>
The section above that describing the different kinds of datasets gets it right
<elvishjerricco>
"bookmark Much like a snapshot, but without the hold on on-disk data. It can be used as the source of a send (but not for a receive)."
<gchristensen>
yeah but still
<elvishjerricco>
Yea it's ridiculous to have a literal lie in the man pages lol
<gchristensen>
it doesn't explain at all how the data transfer is calculated later, which since this is basically the raison d'être seems like a fairly large oversight :)
<gchristensen>
since how the heck do you send an incremental diff you can't compare anymore
<gchristensen>
anyway. elvishjerricco thanks for teaching me today :) I have a PR to send to ZFS... I might sneak a manual patch in too :)
<elvishjerricco>
Heh you have a PR unrelated to this to send?
<gchristensen>
yeah
<elvishjerricco>
Mind if I ask what for? I'm somewhat interested in getting involved in ZFS dev
<gchristensen>
I have a WIP to make `zfs destroy -r` safer by limiting the authority of the filesystem|volume@snap[%snap[,snap[%snap]]]... argument
<elvishjerricco>
What do you mean by "authority"?
<gchristensen>
ie: `zfs destroy -s -r ...` the `-s` saying "only snapshots", and if anything other than a snapshot is provided, the destroy fails
<elvishjerricco>
Ahhh that's a cool idea
<gchristensen>
basically I want to be able to `xargs` destroy some ancient snapshots without accidentallying my pool :P
<elvishjerricco>
gchristensen: Maybe make it more like the `-t` argument of `zfs list`? Specify the type as an argument
<gchristensen>
yeah that would probably be better :)
<elvishjerricco>
The biggest deterrent for me to work on ZFS is that it's written in C.... Man I really really hate C
<gchristensen>
it is an extremely nasty hack at the moment. build/test cycles are super slow and painful, because I can't figure out how to make a `nix-shell` suitable for a `make`-based workflow
<elvishjerricco>
Can't you just nix-shell the ZFS derivation?
<elvishjerricco>
I never had issues hacking on GHC or llvm using that method
<gchristensen>
I assume so, but I wasn't patient enough to figure it out exactly. if you find out, please let me know? :)
<elvishjerricco>
Maybe I'll give that a shot tonight as my first foray into ZFS dev :)
<gchristensen>
for great justice!
<gchristensen>
(I don't like C either)
<elvishjerricco>
gchristensen: Yea, `nix-shell "<nixpkgs> -A`, followed by `patchPhase; autoreconfPhase; configurePhase; buildPhase` worked for me
<elvishjerricco>
One patch didn't apply but it only seems important for `installPhase`
<elvishjerricco>
whoops. s/-A/-A zfs/
<gchristensen>
Nix stuns me sometimes
<infinisil>
I wish this phases business was a bit more uniform/streamlined
<gchristensen>
I'd like to be able to do `runTo buildPhase` and just have everything go until build phase and stop
<infinisil>
Yeah was just thinking the same
<elvishjerricco>
gchristensen: There is `genericBuild`, but that does unpackPhase as well, meaning it doesn't work on your intended tree :P
<gchristensen>
right that wouldn't work at all hehe
<infinisil>
I mean the *ideal* workflow would be something like `nix-develop '<nixpkgs>' -A zfs` and it opens a writable source in your favorite editor with shortcuts set up to `runTo configure`, `runTo build`, etc.
<gchristensen>
"nix-ide"
<infinisil>
Even better with LSP stuff set up already :O
<elvishjerricco>
Hm I can't just set the out, lib, and dev variables to a local directory to see what installPhase would do. Wonder how it's getting the /nix/store output paths...
<gchristensen>
getopt is hilarious.
<pie_[bnc]>
i had a "framework" for that that lets me set shell aliases for a project shell <infinisil> I mean the *ideal* workflow would be something like `nix-develop '<nixpkgs>' -A zfs` and it opens a writable source in your favorite editor with shortcuts set up to `runTo configure`, `runTo build`, etc.
<pie_[bnc]>
but its like, small, and kind of useless
<infinisil>
Soo like `shellHook = "alias foo=bar";`?
<pie_[bnc]>
to be clear there was nothing automatic its just you giving it commands
<pie_[bnc]>
I think i was annoyed by having to re-enter the shell to load the new functions when i was developing on stuff so I added a reload()
<gchristensen>
elvishjerricco: sooo much faster. ./cmd/zfs/zfs destroy -t filesystem rpool/local/old-root@bar => tried to destroy a snapshot, but -t did not allow snapshots to be destroyed.
<elvishjerricco>
Nice!
<elvishjerricco>
gchristensen: Send me the PR when it's up :)
<gchristensen>
ok :)
<elvishjerricco>
I wonder how dangerous it is to use the userspace commands from HEAD while the kernel module is from the distro
<pie_[bnc]>
afaik, not very
<elvishjerricco>
I doubt the abi between them is considered totally stable
<pie_[bnc]>
i ...dont remember the details but it doesnt seem to crash if it cant find something
<elvishjerricco>
Hm neat
<pie_[bnc]>
otoh idk if its *safe*
<pie_[bnc]>
well
<gchristensen>
it isn't safe
<pie_[bnc]>
if its an unprivileged operation it should be fine i guess? but one does not generally stick to unprivileged ops :p
<gchristensen>
I'm not going to run them as root, and presumably ^
<elvishjerricco>
It'd be safe if your local check out was a branch of the release being used in your kernel
<elvishjerricco>
Assuming the abi itself isn't what you're hacking on :P
endformationage has quit [Quit: WeeChat 2.6]
<gchristensen>
okay so docbook is a delight compared to these man pages' format
<gchristensen>
.Oo % Ns Ar snap Ns Oo , Ns Ar snap Ns Oo % Ns Ar snap Oc Oc Oc Ns ...|
<samueldr>
snap crackle and pop?
<gchristensen>
I think Ns means nospace, not sure about Ar
<gchristensen>
also I don't see how to build them, so
<gchristensen>
oh I don't need to
drakonis1 has quit [Quit: WeeChat 2.6]
<ashkitten>
apparently rocket league's linux support is being pulled
<ashkitten>
which pisses me off way more than it maybe should, considering i dont even play that game anymore
<ashkitten>
it's definitely to do with psyonix being bought by epic
<ashkitten>
i wouldn't be surprised if rocket league gets yanked from steam
<samueldr>
I think it already was the plan, no?
<samueldr>
[citation needed]
<ashkitten>
idk
<samueldr>
at least that's the idea I was left in mind
<ashkitten>
i requested a refund because screw that sorta awful business practice
<ashkitten>
i mean, in addition to me not being able to play the game anymore lol
<ashkitten>
i wonder if epic will shove some kinda nasty anticheat in there for good measure, just to make it break for more people
<samueldr>
I kinda expect it at this point :|
<ashkitten>
(rocket league does not have a cheating problem, nor could it really possibly have one. all physics are calculated server side, the client just does interpolation)
<elvishjerricco>
Yet another failure that makes me glad I wrote this silly zfs/pv script. Random power blip in the middle of the night that I was (un)fortunate enough to be awake for. Machine restarted and I had to run the script again. I guess it's paying its dues.
cdepillabout has quit [Remote host closed the connection]
<pie_[bnc]>
worldofpeace: is it immediately streamed over youtube?
<pie_[bnc]>
nevermind, found it
<samueldr>
my cat seems to have a super power, generally silent, up until I'm talking to someone on the phone or via voice chat
<samueldr>
up to knowing even before I speak!
<gchristensen>
well your cat is welcome on the calls too
<ashkitten>
cats are good
KeiraT has quit [Quit: KeiraT]
<sphalerite>
ashkitten: I think you might be biased there.
<ashkitten>
no bias!!
<ashkitten>
cats are just categorically, objectively good
<__monty__>
*Maybe* Chaotic Good, but only just barely.
__monty__ has quit [Quit: leaving]
<pie_[bnc]>
crazy idea: a nix paste tool that creates a paste with a full repro in some current nix environment
<gchristensen>
if any US-based person is at risk for buying hardware for a home lab, don't click this link: https://www.labgopher.com/
<pie_[bnc]>
:OOO
<pie_[bnc]>
sadly neither
<gchristensen>
pie_[bnc]: I have a thing kicking around somewhere in ~/projects which takes a gist and gives you a running NixOS vm for 3h
<pie_[bnc]>
well arguably with the idealized version of this you can just pull the repro to your local machine (or whatever)
<pie_[bnc]>
it just sounds nice, idk how feasible it is in reality
<gchristensen>
aye
<gchristensen>
oh that was totally a boring way to describe it
<gchristensen>
it was a website where you paste the gist and it gets you a VM for free for 3h. for example, to try out NixOS or to get support on a module
<pie_[bnc]>
yeah im pretty sure i understood it correctly, but the collaboration is an interesting idea
<gchristensen>
ah
<pie_[bnc]>
as is the trying it out
<samueldr>
gchristensen: you forgot to say that UK, Canada and Australia are available on the left hand menu
<gchristensen>
oh good
<samueldr>
good thing all offers "in canada" are US auctions with terrible shipping rates
<gchristensen>
heh.
<pie_[bnc]>
samueldr: gotta love spam
<samueldr>
hm?
<samueldr>
not technically spam, they're all auctions that do ship
<samueldr>
compared to all those that don't
<samueldr>
the weight of the servers makes it a non-starter
<pie_[bnc]>
NOOOO I dont have anythin ggood on github yet
<ashkitten>
gchristensen: i have that issue on my phone every time i try to load a twitter link, it doesnt load until i refresh the page
<ashkitten>
i hate twitter lol
<pie_[bnc]>
twitter is pretty simpledevs have to justify getting paid?
<ashkitten>
wish i could like, rewrite twitter links to nitter.net
<pie_[bnc]>
accidentally half a sentence there
<samueldr>
the "new twitter" always does it for me on firefox, both on computer and android firefox
<pie_[bnc]>
what is twitter even doing
<pie_[bnc]>
honestly.
<samueldr>
I don't know that they know
<ashkitten>
they're trying to push people toward decentralized platforms, obviously
<joepie91>
samueldr: best part is that it keeps breaking until you go to Home and then it reinitializes Something(tm) and retries after a failure and then all the pages suddenly work again
<samueldr>
hm, for me a simple refresh fixes it for some amount of time
<pie_[bnc]>
I dont think deventralized systems can compete with a centralized one at scale tbh
<pie_[bnc]>
tho bittorrent
<ashkitten>
pie_[bnc]: dunno what you mean by that, the fediverse is thriving
<pie_[bnc]>
but is it "twitter scale"?
<joepie91>
because it's 2020 and obviously it's reasonable to require that everyone load a specific view (it isn't even a page!) first
waleee-cl has quit [Quit: Connection closed for inactivity]
<pie_[bnc]>
joepie91: im closing irc now i dont want to think about it xd
<ashkitten>
pie_[bnc]: dunno, by some counts there are about 4.4 million users
<joepie91>
pie_[bnc]: there's an error in your reasoning though; you're assuming that a decentralized system *needs* to compete with a startuppy, VC-y centralized system at the same scale
<joepie91>
but the reality is that it doesn't
<ashkitten>
^
<joepie91>
because decentralized systems don't have VCs to satisfy with ridiculous numbers
<ashkitten>
twitter's just a big money sink
<joepie91>
so the point that they need to scale to, to be better than the centralized options, is much much lower than the scale of those centralized options; you reduce the scale to that which is actually needed to create a healthy ecosystem
<joepie91>
by eliminating all the artificial demand
<pie_[bnc]>
eh
<ashkitten>
besides, i can talk to all the people i want to talk to
<pie_[bnc]>
twitter has artificial demand though?
<joepie91>
yes, that's my point
<pie_[bnc]>
i mean
<pie_[bnc]>
what?
<joepie91>
Twitter does, and Twitter needs to, because it has the Growth Mindset(tm)
<joepie91>
and its investors expect to see big numbers
<pie_[bnc]>
besides i guess not being able to comment on stuff without logging in and whatever
<pie_[bnc]>
no i mean i have no idea what twitter does to grow besides just exist
<pie_[bnc]>
i didnt notice
<joepie91>
Mastodon and such do not need to meet that requirement
<joepie91>
because it exists to provide a service, not to produce an ROI for investors
<ashkitten>
i dont care if william shatner can find lance ulanoff on the fediverse, i only care about my friends
<gchristensen>
can mastodon migrate identity yet?
<ashkitten>
gchristensen: yes, sort of
<gchristensen>
ooh?
<ashkitten>
you can choose to migrate followers from an account, but you can't move your posts
<joepie91>
pie_[bnc]: every aspect of Twitter is optimized to maximize 'engagement' and onboarding (ie. acquisition of new users)
<joepie91>
like ~every VC-backed tech thing
<joepie91>
this is why they do the bare minimum to prevent abusive users
<pie_[bnc]>
nooooo D: i just realized i left some food in a bag on my bed and forgot to eat it ;_;
<gchristensen>
ouch, ashkitten
<joepie91>
this is why they work towards integration into all sorts of things
<pie_[bnc]>
mh
<joepie91>
because the #1 priority is growth
<joepie91>
it's not one specific design decision where you can see this; it's all over the design of everything
<pie_[bnc]>
thsts one metalevel more subtle than i was thinking
<joepie91>
for example, if Twitter were interested in providing a quality service, they would probably be adding an invite-esque system like lobste.rs has
<joepie91>
where the invite tree is tracked
<joepie91>
but they won't, because that would interfere with their growth optimization objective
<ashkitten>
gchristensen: it's honestly hard to move posts to another account because you have to rewrite the activitystreams objects as if you posted them, you'd lose any replies, and they wouldn't federate to people unless you boost them
<gchristensen>
yeah
<gchristensen>
I just have no inerest in being graham@somerandomdomain. I'd want to be graham@grahamc.com, but the server is evidently pretty heavy
<ashkitten>
gchristensen: mastodon is fairly heavy, yeah. there are other fediverse software that are lighter
<ashkitten>
pleroma is one that's very light, but i really dislike the developer culture of pleroma (and specific pleroma devs)
<ashkitten>
it does support the mastodon frontend, though
<samueldr>
I really want to see the openid-like idea of having either well-known files / html head elements, or DNS component that allows you to delegate your identity to a federated service
<samueldr>
I have no idea if the current in-vogue federated services even could allow that in their design
<samueldr>
with OpenID I just had to put the right <meta /> in an html page, and that html page delegated to another OpenID service
<samueldr>
I could switch the service and the URL still identified me just the same
<ashkitten>
activitypub doesn't specify user discovery
<ashkitten>
most implementations use webfinger
Synthetica has quit [Quit: Connection closed for inactivity]