thePirateKing has quit [Ping timeout: 250 seconds]
<gchristensen>
hi friends
<samueldr>
hi?
<gchristensen>
I have a big file which I'd like to transfer between some machines, and have each machine hosting it able to assist in seeding its contents. (it is starting to sound a lot like bittorrent). I don't want the file to be publicly downloadable at all. any suggestions on how to do this? ideally the sw would be able to determine some peers are in the same LAN and sync locally as well as externally
<gchristensen>
(big file = 15g)
<gchristensen>
(this file is immutable)
<samueldr>
(if it fits in ram it's not big data)
<samueldr>
sorry, had to say it :)
<gchristensen>
lol
<samueldr>
can't help though :/
<gchristensen>
aye :)
<gchristensen>
I want to do some processing on this file on some transient servers, so getting it downloaded as fast as possible is important
<jackdk>
possibly stupid question - has anyone built an auth layer atop bittorrent?
<gchristensen>
so maybe syncthing is the thing
<gchristensen>
ah, no, it requires configuration and I'd want this to be hands-free
<infinisil>
gchristensen: How about splitting the file up into <number of seeding hosts> pieces
<infinisil>
Ah, not every machine is the same speed, nvm
<gchristensen>
maybe ipfs + a wrapping encryption?
<gchristensen>
nope. not that.
<infinisil>
gchristensen: Would you be content with a stupid simple bash script?
<gchristensen>
what for?
<infinisil>
For that transfer thing between machines
<gchristensen>
sure :)
<gchristensen>
it might not do the business, but maybe it will!
<infinisil>
Well I wrote this idea down which should just be a couple lines of bash
<gchristensen>
awesome
<infinisil>
Split the file into 1MB fragments (`split` command), sequentially numbered. The requesting machine starts a download thread for each available seed machine, getting the next fragment, and writing to a file which fragments have been downloaded, repeat until everything is downloaded
<gchristensen>
it is way past my bed time (what are you doing up? :o) so I'll have to look in the AM
<infinisil>
The idea is so simple, but it might just work
drakonis_ has quit [Read error: Connection reset by peer]
MichaelRaskin has quit [Quit: MichaelRaskin]
<elvishjerricco>
gchristensen: I don't know anything advanced about bittorent, but can it just be bittorrent? If auth is needed, gate it behind VPN or encrypt the file.
Myhlamaeus has quit [Ping timeout: 240 seconds]
disasm has quit [Ping timeout: 246 seconds]
disasm has joined #nixos-chat
dashkal has quit [Ping timeout: 250 seconds]
endformationage has quit [Ping timeout: 246 seconds]
<NinjaTrappeur>
gchristensen, fyi, there's a bittorent auth extension https://libtorrent.org/auth.html . If you want to go the bittorent way, you'll probably need to setup your own tracker. It's not necessary complicated as long as you can spin up an embedded tracker (qbittorent for instance) on the machine having your (almost) big file on the first place.
<NinjaTrappeur>
^ Definitely not 0-config though :)
jackdk has quit [Ping timeout: 250 seconds]
ixxie has joined #nixos-chat
pie_ has joined #nixos-chat
<joepie91>
gchristensen: I dunno if aria2c has a seeding-only mode, but if it does, maybe that might work here?
<joepie91>
dunno if it does auth though
pie_ has quit [Quit: Leaving]
pie_ has joined #nixos-chat
__monty__ has joined #nixos-chat
ixxie has quit [Ping timeout: 255 seconds]
<gchristensen>
interesting
lassulus has quit [Ping timeout: 245 seconds]
drakonis_ has joined #nixos-chat
drakonis has quit [Ping timeout: 268 seconds]
drakonis has joined #nixos-chat
drakonis_ has quit [Read error: Connection reset by peer]
drakonis_ has joined #nixos-chat
drakonis1 has joined #nixos-chat
drakonis has quit [Read error: Connection reset by peer]
drakonis has joined #nixos-chat
drakonis_ has quit [Read error: Connection reset by peer]
drakonis_ has joined #nixos-chat
drakonis1 has quit [Read error: Connection reset by peer]
drakonis1 has joined #nixos-chat
drakonis has quit [Read error: Connection reset by peer]
drakonis_ has quit [Ping timeout: 246 seconds]
drakonis has joined #nixos-chat
drakonis_ has joined #nixos-chat
drakonis1 has quit [Ping timeout: 268 seconds]
drakonis has quit [Ping timeout: 264 seconds]
drakonis_ has quit [Ping timeout: 250 seconds]
drakonis_ has joined #nixos-chat
<gchristensen>
I decided to just put it on Wasabi (s3-like) instead of fretting about p2p stuff
<__monty__>
Is this still about p2p nix caches?
<__monty__>
Oh, or your personal dropbox replacement?
<gchristensen>
I have a big file I want some machines to use during some computation :)
drakonis_ has quit [Ping timeout: 255 seconds]
<gchristensen>
the file is immutable, though, so not really dropbox-y
<gchristensen>
daaaangit. I did a whole system provisioning process and VM bootup to find out I forgot to put my ssh key in either the interior VM or the host system.
drakonis_ has quit [Ping timeout: 252 seconds]
<joepie91>
(tl;dr they send fake vulnerability notifications to market their shit)
<averell>
type out the ssh key into authorized_keys as penance
drakonis1 has quit [Read error: Connection reset by peer]
<averell>
oh hey, i got that email. i have them on my junk list for a long time.
<joepie91>
simpson: the original post sounds like a pretty typical case of exactly this; trying to control too much, and burning out as a result of juggling too many plates and having too few other people to share the decisionmaking load
lassulus has joined #nixos-chat
<sphalerite>
gchristensen: you'll also want to use only one type of id, and make sure to exclude partitions
<sphalerite>
gchristensen: probably using `/dev/sd? /dev/nvme?n?` (with shopt -s nullglob) is better
<simpson>
joepie91: Indeed. I wonder whether it will only get worse as we transition further away from ambient/systemwide package management.
<sphalerite>
gchristensen: also, what was the state of your secure boot stuff and have you published it anywhere?
<joepie91>
simpson: I think packaging plays a relatively small role in it
waleee has quit [Quit: WeeChat 2.4]
drakonis has joined #nixos-chat
<andi->
has anyone used the rust support in nix to build webassembly yet?
<gchristensen>
sphalerite: it works-ish, but is error prone and doesn't handle out of space erros nicely -- which it hits frequently, since the secureboot efi programs are quite large
<samueldr>
(presumably less fire damaged at the start)
<gchristensen>
my goodness
<eyJhb>
And then there is the teeny tiny water hoose
<eyJhb>
_one_ (at least visible)
<__monty__>
Water damage on such an old structure might not be much better than fire damage though?
<samueldr>
they must be working more on ensuring what few buildings that are close are kept safe
<eyJhb>
__monty__ water damage would be easier to fix, as it is slower than .. fire damage
<eyJhb>
Can't remember what buildings it is close to, long time since I last visited it
<__monty__>
That's not true. Fire probably does minor damage to concrete/stone, water may make the entire structure unstable. Putting out fires is more than just pouring water onto things, unfortunately.
<gchristensen>
well ... we'll find out what happens when it is done
<gchristensen>
presumably they have emergency plans for such iconic buildings
<infinisil>
"This release also marks a change in ripgrep's versioning. Where as the previousversion was 0.10.0, this version is 11.0.0. Moving forward, ripgrep'smajor version will be increased a few times per year."