<gchristensen>
say for example I wanted to publish once a day a dump of the hydra database. this file is ~30gb. I imagine people would not be interested in fetching it daily ... so maybe it isn't worth uploading daily... maybe ipfs or something?
<gchristensen>
or dat maybe :)
<samueldr>
are there binary diff formats that could help here?
<gchristensen>
its a .sql file
<samueldr>
(text is still binary, prove me wrong)
<gchristensen>
(fair)
<samueldr>
so I guess the answer is kinda :)
<samueldr>
is it 30GB compressed?
<gchristensen>
I think so
<samueldr>
yeowch
<samueldr>
still, it's not big data, it fits in my RAM
<gchristensen>
:P
<simpson>
gchristensen: It's frustrating that the answer can't be "just open up a socket to your database".
<gchristensen>
yeah
<gchristensen>
so the export to .sql will happen totally offline, from a backup host
<gchristensen>
part of the purpose of the export is seeing if the backup worked. if postgres starts and manages to export a dump over 30g, the backup is probabl ygood
<Irenes[m]>
I guess I'd say reduce the frequency of the data dump to what you expect people to actually want
<gchristensen>
being able to provide the data is only a nice byproduct of validating the backup
<Irenes[m]>
ipfs is an interesting suggestion but it's kind of a lot of work for people to download
<Irenes[m]>
makes sense
<gchristensen>
so at that frequnecy, it would be "do the dump and throw it away" until someone comes asking
<Irenes[m]>
yeah it's honestly not clear to me that there's anyone who really has a need to have their own copy of it
<gchristensen>
I imagine that if it was available, people would be interested in asking questions of it :)
<simpson>
Right. First you have the data, then you have the metrics, then you have the insights. But also, like, at the same time, I have no idea what it'd be useful for.
<gchristensen>
the first build it has a record of is from 2010-03-05
<Irenes[m]>
hm
<cole-h>
Which build is that?
<Irenes[m]>
I don't know if there's budget for this, but notionally you could put it in Google's BigQuery or something. Then people could query it without downloading it.
<Irenes[m]>
That would require additional engineering work since it's a different engine.
<samueldr>
I see it as being useful for someone that wants to help with the page queries slowness
<simpson>
mdash: ^^^ Publication isn't a single action, but an ongoing service!
<samueldr>
get it on a local development hydra instance, and look at your personal instance crawling under the load!
<gchristensen>
cole-h: an RPM for patchelf
<gchristensen>
I mean, I guess I'll just throw it away then :P
<simpson>
gchristensen: I am sorry that I don't have a good use for this dump, but this is probably the purest situation that I'll ever see that aligns with my research: There's gotta be a better way to just, like, publish a few TiB of data without anybody noticing or caring~ I wish I had answers, and not just similar desires.
<gchristensen>
no worries :)
<gchristensen>
I could make the build output public, and the last step be a magic wormhole send, and wait for 20 hours , offering to send
<gchristensen>
I mean I guess I could just put it up as an https page
<gchristensen>
assuming <=1 downloader / day, it would be fine. if I had >1 download/day, it'd be nice to use a thing which lets it not upload from me twice
johnny101m has quit [Read error: Connection reset by peer]
<gchristensen>
"The control you have over what information Twitter shares with its business partners has changed. Specifically, your ability to control mobile app advertising measurements has been removed"
<lovesegfault>
andi-: I'm hacking on firefox with LTO/PGO
<lovesegfault>
I'll try and tackle vaapi later
<samueldr>
gchristensen: bittorrent?
<gchristensen>
might be the thing
<samueldr>
you could end up uploading more than once, by you could also end up with seeder friends
<samueldr>
though in that situation maybe less frequent publications can help in getting seeds?
<lovesegfault>
Can someone remind me how to run NixOS tests? I always forget 😅
<gchristensen>
lovesegfault: nix-build
johnny101m2 has joined #nixos-dev
<cole-h>
`-A tests.testsdhajjkdsa` maybe
<gchristensen>
is there a trivial `seed-this-file-as-a-torrent ./my.file` thing?
<lovesegfault>
Hmm, is my autocomplete fooling me? I don't see the ff test
<lovesegfault>
gchristensen: mktorrent is part of that
<Profpatsch>
There’s also downstream dependencies in there (e.g. packages that require bazel) so we have some sense of whether reverse dependencies still build.
<Profpatsch>
Oh and I just noticed that people moving the files around broke the tests
<Profpatsch>
That’s why it’s imortant that they are run automatically, because people will just merge crap forgetting the tests
<Valodim>
is the GoNoGo meeting open for attendance? I'd be inclined to lurk, if that's allowed :)
<srk>
last one was public (streamed on youtube as well)
<worldofpeace>
Valodim: come watch if you want too ✌︎
<worldofpeace>
that's my style
* etu
listened in to the last one, planning to do it today as well
<Valodim>
cool!
orivej has joined #nixos-dev
<jtojnar>
♥ how open and transparent Nix community is regarding the processes
<worldofpeace>
my throat was like soo dry vhen I did that call, I'm not going to lie. I should have stopped to get water 😁
<domenkozar[m]>
python naming scheme is confusing :D
<gchristensen>
I will never get that right I promise
<FRidh2>
domenkozar[m]: it fetches from pypi, but it takes the Nixpkgs pythonPackages set, empties it, and then inserts its own packages in there
<FRidh2>
that's why you get a pythonPackages.foo mention
<domenkozar[m]>
oh wow
<domenkozar[m]>
that sounds like a lot of trouble
<FRidh2>
it builds upon nixpkgs's buildPythonPackage which is package set dependent
<FRidh2>
I suppose that is the motivation, to just use that function and other functions
<gchristensen>
does that mean you have an undeclared dependency?
<FRidh2>
well somehow it did appear in the poetry lock file
<FRidh2>
which package is it? setuptools_scm?
<domenkozar[m]>
pyramid_retry
<domenkozar[m]>
it's only in the lock file, yes
<domenkozar[m]>
maybe lock file is out of date?
<gchristensen>
worth a try re-locking
<domenkozar[m]>
hmm, it's still there
<FRidh2>
maybe its related, but I think poetry2nix has trouble with packages in setup_requires, which need to end up in nativeBuildInputs. Maybe `strictDeps = false;` would help poetry2nix
<gchristensen>
and maybe open an issue so adis'bladis can look later :)
<domenkozar[m]>
hmm right I'll try to minimize the example
<domenkozar[m]>
if poetry rewrites pythonPackages, that means it's depending on pyramid_retry before it's created
<gchristensen>
how so?
<domenkozar[m]>
otherwise I'd expect pythonPackages.pyramid_retry to exist
<domenkozar[m]>
this works just as well as I believe in python packaging :D
<pie_[bnc]>
stackoverflow.com but infiniterecursion.com
<pie_[bnc]>
aw damn someone already registered that
<xfix>
stack is a rather technical term
<gchristensen>
domenkozar[m]: poetry has worked super well for me, supporting use cases that I've wanted forever... hopefully it is an easy fix technical or docs wise :)
<xfix>
but it's how it is known as, even if a language doesn't officially have a stack
<domenkozar[m]>
gchristensen: oh I know :) but I've spent way too much time fixing python packaging to know how to set my expectations
<bennofs>
the second commit of that PR needs backport to release-20.03
<gchristensen>
bennofs: do they not both need backporting?
<bennofs>
the first one was already merged with another PR
<gchristensen>
oh I see
<gchristensen>
should that commit be removed from the PR then?
<bennofs>
ah well, the first one is actually only on release-20.03 ...
<gchristensen>
oh what :)
<gchristensen>
thrilling :P
<bennofs>
since master had already diverged from release-20.03, just cherry-picking isn't enough for the first commit. that's why there are two versions of the first commit
<gchristensen>
oh samueld\r already approved too
<gchristensen>
bennofs: ping me when it finishes its eval
<bennofs>
gchristensen: ok thanks :)
justanotheruser has joined #nixos-dev
<domenkozar[m]>
it seems like stack overflow happens due to setup_requires as FRidh2 said
<domenkozar[m]>
how do I figure out which dependency it is? :O
<FRidh2>
seems like poetry lists all the extras as well, and the extra depends on the package it is an extra of
<FRidh2>
is this package of yours listed as an extra somewhere? Or does it have an extra?
phreedom_ has joined #nixos-dev
<FRidh2>
I think an underlying issue is that we choose to add the run-time dependencies to the closure when we install the Python package while at least for withPackages we do not actually need to. Thus, we could flatten the dependency tree.
phreedom has quit [Ping timeout: 240 seconds]
<domenkozar[m]>
FRidh2: where do you see circular dependency in that example?
<domenkozar[m]>
I don't see toml has declared dependencies
<FRidh2>
hmm right
<domenkozar[m]>
I guess I'll wait for adisbladis :)
<domenkozar[m]>
sounds like it should be named store_property
<gchristensen>
yup I think so..!
<gchristensen>
I spent a while last night trying to make the property typechecked by mypi and it was tough (this typo is on purpose, just for you domenkozar[m] ;))
<domenkozar[m]>
haha
<srk>
:D types .. my attempt at haskell port of nixops exploded very quickly
<srk>
good that Python has some support for types (finally!)
<domenkozar[m]>
srk: my friend commented on that picture
<domenkozar[m]>
"you can see how the left guy is almost done"
<srk>
:D
<domenkozar[m]>
gchristensen: nice
<domenkozar[m]>
I like how they picked "imprecise"
<domenkozar[m]>
really gets programmers OCD out
<gchristensen>
hehe
<gchristensen>
and it is really harsh. 2.65% imprecise is yellow
<srk>
:D
<domenkozar[m]>
yeah the color palete just reinforces that
<domenkozar[m]>
"I'm never getting away from this traffic light am I"
<gchristensen>
at any rate, python isn't a perfect language, but mypy really takes it to another level
<domenkozar[m]>
+1
<FRidh2>
still surprised that in all the peps related to type hints, they've never mentioned duck typing
<aanderse>
is there another go no-go meeting today?
FRidh has joined #nixos-dev
FRidh2 has quit [Ping timeout: 256 seconds]
<rnhmjoj>
i think i accidentally discovered an issue in the python setup-hook: i had a typo somewhere and instead of getting "makeWrapper doesn't understand the argument" i got "die: command not found"
<rnhmjoj>
it looks like the `die.sh` hook is not sourced
justanotheruser has quit [Ping timeout: 265 seconds]
<cole-h>
After the meeting, I expect there'll be another karma shower :P
<drakonis>
so, if the outcome is go, does it immediatly release?
<cole-h>
I think in the last meeting they said around Thursday?
<cole-h>
Depends on the RMs I guess
v0|d` has joined #nixos-dev
v0|d has quit [Ping timeout: 265 seconds]
<worldofpeace>
If it's a go we can begin the tasks to actually put nixos 20.03 out. so not immediate.
<drakonis>
i see
<worldofpeace>
but yeah, whenever is good for disasm and I to work in realtime, and I believe we need infrastructure people to be available in realtime as well
<aanderse>
sorry worldofpeace just observing, no mic
<drakonis>
y'all should maybe try chromium if you have issues with jitsi
<drakonis>
firefox seems to be unsupported
cole-h has joined #nixos-dev
FRidh has quit [Quit: Konversation terminated!]
cole-h has quit [Client Quit]
cole-h has joined #nixos-dev
<worldofpeace>
oof, going to have to watch the recording to figure out everything we have to do. I hope it worked disasm
<worldofpeace>
if not, I have pretty good memory
<gchristensen>
was there a lot?
<worldofpeace>
no not at all
<worldofpeace>
we decided it was a no go because it wasn't DONE
<gchristensen>
cool
<worldofpeace>
but we're going to meet again on friday same time
<domenkozar[m]>
nice - ever since Nix got CI for PRs there was a PR opened each day
<domenkozar[m]>
worldofpeace: what is left? :)
thonkpod has quit [Quit: WeeChat 2.4]
<worldofpeace>
domenkozar: it's all in the blocker label
<worldofpeace>
I'm going to try to get notes
<worldofpeace>
and a recording will be uploaded
<domenkozar[m]>
ok thanks :)
<worldofpeace>
TLDR; bugfixes and release notes
<cole-h>
worldofpeace: You still want a second pair of eyes (proofreader) for those GNOME docs or whatever it was you posted last week?
<worldofpeace>
samueldr: I tried to talk about the aarch64 issue, but I honstely will/did have no idea what I was talking about 😁
<cole-h>
Or did I miss the train
<samueldr>
oops, missed the go/no-go meeting, I thought it was "one week from the last"
<samueldr>
worldofpeace: which one, the raspberry pi one? it seems to be setup dependent, we're "done" in the sense that it's documented
<worldofpeace>
haha, we're trying to speed it up samueldr
<worldofpeace>
samueldr: yep, that exactly
<samueldr>
worldofpeace: if it's the channel evals, now that the fix has been merged in master too (it got merged to 20.03 beforehand) I think it should be good
<worldofpeace>
samueldr: Yep, I think I merged the one for 20.03. So I could say from aarch64 it is a go
<samueldr>
though we may be lacking publishing of the channel on channels.nixos.org, in case there are divergences in update rates?
<samueldr>
worldofpeace: I think it should be
<gchristensen>
Ideally we'd have a channel created before release
<samueldr>
at least for blocker issues, the other is not a blocker per se, more of a user experience thing
<samueldr>
19.09 didn't have a discrete channel published
<samueldr>
it only had the jobset
<worldofpeace>
That makes perfect sense to me. It doesn't need to be handled or followed up on in the blocker evaluation
<ma27[m]>
sorry, didn't make it today. Is there a tl;dr somewhere?
<cole-h>
NO GO, meet again Friday
<worldofpeace>
same time. I'm going to followup in the thread and there's going to be a video ma27
<ma27[m]>
ack, thanks a lot!
<worldofpeace>
totally, amping it up as much as we can
<ma27[m]>
btw, the broken llvm on 20.03 is a known issue?
<globin>
ma27[m]: broken as in?
<ma27[m]>
tests are failing
<ma27[m]>
had to build it locally
<ma27[m]>
ah, already found the related hydra build %)