<gchristensen>
niksnut: do you have any ideas on how I could approach making nix-daemon emit events and performance stats? like a tick when a build starts, when a build finishes, duration during the build
<andi->
what I constantly think about is a way to specify that an input (lets call it $package) that is just there for say "compress all the images in the src tarball during build" should not cause a recompilation of my package that uses $package for that purpose.. e.g $package would have a version bump because of an security issue during file parsing..
<niksnut>
gchristensen: use perf?
<gchristensen>
I mean, specifically events about what the nix-daemon is doing
<bgamari>
andi-, right, in general it seems like nix is a bit too rigid about hashing
<gchristensen>
so I can track things like "builds per day per system" and "builds per day per architecture" "build is progress" (builds started - builds finished) "builds that failed" "builds that aborted" "builds that succeeded"
<bgamari>
having some a notion of a class of functionally equivalent results would be quite useful
<bgamari>
admittedly it's also dangerous
<bgamari>
but sometimes this is the only workable trade-off
<andi->
bgamari: I do not blame it for that.. it is probably the right approach to assume every input change will change the output. There should be more output caching as in if the same result is produced the same "asset" should be downloaded with the other derivation name..
<andi->
aka: if the binaries are 100% reproducible with two different compilre versions (minor version changes) I'd expect to have some kind of content-addressable binarycache
<gchristensen>
so here is the secret
<gchristensen>
for derivations which are not fixed-output, the hash used by the nix store is not at all related to the files produced
<niksnut>
gchristensen: iirc, perf supports user space trace points, so you can instrument specific points in the daemon
<gchristensen>
the hash is exclusively the hash of the inputs used to produce it
<bgamari>
gchristensen, right
<gchristensen>
niksnut: ooh! that could be very cool, and possibly easy to hook in! thank you :D
<andi->
gchristensen: understood
<andi->
I am just dreaming ;-)
<bgamari>
and I would propose that it might be useful to allow a class of derivation which can compute its own input hash
<andi->
that sounds scary
<bgamari>
yes
<bgamari>
indeed it is
<bgamari>
but in some application domains it's necessary
<bgamari>
I'm working with a dataset which takes literally CPU months to produce
<andi->
can't you decouple it more?
<gchristensen>
I'm not sure that is within the realm of "reasonable to do with nix" :/
<bgamari>
so I don't want my pipeline rerunning if I tweak a help message
<bgamari>
gchristensen, my point is that with a small escape hatch, this goes from being borderline impossible to very feasible
<bgamari>
it's not unreasonable for languages to allow escape hatches to their usual invariants if the user knows they get to keep the pieces when they break
<bgamari>
e.g. look at Haskell
<domenkozar>
+1
<simpson>
Couldn't we find some way to do lightweight builds of data?
<simpson>
So that they don't need to take up "full" Nix store entries?
<bgamari>
simpson, that would also be great
<bgamari>
but currently I just live with having to hash 20 GB outputs since the hashing time is quite small relative to the computation required
<simpson>
Basically, the Nix store is an alright content-addressable store, but it'd be really cool if we could do insertions into that store while we're doing insertions into that store.
<pie_>
im not sure if this is valid but i needed to figure out that QML2_IMPORT_PATH needed to be set to /nix/store/yrwp5m7xazzw1sfys19qhyr0926c46xz-qtdeclarative-5.9.2-bin/lib/qt-5.9/qml/QtQuick.2/ when i was using nix-shell -p qt5.full for a certain application because it would end up loading from the system 5.6.2 version and would fail
<pie_>
so maybe that should be added to the environment?
<gchristensen>
yowz jtojnar, xorg for basic systemd units?
phreedom has quit [Quit: No Ping reply in 180 seconds.]
phreedom has joined #nixos-dev
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 264 seconds]
la_putin has joined #nixos-dev
el_putin has quit [Read error: Connection reset by peer]
orivej has joined #nixos-dev
ma27 has joined #nixos-dev
orivej has quit [Ping timeout: 252 seconds]
orivej has joined #nixos-dev
goibhniu has joined #nixos-dev
phreedom has quit [Ping timeout: 248 seconds]
ma27 has quit [Ping timeout: 265 seconds]
el_putin has joined #nixos-dev
la_putin has quit [Read error: Connection reset by peer]
phreedom has joined #nixos-dev
ma27 has joined #nixos-dev
FRidh has joined #nixos-dev
<srhb>
nixos.org appears to be down
<aminechikhaoui>
yeah I think there was a reboot of the instance today
<aminechikhaoui>
probably niksnut or ikwildrpepper need to redeploy it
<aminechikhaoui>
(scheduled reboot from AWS)
<LnL>
cache bugs
<peti>
Or maybe the server was hit by an attacker exploting Meltdown and Spectre. :-)
<aminechikhaoui>
nah there was an AWS email about this :)
__Sander__ has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-dev
ma27 has quit [Ping timeout: 252 seconds]
orivej has quit [Ping timeout: 260 seconds]
michaelpj_ has joined #nixos-dev
orivej has joined #nixos-dev
ma27 has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 240 seconds]
<gchristensen>
for those not on Twitter: I’m looking for a way to work on public NixOS and Nix community projects full- (or almost full-) time. If your company can help me do something like this, or can sponsor me for some hours a month, lets talk :)
ma27 has quit [Ping timeout: 252 seconds]
<peti>
gchristensen: If you have a concrete project in mind, you might want to try crowd-fund it?
* peti
considered doing that a while ago, but then something else came up.
<peti>
Anyhow, the Typed-Nix project was financed that way quite easily. Shlevy has also that approach.
<domenkozar>
first crowdfunding we did was with shlevy and that worked
<domenkozar>
but it needs to be a very concrete goal
<domenkozar>
for on-going fixes I'd suggest patreon
goibhniu has quit [Remote host closed the connection]
goibhniu has joined #nixos-dev
<gchristensen>
that may be true, but I don't think patreon is the right thing for this. I'm not sure how to sort it exactly, but company-level-money, not user-level-money. like "sponsor a dev for 1 day/mo"
<gchristensen>
4.9.74 doesn't have the patches, domenkozar
<andi->
you need 4.9.75.. currently in review
<andi->
that will only be released after "Fri Jan 5 19:50:44 UTC 2018"
<andi->
that being said we can fetch the patches from the ML already..
<domenkozar>
one would imagine LTS kernels would have a patch ready for the announcement
<domenkozar>
isn't that the whole point? :)
<gchristensen>
they do, for the announcement planned next week :)
<gchristensen>
everybody collectively broke the embargo a week early
<domenkozar>
ah, I missed that
<domenkozar>
so we wait until tomorrow
<andi->
it might not be released tomorrow since they discovered (this morning 8am UTC+1) that there was a patch missing.. so lets see when the next "review round" ends m(
* __Sander__
really does not know what to think about Intel anymore
<__Sander__>
these 2 serious CPU bugs, CEO selling stock
<__Sander__>
the intel management engine with MINIX
<peti>
I bet that wasn't Insider trading at all. I mean, that would be, like, illegal.
<andi->
bugs happen in every system. If you are serious about the issue then support movements such as RISC-V and not just buy AMD ;-)
<gchristensen>
peti: I bet he was just trying to buy some bitcoin!
<domenkozar>
or ada
<gchristensen>
lots of ada
<peti>
gchristensen: Actually, he said he sold the stock "to pay for a new house or buy a rare piece of artwork".
<peti>
So, that explains it! :-)
<domenkozar>
seems like a good priority in these days
<domenkozar>
"our CPU has a major security flaw, let me buy some artwork first"
<gchristensen>
must be a hell of a piece of art
<domenkozar>
I can see a tagline: Intel CEO after failing to do science, retreats to art
<LnL>
lol
<domenkozar>
folks are reporting up to 50% slowdown in real applications
<domenkozar>
scale your clusters :)
<domenkozar>
this feels like onion
<gchristensen>
I'm pouring one out for all the DBAs who can't shard their dataset. I know that feeling.
<domenkozar>
Has moore's law come to an end? Intel CEO goes back to the roots
<domenkozar>
gchristensen: I feel bad for all those python folks with GIL
<andi->
nah they just re-invented moore's law. They can now scale that "feature" to provide performance gains again :D
<domenkozar>
jeff besos is clapping, overnight they sold 30% on AWS :D
<domenkozar>
bezos*
<domenkozar>
or will do.
<niksnut>
I wonder how it's possible that we've never noticed that for years NixOps has generated an invalid grub.cfg, preventing EC2 instances from rebooting
<kgz>
GSoC org applications open today
<kgz>
is anyone looking at applying again this year?
<gchristensen>
niksnut: I know I've brought something like that up before :/
<niksnut>
I wonder if something changed in grub error handling?
<aminechikhaoui>
niksnut: weird I've rebooted EC2 instances many times and don't remember issues
<niksnut>
yes
<niksnut>
something must have changed recently
<gchristensen>
I've seen this exact problem as far back as mid-2016
<niksnut>
yeah, I see that line on an instance that did reboot succesfully
<niksnut>
actually it looks like it eventually continues
<niksnut>
hah, this was probably caused by having another NixOS root disk mounted
<niksnut>
so /dev/disk/by-label/nixos was ambiguous
<gchristensen>
:$
<gchristensen>
this is annoying, the box I run the evaluator on is ext4 and runs out of inodes way before disk space (2% free inodes, 50% disk). I have to set nix-collect-garbage's --max-freed to like 400GB for it to appreciably clean up any drv files
<LnL>
hmm, what about running the eval with store=/tmp/eval-store on a tmpfs?
<LnL>
pretty sure the store uri stuff works for drv files
<gchristensen>
that could help
<gchristensen>
another option could be switching to something without a separate inode limit
<Sonarpulse>
bgamari: ghc 8.2.2 built!
orivej has joined #nixos-dev
yorick has joined #nixos-dev
<yorick>
peti: can you update the hackage packages?
<peti>
yorick: Yes. I regularly do that.
<yorick>
peti: I know, but can you do it today? :)
<yorick>
I wanna update and my arbtt isn't building
<yorick>
peti: ah, well, maybe you could put arbtt in that set? :)
<peti>
yorick: Well, I don't use arbtt, so I have very little incentive to spend my spare time fixing build errors it might have.
<yorick>
peti: it got fixed in hackage yesterday
ma27 has quit [Ping timeout: 240 seconds]
<peti>
yorick: I know. What I was trying to say is that just because a package is built by Hydra it doesn't that it will be fixed whenever it's broken for some reason. Most builds that fail simply get removed,
<yorick>
peti: well, I'm fine with you removing git-annex ;)
ma27 has joined #nixos-dev
__Sander__ has quit [Quit: Konversation terminated!]
<niksnut>
note: hydra builds are disabled pending a meltdown fix
<gchristensen>
are you waiting for it to hit nixos-unstable?
<niksnut>
I'm testing 4.9.75-rc1 atm
<gchristensen>
ah don't want to use _latest either
<gchristensen>
gotcha, I'm of two minds for the Packet builders, (1) you could "take possession" of them, allowing you to deploy to them like you do the rest, or (2) I could make them netbooted like I did for http://github.com/nix-community/aarch64-build-box and then there isn't a nixops network for it, and the config is very simple
<Dezgeg>
probably a Bad Idea to use the netboot until you figure out how to use the disk
<gchristensen>
perhaps so
<fpletz>
domenkozar: not interested right now but out of curiosity: is it hard to find people? :)
<niksnut>
domenkozar: be sure to send it to nix-devel as well
<domenkozar>
fpletz: hard question, not really. It's hard to find good ops
<domenkozar>
niksnut: will do!
<fpletz>
yeah, that is definitely hard, I mean to find people with good nix experience
orivej has quit [Quit: No Ping reply in 180 seconds.]
<domenkozar>
actually lots of people with nix
<fpletz>
cool :)
<domenkozar>
but we're running 31B$ currency, ops mentality is more desired.
orivej has joined #nixos-dev
<cransom>
i know at least one who would pounce on that but they aren't in an eu-friendly timezone.
<domenkozar>
non-eu is also fine
<domenkozar>
we're hiring 2, one eu, one us
<cransom>
oh handy, let me go yell at that person to get in line.
goibhniu has quit [Ping timeout: 248 seconds]
orivej has quit [Ping timeout: 255 seconds]
orivej has joined #nixos-dev
<bgamari>
Sonarpulse, yay!
<Sonarpulse>
bgamari: just need to change generic builder
<Sonarpulse>
so ghc.bootPkgs is not the setup.hs packages
<Sonarpulse>
because can't use prebuilt for that
<Sonarpulse>
bgamari: also, if we could somehow partition your big pr into mass-rebuild and not-mass-rebuild that would be cool
orivej has quit [Ping timeout: 255 seconds]
orivej has joined #nixos-dev
ma27 has quit [Ping timeout: 265 seconds]
pie_ has quit [Ping timeout: 256 seconds]
<Sonarpulse>
bgamari: I do need to backport some changes, however
<Sonarpulse>
not quite sure what to do with phabricator or whatever :D
<bgamari>
Sonarpulse, indeed, I think I discovered that ghc.bootPkgs change like night
<Sonarpulse>
bgamari: don't know that expression. people change it a lot?
<bgamari>
no no, I was working on bringing up cross-ghc in my own environment
<Sonarpulse>
bgamari: still confused :)
<bgamari>
sorry for the confusion
* bgamari
is being torn in about 10 different directions at the moment
<bgamari>
I was working on cross-compiling ghc myself last night
<bgamari>
working from your branch
<Sonarpulse>
bgamari: oh
<Sonarpulse>
bgamari: your sentence parses obviously now :)
<bgamari>
:)
<Sonarpulse>
actually didn't push the working thing yet
<Sonarpulse>
did a different bootPkgs change perhaps
<bgamari>
did you need ac_cv_c_bigendian=no as well?
<Sonarpulse>
bgamari: don't think so!
<Sonarpulse>
bgamari: you seem to need more of those than me :D
<bgamari>
hmm
<bgamari>
indeed
<bgamari>
very odd
<bgamari>
and slightly concerning
<Sonarpulse>
but I didn't actually try running program yet
<bgamari>
in my case ghc wouldn't even configure
<bgamari>
failing with
<bgamari>
checking whether byte ordering is bigendian... unknown
<bgamari>
configure: error: unknown endianness
<bgamari>
presetting ac_cv_c_bigendian=no (or yes) will help