samueldr changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS 19.09 is released! https://discourse.nixos.org/t/nixos-19-09-release/4306 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 19.09 RMs: disasm, sphalerite | https://logs.nix.samueldr.com/nixos-dev
drakonis_ has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 250 seconds]
drakonis_ has joined #nixos-dev
drakonis1 has joined #nixos-dev
drakonis has quit [Ping timeout: 240 seconds]
<clever> Ericson2314: what was left to do with the vc4 PR? i forget
<Ericson2314> clever: vc4?
<Ericson2314> oh videocode
<Ericson2314> checking
<{^_^}> #72657 (by cleverca22, 2 weeks ago, open): initial implementation of vc4 cross-compile
<clever> ah yeah
<Ericson2314> you can just do some ad-hoc thing to switch the default gcc
<Ericson2314> (gcc and gccFun are the two things, iirc)
<Ericson2314> we can generalize it later
<Ericson2314> clever: https://github.com/NixOS/nixpkgs/pull/72657#discussion_r341860643 is also nice, but as matt says not a PR blocker
<clever> 7602 gccFun = callPackage ../development/compilers/gcc/8;
<clever> 7603 gcc = gcc8;
<clever> Ericson2314: throw some vc4 if's in here?
<Ericson2314> clever: yup!
<clever> 7602 gccFun = callPackage (if stdenv.targetPlatform.isVc4 then ../development/compilers/gcc/6 else ../development/compilers/gcc/8);
<clever> 7603 gcc = if stdenv.targetPlatform.isVc4 then gcc6 else gcc8;
<clever> that good?
<Ericson2314> clever: you bet!
<clever> Ericson2314: pushed
<clever> (but not yet tested with gcc6 flags)
<clever> kicking off a build to see what it does...
<Ericson2314> clever: there is a conflict?
<clever> i'll rebase
<clever> Ericson2314: pushed
<Ericson2314> clever: ok!
worldofpeace1 is now known as worldofpeace
worldofpeace has quit [Quit: authenticating]
drakonis1 has quit [Quit: WeeChat 2.6]
<jtojnar> huh we have gnome_user_docs and gnome3.gnome-user-docs
worldofpeace has joined #nixos-dev
<worldofpeace> wait
<worldofpeace> ...
<worldofpeace> yeah that's a wtf
<worldofpeace> probably a good change to move it out to top-level 😁. and remove that attr
<jtojnar> a466941945bc0e55f940853dca3a3343ccc9c559
<worldofpeace> 🤣
<worldofpeace> wait, does this mean gnome3 was moved to top-level in the past, moved back, and now we're moving it again?
<worldofpeace> lol, and there's the end of underscores for dashes. I wonder what the separator of the future will be.
Scriptkiddi has quit [Remote host closed the connection]
das_j has quit [Remote host closed the connection]
Scriptkiddi has joined #nixos-dev
das_j has joined #nixos-dev
<clever> Ericson2314: it still compiles, and the simulator fails the same as before (a printf uses the switch opcode, which isnt supported)
<clever> Ericson2314: i'll test on real hardware in a bit...
<Ericson2314> clever: good!
<jtojnar> do we need autoreconfhook everywhere now?
<jtojnar> weird, we actually reconfigure zeitgeist
<worldofpeace> Jan Tojnar: I think its because its an archive from gitlab
justan0theruser has quit [Ping timeout: 246 seconds]
justanotheruser has joined #nixos-dev
cdepillabout has joined #nixos-dev
justanotheruser has quit [Ping timeout: 240 seconds]
justanotheruser has joined #nixos-dev
cdepillabout has quit [Quit: Leaving]
<{^_^}> firing: RootPartitionNoFreeSpace4HrsAway: https://status.nixos.org/prometheus/alerts
<{^_^}> resolved: RootPartitionNoFreeSpace4HrsAway: https://status.nixos.org/prometheus/alerts
<eyJhb> Woooo, no fun
FRidh has joined #nixos-dev
Jackneill has joined #nixos-dev
FRidh has quit [Quit: Konversation terminated!]
drakonis_ has quit [Ping timeout: 250 seconds]
drakonis_ has joined #nixos-dev
orivej has quit [Ping timeout: 265 seconds]
FRidh has joined #nixos-dev
__monty__ has joined #nixos-dev
<jtojnar> worldofpeace the CVS tarballs should be clean of the generated files so they should just be copied from gettext package
<jtojnar> aargh, missing in configure.ac and m4 just eats everything until the next command, resulting in helpful “./configure: line 15039: syntax error near unexpected token `;;'”
<gchristensen> ehh I think I'll change that alert to not use predict_linear
psyanticy has joined #nixos-dev
drakonis_ has quit [Read error: Connection reset by peer]
drakonis_ has joined #nixos-dev
phreedom_ has quit [Ping timeout: 260 seconds]
dtz has quit [Ping timeout: 250 seconds]
zarel has joined #nixos-dev
roberth has quit [Ping timeout: 250 seconds]
zarel has quit [Quit: Leaving]
phreedom has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.6]
drakonis has joined #nixos-dev
drakonis1 has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 245 seconds]
drakonis1 has quit [Ping timeout: 252 seconds]
drakonis has quit [Ping timeout: 276 seconds]
drakonis has joined #nixos-dev
ixxie has joined #nixos-dev
orivej has joined #nixos-dev
<noonien> is there a way to get an unwrapped gcc and buildtools for cross-compilation?
<noonien> i'm trying to compile a kernel module for another machine and i'm getting invalid modules compiled, my guess is it's happing because of the wrapping
orivej has quit [Ping timeout: 265 seconds]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 240 seconds]
<infinisil> The wrapper is what makes cross comp possible! Pretty sure anyways
<thoughtpolice> It really depends on what you're trying to accomplish. There are various cross compiler things in the package infrastructure but in other cases I've just written my own package to cross-compile GCC and treat it like any other normal package
<thoughtpolice> (This only works in bare metal-ish environments though, or if you're willing to write your own stuff to tie in a libc)
<noonien> well, i used to build this module in a docker container since i didn't know nixpkgs had gcc49 (i'm building for an older kernel)
<noonien> i've setup a nix-shell with the following: http://ix.io/22mt
<noonien> the project builds successfully, however, when i try to insert the module, i get these errors: http://ix.io/22mu
<{^_^}> ikwzm/FPGA-SoC-Linux#1 (by FPtje, 2 years ago, closed): Unknown symbol _GLOBAL_OFFSET_TABLE_
<samueldr> noonien: ^ might be relevant
<noonien> oh man, i tried to pass -fno-pic myself
<noonien> i'm guessing the wrappping removed it
<noonien> samueldr: thanks for the link!
<samueldr> good show of errors
<samueldr> I searched "Unknown symbol _GLOBAL_OFFSET_TABLE_" to understand what it means...
<samueldr> ... still don't but saw that nixos-specific issue in the results :)
<noonien> i, did so as well, i couldn't see anything related to nix in my search results ^_^
<noonien> thanks!
<samueldr> guess my filter bubble is heavily skewed :)
<noonien> yeah,, i'm also not using google anymore, ddg can be pretty bad for these kinda of searches sometimes
<noonien> now i just have to figure how to pass `hardeningDisable = [ "pic" ];` to `buildPackages.gcc49`
<{^_^}> firing: HomepageUpdateStuck: https://status.nixos.org/prometheus/alerts
FRidh has quit [Quit: Konversation terminated!]
<samueldr> I think that would be for your module
<noonien> i just have a shell.nix file atm: http://ix.io/22mt
<noonien> i'm going to turn it into a module once everything works
<samueldr> put it in your mkShell
<noonien> i tried, it doesn't work
<samueldr> not sure, though
<samueldr> ok
<noonien> i'm guessing because i'm using pkgs.mkShell
<samueldr> shouldn't matter much
<noonien> i only use crossPkgs for buildPackages.gcc49
<samueldr> it's more likely to be because nix-shell ends up using the result and not the build environment
<gchristensen> ^ that "UPdateStuck" error: curl --fail -L https://raw.github.com/NixOS/nixpkgs/master/nixos/modules/virtualisation/ec2-amis.nix failed after curl: (22) The requested URL returned error: 503 first byte timeout
<samueldr> let's hope it's not a flappy one
<samueldr> well
<samueldr> too flappy
<samueldr> it is likely flappy
<gchristensen> probably should only alert if it fails for 1h in a row
* samueldr thinks our hydra might have too much data
<gchristensen> oh?
<samueldr> searches using the top bar search are quite long
<gchristensen> heh
Jackneill has quit [Remote host closed the connection]
<{^_^}> resolved: HomepageUpdateStuck: https://status.nixos.org/prometheus/alerts
orivej has joined #nixos-dev
<noonien> WOHOO it worked!
<noonien> i just had to make a stdenv.mkDerivation instead of mkShell as you pointed out
<{^_^}> firing: RootPartitionNoFreeSpace4HrsAway: https://status.nixos.org/prometheus/alerts
<tilpner> ._.
<gchristensen> perfect timing, was just SSHing in to update that alert
<LnL> oh, neat!
<tilpner> gchristensen: How much free space does the root partition have?
<tilpner> Wait, this should be public information
<gchristensen> it is :)
<LnL> all the prometheus data is accessible
<LnL> fyi, you can use e notation for large numbers
<gchristensen> I decided on 10g for that alert based on https://status.nixos.org/prometheus/graph?g0.range_input=4w&g0.expr=node_filesystem_avail_bytes%7Bmountpoint%3D%22%2F%22%7D%20%3C%3D%2010000000000&g0.tab=0 each of these cases being scenarios where an alert would have been useful
drakonis_ has joined #nixos-dev
<tilpner> If it hits 10GB free space, it can then continue for 30min at the fairly fill rate of 5.5MB/s, to eat those remaining 10GB in 10min
<tilpner> *30min
<gchristensen> sure
<tilpner> That's not a lot of time to react, but I don't have a better proposal
<gchristensen> if something breaks it breaks
<gchristensen> the idea is to reduce "huh, I wonder why X hasn't happened in a week ... " "oh lol the disk ran out of space last thurday"
<LnL> that's why linear_predict is nice for this case
<{^_^}> resolved: RootPartitionNoFreeSpace4HrsAway: https://status.nixos.org/prometheus/alerts
<gchristensen> please do send PRs :P
<LnL> predict + for that is
psyanticy has quit [Quit: Connection closed for inactivity]
drakonis_ has quit [Ping timeout: 265 seconds]
ixxie has quit [Ping timeout: 252 seconds]
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
<gchristensen> I thought builds werec anceled after 2d
Jackneill has joined #nixos-dev
<gchristensen> oh cool it is a legit bug
Jackneill has quit [Remote host closed the connection]
<thoughtpolice> gchristensen: ping
<gchristensen> pong
phreedom has quit [Ping timeout: 260 seconds]
phreedom has joined #nixos-dev
<{^_^}> resolved: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
drakonis_ has joined #nixos-dev
drakonis_ has quit [Quit: Leaving]
<gchristensen> ^ is a bug of several builds deadlocking on who receives what inputs
drakonis has quit [Ping timeout: 276 seconds]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 245 seconds]
drakonis has joined #nixos-dev
drakonis_ has joined #nixos-dev
__monty__ has quit [Quit: leaving]
<domenkozar[m]> I wonder if anyone is even using socket activation
<domenkozar[m]> this is completely backwards how it should be
<domenkozar[m]> if service changed, why is socket restarted?
<domenkozar[m]> the whole point is that it should be up :D
<clever> domenkozar[m]: ive seen problems with a .socket based dns service before, where bind ip's where added, but nixos didnt restart the .socket
<clever> so it just didnt listen on the new IP's
<domenkozar[m]> yeah that's a different bug
<domenkozar[m]> there's even a PR for that one
<domenkozar[m]> so in my case, unit changes and socket gets restarted
<domenkozar[m]> in your case, socket changes and nothing happens
<thoughtpolice> Folks, if you ever want to Learn A Lot about how cache.nixos.org will soon work, I wrote up a whole set of guides and several design notes here: https://www.notion.so/aseipp/The-NixOS-Binary-Cache-db97de6ab19b4d7ab9f4a60cb4cdaaf0
<clever> thoughtpolice: major changes coming up?
<thoughtpolice> Yes, and I've enhanced it quite a bit more since the original posting ~3 months ago
<clever> thoughtpolice: what kind of major changes?
<thoughtpolice> The TL;DR is "much faster globally and with much lower latency". One of the changes there is actually already deployed and reduced TTFB significantly for a HKG user (something like ~1.2 seconds to ~250ms)
<thoughtpolice> There are also many more internal changes that will help us monitor problems and troubleshoot issues.
<clever> thoughtpolice: ah, some of that ive tried to mitigate with cachecache
<clever> but streaming hits is something i still need to work on
<thoughtpolice> The second major latency reducing measure (streaming miss) isn't yet live, but that's a major one
<clever> currently, cachecache will read the entire reply from upstream, write it to disk, then serve it to the client
<clever> but i need to look into conduits more, and stream it better
<clever> another thing, is being able to fork a conduit, mid-stream
<clever> thoughtpolice: so overall, the api is the same, but performance changes to make it respond faster?
<thoughtpolice> Yes, nothing user facing is changing, other than "It's a lot better, now"
<clever> i think the only thing that the client needs to change for, is the http2 nar push stuff
<thoughtpolice> Yes, exactly, that's something I'm experimenting with now
<thoughtpolice> But it's not a high priority
<thoughtpolice> And it will be very selectively enabled for compatible clients
<thoughtpolice> But other things are just "fee". For instance, a lot of times `nix ...` seems to "hang" on downloads before they start, which seems to take a while sometimes. You can also see that with `nix-build` with the normal cURL-style bars that don't start until a few seconds after beginning. That's because right now the cache fetches everything from S3 first before sending a byte. That will be fixed.
<thoughtpolice> s/fee/free
<clever> cachecache has the same problem
<thoughtpolice> Unrelated but similar to cachecache it seems, I also added a proxying mode to Eris recently that allows it to serve (and even sign!) objects from upstream caches, meaning one instance can serve both cache.nixos.org objects, and private objects.
<thoughtpolice> Pretty nifty, but I didn't yet work in the ability to actually keep the proxied objects (that aren't local) on disk yet, which would basically be the same feature I suppose.
<clever> i'm just keeping signatures intact as i proxy, but i do merge multiple caches together
<thoughtpolice> (Well, it doesn't cache the literal .narinfo, it regenerates it)
<clever> and i cache every narinfo i see in a single hashmap in ram
<clever> narinfo misses are cached for a short period of time
<clever> line 362 downloads the entire nar at once
<thoughtpolice> Yeah, I do nothing that fancy, but in practice I expect most users of Eris aren't that latency sensitive for low-volume stuff.
<thoughtpolice> And if they are they'll just set up some other HTTP proxy, like Varnish itself or whatever
<clever> 316 writes that entire nar to local disk
<thoughtpolice> I did get Mojolicious's streaming support to work, however. I think.
<clever> then 318 returns it towards the client
<thoughtpolice> (The main thing I need for Eris now is some kind of ACME integration... So utterly tedious but so useful)
<thoughtpolice> Streaming is hard
<clever> ive been thinking about it for months
<clever> the problem, is that i want to start 1 http download
<clever> then stream it to disk
<clever> and stream it to the user
<clever> but, the hard part
<clever> i want a 2nd user to come along, at any point in time, read the partial file on then, then fork the stream in ram, and join in
<clever> on disk*
<thoughtpolice> Yes, we have this feature at work in our platform, and in practice it's extremely useful when you need it, since in basically allows wide fanout at no cost.
<thoughtpolice> "Request collapsing"
<clever> yeah
<thoughtpolice> Hopefully those issues will be a thing of the past soon for cache.nixos.org at least :)
<clever> thoughtpolice: are there multiple instances of cache.nixos.org spread around the world?
<thoughtpolice> No, there's only one. The storage costs would be astronomical with S3, I'm afraid.
<thoughtpolice> In my own personal experiments, I did play with a geo-redundant cache, however
<clever> i meant the cache layer, after the s3 bucket
<thoughtpolice> Oh, yeah, absolutely
<thoughtpolice> Read all the Fastly docs in that page I linked, I detail it extensively
<thoughtpolice> (I work @ Fastly, to be clear)
<clever> ahhh
<clever> its harder to reverse engineer that from the outside, when cloudflare is masking things
<thoughtpolice> You can basically find all the information in that document publicly, if you watch recordings of our customer conferences and read things carefully.
<thoughtpolice> There's nearly nothing there that hasn't been written elsewhere.
<clever> something ive thought about, is how much advantage a custom program that knows nix can give you, over a dumb squid instance
<clever> oh, and there is one problem that would need to be fixed in nix itself, it refuses to download X from the cache, if X depends on Y, and it doesnt have Y yet
<thoughtpolice> Yes, I do leverage some aspects of Nix to enhance the server configuration, since we essentially use Varnish under the hood. You could probably do more though with some improved cooperation.
<clever> thoughtpolice: you could maybe get much better performance (parallism), if you download both X&Y, but dont try to unpack X's nar, until Y has finished
tv has quit [Ping timeout: 240 seconds]
<thoughtpolice> Yeah, that's one way, I've noticed some things like that. Another I was thinking of is whether we could use TLS 1.3 0RTT to reduce initial handshake times once the user has talked to the cache once. 0RTT has some flaws but for us I don't think they (mainly replay attacks) apply very much for strict GET requests.
<thoughtpolice> There are probably several optimizations we can make if we can enhance Nix I bet, but I've only thought of a few. Server Push is the most obvious one.
<clever> ive also found performance issues in nix/hydra-queue-runner, just in terms of how many narinfo/sec they can request
* clever gets image
<thoughtpolice> HTTP/2 multiplexing should help that today a ton though. Though apparently HTTP/2 is currently sometimes problematic for some users...
<thoughtpolice> It's not clear to me how much of that is cURL bugs vs Nix bugs.
tv has joined #nixos-dev
<thoughtpolice> My current logging states indicate HTTP/2 is only about 20% of the traffic hitting my beta service, even though it's been available in Nix for quite a while at this point by default
<thoughtpolice> (I think, at least)
<clever> thoughtpolice: you can see here, that hydra is barely getting even 1 request/second, when hitting a cachecache instance on localhost
<clever> and nearly all of those are cache hits, in a haskell hashmap
<thoughtpolice> Oh, the *queue runner*. Yeah wow that's insanely slow.
<thoughtpolice> You should literally be able to request like thousands of narinfos a second with HTTP/2 multiplexing from the upstream cache lol (or at least, my version)
<clever> i suspect the algo in nix itself though, isnt capable of that
<clever> it starts at the root of the dep graph, and works downwards thru the tree
<clever> so the number of pending queries will grow as it fans out thru the dep tree
<clever> if its even able to queue them up
<clever> $ nix-build channel:nixos-unstable -A hello --dry-run
<clever> these paths will be fetched (0.04 MiB download, 0.20 MiB unpacked): /nix/store/n214akq42as1ckzj605c3s6y42cxby02-hello-2.10
<clever> lol, ok, bad choice in test
<clever> thoughtpolice: hmmm, i think part of it is grafana and 1m averages
<clever> it just requested ~50 narinfo files, in a very short time
<clever> but because of rate(foo[1m]), it renders as 1/sec