ekleog changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html https://r13y.com | 18.09 release managers: vcunat and samueldr | https://logs.nix.samueldr.com/nixos-dev
<worldofpeace> jtojnar: Is the problem with hydra that much a problem currently? I get the point on "it's just for install" but exposure is exposure.
<aszlig> clever, noonien: this patch should fix the issue: https://gist.github.com/aszlig/f303bc1d860a649ad4442598a37a9d75
<aszlig> but i'm going to submit a better patch upstream
<jtojnar> worldofpeace: I have no idea, honestly
<worldofpeace> jtojnar: I think I'll go for it and see if there's resistence.
<jtojnar> 👍
<gchristensen> what is the problem?
<clever> aszlig: nixpkgs master also works, but thats also 4.14!, let me try throwing in boot.kernelPackages = pkgs.linuxPackages_latest
<aszlig> clever: ... plus boot.kernelPatches ;-)
<clever> [clever@amd-nixos:~/apps/nixpkgs]$ nix-build nixos/ -A vm --arg configuration '{ pkgs, ... }: { users.users.root.initialPassword = "root"; boot.kernelPackages = pkgs.linuxPackages_latest; }'
<clever> aszlig: this is enough to break it on nixpkgs master
<aszlig> clever: i know :-/
<clever> [clever@amd-nixos:~/apps/nixpkgs]$ nix-build nixos/ -A vm --arg configuration '{ pkgs, ... }: { users.users.root.initialPassword = "root"; boot.kernelPackages = pkgs.linuxPackages_latest; boot.kernelPatches = [ { name = "foo"; patch = ./overlayfs.patch; } ]; }'
<worldofpeace> gchristensen: Was that question directed to me?
<gchristensen> yeah, worldofpeace
<worldofpeace> Well the problem wasn't mine but others that people had in the past. Not sure if I will get the same response as there was in https://github.com/NixOS/nixpkgs/pull/10288#issuecomment-146539163
<worldofpeace> IIRC opensuse can build whatever ISO they want under the sun. And everyone has these "spins". Now we don't need those because we're cool. But exposing things in a familar way to users is important.
<gchristensen> ah
<gchristensen> I think the concern is the ISOs are fairly large, get rebuilt quite often, and get stored essentially forever
<worldofpeace> Well I think Opensuse can remove them, and produce iso's if only a subset of tests fail.
<gchristensen> right
<worldofpeace> But OpenQA has nice suite for GNOME
<worldofpeace> Only building it for stable isn't option IMHO because people would just expect there to be an unstable iso
<worldofpeace> But can we stop storing install ISO's forever?
<gchristensen> they would? we don't publish unstable ISOs as it is
<worldofpeace> Define publish. I can download an unstable iso from a link on the homepage.
<gchristensen> you can?
<worldofpeace> Unstable releases
<worldofpeace> * Download Page
<gchristensen> oh fascinating
<gchristensen> I don't think I've ever looked that far down the page
<worldofpeace> Some people would be intersted in using unstable exclusively because it's the latest.
<worldofpeace> and GNOME is well supported there.
<gchristensen> (dreaming time) I've long wanted a nixos-image-generator service where I can flip some switches and get a custom iso
<samueldr> didn't you have one at one point? ah no, it was a "get a shell for a small amount of time" one
<worldofpeace> gchristensen: That was a thought i had in the shower yesterday suddenly 🤣
<gchristensen> yeah, mine gave you a running server with your config
* samueldr thinks nix spoils us
* gchristensen too
<worldofpeace> * pretty much
<worldofpeace> Though I'm not fond of installer pr's after https://github.com/NixOS/nixpkgs/pull/42610 but that was pretty fair
<{^_^}> #42610 (by adisbladis, 37 weeks ago, closed): installer: Don't run graphical installer ISOs as root
yl has quit [Ping timeout: 245 seconds]
<clever> aszlig: of note, not even cat can read the file, execute doesnt have much to do with it
<aszlig> clever: yeah, that's what i noted in my initial mail a month ago
<gchristensen> ooh nice catch, clever
<gchristensen> oh :D
<aszlig> anyway, reply posted to linux-unionfs
<gchristensen> "Universal dilettante" :')
<gchristensen> I'm not sure I'd have described you as a dilettante, hah
<aszlig> my father once called me like that, so i kept it :-D
<aszlig> anyway, i think we can just remove the O_NOATIME flag, especially because that's the pre-4.19-behaviour, but let's see whether there is a compelling reason to use it there
<clever> aszlig: what about a qemu flag, to make 9plan strip O_NOATIME?
<clever> overlayfs with an ext4 lowerdir, would benefit from O_NOATIME being left in
<aszlig> clever: that would be a workaround, i just tested NFS before and it has the same problem
<aszlig> so it clearly is a kernel regression
<clever> it only fails with a 9plan lower, and qemu lacking ownership
<clever> ah, nfs would also need the same work-around...
<clever> maybe a mount flag for overlayfs then?
<clever> or a blacklist, dont try to do O_NOATIME when the lowerdir is nfs or 9plan?
<aszlig> clever: ... or AFS, or GFS, or ...
<aszlig> but IIRC there is also a function to check whether it's a networked filesystem... so that *could* be an option
<aszlig> but i don't think they'd want to go into that direction, especially because there is work to have overlayfs to be mounted as user
<aszlig> so the same problem will exist for local file systems
<clever> depends on if overlayfs will respect the user perms or not, a user mounting overlayfs could be an attack to hide atime updates as you upload all the secrets on a box you owned :P
<aszlig> although the latter can be solved by inode_owner_or_capable()
<clever> an attacker could overlayfs / and a tmpfs together, to hide all atime updates as they scan your box
<aszlig> clever: it will and it already does
<aszlig> the superblock is already using the struct cred of the current user
<clever> ah
<aszlig> so i guess they're already preparing for that
<aszlig> ah, i was wrong, it was FUSE that was preparing to be mounted as user
<aszlig> regarding overlayfs: https://lwn.net/Articles/671641/
<jtojnar> worldofpeace: by the way, I think it better to not include bugfix PRs into the gnome super-PR
<jtojnar> I did that last time and it makes finding discussion for those fixes impossible
<jtojnar> speaking mostly about complex things like https://github.com/NixOS/nixpkgs/pull/54584, simple updates like https://github.com/NixOS/nixpkgs/pull/57392 are fine by me
<{^_^}> #54584 (by worldofpeace, 6 weeks ago, closed): gnome3.gnome-settings-daemon: bypass polkit by using an suid wrapper
<{^_^}> #57392 (by dtzWill, 2 days ago, closed): geary: 0.13.2 -> 0.13.3
<clever> aszlig: ive also had trouble with user namespacing, i must create a user namespace to make a mount namespace, and then i lack access to a lot of /proc/ so nix-collect-garbage (in the namespace) fails
<clever> aszlig: so i'm forced to consider adding a pid namespace!
<aszlig> clever: or bind-mount proc
<clever> aszlig: i already bind-mount /proc
<clever> but the user namespace means i lack access to the /proc of any pid outside my namespace
<clever> the uid 1000 in the namespace isnt the real uid 1000 i think
<worldofpeace> jtojnar: You know I did think about that as possibly being a problem. Bugfixes in update pr's are probably not great. But I thought we agreed in a way.
<aszlig> clever: what's in uid_map within that process?
<aszlig> but yeah, depending on the mapping you can't access proc
<jtojnar> worldofpeace: yeah, I noticed the issue when we started discussing the PR there; and remembered the times scrolling the 3.20 issue and clicking load more comments repeatedly
<aszlig> s/proc/some files in &/
<clever> [clever@amd-nixos:~] (namespaced) $ cat /proc/self/uid_map 1000 1000 1
<clever> word-wrap messed up
<clever> ls: cannot read symbolic link '/proc/9354/fd/0': Permission denied
<clever> lrwx------ 1 clever users 64 Mar 14 01:55 /proc/9354/fd/0
<clever> aszlig: i own this symlink, yet i cant readlink it
<clever> because that pid exists in the parent namespace
<aszlig> clever: what do you have in uid_map?
<clever> 1000 1000 1
<worldofpeace> jtojnar: Yeah lets not torture our hands with that. I can reopen the other pr and just merge it right after 3.32 hits?
<clever> aszlig: and 1000 is clever
<aszlig> clever: well, this should actually work, but lemme check...
<jtojnar> worldofpeace: yeah, it is non-critical
<clever> [clever@amd-nixos:~] (namespaced) $ nix-store --query --roots /nix/store/ydna2d28ziaf3yim04bc9l2nnb7bymkd-util-linux-2.31.1-bin
<clever> error: opening file '/proc/325/maps': Permission denied
<clever> aszlig: pid 325 is a chrome proccess on the host
<worldofpeace> jtojnar: right, moving it out 👍
<jtojnar> worldofpeace: I am starting to thing, as much as I hate merge commits, that our rebases make the github history even less legible, we need better tooling
orivej has quit [Ping timeout: 245 seconds]
<worldofpeace> jtojnar: Define legible? git history for package repo's tend to take certain form.
<jtojnar> I mean mostly the comments
<worldofpeace> Yeah , in progress the discussion of what happened to get x change is just completely gone.
<clever> [clever@amd-nixos:~/apps/nixpkgs]$ nix-build nixos/ -A vm --arg configuration '{ pkgs, ... }: { users.users.root.initialPassword = "root"; boot.kernelPackages = pkgs.linuxPackages_latest; boot.kernelPatches = [ { name = "foo"; patch = ./overlayfs.patch; } ]; }'
<clever> aszlig: i can confirm this fixes the problem
<clever> noonien: ^^
<aszlig> clever: (takes a while, trying this in my own sandbox implementation, where i remember that it worked without pid namespaces)
<clever> aszlig: i'm also warry about adding pid namespacing, because then i may have different pid namespaces on the same fake root, and then nix-store --query --root cant see the other pid namespaces
<clever> and it could delete in-use things
<aszlig> clever: "$(nix-build --no-out-link -I nixpkgs=channel:nixos-18.03 -I vuizvui=https://github.com/openlab-aux/vuizvui/archive/9490c5a93a6738b6df503fdf67cd4a5e2ec9c95e.tar.gz -E 'with import <nixpkgs> {}; (import <vuizvui/pkgs> {}).buildSandbox (writeScriptBin "foo" "#!${stdenv.shell}\nexec ${pkgs.coreutils}/bin/cat /proc/$$/maps") {}')/bin/foo"
<aszlig> 9490c5a93a6738b6df503fdf67cd4a5e2ec9c95e because that version doesn't yet use PID namespacing
<clever> /nix/store/920k63py2w97xpfyb5ps9l6wbidvzfjz-coreutils-8.29/bin/cat: /proc/325/maps: Permission denied
<clever> aszlig: $$ is causing the containerized bash pid to be inserted
<aszlig> yep
<clever> aszlig: if i replace it with the pid of something outside the namespace, it fails the same way nix-user-chroot does
<aszlig> clever: aaah...
<aszlig> i thought you meant within the namespace
<aszlig> sorry
<clever> but the uid map claims i still own 325
<aszlig> hm, so why not using a control socket or pipe that handles GC from outside the namespaces?
<clever> aszlig: the container has its own /nix/store
<clever> aszlig: the goal is to have fully working nix, with binary cache support, when i lack write access to /
<aszlig> okay, and you obviously don't want paths to be GCed when they're used from outside the container
<clever> aszlig: nothing outside the container should be using those paths, but there is the tricky issue is using the same pid namespace if i enter it twice
<clever> (if i add pid namespacing)
<aszlig> clever: hm, what about setns?
<clever> its more about finding out of an existing namespace is there, or if one should be made, when entering
<aszlig> clever: and you want to avoid pidfiles and stuff like that, right?
<clever> pid files and such would be fine, though are complicated by pid namespaces, and nesting containers
<aszlig> well, IIUC you just need to make sure it's not written from within the pid namespace
<clever> but if i re-run the enter tool, from inside the namespace, it will fail to find the pid in the file
<clever> and then try to create a namespace, inside the namespace
<aszlig> ah
<clever> but, i have also had EPERM failures, if i try to make a namespace, inside the namespace
<clever> originally, i solved that by detecting my own install util, and unpacking it
<clever> but then i ran into xdg-open problems (steam also has this issue)
<clever> so i added an escape-hatch fifo
<clever> if you print any string to /escape-hatch, the util that created the namespace will just execute that string, outside the namespace!
drakonis_ has joined #nixos-dev
<aszlig> mhm, i saw that earlier
<clever> and if i click a link in steam on nixos:
<clever> [clever@amd-nixos:~]$ /usr/bin/xdg-open: line 810: : command not found
<clever> /usr/bin/xdg-open: line 880: w3m: command not found
<clever> it cant find any browser inside the sandbox, and fails to open the link!
<clever> the steam sandbox could use some escape-hatches
drakonis has quit [Ping timeout: 240 seconds]
* clever heads off to bed
drakonis has joined #nixos-dev
<matthewbauer[m]> clever: how do other containers handle this? the escape hatch thing always seemed like a bad hack to me but I could never think of a better solution
<aszlig> speaking of sandboxing, just updated the description of #57519
<{^_^}> https://github.com/NixOS/nixpkgs/pull/57519 (by aszlig, 1 day ago, open): nixos: Add 'chroot' options to systemd.services
jtojnar has quit [Remote host closed the connection]
<aszlig> matthewbauer[m]: the implementations i know so far are using their own api to pass specific hooks to the outside process
<aszlig> IIRC snapd does it like that
<aszlig> clever: gn8 =)
ajs124 has left #nixos-dev ["Machine going to sleep"]
<matthewbauer[m]> do they document this?
<matthewbauer[m]> i don't think appimage has it though
drakonis has quit [Quit: WeeChat 2.3]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 245 seconds]
disasm has quit [Ping timeout: 246 seconds]
disasm has joined #nixos-dev
johanot has joined #nixos-dev
edef has quit [Read error: error:1408F10B:SSL routines:ssl3_get_record:wrong version number]
edef has joined #nixos-dev
<aszlig> so in summary: they wrap file open dialogs, xdg-open and others and pass it through to snapctl (https://github.com/snapcore/snapd/blob/38e8503ef2e10bedf2372143e88a7da9022e2e1e/cmd/snapctl/main.go#L50) where it prompts the user whether to allow this action
drakonis_ has quit [Remote host closed the connection]
orivej has joined #nixos-dev
pie__ has quit [Remote host closed the connection]
pie__ has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
init_6 has joined #nixos-dev
orivej has joined #nixos-dev
jtojnar has joined #nixos-dev
<aszlig> samueldr: given that we got rid of the overlayfs issue (well, patch still needs to be applied, ideally upstream), do you think we can get it in 19.03?
<aszlig> s/\<it\>/kernel 4.19/
<samueldr> I was lightly thinking about it yesterday evening, and I don't know for sure
<samueldr> we're about halfway in the stabilisation period, so hmm, it'd be annoying if it brought issues
<samueldr> but on the other hand, it does bring issues to default to an older kernel (for newer hardware mostly)
<samueldr> thinking about the installation media
<aszlig> samueldr: well, halfway is better than at the end of that period :-)
<samueldr> definitely
<aszlig> i could however add that patch to master already... so far there hasn't been opposition on the upstream part
<aszlig> and the patch basically gets to the same behaviour as pre-4.19
<samueldr> make the PR with the swtich to LTS so the test suite runs on it, maybe?
<aszlig> omw...
nbp has joined #nixos-dev
<aszlig> samueldr: first just the overlayfs patch for master
<aszlig> samueldr: i'll add a follow-up PR with the cherry-pick of that plus the LTS
<nbp> I think Nix should remove the read-flag of the /nix/store directory, such that we can really claim to have a security by obfuscation model.
<nbp> I think you only need the 'x' flag on a directory to traverse it.
<aszlig> nbp: this was the case for a short period of time already, but got reverted
<aszlig> although it did have read permissions for "others"
<nbp> hum … are our test well written then ?!
<aszlig> nbp: i have no idea what was *exactly* broken as the commit doesn't say so, but I'd imagine it's one of the path-verification subtests
<ekleog> nbp: FWIW your file system probably allows enumeration of paths in a non-readable directory via side-channels
<aszlig> nbp, ekleog: not only that, but you can also enumerate them via /nix/var/nix/db/db.sqlite
<nbp> ekleog: sure, if you are root.
<nbp> aszlig: which is only supposed to be opened by the nix users / nix deamon?
<aszlig> supposed to be opened, but right now it's world-readable
<ekleog> nbp: not only if you are root, I think the -r flag on directories can likely be bypassed via side-channels -- without experimenting precisely with it, though
drakonis has joined #nixos-dev
<aszlig> i do however see an advantage in that, but not from a security perspective
<aszlig> there are a bunch of applications out there that readdir every path component and they become horribly slow when you have a large store
<{^_^}> #57641 (by aszlig, 12 seconds ago, open): linuxPackages: 4.14 -> 4.19
init_6 has quit [Ping timeout: 245 seconds]
<aszlig> anyone here knows of a special shell that *only* allows the "/bin/sh -c something" usage?
<Profpatsch> aszlig: Isn’t that a small wrapper around sh?
<aszlig> Profpatsch: ?
<Profpatsch> Making that special shell
<aszlig> something which _ideally_ just delimits the "something" by space/tab and _nothing_ more
<aszlig> Profpatsch: well, i would like to avoid using a real shell
<aszlig> well, okay, maybe a _bit_ more
<aszlig> like "exec foo"
<aszlig> so covering only the common use cases for that
<aszlig> Profpatsch: haha, i was expecting something like that from you :-)
<Profpatsch> Both tools exec into the rest of their args after they transform their CLI :)
<Profpatsch> Of course, staying true to my brand :P
<aszlig> however, this is not doing what i'm searching for because the -c arg is a _single_ argument
<aszlig> plus it doesn't do any (minimal) parsing
<aszlig> for context, this is for https://github.com/NixOS/nixpkgs/pull/57519
<{^_^}> #57519 (by aszlig, 2 days ago, open): nixos: Add 'chroot' options to systemd.services
<clever> ooo, that sounds fun
<clever> aszlig: dont forget about /usr/bin/env
<aszlig> clever: i'd rather avoid that, because that's easy to patch
<clever> yeah, env can just be patched out in things
<clever> but /bin/sh is a requirement of libc! cant patch that out
<aszlig> clever: the /bin/sh case is much harder
<aszlig> i do agree with niksnut there that maybe it's better to use bash as /bin/sh by default to not confuse users
<aszlig> but i also want to have an *option* to make it more restrictive
<ekleog> aszlig: `assert(argc > ?); argv[2] = NULL; execve(argv[1], argv, envp);` (with ? a number I can never remember because I never remember whether there's an off-by-1 in argc)
<aszlig> aszlig: ah, what's the context?
<clever> if a program is ran with zero arguments, then argc == 1, because argv[0] is the programs own name
<clever> argv will always be one higher then the last valid slot in argv
<aszlig> lol
<aszlig> <aszlig> aszlig: ... yah! talking to myself, retry:
<aszlig> ekleog: ah, what's the context?
<ekleog> aszlig: your question of something that only allows `sh -c something` with `something` that should be something minimal
<ekleog> unless I misunderstood
<aszlig> ekleog: ah, not the "something" should be something minimal, but /bin/sh itself
<aszlig> it shouldn't be a real shell
<clever> which would break #!/bin/sh and force people to patch things even more!
<aszlig> and also exit with failure if it's not used with -c foo
<aszlig> clever: that's the goal
<clever> it could be a muslc binary that only supports -c
<clever> i remember a very weird bug i helped someone diagnose, many years ago
<clever> they had the binary cache disabled, and glibc failed to build
<clever> because when building glibc, no output of glibc is allowed into the nix sandbox
<clever> and then /bin/sh cant find glibc
<aszlig> hm... thinking about that, i could probably defer this...
<aszlig> like eg. confinement.binSh = null; <- for no /bin/sh
<ekleog> aszlig: hm guess I misunderstood the question then
<clever> if sh was a static binary, the above issue cant happen
<aszlig> or binSh = someshell
<aszlig> so the type is nullOr path
ajs124 has joined #nixos-dev
<aszlig> clever: hm, i also think that your point with /usr/bin/env is probably a good idea as well, at least by default
<aszlig> because even though i *personally* would want to not provide it, unconfined services do have access to it
jtojnar has quit [Quit: jtojnar]
johanot has quit [Quit: WeeChat 2.4]
<gchristensen> ekleog, can you send me some dates and times you're available to talk about the RFC?
<ekleog> gchristensen: follow-up in PM if that's OK?
<gchristensen> yep :)
drakonis1 has joined #nixos-dev
orivej has quit [Ping timeout: 250 seconds]
drakonis has quit [Quit: WeeChat 2.3]
drakonis1 is now known as drakonis
orivej has joined #nixos-dev
<worldofpeace> jtojnar: I'm getting https://github.com/NixOS/nixpkgs/compare/master...worldofpeace:vala-updates/03-2019 btw (not sure you saw the PM). Was there an agreement on how many compiler versions we want to keep?
orivej has quit [Ping timeout: 268 seconds]
<thoughtpolice> Anyone have thoughts/opinions on this, re: propagated* inputs? https://github.com/NixOS/nixpkgs/pull/57590#issuecomment-473042198
<LnL> I personally think propagated inputs are overused
<samueldr> isn't that a case that would generally be handled through a wrapper that wraps gdb into its PATH?
<LnL> yeah, I would actually expect this not to do anything for nix-env or systemPackages
<LnL> that's what propagatedUserEnvPkgs is for
<infinisil> thoughtpolice: Yeah I'd rather have a wrapper, this propagated* stuff doesn't work in all situations
<clever> LnL: and propagatedUserEnvPkgs is why QT stuff breaks hard all the time
<LnL> hm?
<{^_^}> #30775 (by cleverca22, 1 year ago, open): nix-env silently collides propagated-user-env-packages
<clever> LnL: i have even seen this cause the entire KDE desktop env to just explode on login, leaving you unable to login at all
<clever> LnL: because you have an "old" qt thing in nix-env, and then a new KDE desktop (due to nixos-rebuild --upgrade)
<LnL> ah yeah
<clever> but if QT apps didnt use propagatedUserEnvPkgs, none of that would happen
<clever> a simpler fix, is to just add more to the path
<clever> currently, the qt libs are in lib/qt-5.9
<clever> but 5.9.0 and 5.9.1 can collide
<clever> so, just puts the libs in lib/qt-5.9.0!
<LnL> I was just talking about the fact that propagatedBuildInputs should not end up in nix-env, etc. like propagatedUserEnvPkgs do IMHO
<clever> and i agree, the propagated stuff is often mis-understood, and over-used way too much
<LnL> exactly, but on the other hand wrappers aren't ideal either they don't compose nicely
Mic92 has quit [Quit: WeeChat 2.3]
<matthewbauer[m]> maybe we need some guidelines on wrapping vs. propagating?
Mic92 has joined #nixos-dev
<matthewbauer[m]> also, does anyone know what's holding up the channels? looks like channel scripts are failing?
<clever> yeah, that is strange, every channel is 3+ days old, yet i see more recent passing things on hydra
<LnL> yeah eg. https://hydra.nixos.org/job/nixos/trunk-combined/tested/latest points to a different commit
<LnL> niksnut: gchristensen: ^ can either of you look at the channel updates?
jtojnar has joined #nixos-dev
<gchristensen> I can take a look
<gchristensen> I have some hint on a fix for the cache bugs, so I'm going to look at that first
<LnL> oh! that's great news
<gchristensen> it'd be really nice if we had a graph of those errors
<LnL> we can for s3 itself right?
<gchristensen> I don't think so, I think the error is not known by s3
<LnL> sure in this case, but I think we could get data for the s3 part
<LnL> we can configure that?
<gchristensen> yeah
<gchristensen> apparently :)
<LnL> I don't think that's unexpected when talking to s3 directly, even more so if it's a fastly endpoint in another region than the bucket
<gchristensen> yeah
<gchristensen> it cleanly explains why EU users hit it more :|
<gchristensen> I wonder about fastly -> cloudfront -> s3 :)
<LnL> was just going to say, I bet cloudfront is a lot smarter about what to do
<gchristensen> it might be more expensive (?)
<LnL> guess it depends, do we have any info on cache hit ratio / bandwidth?
<gchristensen> yea
<gchristensen> 70% cache hit ratio
<LnL> that's more then I was expecting, might be not a big deal then
<gchristensen> LnL:
<LnL> I see a graph for 503's
jtojnar_ has joined #nixos-dev
jtojnar_ has quit [Remote host closed the connection]
<samueldr> gchristensen: :/ I wanted to suggest it, since I've hear US peeps saying they never had 503s, and I have them infrequently
<samueldr> my connection being a more tortuous route through the great white north
<samueldr> (not really north neither white)
<samueldr> (I mean, I wanted to suggest it might be some low timeout value, but it didn't make sense to me they would do that for such a low threshold)
<gchristensen> me too....
<LnL> gchristensen: what about the first one? from the title I'd think those are actual 503 responses from s3
<gchristensen> first one?
<LnL> error rate = fastly 503s, server errors = backend 504s
<gchristensen> oh error rate is the same as the other graph, but %-based
<LnL> I could be totally wrong, but that's how interpret it
<LnL> ah :/
<gchristensen> I'll check this graph again in an hour or so. need to go for a walk.