jybs has joined #nixos-kubernetes
<jybs> haha
<jybs> "not very active
<jybs> :P
<srhb> o/
<jybs> node is just rebuilding now on unstable
<srhb> jybs: We have some single node tests in the tree if you're curious
<srhb> jybs: eg. nix-build nixos/tests/kubernetes/dns.nix -A singlenode
<jybs> Man updating to unstable pushes a lot of data
<srhb> Oh, I forgot, current unstable is broken with vulkan-headers until next bump, I think..
<srhb> 8d1510a is the last that worked for me, until it bumps again.
<jybs> Oh dear
<srhb> But yeah, release bumps are big :P
<srhb> Not a big issue, it just fails to evaluate..
<srhb> Might only be in certain cases.. :)
<srhb> (Like, vulkan sounds graphics related.)
<jybs> On older versions kubectl just worked, but with newer ones I have to export KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig
<jybs> Is that expected?
<srhb> jybs: Yeah, the tests wrap it to set the env var.
<srhb> jybs: Look in base.nix
<jybs> Yeah I can do it myself, just thought it was odd it changed
<srhb> I think it's not the only magical default to go away.
<jybs> Will i need to remove my current config?
<jybs> (/var/lib/...)
<srhb> jybs: Something you made by hand?
<srhb> Building a test node on master now. #yolo
<srhb> At least it has the latest fixes in. :P
<srhb> Maybe that was not a great idea. Sigh, release windows are always such a mess. :P
<jybs> OK here we go
<jybs> Now it seems to just not get anywhere :(
<jybs> /nix/store/dax0d72jklvln5i3f3m2ikbp7zpij0z5-bash-4.4-p23/bin/bash -e /nix/store/dgf2y7bv3f4znbqnkhpa8q60y1y0dv6b-unit-script-kubelet-online-sta
<jybs> seems to be hung
<srhb> No logs?
<jybs> : Condition check resulted in Kubernetes Kubelet Service being skipped.
<srhb> Yeah, those are the systemd deps we dropped in that PR.
<jybs> *sigh*
<srhb> Maybe unstable was bad advice, at least until next bump... I've been testing on master, so..
<srhb> Not that helpful :)
<jybs> :)
<jybs> Is fine
<jybs> Did your test off master work?
<srhb> Yeah! Trying to build a fresh one off 8d1510a + cherry-pick 11e72e547d013894dfce88e68d0ec80a8bf0b545 -m1
<srhb> Because meanwhile, master got broken on some other packages I need. :P
<srhb> Seems to work fine!
<srhb> coredns running happilly.
<jybs> Do you use nix to create your containers?
<jybs> (I ask as I wait for all of kubernetes to compile)
<srhb> jybs: Usually yes :)
<jybs> Oooh cool. Systemd in the containers?
<srhb> Nope. Barebones single executable stuff.
<srhb> If I need an init system I don't run it in docker.
<jybs> It's less needing an init system, more a system which already knows all about how to start things
<jybs> Do you use k8 on nixos in prod?
<srhb> Not until october. :P
<jybs> lol
<srhb> (I left that job a while back, but going back then :))
<srhb> They've been running k8s on NixOS in prod since.. What, january last year? Something like that.
<jybs> OK
<jybs> It seems like a pretty good combo (if I can get it to work)
<srhb> I'm sure we can work it out. fwiw my config was simply masterAddress = "myHostname" and roles = [ "master" "node" ]; with nothing else
<srhb> Though I prefer to run the explicit approach rather than the smarts, but for testing this is nice.
<jybs> And the hostname doesn't have to resolve via DNS right?
<jybs> It can be /etc/hosts only?
<srhb> jybs: Yeah.
<srhb> I'm just on my laptop now, so it was its own hostname.
<jybs> I don't suppose you'd be up for some random non nix k8 questions?
<jybs> :P
<srhb> You can try, but I'm rusty!
<srhb> until october, hopefully xP
<jybs> OK - so my main one is can you run a prod cluster with three machines. I've asked elsehwere but always get a full range of answers
<jybs> so 3 x master+node
<srhb> Yeah, absolutely.
<jybs> with stacked etcd
<srhb> I like to keep things separate, but without any further concerns like extreme resource usage or whatnot, it's fine.
<jybs> They are mid level machines. 28 core, half a terrabte ram, 6TB SSD
<srhb> That doesn't mean anything to me without a gauge of the workload.
<jybs> Hoping to get Prometheus + friends, Elastic + friends+ PostgreSQL + some random small containers on there
<jybs> So high capacity stuff, but not doing much
<srhb> Probably fine. You can always move the master components away to some laptops or whatnot if the need arises.
<srhb> :P
<srhb> Not like they need a lot.
<jybs> I don't so much *need* k8 as want it :P
<srhb> how strange!
<srhb> ;)
<jybs> ha
<jybs> 99% of the world I suppose
<srhb> I guess. I prefer to run everything off k8s basically
<srhb> But it's a good platform to allow people that are not opsy to deploy things on.
<srhb> Though it is fun, sure :P
<srhb> Maybe one day we get a distributed nix thing and I'll change my mind :P
<jybs> For us it's more about getting a foothold in a datacenter with something which we can extend
<jybs> gah master build didn't work
<srhb> What's up?
<srhb> Oh and yeah that's a good point
<jybs> systemd-timesyncd.service: Failed with result 'exit-code'.
<jybs> I see this every so often and never know what causes it
<srhb> That doesn't sound like a bad failure, probably just bad setup of that service.
<jybs> ooh k8 is up anyway
<jybs> Should kubectl get nodes show my node is a master?
<jybs> I get a role of <none>
<srhb> It should just say Ready
<srhb> That's fine
<jybs> Ok that's fine then
<jybs> I think I'm good
<srhb> You might want to do some more networking setup
<srhb> Because probably you don't have a lot of connectivity to services at this point
<jybs> Yeah I think I saw a section on that in the k8 section of the nixos docs?
<srhb> ie. I'm not sure you can curl
<jybs> I'll have a look
<jybs> I can't
<srhb> fwiw I'm not very happy with flannel, but it's what we use in the tests
<jybs> But because unable to get local issuer certificate
<srhb> Ah that's fine
<srhb> just throw -k at it
<jybs> Forbidden
<srhb> That's also fine :P
<srhb> The keys and stuff are in /var/lib/kubernetes/secrets iirc
<jybs> And if I want to join three servers I just do it manually, not via nix right?
<srhb> jybs: Take a look at how the multinode tests do it
<srhb> jybs: It uses the easyCerts mechanism and makes it easy to do the joining.
<srhb> But you can also do it by hand of course.
<jybs> Hangon
<jybs> "Note: Multi-master (HA) clusters are not supported by the easyCerts module."
<jybs> Sorry this is why I assumed I couldn't
<srhb> Right, if you're building a prod cluster, I wouldn't do easyCerts anyway.
<jybs> Of course
<srhb> The test exchanges the token through the shared tmpfs of the vms
<srhb> Which is a bit cheaty, but nice for testing multinode functionality :P
<srhb> For a real setup, by far the most involved part is setting up proper PKI.
<jybs> Which ones are the MN tests? Are they in nixpkgs/nixos/tests/kubernetes/?
<srhb> Which goes for.. Well, anything, not just k8s :P
<srhb> jybs: They're in both dns and rbac, under the multinode name :)
<srhb> None of them are multi-master though
<srhb> iirc.
<jybs> That's fine
<srhb> What I would suggest you do if you want to do multimaster right away is get rid of roles = [ ... ]; and set it up manually through the low level interface
<srhb> From there it's basically following any k8s-the-hard-way guide.
<srhb> And working out errors in the journal as you go along :)
<srhb> The daemons are actually quite informative about what's wrong at any given point.
<jybs> the hard way sounds ominous, but cool
<srhb> It should be called "kubernetes, but you do all the PKI stuff" ;-)
<srhb> The rest is easy.
<jybs> Amazing
<jybs> helm deploying away
<jybs> Thanks so much for the help!
<srhb> You're welcome!
<srhb> Sorry about the so-so state of the module and stuff
<srhb> Hopefully we can fix that real soon, and definitely in time for 20.03
<jybs> To be fair it doesn't matter as long as it works in the end
<jybs> Once it's in it's in
<srhb> I hope you'll stick around or at least come back, more people means more feedback means better modules :)
<jybs> Oh I'll be back
<srhb> Great!
<jybs> most likely tomorrow when I can't work out the PKI
<srhb> xD
<jybs> Gotta go, thanks again
<srhb> See you :)
jybs has quit [Quit: Lost terminal]
johanot has joined #nixos-kubernetes
ixxie has joined #nixos-kubernetes
ixxie has quit [Ping timeout: 245 seconds]
ixxie has joined #nixos-kubernetes
johanot has quit [Quit: WeeChat 2.4]
ixxie has quit [Ping timeout: 258 seconds]