<ixxie>
so johanot, I was wondering about the difference between the way you configured flannel in your example, and simply setting services.kubernetes.flannel.enable
<johanot>
did I do anything special? hmm.. don't remember
<ixxie>
you didn't really... you set the flannel network, you set some firewall rules
<ixxie>
but notably you *dont* set the services.kubernetes.flannel option
<ixxie>
which is why I am asking what is the difference
<johanot>
I think there is no reason to set the flannel network. The default CIDR should be ok on GCE (where my test nodes are). It's only if that cidr clashes with something else in your network topology.
<johanot>
regarding enable; I think it is mkDefault'ed to true by the kubernetes root module
<ixxie>
so really I shouldn't have to set anything to make it work?
<ixxie>
except maybe firewall rules if I enable firewall (which I should)
<ixxie>
I was expecting a lot more configuration but it seems flannel is really simple to setup
<johanot>
ixxie: the flannel module auto-opens the vxlan ports. but other ports like etcd (2379) and apiserver (6443) needs to be opened on the master. 10250 (kubelet) needs to be opened on nodes.
<ixxie>
right, but all that you did on the example
<johanot>
yes :)
<ixxie>
so basically I can remove the flannel.network option and it should still work
<johanot>
yes.. But.. exciting to see whether re-configuring of an existing cluster network happens without pain.
<ixxie>
heh
<ixxie>
so if there are problems I should manually set the network
<johanot>
I mean.. Flannel needs to restart with new setting. The flannel prestart script writes a new docker daemon settings file and restarts the docker daemon.. And everything needs to come online in the right order. I still feel that this "getting network ready"-dance is not 100% predictable/stable.
<ixxie>
the other question I had was whether I could see some examples of your module-wrapped-manifests approach.... I know you said it ain't quite upstream ready, but still sounds better than other approaches to deploying manifests
<ixxie>
hmm
<ixxie>
and I guess the order in which this happens is critical
<ixxie>
maybe if NixOps knew about this it could intelligently deploy to optimize this?
<johanot>
ixxie: so.. the problem is that the addon manager (we call it that; the manifest-wrapper-thingy) we use in my company; is currently completely entangled with internal company configuration and weird references to our company settings, which we template into the yaml files.. I will have to untangle it first, so that I don't push something with our company names and internal ip-addresses all over
<johanot>
it :)
<ixxie>
I understand, I imagined its some work
<ixxie>
I guess I didn't realize how much though!
<johanot>
yeah.. the order is important, and I guess it just lacks testing with a bunch of scenarios, like.. 1) brand new cluster, 2) partly initialized cluster, 3) running cluster with settings A , etc. etc.
<ixxie>
right
<ixxie>
do you actually write tests of the kubernetes module?
<johanot>
there are two current realiable test cases for the kubernetes module - rbac.nix and dns.nix
<johanot>
in /nixos/tests/kubernetes/
<johanot>
but they are very basic and only test a fraction of the features
<ixxie>
never actually seen tests written in Nix before
<ixxie>
interesting
<ixxie>
thanks johanot, very enlightening :)
<ixxie>
I will maybe try a (Very) naive implementation of the addon manager, as practice
<ixxie>
I was also thinking maybe I could contibute by packaging JeninsX
<ixxie>
but I would need some guidance probably
<ixxie>
at some point
<ixxie>
johanot: may I ask which company you work for actually?
<johanot>
ixxie: one sec
<johanot>
ixxie: work for DBC. A Danish publicly owned company specialized in software for public libraries. I have no particular knowledge or expertise in libraries, but I got interested when the job offer had the words "container infrastructure" and "kubernetes" in it. :)
<johanot>
and NixOS
<johanot>
.. although I didn't know NixOS before I got hired.
<ixxie>
cool
<ixxie>
just curious if there are many companies running such setups
<ixxie>
it seems only a handful and mostly in Europe?
<johanot>
I don't know tbh :) .. wasn't srhb trying to compile a list of NixOS companies at some point? :P
<johanot>
after working with NixOS for 9 months now, I honestly don't get why people/companies would use any other distro. heh
<ixxie>
johanot: probably because there are still a bunch of packages missing
<ixxie>
but still I think its worth the extra packaging effort
<ixxie>
my goal for the next year or two is to setup a standalone turnkey kubernixos solution, something with an authentication server (Gluu maybe), CI/CD (JenkinsX), a data science platform (Jupyterhub/lab) and slap a web GUI on it
<ixxie>
that would be a cool setup to consult with
<johanot>
it sure is.. well, there's also the "switching cost".. And.. quite simply, lack of knowledge about the existence of NixOS.
<ixxie>
boom! Data science cluster
<ixxie>
yeah true
<ixxie>
its also this 'oh its functional, must be an academic thing'
<johanot>
sounds cool. We use Dex for authenticating to our kube-cluster through LDAP/AD
<ixxie>
how is the experience with Dex?
<ixxie>
Gluu was kinda confusing
<ixxie>
it may be better to try something different
<johanot>
i've only tried the dex ldap connector. it works like a charm, easy to setup. We have paired it up with kube-login as web interface for login.
<johanot>
forked it to get our own logo and stuff there
<ixxie>
neat
<ixxie>
need to weigh the pros and cons of LDAP vs OpenID Connect
<ixxie>
very different animals I think
<ixxie>
I guess for SMEs LDAP is frequently suffeciant but for enterprise grade stuff and social network interop OpenID would be needed
<srhb>
johanot: hah, I'm glad you did that
<ixxie>
oh it is openID
<ixxie>
srhb: what about you? You work in the same company or?
<srhb>
ixxie: Not any longer. :)
<ixxie>
ah
<ixxie>
you seem happy about that
<srhb>
Oh, that came out wrong that. I enjoyed my time there very much :)
<ixxie>
no I meant like
<ixxie>
you are enjoying your freedom sorta thing
<johanot>
:D
<srhb>
It's an experience for sure!
<ixxie>
you working elsewhere now with similar stuff?
<srhb>
I'm a consultant with Obsidian Systems now. Haskell full-stack and Nix. :)
<ixxie>
Nice :)
<ixxie>
Haskell frontend?
<srhb>
Yup, reflex-frp
<ixxie>
cool
<srhb>
ixxie: What do you work with?
<ixxie>
unfortunately relatively more mundane technologies
* srhb
nods
<srhb>
I have great experience with sneaking in Nix at work.... ;)
<ixxie>
Python on CentOS
<ixxie>
deployment with Jenkins/Docker-Compose/Ansible
<ixxie>
although I work with NLP which is cool
<ixxie>
and a graph database
<ixxie>
ArangoDB
<ixxie>
srhb: well, this is one of my secret plans
<johanot>
srhb: and once you succeed, you're off ;)
<ixxie>
see our company is at the point where its about to deploy kube
<srhb>
johanot: Yeah, that was a bit dumb, wasn't it... :P
<ixxie>
so if I learn how to setup a kube cluster in my free time, maybe I can sell it like that
<srhb>
ixxie: Neat! :D
<ixxie>
I just need to have good arguments against Ansible
<srhb>
I used to say
<srhb>
"If setting up a machine by hand with ssh is making a snowflake, then Ansible is a tool for making a blizzard"
<ixxie>
hah
<ixxie>
that is a nice one
<srhb>
Mutating things faster isn't going to save anyone. :-P
<srhb>
I once set up a ceph cluster with hand crafted ansible. God, that was awful...
<srhb>
The vendored ansible was even worse, of course...
<ixxie>
we only recently started some Ansible so its all very minimal and innocent looking at this point
* srhb
nods
<srhb>
At that scale it's quite tolerable.
<srhb>
#itsatrap
<srhb>
:P
<ixxie>
I think one issue is we are mostly a windows shop
<srhb>
Oh!
<johanot>
you must never go there Simba
<ixxie>
xD
<srhb>
To be quite honest I wouldn't know where to begin in a Windows shop
<srhb>
I have zero competences in that field.
<ixxie>
build a Linux microservice like I did
<johanot>
srhb: try rebooting?
<ixxie>
xD
<ixxie>
looool
<srhb>
:D
<johanot>
i'm just trolling now, sry :D
<ixxie>
well I just ssh into the CentOS machine and work from there
<srhb>
ixxie: That sounds like a good approach :)
<ixxie>
I think I might ask IT for an upgrade and virtualize NixOS
<ixxie>
not officially supported by delivery but still safer than CentOS
<srhb>
Yeah..
<johanot>
if you have some kind of VM-platform, starting with some proof-of-concept virtual machines is a good approach
<ixxie>
Indeed
<ixxie>
but I think to knock the ball out of the park I need to beat the delivery team to a workable Kube deployment
<ixxie>
(I'm a mere data scientist!)
<ixxie>
minimal but workable
<ixxie>
I guess Dex can hook into AD :)))
<johanot>
we have reached a point where we reeeealy need an internal binary cache asap.. we have too many overlayed/custom packages and waste way too much time on building stuff
<srhb>
That's what they do in johanot's shop. :)
<srhb>
johanot: Need help with Hydra?
<ixxie>
you build your cluster from the cluster itself right?
<johanot>
srhb: we recently expanded morph (our deployment tool) to be able to build all hosts. Next up: We need to hook that up with our CI (either hydra or just gitlab CI). Finally, we need to change the build artifact somewhere. pehaps in Ceph S3 :)
<srhb>
Ah. :)
<ixxie>
neat
<ixxie>
does morph deploy itself too?
<ixxie>
or is that a silly question
<ixxie>
I guess those aren't the only options
* ixxie
ponders
<johanot>
ixxie: morph is like nixops, only with a lot less features and without state - written in Golang. Another thing we haven't gotten around open-sourcing yet :(
<ixxie>
johanot: why not use nixops?
<johanot>
mainly because state. nixops uses an sqlite database for storing and retrieving state of all deployments. SQLite databases are notoriously difficult to share between laptops for example.
<ixxie>
aha
<ixxie>
so cool
<ixxie>
so synchronize state
<ixxie>
so you*
<ixxie>
I guess sshing into a shared deployed is quite... insecure?
<johanot>
"morph" has no state, i.e. it does basically 1) nix copy closure to ssh://..., 2) switch-to-configuration X
<srhb>
That's actually better than many other ways of doing it
<srhb>
(re. ssh'ing into a shared deploy)
<ixxie>
right
<ixxie>
johanot: but why does NixOps need state if Morph can do without?
<ixxie>
srhb: can't really think of other ways :P
<srhb>
ixxie: shared state committed into a git repo! :P
<srhb>
NixOps doesn't strictly need state and doesn't use it with the None backend.
<srhb>
It still creates it though.
<ixxie>
hmm
<ixxie>
so state is mainly needed for creating hosts and keeping track of them?
<srhb>
Yeah, it's very necessary for the provisioning stuff
<srhb>
Which arguably is NixOps' strongest point
<srhb>
When I don't need that, I also use a method similar to johanot's morph.
<johanot>
yeah.. morph cannot do provisioning
<johanot>
it was never designed to do that
<ixxie>
terminology wise, provisioning means creating new hosts as opposed to merely configuring them right?
<ixxie>
hmm
<johanot>
creating hosts, partitioning, formatting, installing etc.
<ixxie>
gchristiansen was talking about replacing the backend bits of nixops with terraform
<ixxie>
he was saying there was talk of this since the backends are difficult to maintain
<srhb>
tweag has that pluggable terraform thingy for nixops
<srhb>
I believe graham works there now?
<srhb>
I believe that's a good approach (though i don't like that it integrates "in reverse" right now) and I think most of the provisioning should probably be ripped out of nixops and left for terraform
<ixxie>
that was the idea I thought
<srhb>
It is indeed.
<ixxie>
I guess eventually state could be moved into etcd or some other distributed store
<srhb>
pls no etcd
<srhb>
:D
<ixxie>
ceph?
<ixxie>
I know nothing of these things
<ixxie>
xD
<srhb>
Nah just joking, etcd wouldn't be a horrible choice.
<johanot>
kubernetes ? :P
<johanot>
crds
<srhb>
:P
<johanot>
we store all the thingz :P
<ixxie>
heh
<johanot>
I use nixops for provisioning too.. done it on hetzner.de a couple of times, because they don't offer a virtual terminal and it just becomes over-complicated to get NixOS installed I think.
<ixxie>
johanot: I just setup stuff on Hetzner but I think they have a virtual terminal
<ixxie>
oh maybe its different in cloud
<ixxie>
I am using the cloud
<johanot>
ixxie: I only have a couple of physical servers there. I haven't found a way to get a virtual display, yet. You can boot into the rescue system and mount your partitions from there, but getting an actual view of the screen of the server at boot time (for example) I miss.