<BlessJah> disasm: out of curiosity, why journalbeat and not journal-upload?
<disasm> BlessJah: seems like journal-upload would require some custom coding to get it to ship to kafka, as well as a timer or something to execute it so often. journalbeat does it in realtime.
<BlessJah> kk
<BlessJah> I was more concerned about getting logs to destination server rather than what will receive them.
<BlessJah> Actually I'm still, since journal-upload won't require any additional software on sending end (given recent enough systemd), and graylog or whatever on receiving end can get those out of local journald.
<BlessJah> *nothing I've actually done except for few experiments
<disasm> BlessJah: yeah, granted my local setup everything is on one server, but I want something I can horizontally scale to millions of logs per second. with kafka, I can have N nodes receiving data and queuing it, and N logstash nodes popping it off the queue, and then N elasticsearch nodes sharding that data. I haven't done any benchmarks yet, but I think this setup will be pretty robust.
<disasm> I originally looked at rabbitmq instead of kafka, but I would have had to run logstash on the nodes sending to rabbitmq and on the receiving end, which I really didn't want to do.
* BlessJah wants to gather logs from tens of servers
<BlessJah> I'll likely have one or two (mirrored) receiving servers
<disasm> yeah, tens and thousands use case is very different :)
<disasm> the other nice thing about beat vs journald is it isn't systemd specific. I use the same kafka cluster to ingest logs from winlogbeat and directly from logstash as well for systems that don't run systemd.
<BlessJah> it's journald or rsyslog (it all ends up in journald)
<BlessJah> either journald or rsyslog is present virtually everywhere, so hopefully I won't need any extra software
<disasm> yeah, have numerous clients I did something similar for in my last job using journal2gelf and SystemdJournal2Gelf with graylog. Mainly set this up because that didn't scale to well with 100 servers per environment and 10 environments I wanted to monitor :)
<disasm> It's not the sender that gets backed up, it's the receivers. Putting a queue between the two lets you distribute that load.
<disasm> FYI, graylog can do stuff like this too, I just wanted to use a vanilla elk stack: https://marketplace.graylog.org/addons/113fd1cb-f7d2-4176-b427-32831bd554ee
<BlessJah> I'll need to get my hands dirty first, I have literally no experience on this
<disasm> BlessJah: yeah, if you have a small setup with low volume logs, it's probably overkill, but something to monitor and alert if you start losing log messages.
<BlessJah> how do i know im loosing messages?
<BlessJah> I want to have prometheus and maybe mtail or telegraph on nodes I control