Seriously impressed with Victoria Logs.
We're replacing our ElasticSearch cluster (27 nodes, ~588 CPU Cores, with 4656gb RAM) with a single VL Node (8 CPU Cores, 64gb RAM).
We shipped 100m logs to VL in the last hour - box is basically idling with only a couple of GB of RAM used. Any random search query returns in a fraction of a second.
Seriously seriously impressive.
@jpm Time will tell. We can only store 35d of logs in our ES cluster at the moment, so ask me in about a month 😅
(Although disk usage so far is pretty good too. Extrapolating out the last hour, we should be able to store 35 days in less than 2tb)
@phs Looking at those numbers... that's a hell of a ES cluster
@shlee Logs in the last 24 hours: 10,198,712,296 😜
Here's hoping Victoria Logs will be able to handle that!
@froztbyte We did not. I happened across VL one night in a blog post praising it's virtues, we threw together a POC and said "yes please"
EDIT: This was the blog post: https://chronicles.mad-scientist.club/tales/grepping-logs-remains-terrible/ so thanks to @algernon are due!
@phs nice, sounds like setup effort is relatively low? that’d be a win over loki - latter heavily advocates k8s and once you’re off that golden path _everything_ is more effort
(loki isn’t alone in this crime but..)
@froztbyte Single binary for the server (including webui) + single binary for the CLI client.
We're currently shipping from rsyslog using omelasticsearch (since we already have it configured for ES) with nginx in front of VL doing client cert authentication. Plan is to rip out omelasticsearch and just ship directly using rsyslog.
@phs @froztbyte have you tried to use Grafana via VictoriaLogs datasource? It has a lot of cool features 
See https://github.com/VictoriaMetrics/victorialogs-datasource or install it directly from Grafana store https://grafana.com/grafana/plugins/victoriametrics-logs-datasource/
@dengolius @froztbyte I am not, but have not read a single bit of documentation for VM yet, just focused on VL to start.
Makes sense though given the flexibility VL has for ingestion sources.
I have very little experience with Loki, because I gave up at the "lets set this thing up" step. Combine that with @cks's writeup, and I'd say VL over Loki any time of day.
/cc @froztbyte @phs
@gnomon @phs @algernon We're someday going to have to replace our Loki setup that I'm no longer happy with¹, and if/when we do, I suspect VictoriaLogs will be the leading candidate. For now, our Loki setup (ingesting system logs from ~140 Linux servers on a basic 1U server) just sits there and doesn't explode on us and manages to use trivial disk space for a year+ of logs. (We've had to erase all current Loki logs and restart periodically for reasons).
¹ https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLokiSimpleNotRecommended
@phs And they're the good guys.
@phs Awesome!
Would you say backup and restore are as easy as the the rest of the workflows?
One of the goals for COS Lite is to be able to ingest considerable amount of data on modest hardware. Load testing is useful for gaining insight into how to size observability clusters appropriately. Method The test method is identical to the method used for load-testing on 4cpu8gb. No k8s resource limits set. The per-pod resource requirements are presented below and can be used by admins to set resource limits. MicroK8s 1.27, Juju 3.1.6. 20 virtual SREs (dashboard gazers) “looking” at panel...
fukawi2
I love this, so I
Altan Ozlu
Shlee fooled around &
JP
Denys Holius
small snoring rock formation
"Musty Bits" McGee
Ben Zanin
Chris Siebenmann
VictoriaMetrics
Melroy van den Berg
Hachiko
Oomfie Snoofies
Rihards Olups
Malte