It was a drunk guy, and our neighbor seated at the bar (also a #monitorama attendee!) and the bartender both had our backs.
The dude got cut off, and would have gotten thrown out had he pushed it any further. Good times.
(5) but there's a lot of demand for good product design: large-scale, heterogenous data, and low-latency feedback paths (including over longer periods of time, not just instantaneous), and nobody wants to pay. #monitorama
(3) Use boring technology and combine it in innovative ways. [ed: although I... worry about older storage engines, and think that their cost economics may not be up to snuff]
(4) It's a crowded market, and the "best" product may not win. #monitorama
(2) New query languages are rarely the solution. A new query language is not likely to succeed. Everyone uses SQL, use it too unless you have a good reason. [ed: and this is why I made @honeycombio's builder says VISUALIZE _ WHERE _ GROUP BY _ ORDER BY _ LIMIT _] #monitorama
So the lessons:
(1) logs (inverted indices, little aggregation) and timeseries (data loss tolerant, compresses well, just a bucket of numbers) are different challenges for storage. Often you need separate engines. [ed: although he thinks @honeycombio is interesting] #monitorama
[ed: because I think we're finally at the end of the journey.]
So now we're at Google. So what does Stackdriver use to measure itself/GCP and do planet-scale observability? "mostly good," he says, [ed: and I'd agree based on my 8-month-stale knowledge] #monitorama
So enter InfluxDB, and suddenly being able to measure everything and high cardinality dimensions. (which could have solved loggly's problems)
every generation can monitor the previous generation, but not itself...
[ed: & why I demoed Honeycomb debugging itself] #monitorama
so he wound up going to work at Loggly in 2012. There was a huuuge volume of logs to index, but at least there was partial visibility.
And by 2014 it still wasn't solved :/ #monitorama