Only a minor (~6 hour) yak shave over this three day weekend to get my blog ported over to @observablehq framework.
I wrote up some abbreviated notes about why I did it, what I did, and what remains to restore full functionality to the blog:
https://mclare.blog/posts/thwarted-for-the-last-time/
I’ve had a backlog of things to write about (#cytriangle, @bangbangcon, and #PyconJP to name a few), so I’m relieved to have finally reduced the friction to do so.
The weekend’s Bonus Drop is a bit tardy as what it is based on took a bit longer to get going and was also interrupted by IRL happenings.
Also: just as I had figured out how to get paid subs moved to WordPress, WordPress does a **** move and turns AI-evil, so stay tuned for a move to Buttondown over the coming weeks.
Type your email…
Subscribe
Self-Hosted Observable Framework Sites
I wanted to put together a larger, and more structured introduction to Observable Framework to further demonstrate the potential this new offering has to disrupt “dashboarding” and “reporting”. Before digging into the tech bit, I just want to take a second to note how bold a move this is by the Observable team. I don’t know how many folks are going to opt for the new xyx.observable.cloud hosted reports, which has to be one of the goals for Framework. Making the core technology hosting platform-agnostic is, IMO, a risk. The fact that I was able to throw together a repeatable self-hosted Docker+Caddy setup means that literally anyone else can too. And, at work, our initial Framework-based “data quality dashboard” is working great in our Kubernetes cluster. Truth-be-told, you can just plop the contents of dist/ into any web context, and it’ll “just work”. That being said, lots of folks do not want to be in the hosting business, and if Observable supports custom domains and fully private/authenticated dashboards/reports, then I think they can absolutely succeed in this endeavor.
With that out of the way, let’s dig into the new project example.
Yet-Another CISA KEV Report
I’m still not able to expose some of our internal data to show off the internal site I built, so I turned to the public source data I use fairly often in these Drops, the CISA Known Exploited Vulnerabilities (KEV) catalog. I’m also using Project Discovery’s cvemap utility, which has a lovely -kev CLI parameter, which lets us get CVE metadata for all KEV entries.
The code is on codeberg, and the report is at:
What’s different about this setup?
Digging In
Rather than include a massive number of code blocks, I’ll be using some snippets and linking to the sources on Codeberg.
Here’s the overall structure (unnecessary directories/files hidden):
├── Dockerfile├── dist│ ├── 01-vendor.html│ ├── 02-vendor-vulns.html│ ├── 03-cves.html│ ├── 04-cwes.html│ └── index.html├── docs│ ├── 01-vendor.md│ ├── 02-vendor-vulns.md│ ├── 03-cves.md│ ├── 04-cwes.md│ ├── components│ │ ├── kev-utils.js│ │ ├── queries.js│ │ └── utils.js│ ├── data│ │ ├── cisa-kev.json.sh│ │ ├── kev-cve.json.R│ │ └── kev-cwe.json.R│ ├── index.md│ └── styles.css├── justfile└── observablehq.config.tsLet’s start with the use of “components”.
In queries.js, we have the SQL queries used primarily by kev-utils.js. I find that it makes the code making the queries far more readable, and this also lets me have a library of standard queries I can mix and match. This is one example:
Notice that we take full example of the upgraded SQL experience afforded us by DuckDB.
You’ll see that utils.js is pretty sparse since these are not KEV-specific function. One shortcut function I like to keep around for WASM DuckDB ops is:
This turns the query results — which end up being an array of Proxy object — to something we can use in other contexts. These are used in kev-utils.js like this:
That’s very clean and succinct (at least I think it is).
We also move all calls to Plot into here, like this one, so we can just call it from the Markdown documents.
Data Loaders
Before we get to the markdown bit, I’ll just note that there are three data loaders.
MClare
hrbrmstr's Daily Dropcisa-kev.json.sh just grabs the KEV JSONkev-cve.json.R reads from some JSON I use in other places. It’s generated by calls to the aforementioned cvemap. I have it yank out various fields and turn timestamps into dates.kev-cwe.json.R uses the same files as kev-cve.json.R, but is 100% focused on the CWEs (common weakness info). It also needs to do some cleanup because data in my industry is awful.Observable Markdown Pages
While I could do page organization in the observablehq.config.ts file, Framework groks directory structure, and I can use a trick from Quarto, and use numeric prefixes to get the order I want (hence the 01-, 02-, etc.).
By placing all the Plot code into components, the <div>s are much cleaner:
Note the passing of width downstream.
One of the pages lets the human reviewing the dashport (that term most certainly will not stick) pick a vendor and then displays different details. We first make an select menu:
const selectedVendor = view( Inputs.select(vendors, { value: vendors[0], label: "Select Vendor:" }),);Then we use it:
<div class="grid grid-cols-1"><div class="card">${resize((width) => PlotVendorDetails(selectedVendor, width))}</div></div><div class="grid grid-cols-1"><div class="card" style="padding-left:5%; padding-right: 5%">${deets(selectedVendor)}</div></div>It’s great having the familiar Observable reactivity to lean on.
Jusfile Enough
There are some notable changes to the Justfile.
First, we use variables up top, which makes the Justfile more easily made generic, and also means we don’t have to change multiple values in multiple places:
exposed_port := "9101"container_port := "80"image_version := "0.2.0"image_name := "kev-report"container_name := "kev-report-fwrk"We also added a section to enable deployment to Observable cloud:
# deploy to Observable@deploy: npm run deploy -- --message "cron"Automation
I have a cron job that runs twice daily (KEV has two update windows per M-F) and runs just build-image daemon. It takes less than 15 seconds to build a new report + image and start the new container.
FIN
I was forced to use GoDaddy for the hrbrmstr.app domain (ages ago), but they have a great CLI, so adding a new dashport to this dashporting server is something that can be all done via “infrastructure as code”.
Hopefully, this helps folks ponder their own idioms for working with Framework (though, this approach applies to lots of similar idioms).
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev ☮️
#html #javascript #markdown #observableFramework #observablePlot
Philipp Hartenfeller
Frederik Elwert