pg_fetch_cycle is an experimental PostgreSQL extension built on top of pg_net that orchestrates multi-step HTTP request sequences. It enables things like paginated API scraping, request chaining, and stateful HTTP workflows.
Building a scraper on top of pg_net in PostgreSQL: I found myself in need of a way to chain requests together (which is what you do when you scrape). It is a simple library built on top of pg_net.
PS: I also discovered you can modify the `lnav` behaviour by only showing new logs at the time of using the SSH command above:
```
#!/usr/bin/env bash
export TZ='Europe/Berlin';
NOW="$(date -Iseconds | sed -e 's/+.*$//')"
LOG_FILE="$(ls -t ~/pgdata/data/logs/*.log | head -n 1)"
if [ "$1" = "now" ]; then
lnav "$LOG_FILE" -c ":hide-lines-before $NOW"
else
lnav "$LOG_FILE"
fi
```
#ssh #lnav #postgresql #postgres #trigger #debugging #troubelshooting #pg_net #supabase
I ended up adding an SSH user called `postgreslogs` which serves as a way to SSH directly into the logs using:
- A `ForceCommand` SSH daemon config for the user
- A `sudoer.d` config to allow postgreslogs to elevate (i.e. sudo) to postgres (the DB admin account) without a password
Works really nicely! All you need to do to see the logs is to `ssh [email protected]` or similar!
#ssh #lnav #postgresql #postgres #trigger #debugging #troubelshooting #pg_net #supabase
arabiccola