✧✦Catherine✦✧grebedoc had its highest share yet of serving garbage requests yesterday (a wave peaking at 150 req/sec)
these waves are getting bigger and bigger which is somewhat concerning. it's nowhere near the hardware capacity yet but i'm hitting some software bottlenecks that i've never thought would be relevant
solo@whitequark like what bottlenecks?
@solonovamax i send an S3 request to Wasabi every time there's a cache miss, including for domains that have never been served by grebedoc. if i'm getting, say, 100k requests to 100k domains i've never seen in a row, these start to really plug up in the worker process. i still have good latencies overall but only on most of the waves, not every single one anymore
violetwonder at which point having the domain list cached on each node starts being an optimisation. i assume it'd be more resource usage than it's worth right now
@avioletheart.me @solonovamax the current plan is:
- make a bloom filter out of the domain list (strictly speaking i could use a hashset here but i think using a bloom filter is basically as much work and should serve even very large instances)
- put a marker object with timestamp of a last 'significant update'
- fetch this marker object once per 60s and update the bloom filter if it got stale
- 'significant update' ideally would be 'a new domain was added' but given the (lack of) directory structure that s3 has i'm not sure that's feasible, it might just have to be 'a new site was added' which is basically as good. i would like to avoid refreshing the list on every site update, that seems like it'll cause a lot of traffic on big instances
what is the difference between a new domain and a new site being added?
@avioletheart.me @solonovamax a site is like
domain.tld or domain.tld/directory, you know like github lets you have a main site and project-specific ones on the same domain