MClare
I’m working on a set-it-and-forget-it mapping/data pipeline for some of my previous projects using @observablehq #ObservableFramework

For the first one, open complaints from the NYC Department of #Buildings, it looks like in the 18 months since I built the project, the number has gone from 600k to 20k???
Nov 21, 2024 at 2:28pmWeb
Show threadMClareNov 21, 2024

Going to investigate this further this morning once I download the ~3 million records full set of data, rather than fetching just open complaints with the API.

The original map looked like this in April 2023, with some buildings having more than 30 complaints:

https://mclare.dev/nyc-building-complaints

NYC Department of Buildings Open Complaints

Show threadMClareNov 21, 2024
What I expect to see? There was probably a massive clean out after some of the collapses (parking garage, building) that made the news.
Show threadMClareNov 21, 2024

So this is interesting...

2023-04-23 Original data dump for map: 3,395,409 rows
2024-11-21 Pipeline data dump: 2,909,859 rows

That's 485,550 records now unaccounted for!

The DOBRunField is now a consistent current date as well, 2/3 of records in the old set said the query was run in 2018.

Show threadMClareNov 21, 2024

I'll look at this further, but my guess is that there was an issue with their data pipelines and records weren't being kept up to date.

Very curious if I'll find that those missing records are all the ones that were open that I mapped...