@pbrass Agreed! Almost its own book in its own right

@dkiesow Giving it a read today!

@dkiesow That’s the question! I’ve gotten pretty good at it over the years but I’m sure there’s a ton of different theories and techniques out there.

@dkiesow things like link path tracing, rescrape timing, detection of index pages vs (what I call) "terminal pages" (think an article on a news site which will basically never change vs a section homepage that is constantly updated). Content extraction from arbitrary pages, also things such as building a scraper that won't accidentally overwhelm a page.

@cguess @eNBeWe Hopefully my talk gets chosen I guess.