Per VognsenFor the love of all that is good, if you have a website with an URL scheme like school.edu/~person/course123/lecture6/notes.pdf then school.edu/~person/course123/ and school.edu/~person/course123/lecture6/ should not give 404 responses. It's surprisingly common to see that and have to do a forward crawl starting from school.edu/~person (and you hope the page isn't orphaned).
Although now I'm wondering if there's a way to easily get the archived URL hierarchy via the Wayback Machine.
tbodt@pervognsen yeah just put the prefix in the search box and add a *