when something almost works - wtf? i need to figure it out since i really need to index this as fast as possible, all these proxy layer slow things down. it has to be something simple - site will browse no prob but a crawler stops, most of the time #intermittent bad gateway?