Titus 🇵🇸Dec 4
@andrewnez hey just wanted to share that I am crawling ecosytems right now (politely), but let me know if it starts appearing as abuse on your side (I sometimes get internal server errors back). As this is the first time using it, I am not sure how many hours it takes. With npm it took days but I am guessing it’ll be faster here
Show threadAndrew Nesbitt
@wooorm thanks for letting me know, it should be able to handle you crawling it, you’ll get 429 responses if you hit it too hard, also do open issues with any problems or improvements you think would be helpful
Dec 4 at 7:33pmIvory for iOS
Show threadTitus 🇵🇸Dec 4
@andrewnez all done! Normal (400 per page) should be 30minutes or so, but I got several internal server errors, which magically resolved themselves after retrying again (repeatedly) after 10min intervals!
Show threadAndrew NesbittDec 4
@wooorm yeah the servers get hammered by AI bots so sometimes the backends get overloaded
Show threadTitus 🇵🇸Dec 4
@andrewnez I don’t have concrete issues other than the opaque server errors; API is clear and very usable! I appreciate the mailto friendly protocol!