If I was looking to pull the RDAP and Whois data for about 250K malicious domains (mixed TLD, ccTLD, gTLD, etc) -- is there anyone who can help me with the data? Doing it with `whois` and https calls and going slow to avoid rate limits is kicking my ass!
@AndrewMohawk Have you thought about writing a module for BBoT (https://github.com/blacklanternsecurity/bbot) to do this recursively?
GitHub - blacklanternsecurity/bbot: The recursive internet scanner for hackers. 🧡

The recursive internet scanner for hackers. 🧡. Contribute to blacklanternsecurity/bbot development by creating an account on GitHub.

GitHub
@Aconite33 I understand your account seems to predominantly exist to promote your tool but I can't figure out anyway that it being 'recursive' would solve the problem over my scaling docker workers? How does writing a bbot module help fetch whois and rdap for 260k domains, I'd still need to actually make those queries?
@AndrewMohawk if you have list of targets, you could feed it to bbot module to do a whois lookup, then consume the event to then use it for rdap query. You could also rate limit it so you never go over the limit. We've used similar modules when doing brute force DNS and other mechanisms so we don't abuse API limits. Also, my account doesn't exist to promote bbot, but the company that built it as I helped start it. You can view the over 50 modules we've built for variety of functions to do complex queries to pull back data about multiple targets lists.
GitHub - blacklanternsecurity/bbot: The recursive internet scanner for hackers. 🧡

The recursive internet scanner for hackers. 🧡. Contribute to blacklanternsecurity/bbot development by creating an account on GitHub.

GitHub
@AndrewMohawk Also, I didn't see the docker worker requirement in the original post, but bbot could easily work within a docker if is a specific requirement. I don't know the specifics of your project, but the PoC can poll the API (whodat) and make the whois/rdap query.