@heidilifeldman @guardian

#USpol #TheBrownSpiderWeb

(3/n)

...connecting every dataset with every other dataset,... coordinating the #SurveillanceState. #HomelandSecurity is already #spidering through #IRS data to locate undocumented #immigrants through their tax filings..."
"While the shock troops do the dirty work and the marching bands inspire the masses, #Doge may expand from enforcing fealty in the federal workforce to exacting it from everyone....

Via his newsletter, Ed Yong profiled for the Cornell ornithology mag: https://www.allaboutbirds.org/news/ed-yong-writer-bird-immersion/

> Yong, who describes himself as a “quite restless person,” extols the power of birding and bird photography for mental health.
>
> “I’m always thinking about stuff. I have anxieties. I have a lot of things in my mind. And when I’m out in the field looking through a [camera] lens or binoculars, a lot of that just falls away,” he says. “I’ve described it as being more meditative than actual meditation. It really does focus the mind. … You are putting all of your attention on this other creature, often a very innocuous, honest, unassuming creature. We all joke about like the little brown jobs, right?
>
> “So you focus all of your energy and attention on this tiny brown sparrow, and it is the center of your world, for this fleeting moment.”

Like he writes in his NYT column from last year (https://www.nytimes.com/2024/03/30/opinion/birding-spring-merlin-ebird.html // https://archive.ph/dlIqZ): "While birding, I seem impervious to heat, cold, hunger and thirst."

I feel exactly the same looking for bugs. One minute I am sweaty, hot, annoyed—then I spot a particularly interesting spider or mite and everything else disappears.

Also this photo—!!! See it full size here: https://buttondown.com/edyong209/archive/the-eds-up-on-doing-something/

#birding #birds #spidering

How a Prizewinning Science Writer Found Himself Immersed in Birding

A few years ago, the Pulitzer-winning science journalist Ed Yong discovered birds could be a balm for modern anxieties.

All About Birds

For a month now I've been writing articles in Italian and English trying to make people understand the differences between #scraping, #indexing, #crawling and #spidering. I didn't realize how much confusion there was!

The real problem is that actions with very different goals use the same technical words (which are also misused in the debate).

Have you ever read something addressing this mess?

Released spidr 0.7.0. Added a `Spidr.domain` method for spidering the domains and any sub-domains.
https://github.com/postmodern/spidr/blob/master/ChangeLog.md#070--2022-12-31
https://github.com/postmodern/spidr
#ruby #webspider #webcrawler #spidering
spidr/ChangeLog.md at master · postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. - spidr/ChangeLog.md at master · postmode...

GitHub
Coming up with the options for a web spider command and which options are mutually exclusive, is really difficult. Like obviously such a common should print the URLs by default. However, what if the user also wants to scrape HTML nodes out of each webpage using an XPath? Should you print the URLs and the matched content, or disable printing of URLs if --xpath is specified, or have a separate option called like --no-print-urls to explicitly disable printing the URLs if you only want to pipe the matched HTML into some other util.
#webspidering #webspider #spidering #cli #recon
If you were spidering a website and extracting HTML comments (ex: <!--- comment here -->), should you automatically strip any leading/trailing whitespace or leave it in? Apparently libxml2 preserves the spaces between the <!--/--> and the text.
#webspider #spidering #recon
automatically strip
63.6%
preserve it
36.4%
Poll ended at .
What would be some good infosec/bughunter web spidering enumeration functions? So far I have hooks for enumerating every cert, favicon, hostname, JavaScript source, and JS strings. Trying to think of more.
#infosec #bugbounty #spidering #recon