Screaming Frog

@[email protected]
302 Followers
44 Following
53 Posts
Run by Founder Dan Sharp of Screaming Frog, a UK search marketing agency & developer of the SEO Spider. Gooner. SEO. & Philanthropist.
Screaming FrogMay 21, 2024

Just released a small update to version 20.1 of the @screamingfrog SEO Spider - https://www.screamingfrog.co.uk/seo-spider-20/#20.1.

Mostly small improvements & bug fixes from the recent release.

It also includes new JS snippets to be able to crawl with Google Gemini.

Screaming Frog SEO Spider Update – Version 20.0 - Screaming Frog

We’re delighted to announce Screaming Frog SEO Spider version 20.0, codenamed internally as ‘cracker’. It’s incredible to think this is now our 20th major release of the software, after it started as a side project in a bedroom many years ago. Now is not the time for reflection though, as...

Screaming Frog
Screaming FrogMay 7, 2024

Just released @screamingfrog SEO Spider v.20:

https://www.screamingfrog.co.uk/seo-spider-20/

Includes -

🧙‍♂️ Custom JavaScript
🤖 Crawl with ChatGPT
📱 Mobile Usability
🔑 N-grams Analysis
🌳 Carbon Footprint

And lots more. Enjoy!

Screaming Frog SEO Spider Update – Version 20.0 - Screaming Frog

We’re delighted to announce Screaming Frog SEO Spider version 20.0, codenamed internally as ‘cracker’. It’s incredible to think this is now our 20th major release of the software, after it started as a side project in a bedroom many years ago. Now is not the time for reflection though, as...

Screaming Frog
Screaming FrogFeb 20, 2024

Just released a small update to 19.6 of the @screamingfrog SEO Spider - https://www.screamingfrog.co.uk/seo-spider-19/#19.6

Mostly small updates & bug fixes.

Screaming Frog SEO Spider Update – Version 19.0 - Screaming Frog

We’re delighted to announce Screaming Frog SEO Spider version 19.0, codenamed internally as ‘Peel’. This update contains a number of significant updates, new features and enhancements based upon user feedback and a little internal steer. Let’s take a look at what’s new. 1) Updated Design While subtle, the GUI appearance...

Screaming Frog
Screaming FrogFeb 12, 2024

Just released a small update to 19.5 of the @screamingfrog SEO Spider -

https://www.screamingfrog.co.uk/seo-spider-19/#19.5

Mostly small updates & bug fixes.

FID has also been replaced with INP in our Core Web Vitals Assessment, in preparation for the change in a month.

Screaming Frog SEO Spider Update – Version 19.0 - Screaming Frog

We’re delighted to announce Screaming Frog SEO Spider version 19.0, codenamed internally as ‘Peel’. This update contains a number of significant updates, new features and enhancements based upon user feedback and a little internal steer. Let’s take a look at what’s new. 1) Updated Design While subtle, the GUI appearance...

Screaming Frog
Screaming FrogJan 10, 2024

Just released @screamingfrog Log File Analyser v.6.0:

https://www.screamingfrog.co.uk/log-file-analyser-6-0/

Includes -

⚙️Updated Design & Unified Settings
🔎Bytes Tab
🌳CO2 Calculation
📊New Overview Charts
😱In-App Updates
🐸Translated UI

And more. Enjoy!

Screaming Frog Log File Analyser Update – Version 6.0 - Screaming Frog

We’re delighted to announce the release of the Screaming Frog Log File Analyser 6.0, codenamed ‘Cookie’. If you’re not already familiar with the Log File Analyser, it allows you to upload your server log files, verify search engine bots, and get valuable insight into search bots crawling your website. We’ve...

Screaming Frog
Screaming FrogDec 13, 2023
Show threadDave Smart Dec 12, 2023

And let's never forget the ever awesome @screamingfrog https://www.screamingfrog.co.uk/robots-txt-tester/

So, sad as it is to see the old tool go, there's a world of choice out there!

Robots.txt Testing In The SEO Spider

View URLs blocked by robots.txt, the disallow lines & use the custom robots.txt to check & validate a sites robots.txt thoroughly, and at scale.

Screaming Frog
Screaming FrogDec 13, 2023
Dave Smart Dec 12, 2023

So the old robots.txt tool from google is truly gone, it now redirects to the new report. Quick reminder that if you're missing the ability to test rules & urls, @jammer_volts & I did this tool https://tamethebots.com/tools/robotstxt-checker that uses the official parse and lets you test one or many urls, and export the results

A more familiar to the depreciated tool is the excellent https://www.realrobotstxt.com/ from Will Crichlow.

Merkle's https://technicalseo.com/tools/robots-txt/ allows you to test a url & its resources #seo #robotstxt

robots.txt Testing Tool

Test a list of URLs against a live or custom robots.txt file, using Google's open source parser.

Tame The Bots
Screaming FrogNov 16, 2023
Little tip - You can test robots.txt in @screamingfrog via 'Config > Robots.txt'. You can download the robots.txt, customise and test rules - which work with your selected user-agent. It uses code from Google's published OS robots.txt parser.
Screaming FrogNov 16, 2023
Eric HeikenNov 16, 2023

What happens when you do SEO for a couple huge sites and you haven't cleared out your @screamingfrog crawls in 6 months.

#seo

Screaming FrogNov 1, 2023
Simon Cox Nov 1, 2023

@screamingfrog updated to 19.3 for the #seo fans out there.

"Small Update – Version 19.3 Released 31th October 2023"

Too many sweeths last night Dan?

https://www.screamingfrog.co.uk/seo-spider-19/#19.3

Screaming Frog SEO Spider Update – Version 19.0 - Screaming Frog

We’re delighted to announce Screaming Frog SEO Spider version 19.0, codenamed internally as ‘Peel’. This update contains a number of significant updates, new features and enhancements based upon user feedback and a little internal steer. Let’s take a look at what’s new. 1) Updated Design While subtle, the GUI appearance...

Screaming Frog