Screaming Frog.

What is Screaming Frog?

Let’s start from the beginning. Screaming Frog is a tool which allows you to take a crawl of a website. The tool then displays key items in a spreadsheet format which you can then export as you choose. The free version allows you to crawl up to 500 URLs whereas the paid version removes this limit and allows further customisation.

Whilst they are constantly developing new features, Screaming Frog is mainly an investigation tool. It has a lot of customisable capabilities which the larger analytics platforms do not offer, but does not give the same level of advice as the likes of SEMrush or Moz. You have to draw conclusions and actions from the data you are provided with.

Screaming Frog Modes

The tool allows you to customise the way in which you crawl the site, with three main options:

1. Spider

This is the main mode for Screaming Frog. It works like any other spider or bot, visiting the site, reading the information and then providing this back to you. There are different way in which you can utilise the spider by going to Spider Configuration and selecting/deselecting options (such as image crawl).

2. List

In this configuration, the tool will crawl a list of URLs that you have entered manually or uploaded via a CSV (or sitemap). You can also use it to crawl backlinks taken from Search Console which can be useful for identifying 404s.

3. SERP

This doesn’t actually involve crawling the website. Instead, it allows you to upload and amend Title Tags and Meta Descriptions to see how they would fit into the SERPs.

Internal & External Crawling

The internal and external tabs show, well, what you might expect given their names! The internal tab shows all of the URLs on the domain that you have entered, whereas the external tab will show sub domains and domains to which your website has linked.

Spider Configuration

It’s important to note that Screaming Frog will use your computer to conduct the crawl. As such, it can take up a significant amount of the computing power available to you. For example, 8GB of RAM allows you to crawl up to 5 million URLs. However, this RAM will not be reallocated whilst it is crawling which means that unless you pause the crawl or close down the tool, it’s likely to have an effect on your computer’s performance.

It is therefore prudent if you are working on a large site to configure how you crawl websites. Not only will this improve the speed at which you can crawl sites but will also help you hone in on key issues without being inundated with what could be irrelevant data at the time.

Note that the majority of the configurations are only available on the paid version of the tool (a.k.a licensed users).

Basic

The basic settings allow you dictate what the spider will crawl and return within the tool. As can be seen from the screen shot below, you can select/deselect options including ‘check images’, ‘check CSS’, ‘Check JavaScript’ and ‘Crawl All Subdomains’.

Limits

You can also set limits for the crawler in terms of the total amount of URLs crawled as well as the folder depth to which the spider will crawl.

Rendering

The spider is configured to be render text-only by default. This means that when you try to crawl a website with a JavaScript front end (e.g built with React.js) the crawl will not work. On these occasions you need to alter this setting to crawl JavaScript.

* Remember that Googlebot conducts two crawls of a website, one without JavaScript and then with JavaScript, depending on their load.

Finally, if there are dynamic URLs within the website structure you may find that the total amount of URLs the spider is trying to crawl will continue to increase as the crawl occurs. Clearly this can extend the time it takes to crawl so you will want to adjust your settings accordingly.

Key Features of a Report

The tool can be used in a variety of ways from providing an overview of SEO items for an entire website, to reporting on specific issues and URLs (depending on how you configure it). If you’re looking at a standard report for the purposes of an overarching onsite optimisation audit (or some competitor research), we would recommend looking at the following key features:

Status Codes, Indexability and Redirects

Are the pages that you want to be indexed available to be indexed? If they are redirected, are they part of a redirect chain? Any chains will reduce your link equity.

Canonicals

Match up your canonicals with the intended URL. If there are discrepancies here, you’ll want to amend these so that robots are clear on what the original URL of the page is.

Title Tags and Meta Descriptions

Are your title tags and meta descriptions unique to each page and aligned to target search terms? We want to avoid duplication and importantly, stay within character/pixel width recommendations.

Headings

Again, are H1 tags unique to each page and aligned to keywords? Look to avoid duplication throughout.

Word Count

This may help you with low text to html ratios but remember that this is somewhat of a ‘horses for courses’ item. If you’re looking to answer a question which relies on considerable amounts of data or an in depth answer, you will likely need a higher word count. Whereas if the solution to the user intent is very short, trying to fulfil a certain word count may result in thin content.

Crawl Depth, Sitemaps and Visualisation

This function is really useful in understanding how ‘deep’ a website is. You can see this via crawl depth or the visualisation tool which provides a visual representation of the sitemap and user flow. The consensus is that a website should avoid having excessive depth, although for e-commerce sites this can become more difficult due to categories, filters and inner products.

Screaming Frog also allows you to generate an XML sitemap, although we would recommend checking/editing prior to uploading to Search Console.

Hreflang

Great for international sites and should be cross referenced with canonicals to ensure that your hreflang tags have been set up correctly.

Response Time

Load speeds are incredibly important for user experience on both desktop and mobile, and therefore important for SEO. The response time may highlight areas for improvement here.

Link Data (Inlinks/Outlinks)

A useful comparison of how many links are pointing out of a page and how many are pointing in. Whilst PageRank Sculpting is very much a thing of the past, this may help you identify pages that have a large discrepancy in this area.

Exporting & Saving

If you are licensed user (i.e a paying customer), Screaming Frog will automatically save your crawls which you can the re-access. However, we would highly recommend that you export your crawls and save them either on your computer or in the cloud, such as on Google Drive.

As you would expect given the customisable nature of Screaming Frog, you can also customise how and what you export or save. CSVs are probably your go to format but you can also save the files as a .seospider file.

That’s all from us at this point. We’re really big fans of the Screaming Frog tool and considering the costs and customisable nature, it does really have to be part of your SEO arsenal!