How to Use Screaming Frog to Get the Most Out Of A Website Audit

When you’re mapping out a plan to improve the results from your website, knowing where to look for data is often half the battle.

You’re probably already familiar with Google Analytics to some degree, and maybe you’re already using tools like HubSpot, SEMRush, and heatmapping tools. But have you met the Screaming Frog?

[cmtoc_table_of_contents]

What is Screaming Frog and how can I use it for my marketing?

Screaming Frog is a “technical SEO” tool that can bring even deeper insights and analysis to your digital marketing program. With Screaming Frog, you can extract data and audit your website for common SEO and technical issues that might be holding back performance. It is one of the most popular tools among marketing experts to crawl a website and gain useful insights into SEO and website performance.

Screaming Frog offers both a paid and free version, which you can choose depending on the scope of your project. The free version is capable of crawling a website’s first 500 URLs.

Combining Screaming Frog and Google’s marketing tools for enhanced reporting

In this article, we’ll show you how you can connect Google Analytics, Google Search Console (GSC), and the Google PageSpeed API with Screaming Frog to generate a report that contains insights and data beyond what you can gather from any one of these tools alone. With the enhanced Screaming Frog report you’ll be able to quickly identify how your website is performing and what areas of the website you can impact the most with additional marketing efforts.

By combining the power of Screaming Frog and Google’s host of powerful web and marketing tools, it’s possible to cut through the noise, quickly identify areas you can target for improvement, and build a stronger framework for your digital marketing strategy.

Connecting Google Analytics with Screaming Frog

Connecting your Google Analytics account is a straightforward process with Screaming Frog. You’ll navigate to the Screaming Frog menu and under the ‘Configuration’ dropdown, you’ll see an API Access option. Within that menu, you’ll select ‘Google Analytics.’

This will open up a window where you’ll go through the process of connecting your Google Analytics account. After your account is successfully connected, the Google Analytics menu will allow you to select a specific account, property (the website), and view.

You’ll want to pay close attention to the ‘Segment’ dropdown. By default this is set to ‘organic,’ but you may want to adjust this based on needs. Screaming Frog has a large list of options for filtering data if you want to perform deeper analysis. For the purposes of this article, we will keep things at a general level, but keep in mind that you can adjust the data pulled back from Google Analytics.

You’ll also have the option to filter the date range and additional settings on the remaining tabs. The most important are the ‘Date Range’ tab, which is straightforward, and the ‘Metrics’ tab. The metrics tab gives you the option to select up to 30 items; the structure of your site will help determine what data you’ll want to pull back. Some of the most critical items we can gain insight from include:

• User
• Page Tracking
• Goals (if configured on your site)
• Site Speed
• Social Interactions

Several other metrics may be critical to your application, as additional data can be pulled back for Google advertising, e-commerce, and additional metrics.

Connecting Google Search Console with Screaming Frog

Connecting your Google Search Console account to Screaming Frog is an almost identical repeat of the Google Analytics connection process. You’ll navigate to the Screaming Frog menu, and under the ‘Configuration’ dropdown, you’ll see an API Access option. Within that menu, you’ll select ‘Google Search Console.’

This will open up a window where you’ll go through the process of connecting your Google Search Console account. After your account is successfully connected, the Google Search Console menu will allow you to select a specific property.

The additional options for Google Search Console are less complex than the Google Analytics options. The date range and dimension filters are the two key tabs for this tool. Usually, you can leave these to default settings, but the option is there to limit this data to specific devices or countries if you need a more specific data set in your Screaming Frog report.

Connecting PageSpeed Insights with Screaming Frog

The last Google tool we will connect is perhaps the most meaningful. Google’s PageSpeed Insights tool provides data on page performance (including metrics on Core Web Vitals) on both mobile and desktop devices, along with suggestions on how to improve any areas of concern that may be negatively impacting your site. As most marketers know, page speed is one of Google’s key metrics for how they rank a website in search results, and of course, fast-loading websites provide the best experience for your users.

The Google PageSpeed Insights API allows us to run the PageSpeed test on multiple pages concurrently and quickly get a report that covers the entire site. It’s a largely overlooked but invaluable tool when analyzing a website that is new to you, a situation that many incoming marketers and web developers face.

Connecting the Google Pagespeed Insights API will require an API key from the Google Developers Console. You’ll navigate to the Screaming Frog menu and under the ‘Configuration’ dropdown, you’ll see an API Access option. Within that menu you’ll select ‘Pagespeed Insights.’

This will open up a window where you’ll enter a key generated from the Developers Console. After your account is successfully connected, you’ll need to configure the metrics that you want to pull back on the crawl.

You’ll have the option to select Desktop or Mobile for the device type, and you may want to run a report on each device. We recommend starting on desktop, and once those metrics are acceptable, moving on to mobile and ensuring all pages are performing on mobile as well as desktop.

There are a lot of options for the Pagespeed metrics and you may want to consult a developer if you have one available, but at a minimum, you’ll want to utilize the default settings and be aware of the device selected.

Crawling a site with Google Marketing Tools and Screaming Frog

Once you’ve completed connecting the various Google tools from which you’d like to pull data, you’ll run the crawl. For crawl settings, most users will want to utilize the guide from Screaming Frog on crawling a larger website However, if you have a smaller website that is less than 10 pages, the default settings would be fine to leave in place. In both cases, we recommend adjusting the speed of the crawl to “1 thread and 2 URLs” so that it does not have a significant impact on your hosting performance.

After the crawl is complete, results for each individual tool will display in specific columns in the crawl results, and you’ll also see data in the overview tab for each tool.

The analytics and search console data are excellent for getting a quick overview on areas of concern. The initial analysis on these overview tabs will show you the total number of pages crawled, along with specific numbers depending on the tool being used.

For Google Analytics, these data points include:

• Sessions Above 0
• Bounce Rate Above 70%
• No GA Data
• Non-Indexable With GA Data
• Orphan URLs

For Google Search Console, the data points are:

• Clicks Above 0
• No GSC Data
• Non-Indexable with GSC Data
• Orphan URLs

These are excellent starting points for identifying areas of focus for your digital marketing efforts, but you’ll learn even more by going deeper into the Screaming Frog data that is returned.

Diagnosing an enhanced Screaming Frog crawl

You connected all the tools, ran the crawl, and now you have a spreadsheet full of data. What does all this data from Google, Screaming Frog, and your website mean?

After you’ve run your report with the various tools connected, you’ll have a plethora of data to filter through and find opportunities for focusing your marketing efforts. We will look at a few specific types of data that allow you to identify areas of your website that need attention, whether in terms of content or how the website is built. The data can guide you on making the most cost-effective marketing decisions.

Screaming Frog lets you sort any column from high to low, so you can quickly identify a group of pages to focus on. The areas we recommend initially are:

GA Bounce Rate: Depending on how your site is performing, you may need to start at 90% and above and work your way down. Focus the most on the pages with a high bounce rate that are most critical to your marketing mission. (For example, if your homepage and product/service pages have high bounce rates, improving these would be a higher priority than improving old blog articles).

GA Page Views Per Session: This can be a great way to identify pages that aren’t clearly aligned with your overall positioning and marketing goals, as it indicates users aren’t spending much time on the website after the initial page visit.

GA Avg. Time on Page: This metric is a clue to how engaged visitors are and whether the page is capturing their attention.

GA Exit Rate: This is an excellent way to determine where you are losing users in the funnel. For example, if the website being tested has a multi-step quote form and the exit rate for step 3 is extremely high compared to the other steps, it would tell us something on that page is happening to push users away and needs further evaluation.

GA Avg Page Load Time: Pages with a higher load time are likely candidates for a development performance review. Perhaps an image on the page isn’t optimized. This metric doesn’t uncover exactly what the issue is, but you’ll know something isn’t right on the page and it needs to be inspected more closely.

The GSC data on the report will be far less intrinsic but still carries value. The clicks, impressions, and click-through rate (CTR) are incredibly valuable, and you can sort by the CTR to identify pages that might need some adjustments to their title and/or meta tags to improve the display of the page in search engine result pages.

Lastly, the Google Pagespeed Insights API contains a wealth of data. It focuses more specifically on the development of the website, how it performs, and how it is structured. In short, is your site built well for users or are there areas that can be improved upon?

The most critical elements of the PageSpeed report to focus on are:

• Performance Score
• FCP (First Contentful Paint) Score
• Speed Index Score
• LCP (Largest Contentful Paint) Score
• Time to Interactive Score
• CLS (Cumulative Layout Shift) Score

Taking the lowest-performing pages in each of these metrics will give you a good base of pages for your development team to work on. It’s also worth noting here that some issues are universal and resolving issues on one page will have a cascading effect and the issue will be fixed across the site. If you see the same issue appearing on multiple pages of a scan or across an entire site, it’s important to prioritize these issues as resolving them will have a large impact across your entire site.

Continual performance monitoring

As always, there is no one-size-fits-all approach to online marketing, and that is why Screaming Frog thrives. Whether you have an e-commerce site or a simple blog, Screaming Fron has the capability to generate reports with the data you need to identify those areas of the site that need improvement.

We recommend running a report like this on a regular basis and continually monitoring the performance of your website. Over time, the requirements to achieve a high ranking will change,as they always have, but by combining the power of Google’s marketing tools with Screaming Frog, you can identify the areas of a website that aren’t meeting performance expectations and focus on continually improving.