How To Scrape ‘People Also Asked’ Boxes With Screaming Frog

Recently whilst working on a keyword research project I was trying to find a way to get access to more questions and queries thrown up by Google within the ‘People Also Asked’ boxes.

Fortunately, after a little research and testing, I was able to find a way to do exactly that using Screaming Frog, a free Chrome extension and a pretty straightforward XPATH string.

This allowed me to list of a load of additional keywords that I could potentially use to optimise for featured snippets. So, this is how I scrape ‘People Also Asked’ boxes in organic search results.

How To Scrape People Also Asked Boxes

  • Setup Screaming Frog’s configuration correctly
  • Set the correct User-Agent
  • Create a template to refer to when scraping ‘People Also Asked’ boxes
  • Add in your preferred keyword search queries
  • Download the ‘Scraper’ Chrome extension
  • Find a SERP where ‘People Also Asked’ boxes appear
  • Identify the correct <div> class
  • Create an XPATH string to use when scraping
  • Add the XPATH string into Screaming Frog
  • Import the SERP URLs and export your ‘People Also Asked’ queries

Setup Screaming Frog’s configuration correctly

If you haven’t done it already, you’ll need to download the Screaming Frog SEO Spider. Unfortunately, you will need to have a licence key in order to use the configuration features required for this task.

Setting Up Screaming Frog

After you’ve downloaded and purchased a licence key you’ll need to update the crawlers configuration settings. To do this, go to Configuration > Speed, change the Max Threads to 1, ensure that the Limit URI/s box is checked and that the Max URI/s is set to 0.5.

Setting Up The Speed Configuration

Setting the correct User-Agent

Now that the speed configuration settings have been updated, we need to set the User-Agent to Chrome. To change the User-Agent go to:

Configuration > User-Agent using the top toolbar.

Update The User Agent

Using the dropdown menu, change the User-Agent from the default to Chrome.

Setting The User Agent

Create an Excel or Google Sheet template to refer to

Before you can start to scrape the ‘People Also Asked’ boxes you’ll need to get a list of search result URLs to paste into Screaming Frog.

I find that the best way to do this is to create a basic template using Excel or Google Sheets that contains the following information:

  • Google Address
  • Search Queries
  • URL Parameters
  • The concatenated URL

Excel Template Example

You’ll need to use the relevant Google search engine based on your location. For example, for me, the most relevant engine would be Google.co.uk or Google.com. You’ll also need to add the search parameter used to the end of the URL as you can see in the screenshot above.

Add in your preferred search queries

When you add the search queries, you can type them in using spaces, however, you will need to complete a Find & Replace to replace any spaces with an ‘+’.

Use the following formula in the Concatenated String column to combine all of the elements together to create a new URL to paste into Screaming Frog:

=CONCATENATE(A2,B2,C2) and hit Enter/Return.

Download the ‘Scraper’ Chrome extension

To get the information we need from the SERPs we need to scrape some information for the ‘People Also Asked’ boxes.

To do that, we need to download a free Chrome extension called ‘Scraper

Scraper Chrome Extension

Find a SERP where the ‘People Also Asked’ boxes appear

Before we can add the SERP URLs into Screaming Frog, we need to get some more information. Complete a search that produces the People Also Asked boxes. Most popular questions will do this.

Identify the <div> class by using the ‘Inspect’ feature in Chrome

After you’ve completed the search and a suggested question box has appeared, use the ‘Inspect Element’ feature in Chrome to view the code and identify the <div> class.

Inspect SERP

This is what we will use as part of the XPATH string to gather the data that we need.

Hover over one of the questions in the People Also Asked box and identify the <div> class:

DIV Class

Use the <div> class to create an XPATH string when scraping

Now that you know the class, we can create a basic XPATH string to use within the Scraper extension to pull the additional questions and queries within the SERP box feature.

To enter the XPATH string, right click one of the questions and select the ‘Scraper’ tool and put in the following XPATH string – //div[@class=”INSERT-DIV-CLASS-HERE”]:

Add XPATH String

You’ll need to check the <div> class before using the same one over and over again as they change frequently. Once you have added the class in, hit Scrape and you will find a list of queries that appear within the People Also Asked box.

Add the XPATH string into Screaming Frog

Now we know that the XPATH string works, we can add it to Screaming Frog and import our SERP URLs.

To add the XPATH string into Screaming Frog go to Configuration > Custom > Extraction:

Add XPATH String Screaming Frog

The default extraction name is ‘Extractor 1’. I’d recommend adding a personalised title to make it easier to identify when you export the report. You can then select the extraction type (XPATH) and paste the string into the input area.

Once you have added the extraction details, click OK.

Add your search result URLs from your template and crawl

Before you start a new crawl using the extraction details we have added, you need to copy and paste the search result URLs from your Excel or Google Sheets template.

You want to copy the concatenated URL. It should look something like this:

https://www.google.com/search?q=what+are+redirects&ion=0&num=10&source=Int&cr=countryGB

To paste multiple URLs into Screaming Frog you can go to:

Mode > List > Upload > Enter Manually OR Paste URLs

A Guide To Keyword Research For Beginner SEO’s

What is keyword research?

Keyword research is the process of researching and identifying specific words and phrases associated with a specific topic that people may use on a search engine such as Google or Bing and is considered a key SEO task.

The keyword research process is typically completed by website owners, bloggers and digital marketing professionals to help them optimise their websites. By optimising a website correctly and by targeting the right keywords, search engines such as Google will be able to understand the content on the pages better.

A well optimised page on a website has the potential to attract new visitors and traffic, as well as improve it’s organic ranking with a view to increasing online conversions in the long run.

However, the research process isn’t just about trying to find the keywords with the most search volume. If you think your going to start making money off the back of a research task that focuses on quantity, not quality, then you may want to have a quick re-think before going any further.

Why keyword research is so important?

A well structured keyword research document is hugely important for any website owner or digital marketing professional. But why?

Let me put it this way; you wouldn’t voluntarily choose to watch a movie with no sound would you? And the reason for that would be because you’d want to hear what’s being said so you know what’s happening – right?

Google and other search engines, see your website exactly like this. It needs to understand and hear what the content is about so it can crawl it, index it and rank it within the results pages.

So why is keyword research so important?

If you didn’t conduct any keyword research and decided to launch a website or post a new blog article then it would eventually get crawled and indexed. However, the search engines will have a hard time determining what your content is about.

Your keyword research is your route to understanding what potential customers and website visitors are looking for. Once you have the research documents in place, you’ll be in a much better position to optimise your website for search and conversions.

How to do keyword research for beginners

If you’re starting a keyword research project for the first time, you can use this method to help get you started. This is a keyword research process for beginners which can be easily tweaked as you see fit.

Not all research projects are the same so flexibility is key.

Step 1: Create a initial list of keywords that your website is about

Before you dive into researching tools, create your own list of keywords that you think that your websites products and services are best associated with. This is called a ‘seed keyword list’ and helps you in the very early stages of keyword research.

Here’s an example of some initial seed keywords that I may use for my website:

  • seo consultant
  • keyword research
  • google ads
  • getting to the top of google

The main thing about your seed list is quantity, not quality at this stage. The idea is to get as many ideas down on paper as possible, no matter how good or bad they are.

Here’s another example of some seed keywords that a local heating engineer might consider:

  • boilers
  • central heating
  • plumbing

Step 2: Brainstorm additional variants of your seed keywords

Now you have your seed list it’s time to start getting creative and think about what could be a good topic to optimise for or write about. Think about what modifiers you could add to the beginning or end of your seed keywords to create some new ones.

For example:

  • seo consultant in [location name]
  • keyword research tools
  • create better google ads
  • getting to the top of google without spending money

Or if we go back to our heating example:

  • boiler installations
  • central heating servicing
  • plumbing services in [location]

Step 3: Start researching using tools, open source platforms and customers

I’ll talk about specific keyword research tools a little later in this guide. However, for now I’ll be staying quite broad. You can use a selection of research tools to highlight potential queries and phrases that you may not necessarily have thought about before.

Something I like to do when completing keyword research is to take advantage of open source platforms such as Google Search Console to gather keyword data. This is a great way to find some additional terms, and the best bit?

These are keywords that are generating clicks to your site!

Google Search Console Step One

Go to Google Search Console, make sure that you’re using the new interface and select ‘Performance’

Google Search Console Step Two

If you want to get site wide data then you can skip this step and go straight to the next. If you want keyword data on specific pages, you can select the ‘Page’ filter and enter the specific URL that you want data for.

Google Search Console Step Three

In the old Search Console you were limited to 90 days of data. But in the new Search Console, you can now get up to 16-months of keyword data!

Google Search Console Step Four

Make sure that the ‘Queries’ tab is selected and scroll down to see your keywords. You can download these in a CSV by selecting the export option in the right corner.

Step 4: Gather search volumes and start prioritising your keywords

The easiest way to get search volume data for keywords is by using a free Chrome extension called ‘Keywords Everywhere’.

Simply add the extension and you can then upload up to 2000 unique keywords and get some awesome data such as search volume, average cost per click and the level of competition for that query.

Step 5: Compile your findings into an Excel spreadsheet to refer back to

There is nothing worse than finding some absolute gems and forgetting to write them down or store them somewhere (yes, it happens). I find that creating a simple Excel spreadsheet does this nicely.

Alternatively, you can use Google Sheets; either is absolutely fine.

Step 6: Review and go again!

Congratulations! You’ve completed your very first round of keyword research.

As a beginner, it will take some getting used to, but once you become familiar with some handy tools and best practices, keyword research will start to come naturally to you and you’ll be considering everything at every opportunity.

Remember to review your keywords on a regular basis. Search patterns change, as do volumes. You want to stay on top of your target audiences search habits and keep your site up to date.

How to use your keyword research

Now you have your keyword research it’s time to put it to use.

For beginners, the majority of your research will be used to optimise basic landing pages, product pages and blog posts. However, your research can provide a lot more than that.

You can use it to target specific searchers by using paid advertising services such as Google Ad’s (formally AdWords) and Bing Ad’s.

But the best way to use your keyword research data is to create a keyword map and optimise your site correctly. You’ll need to finalise a list of words you want to target and then ensure that they are in the following:

  • Titles
  • Meta descriptions
  • H1’s
  • URLs (where possible)
  • Images (file names and ALT text where possible)
  • Body content
  • Anchor text for internal linking

Keyword research tools for SEO

As well as open source platforms and brainstorming there are other ways to get keyword ideas, the most popular being keyword research tools.

Keyword research tools are pieces of software that help identify new keywords based around and initial ‘seed’ word or phrase. There are many tools out there to use, both paid and free.

Of course, generally speaking the more you pay for a keyword research tool, the more features you have access to. Although that doesn’t always mean to get access to the very best tools. Some of the best tools available for finding keywords are actually free.

Free keyword research tools

Free keyword research tools are great for beginner SEO’s when budget is an issue, and more often than not, that is always the case for those who are just starting out.

One of the best free tools for getting keyword ideas is Google’s Keyword Planner. You’ll need to sign up for a Google Ad’s account but you don’t need to spend any money to get access to the tool.

Once you have access, simply type in a selection of keywords to get suggestions and data directly from Google.

Keyword Planner Example

Other free keyword research tools include:

  • Google Search Console
  • Infinite Suggest
  • Answerthepublic
  • Google Trends
  • Keyword Shitter
  • Wordtracker
  • Keyword Finder
  • Ubersuggest
  • Search Volume.io

Paid keyword research tools

When you get to grips with keyword research you might want to consider a paid tool for finding new terms, and there are plenty out there.

My personal paid keyword research tool of choice is Moz’s Keyword Explorer.

It provides a ton of great metrics and keyword data, and even provides you with additional filtering options to identify keywords at different stages of a users journey.

Unfortunately, at the moment you can’t filter out location specific terms.

But that might change:

Here are some other paid keyword research tools that you might want to consider:

  • SEMrush
  • SEO Profiler
  • Ahrefs
  • SEO Powersuite

Different types of keywords and research

The type of keywords your looking for will vary, so understanding the variation of keywords and their potential meanings is important. There are predominantly, two main types of keywords.

Long tail and short tail.

These keywords also come with different types of keyword intent. The way I structure my keywords focuses heavily on the users intent, and what they might actually be looking for.

I tend to split my keywords into three main stages:

  • Awareness
  • Consideration
  • Decision

Long tail keyword research

Long tail keyword research is the process of identifying phrases used by people in search engines instead of single words. Up until recently, and until the introduction of content marketing a lot of businesses attempted to rank for very short, broad keywords.

The downside of that?

They were wasting their time. The beauty of long tail keywords, is that it gives you a greater opportunity to appear in organic search results as the query is far more detailed and specific.

So as long as you’ve done your keyword research correctly, and optimised your content, you stand a very good chance of getting some organic clicks to your site.

If you’re completing keyword research as a beginner, I would urge you to consider targeting long tail key phrases in the short term until your site becomes authoritative enough to compete with the larger websites.

An example of a good long tail keyword would be something that includes:

  • At least four words within the phrase
  • Has a search volume of at least 100
  • Is low in competition
  • And has an affordable estimated CPC (if you’re researching for ad’s)

How to do long tail keyword research

An easy way to find long tail keywords is to take advantage of Google.

Head over to Google.com and complete a search for a chosen term but don’t hit the return key or search just yet.

Select the end of the last word in your search bar and hit space to see what happens.

Google Search Example

Google with automatically suggest new keywords based around your initial search. Use the arrow keys to navigate down to a keyword that takes your fancy, allow it to show in the search bar and do exactly the same again.

You can keep repeating this process to help identify some new long tail phrases and terms.

Keyword research tips and best practices to take away

Hopefully if you’ve got to the end of this keyword research guide for beginners, you’ll have discovered some new ways to conduct keyword research, whether that be in an agency or as an in house digital marketer.

So to quickly summarise, here’s my advice for get started with keyword research for beginners and best practices for newbie SEO’s:

  • Make sure that you understand the importance for thorough keyword research
  • When starting a keyword research project begin by making a ‘seed’ list
  • Brainstorm around your ideas to create new phrases
  • Review your Google Search Console, Analytics and Ad’s data to find converting terms
  • Stay on top of your website and structure your content correctly with a keyword map
  • Review your keywords regularly to check for changes to habits and search volumes
  • Start with free keyword research tools until you’re comfortable with the process
  • If you’re considering a paid tool; Moz and SEMrush are your best options
  • Target long tail keywords to help increase your chances of ranking

You can find more information on my SEO and keyword research services here.