How To Scrape ‘People Also Asked’ Boxes With Screaming Frog

Recently whilst working on a keyword research project I was trying to find a way to get access to more questions and queries thrown up by Google within the ‘People Also Asked’ boxes.

Fortunately, after a little research and testing, I was able to find a way to do exactly that using Screaming Frog, a free Chrome extension and a pretty straightforward XPATH string.

This allowed me to list of a load of additional keywords that I could potentially use to optimise for featured snippets. So, this is how I scrape ‘People Also Asked’ boxes in organic search results.

How To Scrape People Also Asked Boxes

  • Setup Screaming Frog’s configuration correctly
  • Set the correct User-Agent
  • Create a template to refer to when scraping ‘People Also Asked’ boxes
  • Add in your preferred keyword search queries
  • Download the ‘Scraper’ Chrome extension
  • Find a SERP where ‘People Also Asked’ boxes appear
  • Identify the correct <div> class
  • Create an XPATH string to use when scraping
  • Add the XPATH string into Screaming Frog
  • Import the SERP URLs and export your ‘People Also Asked’ queries

Setup Screaming Frog’s configuration correctly

If you haven’t done it already, you’ll need to download the Screaming Frog SEO Spider. Unfortunately, you will need to have a licence key in order to use the configuration features required for this task.

Setting Up Screaming Frog

After you’ve downloaded and purchased a licence key you’ll need to update the crawlers configuration settings. To do this, go to Configuration > Speed, change the Max Threads to 1, ensure that the Limit URI/s box is checked and that the Max URI/s is set to 0.5.

Setting Up The Speed Configuration

Setting the correct User-Agent

Now that the speed configuration settings have been updated, we need to set the User-Agent to Chrome. To change the User-Agent go to:

Configuration > User-Agent using the top toolbar.

Update The User Agent

Using the dropdown menu, change the User-Agent from the default to Chrome.

Setting The User Agent

Create an Excel or Google Sheet template to refer to

Before you can start to scrape the ‘People Also Asked’ boxes you’ll need to get a list of search result URLs to paste into Screaming Frog.

I find that the best way to do this is to create a basic template using Excel or Google Sheets that contains the following information:

  • Google Address
  • Search Queries
  • URL Parameters
  • The concatenated URL

Excel Template Example

You’ll need to use the relevant Google search engine based on your location. For example, for me, the most relevant engine would be Google.co.uk or Google.com. You’ll also need to add the search parameter used to the end of the URL as you can see in the screenshot above.

Add in your preferred search queries

When you add the search queries, you can type them in using spaces, however, you will need to complete a Find & Replace to replace any spaces with an ‘+’.

Use the following formula in the Concatenated String column to combine all of the elements together to create a new URL to paste into Screaming Frog:

=CONCATENATE(A2,B2,C2) and hit Enter/Return.

Download the ‘Scraper’ Chrome extension

To get the information we need from the SERPs we need to scrape some information for the ‘People Also Asked’ boxes.

To do that, we need to download a free Chrome extension called ‘Scraper

Scraper Chrome Extension

Find a SERP where the ‘People Also Asked’ boxes appear

Before we can add the SERP URLs into Screaming Frog, we need to get some more information. Complete a search that produces the People Also Asked boxes. Most popular questions will do this.

Identify the <div> class by using the ‘Inspect’ feature in Chrome

After you’ve completed the search and a suggested question box has appeared, use the ‘Inspect Element’ feature in Chrome to view the code and identify the <div> class.

Inspect SERP

This is what we will use as part of the XPATH string to gather the data that we need.

Hover over one of the questions in the People Also Asked box and identify the <div> class:

DIV Class

Use the <div> class to create an XPATH string when scraping

Now that you know the class, we can create a basic XPATH string to use within the Scraper extension to pull the additional questions and queries within the SERP box feature.

To enter the XPATH string, right click one of the questions and select the ‘Scraper’ tool and put in the following XPATH string – //div[@class=”INSERT-DIV-CLASS-HERE”]:

Add XPATH String

You’ll need to check the <div> class before using the same one over and over again as they change frequently. Once you have added the class in, hit Scrape and you will find a list of queries that appear within the People Also Asked box.

Add the XPATH string into Screaming Frog

Now we know that the XPATH string works, we can add it to Screaming Frog and import our SERP URLs.

To add the XPATH string into Screaming Frog go to Configuration > Custom > Extraction:

Add XPATH String Screaming Frog

The default extraction name is ‘Extractor 1’. I’d recommend adding a personalised title to make it easier to identify when you export the report. You can then select the extraction type (XPATH) and paste the string into the input area.

Once you have added the extraction details, click OK.

Add your search result URLs from your template and crawl

Before you start a new crawl using the extraction details we have added, you need to copy and paste the search result URLs from your Excel or Google Sheets template.

You want to copy the concatenated URL. It should look something like this:

https://www.google.com/search?q=what+are+redirects&ion=0&num=10&source=Int&cr=countryGB

To paste multiple URLs into Screaming Frog you can go to:

Mode > List > Upload > Enter Manually OR Paste URLs