How to Crawl a Sitemap in Screaming Frog

You’ve heard it a hundred times — “your sitemap is the roadmap to your website.” But here’s the thing most people skip: auditing that roadmap regularly is the difference between ranking and getting buried on page 5.

Screaming Frog is the go-to SEO tool for crawling websites and auditing XML sitemaps. Over 97% of professional SEO audits involve some form of crawl analysis, and Screaming Frog is the most widely used desktop crawler in the industry.

The problem? A lot of people install it and never figure out how to crawl a sitemap specifically. They run a basic crawl, skim the tabs, and call it a day. This guide fixes that.

In the next few minutes, you’ll know exactly how to crawl a sitemap in Screaming Frog — step by step — so you can spot broken URLs, orphan pages, and indexation issues before they tank your rankings.

Let’s get into it.

How to Crawl a Sitemap in Screaming Frog

There are two ways to crawl a sitemap in Screaming Frog. Both are useful. Knowing when to use which one is what separates a surface-level audit from a deep, actionable SEO analysis.

Method 1: List Mode (Crawl Just the Sitemap URLs)

This is the fastest way to audit a specific XML sitemap. You’re uploading the sitemap directly and letting Screaming Frog crawl only those URLs — nothing extra.

When to use it: You want to audit a specific sitemap in isolation. Great for checking status codes, redirect chains, and indexability of sitemap URLs quickly.

Step-by-step:

Step 1 — Switch to List Mode Open Screaming Frog. In the top navigation bar, click Mode → List. This tells the tool you’re providing the URLs to crawl, not asking it to spider the whole site.

Step 2 — Upload Your XML Sitemap Click Upload in the toolbar. You’ll see two options:

• Upload a file — if you have a saved .xml file locally
• Download XML Sitemap — if your sitemap is live

Choose Download XML Sitemap, paste in your sitemap URL (e.g., https://yourdomain.com/sitemap.xml), and click OK. If you have a sitemap index file, this single step will pull in all child sitemaps automatically — which is a massive time saver.

Step 3 — Start the Crawl Hit the green Start button. Screaming Frog will begin crawling every URL listed in your sitemap. Depending on your sitemap size, this can take anywhere from seconds to a few minutes.

Step 4 — Analyse the Results Once the crawl hits 100%, check:

• Internal tab → filter by status code (look for 3xx redirects, 4xx errors, 5xx server errors)
• Response Codes tab → sort by status to find broken pages
• Sitemaps tab → review URL coverage

Pro tip: Export the results to a CSV or Excel file via Export in the top menu. This makes it easy to share findings with your dev team or client.

Method 2: Spider Mode with Sitemap Configuration (Full Site Crawl + Sitemap Comparison)

This is the more powerful method. You crawl the entire site and your sitemap at the same time — then Screaming Frog compares both to surface issues that Method 1 can’t catch.

When to use it: You want to find orphan URLs (pages in the sitemap but not linked internally), or URLs that exist on the site but are missing from the sitemap. This is a paid feature (requires Screaming Frog licence — £209/year as of 2024).

Step-by-step:

Step 1 — Configure Sitemap Crawling Go to Configuration → Spider → Crawl → XML Sitemaps. Tick the box that says “Crawl Linked XML Sitemaps.” In the “Crawl These Sitemaps” field, paste your sitemap URL. If you don’t know it offhand, check your robots.txt file at https://yourdomain.com/robots.txt — it’s almost always listed there.

Step 2 — Start the Full Site Crawl Go back to the main screen. Enter your root domain URL in the “Enter URL to spider” field and click Start. Screaming Frog will crawl the site and pull your sitemap simultaneously.

Step 3 — Run Post-Crawl Analysis Here’s where most people miss a step. Once the crawl finishes, five of the seven sitemap filters require post-crawl analysis to populate. You’ll see a “Crawl Analysis Required” message in the overview panel.

To run it: Go to Crawl Analysis → Start. Wait for the progress bar to hit 100%. Now all your sitemap filters are live.

Step 4 — Review the Sitemaps Tab The Sitemaps tab has 7 filters to help you identify issues:

• URLs In Sitemap — all URLs found in your XML sitemap
• URLs Not In Sitemap — pages discovered during the crawl that are missing from your sitemap
• Non-Indexable URLs In Sitemap — pages in your sitemap that are blocked, noindexed, or canonicalised elsewhere (these shouldn’t be in your sitemap)
• Redirects In Sitemap — 3xx URLs that should be updated to their final destination
• Non-200 URLs In Sitemap — broken or error pages incorrectly included in the sitemap
• Orphan URLs — URLs in the sitemap with no internal links pointing to them
• Missing From Sitemap — crawled pages that should be in the sitemap but aren’t

Step 5 — Export and Prioritise Use Bulk Export → Sitemaps to export any filter as a CSV. This gives you a clean, actionable list of every sitemap issue, ready to prioritise and hand off.

What to Look For in Your Sitemap Audit

Once you’ve run either method, here are the key issues worth fixing immediately:

Broken URLs (4xx or 5xx) These are dead pages still listed in your sitemap. Google will attempt to crawl and index them, which wastes crawl budget and can trigger indexation issues in Google Search Console. Remove them or fix the underlying pages.

Redirects in the Sitemap If your sitemap lists URLs that redirect to another page, that’s a wasted hop. Update your sitemap to point directly to the final destination URL. Studies show that redirect chains can reduce PageRank flow by up to 15%.

Non-Indexable URLs Pages with a noindex tag or a canonical pointing elsewhere have no business being in your sitemap. Including them confuses crawlers and dilutes your sitemap quality.

Orphan URLs These are pages in your sitemap that have zero internal links pointing to them. Google technically can find them via the sitemap, but without internal links, they carry minimal authority. Fix this by linking to them from relevant pages.

Missing Pages Important pages that exist on the site but aren’t in the sitemap won’t get the crawl priority they deserve. Audit these and add them manually or update your CMS sitemap settings.

Quick Reference: Free vs Paid Screaming Frog

If your site has fewer than 500 pages and you only need basic sitemap checks, the free version handles it. For anything beyond that — especially client audits or large e-commerce sites — the paid licence pays for itself in the first audit.

Common Mistakes to Avoid

Not running Crawl Analysis after a Spider Mode crawl Five of the seven sitemap filters won’t populate without it. Don’t skip this step or you’ll have an incomplete picture.

Assuming your sitemap is already clean A study by Ahrefs found that over 53% of websites have URLs in their sitemap that return a non-200 status code. Don’t assume — always verify.

Crawling too fast on a live site Screaming Frog’s default crawl speed is 5 requests per second. On a smaller server, this can cause slowdowns. Reduce it under Configuration → Speed if you’re auditing a production site.

Ignoring orphan URLs Pages with zero internal links earn significantly less PageRank, regardless of whether they’re in the sitemap. Always check the orphan URLs filter and build internal links to important pages.

Conclusion

Crawling a sitemap in Screaming Frog is one of the highest-leverage things you can do in an SEO audit. It takes less than 10 minutes and immediately surfaces broken pages, indexation issues, redirect chains, and orphan URLs that could be costing you rankings right now.

Here’s the quick recap:

• Use List Mode to crawl a sitemap in isolation — fast, clean, and works on the free version
• Use Spider Mode with sitemap configuration for a full comparison of crawl vs. sitemap — requires a paid licence
• Always run post-crawl analysis after a Spider Mode crawl to populate all seven sitemap filters
• Fix broken URLs, remove non-indexable pages, and address orphan URLs first — these have the highest SEO impact

Once your site health is dialled in, the next step is making sure you’re getting the right traffic. Clean technical SEO creates the foundation — but outbound pipeline fills the calendar.