A Screaming Frog content audit is the quickest way to stop guessing why your site feels “big” but performs small. You run one crawl, and suddenly the usual suspects show themselves: thin pages that do not pull their weight, duplicates that quietly compete with each other, and internal links that send both users and crawlers down the wrong paths.
When you do this well, you do not end up with a scary spreadsheet that nobody touches. You end up with a short list of decisions you can ship: which URLs deserve a refresh, which should be merged, which need redirects or canonicals, and which pages should be kept out of the index entirely. That is what we are going to build together.
Why a Screaming Frog content audit works when your site is too big to “just check it”
If you have ever tried to audit content by clicking around your own navigation, you know the feeling. You see the same 30 pages over and over, while the real problems hide in the corners: old blog tags, half-built pages, parameter URLs, and outdated promos that are still linked from somewhere.
Screaming Frog SEO Spider gives you a crawl that behaves like a search engine would, then organizes what it finds into views you can sort and export. It is one of those tools that feels simple until you realize how much it surfaces in one pass. Screaming Frog’s team lays out the bigger idea in its guide on mastering SEO audits with Screaming Frog, and it matches what we see with Nina customers: the crawl is what turns “I think” into “I know.”
One quick note on limits: the free version caps at 500 URLs. If you are auditing anything beyond a small brochure site, the paid license makes your life easier because it removes the cap and unlocks features like JavaScript rendering and near-duplicate detection.
Pre-crawl setup for a Screaming Frog content audit you can trust
Most audit mistakes happen before the crawl even starts. The crawler only reports what it can see, so you want to set it up to match how your site actually loads.
- Use a sensible user agent: pick a standard crawler user agent so your results are consistent run to run.
- Enable JavaScript rendering if your pages load key content client-side. If your headings and body copy “appear” after the initial HTML, you want the crawler to see that.
- Turn on “Store HTML” so near-duplicate analysis has enough data to work with.
- Set boundaries: block staging sites, infinite calendars, and spammy parameter patterns unless you are intentionally auditing them.
Cadence-wise, keep it practical. If you ship content weekly or have thousands of URLs changing often, crawl weekly. If your site changes slowly, monthly is usually plenty. The point is to catch drift before it becomes a cleanup project.
Screaming Frog content audit workflow: find thin content without falling into the “just add words” trap
Thin content is rarely about length. It is about usefulness. A 150-word page can be perfect if it answers a simple question. A 1,500-word page can still be thin if it says the same thing as three other URLs on your site.
Inside Screaming Frog, go to the Content area and look for Low Content Pages. Then sort by Word Count so the thinnest pages float to the top. Treat that list as a set of candidates, not a list of pages you automatically rewrite.
We recommend giving each thin page a destination before anyone opens a doc.
| Thin page pattern | What it usually looks like | What you do next |
|---|---|---|
| High-intent but underbuilt | A service subpage with vague copy and no proof | Expand it with examples, FAQs, outcomes, and clear next steps |
| Overlapping topic | Two posts that answer the same query in slightly different ways | Merge into one stronger URL and redirect the weaker one |
| Utility or system page | Internal search results, login screens, thank-you pages | Noindex, and keep it out of main navigation |
| Outdated or irrelevant | An old promo page that no longer matches what you sell | Remove it and redirect if there is a close replacement |
If you are staring at a long list of thin pages and wondering how to turn it into a real plan, clusters help. You pick a few pages that should be your “home base” for a topic, then you build supporting content that answers narrower questions and links back intentionally. We walk through that approach in SEO content strategy: from service page to topic cluster, and it pairs nicely with a crawl because you can spot where topics are fragmented across too many half-pages.
Screaming Frog content audit workflow: spot duplicates that split signals and waste crawl time
Duplicate content is one of those issues that teams underestimate because it is rarely obvious on the surface. The duplication often comes from URL variants, tags, filters, pagination, and CMS habits. Google’s own documentation is clear that consolidating duplicate URLs helps search engines understand which version should be treated as the main one, which is exactly what you are trying to accomplish in an audit.
In Screaming Frog, you will typically tackle duplicates in two passes: exact duplicates first, then near duplicates.
Duplicate content audit in Screaming Frog: fix exact duplicates first
Exact duplicates are the easy wins. They are often the same content accessible through multiple URL versions, like trailing slash variants, parameter versions, or duplicated paths created by tags and category archives.
In Screaming Frog, open the Content section and filter for Exact Duplicates. Then make a call page by page:
- Choose the primary URL that should be indexed and treated as the “real” page.
- Use a canonical when multiple versions need to exist but you want one preferred version for indexing.
- Use a 301 redirect when the duplicate should not exist for users at all.
- Noindex utility pages that add no unique value and are only creating noise.
While you are here, export canonical reports and scan for contradictions. A common mess is a page that is canonicalized to one URL, but internal links point to another, or the canonical points to a URL that is not indexable. Those are the kinds of small misalignments that add up on large sites.
Duplicate content audit in Screaming Frog: use near-duplicate checks to find pages that feel the same
Near duplicates show up when pages share a template, a format, or a strategy that went a bit too far. Think location pages, tag archives, e-commerce categories, and “we serve X city” pages that only swap out a place name.
To use near-duplicate detection, you will need to enable it before crawling and make sure you stored HTML. After the crawl, run the analysis and start with a high similarity threshold, like 90%, to catch the most obvious cases. Lower it if you want to uncover softer overlaps.
The key is judgment. Similar does not always mean wrong. Sometimes it just means you have a templated section that needs a unique layer: examples, pricing context, FAQs, or a clearer angle that matches the query. Other times, two pages are truly competing, and consolidation is the cleanest move.
Screaming Frog content audit workflow: internal link fixes that actually change outcomes
Content does not rank in isolation. Internal links are how you show relationships, priority, and paths through the site. They also decide whether a page is discoverable at all.
Here are the internal link issues we see most often, and how you pull them from Screaming Frog:
- Broken internal links: go to Response Codes and filter for 4xx client errors. Use the Inlinks tab to see which pages are linking to the broken URL so you can update the links or add the right redirect.
- Redirect chains: export redirect chains and prioritize anything with three or more hops. Chains are a time sink for crawlers and a drag on user flow.
- Nofollow internal links: review follow status so you do not accidentally cut off important pages from internal signals.
- Orphan pages: connect your crawl with XML sitemaps and Search Console exports to find URLs that exist but have no internal links pointing to them.
If you want a clean order of operations, use this sequence:
- Fix broken internal links.
- Remove redirect chains.
- Address orphan pages.
- Improve anchor text and reduce unnecessary depth for key pages.
You will notice this is less about “SEO tricks” and more about housekeeping that keeps your site easy to crawl and easy to navigate. That is the point.
Turn your Screaming Frog content audit exports into a backlog your team can ship
Audit data gets ignored when it feels like homework. The fastest way to make it real is to turn exports into a short backlog with owners, a decision label, and a definition of done.
- Export three buckets: thin pages, duplicates, and internal link issues.
- Add one decision column: Improve, Merge, Redirect, Canonicalize, Noindex, Remove, or Re-link.
- Prioritize with context: start with pages that already get impressions, sit close to revenue, or are linked from high-authority sections of your site.
- Batch by type: group technical fixes into one sprint, then content work into another so you do not context-switch every hour.
If you are thinking, “Cool, but we do not have the bandwidth,” you are not alone. That is where we fit. Nina helps you turn those decisions into drafts that are ready to review: structured, linked, sourced, and formatted for publishing. If you want to see the workflow end to end, start with how Nina works.
How you can use audit data with Nina to scale updates without losing your voice
Screaming Frog tells you what is happening on the site. Nina helps you do something about it, consistently.
- Thin pages become refresh briefs: you pick the outcome, we generate a draft built around intent, missing sections, and the internal links that should support it.
- Near duplicates become consolidation maps: you decide the winning URL, then you fold the best parts of the overlapping pages into one publishable update and redirect the rest.
- Internal link issues become an SOP: you set simple rules for how new posts link to pillars, service pages, and supporting articles so the site gets stronger as it grows.
If you want to plug this into your publishing cadence, you can start at Nina’s homepage and see what it looks like in practice.
FAQ: Screaming Frog content audits
How often should you run a Screaming Frog content audit?
If your site changes constantly, weekly crawls help you catch regressions early. If your site is steady, monthly is usually enough. What matters is picking a cadence you will actually stick to.
What word count counts as thin content?
There is no magic number. Screaming Frog flags low word count as a signal, but “thin” really means the page does not satisfy intent or it offers nothing unique compared to other URLs.
What is the difference between exact duplicates and near duplicates?
Exact duplicates are identical content. Near duplicates are highly similar pages that share most of the same text or structure. Near duplicates are common with templated pages and often need differentiation or consolidation.
How do you fix duplicates without creating ranking chaos?
Pick a primary URL, keep it indexable, and make internal links consistent. Use 301 redirects when the duplicate should not exist. Use canonicals when variants need to exist. Use noindex for utility pages that do not need to appear in search.
Which internal link issues matter first?
Broken internal links and redirect chains are the best first fixes because they waste crawl resources and frustrate users. Then focus on orphan pages and depth so your important pages are easy to discover.
Conclusion
A Screaming Frog content audit gives you a grounded view of what is holding your site back: thin pages that need a real purpose, duplicates that compete for attention, and internal link issues that quietly weaken discovery. Run the crawl with the right settings, turn the exports into a decision backlog, and you will have a plan you can execute instead of a report you regret. When you are ready to move faster on the content side without losing human review and brand voice, Nina is built to help you ship those updates consistently.
