How to Find All Existing and Archived URLs on a web site

How to Find All Existing and Archived URLs on a web site

Blog Article

There are lots of motives you might have to have to search out each of the URLs on an internet site, but your actual aim will determine That which you’re trying to find. For instance, you may want to:

Recognize each and every indexed URL to analyze difficulties like cannibalization or index bloat
Accumulate recent and historic URLs Google has viewed, specifically for web-site migrations
Come across all 404 URLs to Get better from article-migration glitches
In Just about every circumstance, just one tool received’t Present you with every little thing you will need. Sadly, Google Research Console isn’t exhaustive, as well as a “web site:case in point.com” lookup is proscribed and hard to extract knowledge from.

Within this article, I’ll stroll you through some instruments to build your URL list and ahead of deduplicating the information utilizing a spreadsheet or Jupyter Notebook, based upon your website’s size.

Old sitemaps and crawl exports
Should you’re seeking URLs that disappeared within the Are living web site just lately, there’s a chance someone with your staff could have saved a sitemap file or maybe a crawl export prior to the adjustments were designed. Should you haven’t now, check for these documents; they could often supply what you require. But, if you’re reading through this, you most likely didn't get so lucky.

Archive.org
Archive.org
Archive.org is an invaluable Device for Web optimization jobs, funded by donations. When you try to find a site and select the “URLs” option, you'll be able to accessibility approximately ten,000 shown URLs.

On the other hand, there are a few limits:

URL Restrict: You may only retrieve around web designer kuala lumpur 10,000 URLs, which happens to be inadequate for bigger internet sites.
Top quality: Quite a few URLs may very well be malformed or reference source information (e.g., illustrations or photos or scripts).
No export solution: There isn’t a crafted-in solution to export the checklist.
To bypass The dearth of the export button, use a browser scraping plugin like Dataminer.io. However, these restrictions signify Archive.org may well not provide a complete Resolution for bigger internet sites. Also, Archive.org doesn’t point out no matter whether Google indexed a URL—but when Archive.org observed it, there’s a very good chance Google did, too.

Moz Pro
Though you may perhaps generally make use of a backlink index to search out exterior web pages linking to you personally, these applications also uncover URLs on your website in the process.


How to utilize it:
Export your inbound one-way links in Moz Professional to acquire a speedy and easy list of concentrate on URLs out of your site. For those who’re handling a large Web page, think about using the Moz API to export information outside of what’s manageable in Excel or Google Sheets.

It’s crucial that you Notice that Moz Pro doesn’t verify if URLs are indexed or identified by Google. Even so, due to the fact most internet sites utilize a similar robots.txt policies to Moz’s bots since they do to Google’s, this method normally works properly for a proxy for Googlebot’s discoverability.

Google Search Console
Google Search Console gives many precious resources for developing your listing of URLs.

Inbound links stories:


Much like Moz Professional, the Hyperlinks part supplies exportable lists of target URLs. Sad to say, these exports are capped at one,000 URLs Each and every. You can use filters for distinct web pages, but since filters don’t apply for the export, you would possibly should trust in browser scraping equipment—limited to five hundred filtered URLs at any given time. Not best.

Functionality → Search engine results:


This export will give you an index of webpages obtaining research impressions. While the export is limited, You need to use Google Look for Console API for larger datasets. You will also find free Google Sheets plugins that simplify pulling more in depth data.

Indexing → Web pages report:


This segment presents exports filtered by concern form, although these are also restricted in scope.

Google Analytics
Google Analytics
The Engagement → Webpages and Screens default report in GA4 is an excellent resource for amassing URLs, using a generous limit of a hundred,000 URLs.


Better yet, you may use filters to make distinctive URL lists, proficiently surpassing the 100k limit. Such as, if you need to export only weblog URLs, observe these measures:

Action one: Add a phase towards the report

Stage 2: Simply click “Develop a new phase.”


Action 3: Define the phase with a narrower URL pattern, including URLs made up of /blog/


Take note: URLs present in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they offer useful insights.

Server log data files
Server or CDN log information are Potentially the ultimate Instrument at your disposal. These logs seize an exhaustive list of every URL path queried by consumers, Googlebot, or other bots in the recorded period.

Considerations:

Data size: Log documents could be enormous, a lot of websites only retain the final two months of data.
Complexity: Analyzing log information is often difficult, but a variety of instruments are offered to simplify the process.
Combine, and great luck
As soon as you’ve gathered URLs from each one of these sources, it’s time to mix them. If your internet site is small enough, use Excel or, for greater datasets, equipment like Google Sheets or Jupyter Notebook. Make sure all URLs are continually formatted, then deduplicate the list.

And voilà—you now have a comprehensive list of existing, outdated, and archived URLs. Great luck!