Crawling seo tools
robots.txt, sitemaps, and canonical URLs.
Robots.txt Generator
Compose a robots.txt with allow/disallow and sitemap lines.
Crawling
Sitemap Generator
Turn a list of URLs into a sitemap.xml snippet.
Crawling
Canonical URL Generator
Normalize a URL and output a rel=canonical tag.
Crawling
HTTP Status Code Checker
Look up status codes. This tool does not fetch remote URLs.
Crawling