TL;DR: Using Robots.txt Effectively
Remember that the longest matching rule takes priority
In cases where both Allow and Disallow rules apply, the longest rule wins.
Use wildcards wisely
The * wildcard is automatically applied to rules unless specified otherwise
Create separate rules for subdomains
Each subdomain requires its own robots.txt file. Rules do not automatically apply across subdomains
Use absolute Sitemap URLs
Relative paths won’t be respected by search engines.
Pay attention to case sensitivity
Robots.txt rules are case-sensitive, so make sure to include all necessary variations (e.g., /path and /Path).
Match encoded URLs correctly
Keep your robots.txt file unencoded to avoid issues.
Repeat general user-agent directives for specific crawlers
When defining rules for specific bots, repeat general user-agent rules. This ensures that the more specific bots don’t miss important directives.
Test your rules with a robots.txt validator
Always verify your rules using a robots.txt tester (available in Google Search Console and other SEO tools).
Understanding the Role of a Robots.txt
The robots.txt file prevents crawlers from accessing certain pages/scripts/resource files that don’t need to be indexed.
For example, files like /ads-beacon.js or /location.json in Searchenginejournal com are likely related to advertisements and bidding mechanisms.
Crawlers do not need to index these files, so blocking them helps reduce the load on the server and keeps crawlers focused on important pages. For a more comprehensive approach to optimizing your website’s crawlability and ensuring essential pages are prioritized, consider investing in a technical SEO audit service to identify and resolve issues.
(!) Important — don’t rely on robots.txt to hide pages from crawlers. Even if you block a page from being crawled, search bot may still index the page if other sites link to it with descriptive text. Instead, use methods like noindex tags or password protection if you want to prevent indexing completely.
Robots.txt specification
The robots.txt file must be located in the TOP-level directory of your site and be accessible through supported protocols.
Correct URL: https://example.com/robots.txt
Incorrect URL: https://example.com/folder/robots.txt
When Google's crawlers request a robots.txt file, the server's HTTP response status code affects how the file is processed:
- 2xx (OK). The file is processed as received.
- 3xx (Redirection). Crawler follows up to five redirects, after which it treats the response as a 404 if the rules couldn't be retrieved.
- 4xx (Client Errors). Treated as though no robots.txt file exists, except for a 429 status code.
- 5xx (Server Errors). Temporarily, crawler assumes the site is disallowed. If the file is unreachable for over 30 days, the last cached version is used, or no crawl restrictions are assumed.
Google caches the contents of the robots.txt file for up to 24 hours, potentially longer if the file remains unreachable. The cache can be shared among different crawlers.Valid lines in a robots.txt file consist of a field, a colon, and a value. Comments can be added using the # character. Google supports the following fields:
User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Sitemap: https://yourwebsite.com/sitemap.xml
- User-agent — the crawler to which the rules apply.
- Allow — paths that may be crawled.
- Disallow — paths that must not be accessed.
- Sitemap — URL of a sitemap
Google accesses web page resources similarly to a browser. Blocking certain CSS or JavaScript files can affect how search engines view the page’s layout, potentially resulting in the page not being deemed mobile-friendly or causing important content to be overlooked, especially on sites that heavily depend on JavaScript. Each rule blocks or allows access for all or a specific crawler to a specified file path on the domain or subdomain where the robots.txt file is hosted. The default behaviour is that user agents are allowed to crawl the entire site.
Sounds complicated?
Hire SEO consultant to help you
Get a Consultation
To wrap up, using robots.txt effectively requires a good understanding of its syntax, rules, and limitations. Take the time to craft robots.txt rules that suit your site’s needs, ensuring crawlers can access the right content while avoiding pages and resources that shouldn't be indexed. Testing your robots.txt file is crucial, so make sure to validate your rules using tools to avoid unintended access or blocking.
