AI Crawler Traffic and Cloudflare 524 Errors on WordPress: robots.txt, Cache Rules and Rate Limits

In short: A Cloudflare 524 means the origin exceeded a Cloudflare timeout, usually the default 125-second read limit, and it does not identify which crawler asked. Match the 524s to crawler bursts in the origin log, keep robots.txt cached, and cache anonymous HTML before blocking AI crawlers by purpose.

A WordPress site behind Cloudflare starts showing error 524 pages, the access log is full of GPTBot, ClaudeBot and Meta-ExternalAgent, and the obvious conclusion is that AI crawlers are knocking the site over. Sometimes that is true. Often it is only half true, and the fix people reach for first, blocking every AI user agent in robots.txt, can remove a site from AI search answers while leaving the real problem in place.

A 524 tells you that your origin was too slow to answer. It does not tell you who asked. This guide works through the problem in the order that tends to fix it: confirm what the 524 means and who caused it, make sure robots.txt itself cannot fail, cache HTML at the edge so that most bursts are answered without PHP, and only then decide which crawlers to turn away and how to enforce that. Everything here can be done on Cloudflare's Free plan unless stated otherwise.

What does a Cloudflare 524 error mean?

Cloudflare defines error 524 precisely: Cloudflare connected to the origin, but the origin "did not provide an HTTP response before the default 125 seconds Proxy Read Timeout".^[1] Cloudflare's connection limits table gives the same 125 seconds, and only Enterprise zones can raise it.^[2] A 524 can also come from the 30-second Proxy Write Timeout, which cannot be changed at all.^[1]

That definition rules some things in and others out:

• Cloudflare reached your server. A 522 means Cloudflare could not complete a connection or got no acknowledgement in time, and its most common cause is a host or firewall blocking Cloudflare's IP addresses.^[3] A 524 is not a block. The connection worked, and the response never came.
• The slowness is at the origin. Cloudflare's own list of causes is a long-running process or an overloaded origin server.^[1]
• The request that failed is not necessarily the request that caused the problem. A human visitor's page view can time out because forty crawler requests are already queued ahead of it.

It also matters that the usual safety nets may not catch it. Cloudflare's stale-if-error handling is documented as triggering on 500, 502, 503 and 504 responses, and 524 is not on that list.^[4] Do not assume that a stale copy will paper over the problem; check it on your own zone.

Why can AI crawlers cause 524 errors on WordPress?

Three facts combine.

Cloudflare does not cache HTML by default. Its documentation says plainly that "The Cloudflare CDN does not cache HTML or JSON by default."^[5] Images, CSS and JavaScript are cached by file extension; WordPress pages are not. Unless you have added a rule, every page request a crawler makes is passed to your server.

Pages that miss both Cloudflare's cache and any origin page cache need PHP, and PHP has a fixed number of workers. On most WordPress hosts PHP runs under PHP-FPM, where pm.max_children "sets the limit on the number of simultaneous requests that will be served".^[6] Requests beyond that limit wait in a queue for a free process; the FPM status page reports this as the "listen queue".^[7] If a burst of crawler requests occupies every worker for long enough, the queue grows until requests at the back pass Cloudflare's timeout. WordPress's own performance guidance describes the same failure from the other side: without a caching solution, requests "come in and stack up".^[8] By default, WordPress also checks for due WP-Cron tasks on page load.^[9]

AI crawling is heavy and lopsided. Cloudflare's study of AI crawler traffic, published on 28 August 2025, found that training accounted for nearly 80% of crawling by AI bots. It also compared crawling with the visits each platform refers back: by default, Anthropic crawled nearly 50,000 pages for each referral, OpenAI 887 and Perplexity 118.^[10] Those ratios are network-wide averages for one period, not a forecast for any site, but they explain why site owners see so much crawling and so little traffic in return.

Crawlers also go where people rarely do. Search result URLs, tag and archive pagination, feeds and filtered product listings all render through PHP, and many of them are effectively unlimited. Google's guidance on faceted navigation warns that "crawling faceted URLs tends to cost sites large amounts of computing resources".^[11] WordPress marks search results noindex, but only after it has run the query and rendered the page, so the tag does nothing for load.

Are AI crawlers really causing your 524s?

The point of this step is to find out whether crawlers caused the 524s, or merely appeared in the same logs. The method is correlation, and it needs data from both sides of Cloudflare.

1. List the 524s. In the Cloudflare dashboard, filter the error analytics by edge status code 524 and note the times and paths. Cloudflare's error analytics are a 1% sample, so they show the pattern, not every request.^[12]
2. Line them up with the origin log. Take the access log for the same minutes and count requests by user agent and by path. You are looking for a burst of uncached requests that starts just before the 524s and ends when they end. If your host gives you the PHP-FPM status page, look at the listen queue and the "max children reached" counter for the same period.^[7]
3. Check that the crawler is who it says it is. A user agent string costs nothing to fake. Google documents a reverse DNS lookup followed by a forward lookup, and publishes its IP ranges as JSON.^[13] The other major operators publish IP lists, covered below. Traffic that claims to be a known crawler but comes from elsewhere is not that crawler, and it needs different handling.
4. Read the dashboard counts carefully. AI Crawl Control, Cloudflare's crawler dashboard, is available on all plans^[14] and shows allowed and unsuccessful requests per crawler, with a status code chart.^[15] "Unsuccessful" is broad. Cloudflare notes that unsuccessful requests "may come from any rule or response error, not just the block action in AI Crawl Control".^[16] Open the status code distribution before drawing a conclusion.

A field note shows why that last step matters. On one WordPress site behind Cloudflare, AI Crawl Control listed hundreds of unsuccessful AI crawler requests over a week. Every one was a 4xx response, mostly 403 and 429, and none was a 5xx. Those counts do not prove where the 403 and 429 responses came from; that would need request-level evidence from both Cloudflare and the origin. Our guide on finding which layer refused an AI crawler shows how to read that evidence. What the status chart does show is that none of them was a 524. A dashboard full of failed crawler requests can therefore mean a 524 problem, a host refusing crawlers, or your own rules working as intended, and only the status codes and the origin log tell them apart.

If the correlation is not there, keep looking at the origin: slow plugins, database queries and scheduled jobs produce 524s without any help from crawlers. If it is there, the remaining steps apply.

What happens when robots.txt times out?

Before changing who may crawl, make sure the file that tells them is always available, because robots.txt failures have an outsized effect.

The Robots Exclusion Protocol, RFC 9309, says that if robots.txt is unreachable because of a server error, the crawler "MUST assume complete disallow". A 4xx response is treated the other way: the crawler "MAY access any resources on the server".^[17] Google spells out its own version. For the first 12 hours it stops crawling the site while it keeps trying to fetch the file, then it uses the last good copy for up to 30 days, or assumes no restrictions if it has no copy. Timeouts count as server errors.^[18] So a robots.txt request that ends in a 524 can make a compliant crawler treat the whole site as off limits, whatever the file says.

On WordPress this is a real risk, because robots.txt is usually not a file. When there is no physical robots.txt in the web root, WordPress generates one on each request through do_robots(), which prints a default set of lines and lets plugins change them through the robots_txt filter.^[19] Yoast SEO, for example, edits that virtual file, and its help pages explain how to create a physical one instead.^[20] A virtual robots.txt sits behind the same PHP workers as everything else, so when the origin is saturated, robots.txt times out with it.

There are two ways out:

• Serve robots.txt from the edge. Cloudflare says that by default it caches a site's robots.txt.^[5] It will not cache a response that sets a cookie or sends Cache-Control values such as no-cache or private, which some plugins and hosts add, so request the file and check that the cf-cache-status response header says HIT. If it does not, a Cache Rule that matches /robots.txt and ignores the origin's cache headers will hold a copy at the edge.
• Use a physical file. A static robots.txt in the web root is served by the web server without starting WordPress. The trade-off is that plugins can no longer change it, so every change is made by hand.

Either way, the file should answer in milliseconds even while the rest of the site is struggling.

How do you cache WordPress pages at Cloudflare's edge?

Caching suitable HTML at the edge can take much of the crawler load off the origin, because a page served from Cloudflare's cache needs no response from PHP. It also helps every human visitor.

Cache Rules are available on the Free plan, which allows ten of them.^[21] A rule for WordPress has three parts.

What to cache. Set the rule's cache eligibility to "Eligible for cache" for anonymous GET requests to your pages. Cloudflare warns that caching all HTML can show visitors "information not intended for them", and suggests excluding requests by cookie or by path.^[22]

What to leave alone. Exclude:

• Requests that carry WordPress's logged-in cookie, wordpress_logged_in_ followed by a hash, and the commenter cookies that begin comment_author_.^[23] Also exclude the post-password cookie, which begins wp-postpass_, if you use password-protected posts.^[24]
• The admin area, the login page, wp-cron.php, previews and the REST API under /wp-json/.
• On WooCommerce, the cart, checkout and my account pages, requests that add an item to the cart through an add-to-cart parameter, API requests through wc-api, AJAX requests through wc-ajax, and requests carrying WooCommerce's session and cart cookies, such as woocommerce_items_in_cart, woocommerce_cart_hash and wp_woocommerce_session_.^[25][26] Use the page addresses your store actually has, which may differ from the defaults.

How long to keep it. The Edge TTL setting can follow the origin's Cache-Control header or ignore it and use a fixed time.^[27] Start with a short TTL, and expect published changes to take up to the TTL to appear unless you purge. If a purge does not show the change, a cache behind Cloudflare may still hold the old copy. You may also want visitors and crawlers to keep getting the old copy while Cloudflare fetches a new one after it expires. Cloudflare does this only when the origin's Cache-Control header includes stale-while-revalidate; without it, "visitors wait for the origin to respond before receiving content". Directives such as must-revalidate and s-maxage switch stale serving off, and so does turning off the Cache Rule setting Serve stale content while revalidating.^[28] Check for an UPDATING cache status after a copy expires before relying on it.

Cloudflare also collapses identical cache misses. When many requests for the same uncached URL arrive at one data centre at once, it sends only the first to the origin.^[5] That protects the origin from a burst on one page, but not from a crawler requesting thousands of different pages, which is why the TTL needs to be long enough for the cache to stay warm across your popular URLs.

If you would rather not build the rule yourself, Cloudflare's Automatic Platform Optimization for WordPress caches HTML at the edge and bypasses the cache for logged-in users and for WooCommerce and session cookies.^[29] It is a paid add-on on the Free plan and included on paid plans.^[30]

After deploying either, test it: request the same page several times from a logged-out browser and check for a HIT, then repeat while logged in and confirm the page is not cached.

Should you block GPTBot and other AI crawlers?

Once HTML is cached, crawler traffic is far cheaper, and the question becomes one of policy rather than survival. The most useful way to think about AI crawlers is by purpose, because the major operators now split their bots that way, with a separate robots.txt user agent for each.

Purpose	User agent or token	What blocking it does
Training and datasets	GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent	Asks that crawler not to collect your pages for those uses
Content-use controls	Google-Extended, Applebot-Extended	Google-Extended covers Gemini training and grounding; Applebot-Extended covers Apple model training. Neither stops the crawler that collects the pages
AI search	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Removes or reduces your site in that assistant's search answers
User-triggered fetch	ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher	Asks the assistant not to fetch your page for a user; several of these agents may ignore it

The sources behind each row:

• OpenAI treats its bots as independent. Blocking GPTBot signals that content should not be used for training; blocking OAI-SearchBot means the site "will not be shown in ChatGPT search answers", apart from navigational links. ChatGPT-User acts for users, so "robots.txt rules may not apply". OpenAI says robots.txt changes take about 24 hours to reach search.^[31]
• Anthropic describes ClaudeBot as collecting content that could contribute to training, Claude-SearchBot as improving search results, and Claude-User as fetching pages when people ask Claude a question. It says disabling either of the last two may reduce a site's visibility in user search.^[32]
• Google-Extended is not a crawler. Google calls it "a standalone product token" with no user agent of its own, used to manage whether content Google crawls may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI, and says it "does not impact a site's inclusion in Google Search".^[33] Applebot-Extended is the same idea: it "does not crawl webpages" and only controls how Apple uses what Applebot collects.^[34]
• Perplexity says PerplexityBot surfaces sites in its search and is not used for training, while Perplexity-User "generally ignores robots.txt rules".^[35] Meta says Meta-ExternalAgent crawls for uses such as training models, and that Meta-ExternalFetcher "may bypass robots.txt rules" because users request its fetches.^[36]

So the answer to "should I block GPTBot?" depends on what you are trying to achieve. Blocking it is a training opt-out. It does not remove the site from ChatGPT search, and it does not by itself reduce load from OpenAI's other bots. If load is the problem, the training crawlers are usually the cheapest to turn away, because training does not send visitors back. If visibility in AI answers matters, leave the search crawlers alone; the earlier iLogr article on getting a store into ChatGPT and AI Overviews covers which crawlers each surface depends on.

Two further cautions. First, a blanket group that disallows every AI user agent blocks the search crawlers too. Second, Crawl-delay is not part of RFC 9309.^[17] Some operators honour it anyway: Anthropic supports it as a "non-standard" extension^[32], and Common Crawl says CCBot obeys it.^[37] Others do not: Google does not support it^[18], Apple says Applebot "does not follow crawl-delay"^[34], and Amazon says its bots "do not support the crawl-delay directive".^[38] Treat Crawl-delay as a request to a few operators, not a throttle.

How do you block AI crawlers in Cloudflare?

robots.txt is a request. Cloudflare says as much about its own managed version: compliance "is voluntary".^[39] When a crawler ignores it, or claims to be a crawler it is not, enforcement has to happen at the edge.

Managed robots.txt. On every plan, Cloudflare can generate robots.txt entries that tell known AI training crawlers to stay away, and it adds them in front of your existing file.^[39] It is convenient, but it is still robots.txt.

Blocking named crawlers. AI Crawl Control lets you allow or block each listed crawler. On the Free plan it identifies them by their user agent strings, and a block creates or updates a WAF custom rule on the zone.^[16] Blocking any number of crawlers this way uses one custom rule, and the Free plan has five in total.^[40][41]

Blocking by purpose. Cloudflare's AI bot policies can block Training or Agent traffic on every page or only on pages that display ads, while allowing Search. Crawlers that combine Search and Training count as Training, so a training block can also stop a crawler that feeds a search product. The policies cover verified bots in those classes and some unverified bots that behave similarly. Its documentation says that from 15 September 2026 new domains get updated defaults: Training and Agent bots blocked on pages with ads, and Search allowed.^[42] If your zone predates that change, check what it is set to rather than assuming.

Blocking impostors. Cloudflare's verified bot programme confirms an operator's identity through a published IP list, reverse DNS or a cryptographic Web Bot Auth signature, and expects the bot to obey robots.txt and keep reasonable request rates.^[43] The rules field cf.client.bot is true for requests from a known good bot.^[44] A custom rule can block requests that claim a major crawler's user agent, such as Googlebot, GPTBot or ClaudeBot, but are not recognised as verified bots. Treat those requests as suspected impostors rather than proven ones: check a sample of their addresses against the operator's own verification method before blocking, and watch the rule for genuine traffic it catches. If you write your own allow or block lists instead, use the operators' published IP files: OpenAI publishes one per bot^[31]; Anthropic's crawler page now links a single list that covers ClaudeBot, Claude-SearchBot and Claude-User alike^[32]; Perplexity, Apple and Google publish theirs.^[35][34][13] Meta's crawler page publishes no IP list.^[36]

Bot Fight Mode. The Free plan's Bot Fight Mode challenges likely bots, but it cannot be customised, and Cloudflare says you "cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules".^[45] If you need exceptions, use custom rules instead.

Can Cloudflare's Free plan rate-limit AI crawlers?

A rate limit sounds like the natural answer to crawler bursts. On the Free plan it is a small tool. You get one rate limiting rule; it counts by IP address only, over a 10-second period, applies its action for 10 seconds, and can match only on path and verified bot status. Cached requests still count towards the limit.^[46] Counters are kept per data centre, not across Cloudflare's network.^[47]

That has consequences. A crawler that spreads its requests across many addresses and data centres may barely register against a per-IP limit. And because cached hits count, a limit tight enough to matter can block a reader browsing quickly. Cloudflare also warns that applying rate limits to verified bots "might affect Search Engine Optimization".^[46]

The best use of the single Free rule is an expensive endpoint with a path of its own, such as the login page. It cannot single out WordPress's site search, because searches use the home page address with an s query parameter, and a Free rule matches on path alone. Paid plans add query matching, more rules and longer periods.^[46]

When you do refuse a crawler because of load, the status code matters. Google treats 429 and 5xx responses as a sign of overload and slows down; it says 401 and 403 should not be used to limit crawl rate, because 4xx codes other than 429 have no effect on it.^[48] Even 429 or 503 is an emergency measure: Google advises against serving them for longer than one or two days, because URLs that keep returning them can be dropped from its index.^[49] Other operators document their own behaviour, if at all, so a refusal intended to slow a crawler may simply stop it.

When is the hosting plan the real fix?

Caching and a smaller crawler footprint reduce how many requests need PHP, but they cannot raise the number of PHP workers a plan allows.^[6] If requests still queue at ordinary traffic levels with anonymous HTML cached, compare FPM status samples over that period and look for slow PHP requests and database queries first, because the listen queue shows waiting requests but not why they wait. The "max children reached" figure only records that the limit was hit at some point.^[7] If normal visitor traffic still exhausts the workers after those checks, ask the host what its PHP worker limit is and whether it can be raised, then compare a plan with more workers, or a page cache on the server, against the traffic you actually need to serve. Blocking more crawlers will not fix capacity that visitors alone use up.

A short checklist

1. Confirm that the 524s line up with crawler bursts in the origin log, and check the status codes behind any "unsuccessful" crawler counts.
2. Make robots.txt a cached response or a physical file, so it answers even when PHP cannot.
3. Add a Cache Rule for anonymous HTML that excludes logged-in, commenter and WooCommerce cookies, the admin, login, cron, preview and REST paths, and WooCommerce's cart, checkout, account, add-to-cart, API and AJAX requests. If you want stale copies served during revalidation, check that the origin sends stale-while-revalidate.
4. Write robots.txt groups by purpose: training crawlers separately from search crawlers, and none of it relying on Crawl-delay.
5. Enforce at the edge with AI Crawl Control or the AI bot policies, plus a custom rule for suspected impostors that you check before trusting.
6. Keep the Free plan's rate limiting rule for one expensive endpoint with its own path, and do not use 403 to slow a search crawler.

The first three steps reduce crawler load on the origin and make any remaining failures easier to read. If 524s continue, look at the requests that still miss the cache, the paths you excluded and slow work at the origin before treating crawler policy as the answer.

References

Numbers in square brackets in the text link to these sources.

1. Cloudflare docs: Error 524 (opens in a new tab).Used for: what a 524 means and its 125 second default timeout.
2. Cloudflare docs: Connection limits (opens in a new tab).Used for: the timeout limits and which plans can change them.
3. Cloudflare docs: Error 522 (opens in a new tab).Used for: how a 522 differs from a 524.
4. Cloudflare docs: Origin Cache Control (opens in a new tab).Used for: which status codes trigger stale-if-error.
5. Cloudflare docs: Default cache behavior (opens in a new tab).Used for: HTML is not cached by default, and collapsing of identical cache misses.
6. PHP manual: FPM configuration (opens in a new tab).Used for: pm.max_children as the limit on simultaneous PHP requests.
7. PHP manual: FPM status page (opens in a new tab).Used for: the listen queue and max children reached counters.
8. WordPress Advanced Administration Handbook: Optimization (opens in a new tab).Used for: requests stacking up without a caching solution.
9. WordPress Plugin Handbook: Cron (opens in a new tab).Used for: WP-Cron checking for due tasks on page load.
10. Cloudflare blog: A deeper look at AI crawlers: breaking down traffic by purpose and industry (28 August 2025) (opens in a new tab).Used for: training share of crawling and crawl-to-referral ratios.
11. Google crawling documentation: Managing crawling of faceted navigation URLs (opens in a new tab).Used for: the compute cost of crawling faceted URLs.
12. Cloudflare docs: Cloudflare 5xx errors (opens in a new tab).Used for: error analytics being a sample.
13. Google crawling documentation: Verify requests from Google crawlers and fetchers (opens in a new tab).Used for: verifying Googlebot by reverse DNS and published IP ranges.
14. Cloudflare docs: AI Crawl Control (opens in a new tab).Used for: AI Crawl Control being available on all plans.
15. Cloudflare docs: AI Crawl Control, Analyze AI traffic (opens in a new tab).Used for: the per-crawler counts and status code chart.
16. Cloudflare docs: AI Crawl Control, Manage AI crawlers (opens in a new tab).Used for: what unsuccessful means, and how blocks create a WAF rule.
17. RFC 9309: Robots Exclusion Protocol (opens in a new tab).Used for: robots.txt failure semantics and Crawl-delay not being in the standard.
18. Google crawling documentation: How Google interprets the robots.txt specification (opens in a new tab).Used for: Google handling of robots.txt errors, timeouts and Crawl-delay.
19. WordPress code reference: do_robots() (opens in a new tab).Used for: WordPress generating robots.txt through PHP.
20. Yoast: The robots.txt file in Yoast SEO (opens in a new tab).Used for: creating a physical robots.txt that replaces the virtual file.
21. Cloudflare docs: Cache Rules (opens in a new tab).Used for: Free plan Cache Rules allowance.
22. Cloudflare docs: Cache Level (Cache Everything) example (opens in a new tab).Used for: the warning about caching all HTML.
23. WordPress Advanced Administration Handbook: Cookies (opens in a new tab).Used for: the logged-in and commenter cookie names.
24. WordPress source: wp-login.php (post-password cookie) (opens in a new tab).Used for: the post-password cookie name.
25. WooCommerce developer docs: How to configure caching plugins for WooCommerce (opens in a new tab).Used for: which WooCommerce pages and cookies to exclude from caching.
26. WooCommerce Developer Blog: Custom AJAX Endpoints in 2.4 (30 July 2015) (opens in a new tab).Used for: the wc-ajax endpoint.
27. Cloudflare docs: Cache Rules, available settings (opens in a new tab).Used for: Edge TTL and the origin header options.
28. Cloudflare docs: Revalidation (opens in a new tab).Used for: when stale content is served during revalidation.
29. Cloudflare docs: Automatic Platform Optimization, About (opens in a new tab).Used for: what APO caches and bypasses.
30. Cloudflare: Plans (opens in a new tab).Used for: APO being a paid add-on on the Free plan.
31. OpenAI: Overview of OpenAI crawlers (opens in a new tab).Used for: the roles of GPTBot, OAI-SearchBot and ChatGPT-User, and IP lists.
32. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (opens in a new tab).Used for: the roles of ClaudeBot, Claude-SearchBot and Claude-User.
33. Google crawling documentation: Google's common crawlers (opens in a new tab).Used for: Google-Extended as a standalone product token.
34. Apple: About Applebot (opens in a new tab).Used for: Applebot-Extended, and Applebot ignoring crawl-delay.
35. Perplexity: Perplexity Crawlers (opens in a new tab).Used for: PerplexityBot, Perplexity-User and their IP lists.
36. Meta: Meta Web Crawlers (opens in a new tab).Used for: Meta-ExternalAgent and Meta-ExternalFetcher behaviour.
37. Common Crawl: FAQ (opens in a new tab).Used for: CCBot obeying Crawl-delay.
38. Amazon: Amazonbot (opens in a new tab).Used for: Amazonbot not supporting crawl-delay.
39. Cloudflare docs: robots.txt setting (managed robots.txt) (opens in a new tab).Used for: managed robots.txt for AI crawlers.
40. Cloudflare docs: AI Crawl Control with Cloudflare WAF (opens in a new tab).Used for: one custom rule covering many blocked crawlers.
41. Cloudflare docs: WAF custom rules (opens in a new tab).Used for: the Free plan custom rule allowance.
42. Cloudflare docs: Block AI Bots (opens in a new tab).Used for: blocking AI bots by purpose, and the September 2026 defaults.
43. Cloudflare docs: Verified bots (opens in a new tab).Used for: how verified bots are confirmed.
44. Cloudflare Rules language: cf.client.bot (opens in a new tab).Used for: the cf.client.bot field.
45. Cloudflare docs: Bot Fight Mode (opens in a new tab).Used for: Bot Fight Mode not being customisable.
46. Cloudflare docs: Rate limiting rules (opens in a new tab).Used for: Free plan rate limiting limits and the SEO warning.
47. Cloudflare docs: Request rate calculation (opens in a new tab).Used for: counters being kept per data centre.
48. Google crawling documentation: HTTP status codes and Google's crawlers (opens in a new tab).Used for: how Google treats 429, 5xx and other 4xx codes.
49. Google crawling documentation: Reduce the Google crawl rate (opens in a new tab).Used for: limits on using 429 or 503 to slow Google.