Skip to content
Cloudflare API
Start here
API Reference
AI Audit
Robots

Bulk get robots.txt rules

ai_audit.robots.bulk_get(RobotBulkGetParams**kwargs) -> RobotBulkGetResponse
POST/zones/{zone_id}/ai-audit/robots/bulk

Fetches and parses robots.txt files for multiple domains within a zone in a single request. Each domain must belong to the specified zone. Results are keyed by hostname.

Security
API Token

The preferred authorization scheme for interacting with the Cloudflare API. Create a token.

Example:Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY
API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

Example:X-Auth-Email: user@example.com

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

Example:X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194
Accepted Permissions (at least one required)
Zone Settings WriteZone Settings Read
ParametersExpand Collapse
zone_id: str
body: Sequence[str]

Array of domain hostnames to fetch robots.txt for. Each domain must end with the zone name. Maximum 25 domains per request.

ReturnsExpand Collapse
Dict[str, RobotBulkGetResponseItem]

Map of hostname to parsed robots.txt rules.

user_agents: Dict[str, RobotBulkGetResponseItemUserAgents]

Map of user-agent string to its parsed rules.

allow: List[str]

List of allowed path patterns.

disallow: List[str]

List of disallowed path patterns.

content_signals: Optional[RobotBulkGetResponseItemUserAgentsContentSignals]

Content signal directives from robots.txt.

ai_input: Optional[Literal["yes", "no"]]

Whether AI input usage is permitted.

One of the following:
"yes"
"no"
ai_train: Optional[Literal["yes", "no"]]

Whether AI training is permitted.

One of the following:
"yes"
"no"
One of the following:
crawl_delay: Optional[float]

Crawl delay in seconds.

sitemaps: Optional[List[str]]

List of sitemap URLs found in robots.txt.

status: Optional[int]

HTTP status code from fetching the robots.txt file.

Bulk get robots.txt rules

import os
from cloudflare import Cloudflare

client = Cloudflare(
    api_token=os.environ.get("CLOUDFLARE_API_TOKEN"),  # This is the default and can be omitted
)
response = client.ai_audit.robots.bulk_get(
    zone_id="zone_id",
    body=["example.com", "blog.example.com"],
)
print(response)
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}
Returns Examples
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}