Skip to content
Cloudflare API
Start here
API Reference
AI Audit

Robots

Get robots.txt rules
ai_audit.robots.get(RobotGetParams**kwargs) -> RobotGetResponse
GET/zones/{zone_id}/ai-audit/robots
Bulk get robots.txt rules
ai_audit.robots.bulk_get(RobotBulkGetParams**kwargs) -> RobotBulkGetResponse
POST/zones/{zone_id}/ai-audit/robots/bulk
ModelsExpand Collapse
class RobotGetResponse: …

Parsed robots.txt rules for a single domain.

user_agents: Dict[str, UserAgents]

Map of user-agent string to its parsed rules.

allow: List[str]

List of allowed path patterns.

disallow: List[str]

List of disallowed path patterns.

content_signals: Optional[UserAgentsContentSignals]

Content signal directives from robots.txt.

ai_input: Optional[Literal["yes", "no"]]

Whether AI input usage is permitted.

One of the following:
"yes"
"no"
ai_train: Optional[Literal["yes", "no"]]

Whether AI training is permitted.

One of the following:
"yes"
"no"
One of the following:
crawl_delay: Optional[float]

Crawl delay in seconds.

sitemaps: Optional[List[str]]

List of sitemap URLs found in robots.txt.

status: Optional[int]

HTTP status code from fetching the robots.txt file.

Dict[str, RobotBulkGetResponseItem]

Map of hostname to parsed robots.txt rules.

user_agents: Dict[str, RobotBulkGetResponseItemUserAgents]

Map of user-agent string to its parsed rules.

allow: List[str]

List of allowed path patterns.

disallow: List[str]

List of disallowed path patterns.

content_signals: Optional[RobotBulkGetResponseItemUserAgentsContentSignals]

Content signal directives from robots.txt.

ai_input: Optional[Literal["yes", "no"]]

Whether AI input usage is permitted.

One of the following:
"yes"
"no"
ai_train: Optional[Literal["yes", "no"]]

Whether AI training is permitted.

One of the following:
"yes"
"no"
One of the following:
crawl_delay: Optional[float]

Crawl delay in seconds.

sitemaps: Optional[List[str]]

List of sitemap URLs found in robots.txt.

status: Optional[int]

HTTP status code from fetching the robots.txt file.