Skip to content
Cloudflare API
Start here
API Reference

AI Audit

AI AuditRobots

Get robots.txt rules
GET/zones/{zone_id}/ai-audit/robots
Bulk get robots.txt rules
POST/zones/{zone_id}/ai-audit/robots/bulk
ModelsExpand Collapse
RobotGetResponse object { userAgents, sitemaps, status }

Parsed robots.txt rules for a single domain.

userAgents: map[object { allow, disallow, contentSignals, crawlDelay } ]

Map of user-agent string to its parsed rules.

allow: array of string

List of allowed path patterns.

disallow: array of string

List of disallowed path patterns.

contentSignals: optional object { "ai-input", "ai-train", search }

Content signal directives from robots.txt.

"ai-input": optional "yes" or "no"

Whether AI input usage is permitted.

One of the following:
"yes"
"no"
"ai-train": optional "yes" or "no"

Whether AI training is permitted.

One of the following:
"yes"
"no"
One of the following:
crawlDelay: optional number

Crawl delay in seconds.

sitemaps: optional array of string

List of sitemap URLs found in robots.txt.

status: optional number

HTTP status code from fetching the robots.txt file.

RobotBulkGetResponse = map[object { userAgents, sitemaps, status } ]

Map of hostname to parsed robots.txt rules.

userAgents: map[object { allow, disallow, contentSignals, crawlDelay } ]

Map of user-agent string to its parsed rules.

allow: array of string

List of allowed path patterns.

disallow: array of string

List of disallowed path patterns.

contentSignals: optional object { "ai-input", "ai-train", search }

Content signal directives from robots.txt.

"ai-input": optional "yes" or "no"

Whether AI input usage is permitted.

One of the following:
"yes"
"no"
"ai-train": optional "yes" or "no"

Whether AI training is permitted.

One of the following:
"yes"
"no"
One of the following:
crawlDelay: optional number

Crawl delay in seconds.

sitemaps: optional array of string

List of sitemap URLs found in robots.txt.

status: optional number

HTTP status code from fetching the robots.txt file.