# AI Audit # Robots ## Get robots.txt rules `ai_audit.robots.get(RobotGetParams**kwargs) -> RobotGetResponse` **get** `/zones/{zone_id}/ai-audit/robots` Fetches and parses the robots.txt file for a zone or a specific subdomain within the zone. Returns parsed user-agent rules, content signals, and sitemaps. ### Parameters - `zone_id: str` - `subdomain: Optional[str]` Optional subdomain to fetch robots.txt for. If omitted, fetches robots.txt for the zone apex domain. ### Returns - `class RobotGetResponse: …` Parsed robots.txt rules for a single domain. - `user_agents: Dict[str, UserAgents]` Map of user-agent string to its parsed rules. - `allow: List[str]` List of allowed path patterns. - `disallow: List[str]` List of disallowed path patterns. - `content_signals: Optional[UserAgentsContentSignals]` Content signal directives from robots.txt. - `ai_input: Optional[Literal["yes", "no"]]` Whether AI input usage is permitted. - `"yes"` - `"no"` - `ai_train: Optional[Literal["yes", "no"]]` Whether AI training is permitted. - `"yes"` - `"no"` - `search: Optional[Literal["yes", "no"]]` Whether search indexing is permitted. - `"yes"` - `"no"` - `crawl_delay: Optional[float]` Crawl delay in seconds. - `sitemaps: Optional[List[str]]` List of sitemap URLs found in robots.txt. - `status: Optional[int]` HTTP status code from fetching the robots.txt file. ### Example ```python import os from cloudflare import Cloudflare client = Cloudflare( api_token=os.environ.get("CLOUDFLARE_API_TOKEN"), # This is the default and can be omitted ) robot = client.ai_audit.robots.get( zone_id="zone_id", ) print(robot.user_agents) ``` #### Response ```json { "errors": [], "messages": [ { "code": 1000, "message": "message", "documentation_url": "documentation_url", "source": { "pointer": "pointer" } } ], "success": true, "result": { "userAgents": { "foo": { "allow": [ "string" ], "disallow": [ "string" ], "contentSignals": { "ai-input": "yes", "ai-train": "yes", "search": "yes" }, "crawlDelay": 0 } }, "sitemaps": [ "string" ], "status": 0 } } ``` ## Bulk get robots.txt rules `ai_audit.robots.bulk_get(RobotBulkGetParams**kwargs) -> RobotBulkGetResponse` **post** `/zones/{zone_id}/ai-audit/robots/bulk` Fetches and parses robots.txt files for multiple domains within a zone in a single request. Each domain must belong to the specified zone. Results are keyed by hostname. ### Parameters - `zone_id: str` - `body: Sequence[str]` Array of domain hostnames to fetch robots.txt for. Each domain must end with the zone name. Maximum 25 domains per request. ### Returns - `Dict[str, RobotBulkGetResponseItem]` Map of hostname to parsed robots.txt rules. - `user_agents: Dict[str, RobotBulkGetResponseItemUserAgents]` Map of user-agent string to its parsed rules. - `allow: List[str]` List of allowed path patterns. - `disallow: List[str]` List of disallowed path patterns. - `content_signals: Optional[RobotBulkGetResponseItemUserAgentsContentSignals]` Content signal directives from robots.txt. - `ai_input: Optional[Literal["yes", "no"]]` Whether AI input usage is permitted. - `"yes"` - `"no"` - `ai_train: Optional[Literal["yes", "no"]]` Whether AI training is permitted. - `"yes"` - `"no"` - `search: Optional[Literal["yes", "no"]]` Whether search indexing is permitted. - `"yes"` - `"no"` - `crawl_delay: Optional[float]` Crawl delay in seconds. - `sitemaps: Optional[List[str]]` List of sitemap URLs found in robots.txt. - `status: Optional[int]` HTTP status code from fetching the robots.txt file. ### Example ```python import os from cloudflare import Cloudflare client = Cloudflare( api_token=os.environ.get("CLOUDFLARE_API_TOKEN"), # This is the default and can be omitted ) response = client.ai_audit.robots.bulk_get( zone_id="zone_id", body=["example.com", "blog.example.com"], ) print(response) ``` #### Response ```json { "errors": [], "messages": [ { "code": 1000, "message": "message", "documentation_url": "documentation_url", "source": { "pointer": "pointer" } } ], "success": true, "result": { "foo": { "userAgents": { "foo": { "allow": [ "string" ], "disallow": [ "string" ], "contentSignals": { "ai-input": "yes", "ai-train": "yes", "search": "yes" }, "crawlDelay": 0 } }, "sitemaps": [ "string" ], "status": 0 } } } ``` ## Domain Types ### Robot Get Response - `class RobotGetResponse: …` Parsed robots.txt rules for a single domain. - `user_agents: Dict[str, UserAgents]` Map of user-agent string to its parsed rules. - `allow: List[str]` List of allowed path patterns. - `disallow: List[str]` List of disallowed path patterns. - `content_signals: Optional[UserAgentsContentSignals]` Content signal directives from robots.txt. - `ai_input: Optional[Literal["yes", "no"]]` Whether AI input usage is permitted. - `"yes"` - `"no"` - `ai_train: Optional[Literal["yes", "no"]]` Whether AI training is permitted. - `"yes"` - `"no"` - `search: Optional[Literal["yes", "no"]]` Whether search indexing is permitted. - `"yes"` - `"no"` - `crawl_delay: Optional[float]` Crawl delay in seconds. - `sitemaps: Optional[List[str]]` List of sitemap URLs found in robots.txt. - `status: Optional[int]` HTTP status code from fetching the robots.txt file. ### Robot Bulk Get Response - `Dict[str, RobotBulkGetResponseItem]` Map of hostname to parsed robots.txt rules. - `user_agents: Dict[str, RobotBulkGetResponseItemUserAgents]` Map of user-agent string to its parsed rules. - `allow: List[str]` List of allowed path patterns. - `disallow: List[str]` List of disallowed path patterns. - `content_signals: Optional[RobotBulkGetResponseItemUserAgentsContentSignals]` Content signal directives from robots.txt. - `ai_input: Optional[Literal["yes", "no"]]` Whether AI input usage is permitted. - `"yes"` - `"no"` - `ai_train: Optional[Literal["yes", "no"]]` Whether AI training is permitted. - `"yes"` - `"no"` - `search: Optional[Literal["yes", "no"]]` Whether search indexing is permitted. - `"yes"` - `"no"` - `crawl_delay: Optional[float]` Crawl delay in seconds. - `sitemaps: Optional[List[str]]` List of sitemap URLs found in robots.txt. - `status: Optional[int]` HTTP status code from fetching the robots.txt file.