# Robots

## Get robots.txt rules

`client.AIAudit.Robots.Get(ctx, params) (*RobotGetResponse, error)`

**get** `/zones/{zone_id}/ai-audit/robots`

Fetches and parses the robots.txt file for a zone or a specific subdomain within the zone. Returns parsed user-agent rules, content signals, and sitemaps.

### Parameters

- `params RobotGetParams`

  - `ZoneID param.Field[string]`

    Path param: Identifier of the zone.

  - `Subdomain param.Field[string]`

    Query param: Optional subdomain to fetch robots.txt for. If omitted, fetches robots.txt for the zone apex domain.

### Returns

- `type RobotGetResponse struct{…}`

  Parsed robots.txt rules for a single domain.

  - `UserAgents map[string, RobotGetResponseUserAgent]`

    Map of user-agent string to its parsed rules.

    - `Allow []string`

      List of allowed path patterns.

    - `Disallow []string`

      List of disallowed path patterns.

    - `ContentSignals RobotGetResponseUserAgentsContentSignals`

      Content signal directives from robots.txt.

      - `AIInput RobotGetResponseUserAgentsContentSignalsAIInput`

        Whether AI input usage is permitted.

        - `const RobotGetResponseUserAgentsContentSignalsAIInputYes RobotGetResponseUserAgentsContentSignalsAIInput = "yes"`

        - `const RobotGetResponseUserAgentsContentSignalsAIInputNo RobotGetResponseUserAgentsContentSignalsAIInput = "no"`

      - `AITrain RobotGetResponseUserAgentsContentSignalsAITrain`

        Whether AI training is permitted.

        - `const RobotGetResponseUserAgentsContentSignalsAITrainYes RobotGetResponseUserAgentsContentSignalsAITrain = "yes"`

        - `const RobotGetResponseUserAgentsContentSignalsAITrainNo RobotGetResponseUserAgentsContentSignalsAITrain = "no"`

      - `Search RobotGetResponseUserAgentsContentSignalsSearch`

        Whether search indexing is permitted.

        - `const RobotGetResponseUserAgentsContentSignalsSearchYes RobotGetResponseUserAgentsContentSignalsSearch = "yes"`

        - `const RobotGetResponseUserAgentsContentSignalsSearchNo RobotGetResponseUserAgentsContentSignalsSearch = "no"`

    - `CrawlDelay float64`

      Crawl delay in seconds.

  - `Sitemaps []string`

    List of sitemap URLs found in robots.txt.

  - `Status int64`

    HTTP status code from fetching the robots.txt file.

### Example

```go
package main

import (
  "context"
  "fmt"

  "github.com/cloudflare/cloudflare-go"
  "github.com/cloudflare/cloudflare-go/ai_audit"
  "github.com/cloudflare/cloudflare-go/option"
)

func main() {
  client := cloudflare.NewClient(
    option.WithAPIToken("Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY"),
  )
  robot, err := client.AIAudit.Robots.Get(context.TODO(), ai_audit.RobotGetParams{
    ZoneID: cloudflare.F("zone_id"),
  })
  if err != nil {
    panic(err.Error())
  }
  fmt.Printf("%+v\n", robot.UserAgents)
}
```

#### Response

```json
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}
```

## Bulk get robots.txt rules

`client.AIAudit.Robots.BulkGet(ctx, params) (*RobotBulkGetResponse, error)`

**post** `/zones/{zone_id}/ai-audit/robots/bulk`

Fetches and parses robots.txt files for multiple domains within a zone in a single request. Each domain must belong to the specified zone. Results are keyed by hostname.

### Parameters

- `params RobotBulkGetParams`

  - `ZoneID param.Field[string]`

    Path param: Identifier of the zone.

  - `Body param.Field[[]string]`

    Body param: Array of domain hostnames to fetch robots.txt for. Each domain must end with the zone name. Maximum 25 domains per request.

### Returns

- `type RobotBulkGetResponse map[string, RobotBulkGetResponseItem]`

  Map of hostname to parsed robots.txt rules.

  - `UserAgents map[string, RobotBulkGetResponseItemUserAgent]`

    Map of user-agent string to its parsed rules.

    - `Allow []string`

      List of allowed path patterns.

    - `Disallow []string`

      List of disallowed path patterns.

    - `ContentSignals RobotBulkGetResponseItemUserAgentsContentSignals`

      Content signal directives from robots.txt.

      - `AIInput RobotBulkGetResponseItemUserAgentsContentSignalsAIInput`

        Whether AI input usage is permitted.

        - `const RobotBulkGetResponseItemUserAgentsContentSignalsAIInputYes RobotBulkGetResponseItemUserAgentsContentSignalsAIInput = "yes"`

        - `const RobotBulkGetResponseItemUserAgentsContentSignalsAIInputNo RobotBulkGetResponseItemUserAgentsContentSignalsAIInput = "no"`

      - `AITrain RobotBulkGetResponseItemUserAgentsContentSignalsAITrain`

        Whether AI training is permitted.

        - `const RobotBulkGetResponseItemUserAgentsContentSignalsAITrainYes RobotBulkGetResponseItemUserAgentsContentSignalsAITrain = "yes"`

        - `const RobotBulkGetResponseItemUserAgentsContentSignalsAITrainNo RobotBulkGetResponseItemUserAgentsContentSignalsAITrain = "no"`

      - `Search RobotBulkGetResponseItemUserAgentsContentSignalsSearch`

        Whether search indexing is permitted.

        - `const RobotBulkGetResponseItemUserAgentsContentSignalsSearchYes RobotBulkGetResponseItemUserAgentsContentSignalsSearch = "yes"`

        - `const RobotBulkGetResponseItemUserAgentsContentSignalsSearchNo RobotBulkGetResponseItemUserAgentsContentSignalsSearch = "no"`

    - `CrawlDelay float64`

      Crawl delay in seconds.

  - `Sitemaps []string`

    List of sitemap URLs found in robots.txt.

  - `Status int64`

    HTTP status code from fetching the robots.txt file.

### Example

```go
package main

import (
  "context"
  "fmt"

  "github.com/cloudflare/cloudflare-go"
  "github.com/cloudflare/cloudflare-go/ai_audit"
  "github.com/cloudflare/cloudflare-go/option"
)

func main() {
  client := cloudflare.NewClient(
    option.WithAPIToken("Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY"),
  )
  response, err := client.AIAudit.Robots.BulkGet(context.TODO(), ai_audit.RobotBulkGetParams{
    ZoneID: cloudflare.F("zone_id"),
    Body: []string{"example.com", "blog.example.com"},
  })
  if err != nil {
    panic(err.Error())
  }
  fmt.Printf("%+v\n", response)
}
```

#### Response

```json
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}
```