# Robots.txt Configuration

The `RobotsController` is responsible for generating the `/robots.txt` file dynamically for the application. It
automatically appends the Sitemap URLs based on the configured domains and locales.

Additionally, it allows for flexible and extensible configuration of `User-agent`, `Allow`, and `Disallow` rules through
route parameters.

## Default Behavior

By default, if no custom configuration is provided, the controller generates a standard permissive `robots.txt` file:

```text
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
...
```

## Customizing Rules

You can customize the rules by overriding the route definition in your project's routing configuration (usually
`config/routes.yaml`).

The controller accepts a `robotsBlocks` array parameter. Each item in this array represents a block in the `robots.txt`
file and can contain:

- `user_agents`: An array of User-Agent strings.
- `rules`: An array of rule strings (e.g., `Disallow: /path`, `Allow: /path`, `Crawl-delay: 10`).

### Example: Disallowing PDF files

To disallow crawling of PDF files for all user agents, add the following configuration to your `config/routes.yaml`:

```yaml
aa_seo_robots_txt:
    path: /robots.txt
    controller: 'AutoAdmin\Bundle\Seo\Controller\RobotsController::index'
    defaults:
        robotsBlocks:
            -   user_agents: [ '*' ]
                rules:
                    - 'Disallow: /*.pdf'
```

### Example: Multiple User-Agents and Complex Rules

You can define multiple blocks to target specific bots:

```yaml
aa_seo_robots_txt:
    path: /robots.txt
    controller: 'AutoAdmin\Bundle\Seo\Controller\RobotsController::index'
    defaults:
        robotsBlocks:
            # Block for all bots
            -   user_agents: [ '*' ]
                rules:
                    - 'Disallow: /admin/'
                    - 'Disallow: /*.pdf'

            # Specific block for Googlebot
            -   user_agents: [ 'Googlebot' ]
                rules:
                    - 'Allow: /public/images/'
                    - 'Disallow: /private/'
```

This configuration will generate the following `robots.txt` structure before appending the Sitemaps:

```text
User-agent: *
Disallow: /admin/
Disallow: /*.pdf

User-agent: Googlebot
Allow: /public/images/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
...
```
