> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anysite.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Batch Processing

> Process thousands of inputs in parallel with rate limiting and error handling

## Overview

Batch processing lets you run the same API endpoint against multiple inputs — from a file or stdin — with parallel execution, rate limiting, and configurable error handling.

## Basic Syntax

```bash theme={null}
anysite api <endpoint> --from-file <file> --input-key <key> [OPTIONS]
```

## Input Sources

### From File

Provide a list of inputs in a text file (one per line):

```bash theme={null}
# users.txt
satyanadella
jeffweiner08
billgates
```

```bash theme={null}
anysite api /api/linkedin/user --from-file users.txt --input-key user
```

### From CSV/JSON Files

Batch from structured files — the CLI auto-detects the format:

```bash theme={null}
# From CSV (uses specified column)
anysite api /api/linkedin/company --from-file companies.csv --input-key company

# From JSONL (uses specified field)
anysite api /api/linkedin/user --from-file users.jsonl --input-key user
```

### From Stdin

Pipe data directly from other commands:

```bash theme={null}
cat urls.txt | anysite api /api/linkedin/user --input-key user --stdin

# Chain with other CLI commands
anysite api /api/linkedin/search/users keywords="CTO" count=50 -q --format jsonl | \
  jq -r '.urn.value' | \
  anysite api /api/linkedin/user --input-key user --stdin
```

## Parallel Execution

Control the number of concurrent workers:

```bash theme={null}
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5
```

<Warning>
  Higher parallelism increases throughput but also API token consumption. Start with 3-5 workers and adjust based on your plan limits.
</Warning>

## Rate Limiting

Set a maximum request rate to stay within limits:

```bash theme={null}
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --rate-limit "10/s"
```

Rate limit formats:

* `10/s` — 10 requests per second
* `100/m` — 100 requests per minute

## Error Handling

Configure behavior when individual requests fail:

```bash theme={null}
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --on-error skip
```

| Strategy | Behavior                                         |
| -------- | ------------------------------------------------ |
| `stop`   | Stop the entire batch on first error (default)   |
| `skip`   | Skip the failed input and continue with the rest |
| `retry`  | Retry failed requests with exponential backoff   |

## Progress and Statistics

### Progress Tracking

Show a real-time progress bar:

```bash theme={null}
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --progress
```

### Batch Statistics

Display summary statistics after completion:

```bash theme={null}
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --stats
```

Output includes total processed, succeeded, failed, and elapsed time.

## Output Options

All [output format options](/cli/api-calls#output-formats) work with batch processing:

```bash theme={null}
# Save batch results to CSV
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --format csv --output results.csv

# JSONL for streaming
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --format jsonl --output results.jsonl

# Quiet mode for piping
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 -q --format jsonl | anysite db insert mydb --table profiles --stdin
```

## Complete Example

Enrich a list of LinkedIn profiles with parallel processing, rate limiting, and error recovery:

```bash theme={null}
anysite api /api/linkedin/user \
  --from-file linkedin_urls.txt \
  --input-key user \
  --parallel 5 \
  --rate-limit "10/s" \
  --on-error skip \
  --progress \
  --stats \
  --format csv \
  --output enriched_profiles.csv
```

## Options Reference

| Option         | Description                                               |
| -------------- | --------------------------------------------------------- |
| `--from-file`  | Input file path (TXT, CSV, JSON, JSONL)                   |
| `--input-key`  | Parameter name to map each input value to                 |
| `--stdin`      | Read inputs from stdin instead of file                    |
| `--parallel`   | Number of concurrent workers (default: 1)                 |
| `--rate-limit` | Maximum request rate (e.g., `10/s`, `100/m`)              |
| `--on-error`   | Error handling: `stop`, `skip`, `retry` (default: `stop`) |
| `--progress`   | Show real-time progress bar                               |
| `--stats`      | Display batch statistics after completion                 |

## Next Steps

<CardGroup cols={2}>
  <Card title="Dataset Pipelines" icon="diagram-project" href="/cli/datasets/overview">
    Build multi-source workflows with dependency chains and scheduling
  </Card>

  <Card title="Database Loading" icon="database" href="/cli/database/operations">
    Load batch results directly into databases
  </Card>
</CardGroup>