> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Speech to Text setup

> Configure the Speech to Text service for audio transcription and text-to-speech capabilities.

The Speech to Text service is a Python-based microservice that provides audio transcription using OpenAI Whisper (local or API) and text-to-speech via the OpenAI TTS API. It supports both synchronous REST calls and asynchronous Kafka job processing for long-running audio files.

## Dependencies

The Speech to Text service connects to standard FlowX infrastructure services that should already be configured in your environment:

* [**Kafka**](/5.9/setup-guides/kafka-authentication-config) — async job processing for transcription requests from Integration Designer, plus subscription to the AI Providers cache invalidation topic
* **S3-compatible storage** (MinIO or AWS S3) — audio file retrieval and result storage
* **organization-manager** — resolves OpenAI credentials and model assignments from the AUDIO capability configured in [AI Providers](/5.9/ai-platform/ai-providers)
* **OpenAI API key** (fallback) — used for local development, MCP usage, or when no tenant context is available on a request
* **ffmpeg** — required for audio decoding (included in Docker image)

<Info>
  The local Whisper transcription engine runs on-device and does not require any OpenAI credentials. Only cloud transcription (`/transcribe/openai`) and TTS (`/tts/openai`) need credentials, which are resolved from the organization's AUDIO capability when tenant headers are present on the request.
</Info>

***

## Capabilities

### What it can do

* **Local transcription**: On-device transcription using any Whisper model (`base`, `small`, `medium`, `large`, `turbo`) with automatic language detection
* **Cloud transcription**: OpenAI Whisper API with automatic chunking of long audio into 10-minute segments
* **Text-to-speech**: OpenAI TTS API with multiple models (`tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`) and voice options
* **Segment-level timing**: Returns word-level segments with start/end timestamps
* **Language detection**: Automatic source language detection with confidence probabilities
* **Long audio support**: Kafka-based async processing for files over 15 minutes

### Supported audio formats

| Format | Extensions     |
| ------ | -------------- |
| MP3    | `.mp3`         |
| WAV    | `.wav`         |
| FLAC   | `.flac`        |
| AAC    | `.aac`, `.m4a` |
| OGG    | `.ogg`         |

***

## API endpoints

All endpoints are prefixed with the `URL_PREFIX` value (default: `/speech-to-text`).

| Endpoint                                   | Method | Purpose                                    |
| ------------------------------------------ | ------ | ------------------------------------------ |
| `/api/v1/speech-to-text/transcribe`        | POST   | Transcribe audio using local Whisper model |
| `/api/v1/speech-to-text/transcribe/openai` | POST   | Transcribe audio using OpenAI Whisper API  |
| `/api/v1/speech-to-text/tts/openai`        | POST   | Text-to-speech using OpenAI TTS API        |
| `/api/v1/speech-to-text/info/health`       | GET    | Health check                               |

***

## Configuration

### Server configuration

| Environment Variable | Description                                 | Default Value     |
| -------------------- | ------------------------------------------- | ----------------- |
| `URL_PREFIX`         | FastAPI root path for reverse proxy routing | `/speech-to-text` |
| `PORT`               | API port                                    | `8000`            |
| `GUNICORN_WORKERS`   | Number of Gunicorn worker processes         | `2`               |
| `GUNICORN_TIMEOUT`   | Worker timeout in seconds                   | `120`             |

<Tip>
  Increase `GUNICORN_TIMEOUT` for long audio files. The default `120` seconds works for most files, but transcriptions of 30+ minute recordings may need `600` or higher.
</Tip>

***

### Whisper model configuration

| Environment Variable   | Description                                                                                                                                                    | Default Value |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `WHISPER_MODEL`        | Local Whisper model name                                                                                                                                       | `turbo`       |
| `OPENAI_API_KEY`       | OpenAI API key — fallback for local dev, MCP, or requests without tenant context. In production, credentials resolve from the organization's AUDIO capability. | -             |
| `OPENAI_WHISPER_MODEL` | Fallback model for OpenAI cloud transcription when no tenant context is present or the resolver returns no model                                               | `whisper-1`   |

<Info>
  The primary source for OpenAI credentials and the transcription model is the **AUDIO capability** configured under **Organization Settings** → **AI Settings** → **Defaults & Fallbacks**. The env vars above are used only when tenant context is missing or AUDIO is not configured for the organization. See [AI providers and model configuration](/5.9/ai-platform/ai-providers).
</Info>

Available local Whisper models:

| Model    | Size    | Speed   | Accuracy | Use case                                            |
| -------- | ------- | ------- | -------- | --------------------------------------------------- |
| `base`   | 74 MB   | Fastest | Low      | Quick prototyping, non-critical transcription       |
| `small`  | 244 MB  | Fast    | Medium   | General use with limited resources                  |
| `medium` | 769 MB  | Medium  | Good     | Balanced speed and accuracy                         |
| `large`  | 1.55 GB | Slow    | High     | High-accuracy requirements                          |
| `turbo`  | 809 MB  | Fast    | High     | Recommended default — best speed/accuracy trade-off |

<Warning>
  Larger Whisper models require significantly more memory. The `large` model needs at least 4 Gi RAM per worker. Plan memory limits accordingly and keep `GUNICORN_WORKERS` low when using larger models.
</Warning>

***

### Text-to-speech configuration

TTS is configured per-request via the API. Available options:

| Parameter | Options                                                                     | Default |
| --------- | --------------------------------------------------------------------------- | ------- |
| Model     | `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`                                      | `tts-1` |
| Voice     | `alloy`, `ash`, `coral`, `echo`, `fable`, `onyx`, `nova`, `sage`, `shimmer` | `alloy` |

<Info>
  The `gpt-4o-mini-tts` model supports additional voices: `ballad`, `verse`, `marin`, `cedar`.
</Info>

***

### Tenant context

The Speech to Text service resolves OpenAI credentials per organization via the `llm-config` library. To route a request to the correct organization's AUDIO capability, the caller must propagate tenant context.

#### REST requests

Integration Designer and other upstream services must include these headers on every call to `/transcribe/openai` and `/tts/openai`:

| Header               | Purpose                                                     |
| -------------------- | ----------------------------------------------------------- |
| `Fx-Organization-Id` | Organization UUID — used to resolve the AUDIO capability    |
| `Fx-Workspace-Id`    | Workspace UUID — used when workspace-scoped overrides exist |

The `TenantContextMiddleware` reads these headers and sets the tenant context for the duration of the request. When neither header is present, the service falls back to the `OPENAI_API_KEY` env var.

#### Kafka jobs

Transcription requests published to `ai.flowx.stt.job.request` must include `organizationId` and `workspaceId` fields in the payload. The job consumer reads these fields and sets the tenant context before invoking the transcription resolver.

<Info>
  Cache invalidation for AI provider configuration changes is delivered via Kafka. The service subscribes to `ai.flowx.llm.config.changed.v1` automatically on startup — no manual topic creation is required for invalidation, but the topic must exist in the cluster.
</Info>

***

### Kafka configuration (optional)

The Speech to Text service supports Kafka for async job processing of long-running audio files (15+ minutes) that would timeout via REST. Kafka topics are not pre-created in standard deployments and must be provisioned manually if needed.

<Warning>
  Kafka integration for Speech to Text requires manual topic creation. Ensure the request and response topics exist before enabling the consumer.
</Warning>

#### Core Kafka settings

| Environment Variable         | Description                                    | Default Value |
| ---------------------------- | ---------------------------------------------- | ------------- |
| `KAFKA_BOOTSTRAP_SERVERS`    | Kafka broker address                           | `kafka:9092`  |
| `KAFKA_CONSUMER_ENABLED`     | Enable Kafka consumer (`0` to turn off)        | `1`           |
| `KAFKA_CONSUMER_GROUPID`     | Consumer group ID                              | `ai-services` |
| `KAFKA_SECURITY_ENABLED`     | Enable SASL/OAUTHBEARER auth (`0` to turn off) | `0`           |
| `KAFKA_MAX_POLL_INTERVAL_MS` | Max poll interval (ms)                         | `600000`      |

<Tip>
  The default `KAFKA_MAX_POLL_INTERVAL_MS` is set to 10 minutes (600000 ms) to prevent consumer rebalancing during long transcriptions. Increase this value if you regularly process audio files longer than 10 minutes.
</Tip>

#### Topic configuration

| Environment Variable         | Description                                                                                                                                                                    | Default Value                    |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------- |
| `KAFKA_JOB_REQUEST_TOPIC`    | Incoming transcription requests (from Integration Designer)                                                                                                                    | `ai.flowx.stt.job.request`       |
| `KAFKA_JOB_RESPONSE_TOPIC`   | Outgoing transcription responses (to Integration Designer)                                                                                                                     | `ai.flowx.stt.job.response`      |
| *(subscribed automatically)* | AI Providers cache invalidation — consumed by the `llm-config` resolver to refresh cached AUDIO capability assignments when an organization admin updates Defaults & Fallbacks | `ai.flowx.llm.config.changed.v1` |

<Info>
  The Speech to Text service integrates with [flx-job-lib](https://github.com/flowx-ai/flowx-job-lib) for async job processing. On startup, the service verifies Kafka connectivity and starts a background consumer.
</Info>

***

### Storage configuration (MinIO / S3)

The Speech to Text service reads audio files from and writes results to S3-compatible storage.

| Environment Variable         | Description           | Default Value |
| ---------------------------- | --------------------- | ------------- |
| `MINIO_DOCUMENTS_URL`        | MinIO/S3 endpoint URL | -             |
| `MINIO_DOCUMENTS_ACCESS_KEY` | MinIO access key      | -             |
| `MINIO_DOCUMENTS_SECRET_KEY` | MinIO secret key      | -             |
| `MINIO_DOCUMENTS_BUCKET`     | MinIO bucket name     | -             |

***

### Observability (optional)

| Environment Variable | Description                                         | Default Value |
| -------------------- | --------------------------------------------------- | ------------- |
| `USE_OBSERVATORY`    | Enable FlowX Observatory middleware (`1` to enable) | `0`           |

***

### FLEURS evaluation (optional)

The service can optionally download Google FLEURS datasets at startup for language evaluation and benchmarking.

| Environment Variable | Description                           | Default Value             |
| -------------------- | ------------------------------------- | ------------------------- |
| `FLEURS_LANGS`       | Comma-separated FLEURS language codes | `en_us,hu_hu`             |
| `FLEURS_SPLIT`       | FLEURS dataset split                  | `test`                    |
| `FLEURS_CACHE_DIR`   | Dataset cache directory               | `/flowx/code/data/fleurs` |

<Info>
  FLEURS datasets are used for evaluating transcription accuracy across languages. This is not required for production use — skip this configuration unless you are benchmarking transcription quality.
</Info>

***

## Kafka job processing

The Speech to Text service processes transcription requests asynchronously via Kafka. Integration Designer sends requests to the job request topic, and the service writes results to S3 and responds through the job response topic.

### Request payload

| Field             | Type     | Default                            | Description                                                                                    |
| ----------------- | -------- | ---------------------------------- | ---------------------------------------------------------------------------------------------- |
| `fileStoragePath` | `string` | *(required)*                       | Path to audio file in MinIO/S3                                                                 |
| `provider`        | `string` | `local`                            | `local` for on-device Whisper, `openai` for OpenAI Whisper API                                 |
| `organizationId`  | `string` | *(required for `openai` provider)* | Organization UUID — sets tenant context so the resolver can fetch AUDIO capability credentials |
| `workspaceId`     | `string` | -                                  | Workspace UUID — used for workspace-scoped overrides                                           |

### Response flow

1. Integration Designer publishes a transcription request to `ai.flowx.stt.job.request`
2. Speech to Text downloads the audio file from S3, transcribes it, and stores the result in S3
3. Speech to Text publishes a response with the result path to `ai.flowx.stt.job.response`
4. Integration Designer retrieves the result (text, language, segments with timestamps) from S3

***

## Deployment and sizing

### Docker

* Base image: Python 3.13
* Port: `8000`
* Health check: `/speech-to-text/api/v1/speech-to-text/info/health`
* Requires ffmpeg (included in Docker image)

### Kubernetes configuration

<Tabs>
  <Tab title="Turbo model (recommended)">
    | Setting            | Value   |
    | ------------------ | ------- |
    | Replicas           | 2       |
    | CPU requests       | 2 cores |
    | CPU limits         | 4 cores |
    | RAM requests       | 2 Gi    |
    | RAM limits         | 4 Gi    |
    | `WHISPER_MODEL`    | `turbo` |
    | `GUNICORN_WORKERS` | `2`     |
  </Tab>

  <Tab title="Large model (high accuracy)">
    | Setting            | Value   |
    | ------------------ | ------- |
    | Replicas           | 2       |
    | CPU requests       | 4 cores |
    | CPU limits         | 8 cores |
    | RAM requests       | 4 Gi    |
    | RAM limits         | 8 Gi    |
    | `WHISPER_MODEL`    | `large` |
    | `GUNICORN_WORKERS` | `1`     |
  </Tab>
</Tabs>

<Warning>
  The Whisper model is loaded into memory at startup. Each Gunicorn worker loads its own copy of the model. With the `large` model (\~1.55 GB), keep `GUNICORN_WORKERS` at `1` to avoid OOM kills. Scale horizontally with more pods instead.
</Warning>

***

## Verify your setup

<Check>
  The Speech to Text pod is running: `kubectl get pods -l app=speech-to-text`
</Check>

<Check>
  The health endpoint returns HTTP 200: `curl http://speech-to-text:8000/api/v1/speech-to-text/info/health`
</Check>

<Check>
  Whisper model loaded successfully — check pod logs for model initialization messages at startup
</Check>

<Check>
  Kafka consumer is connected — check pod logs for `Kafka consumer started` message at startup
</Check>

<Check>
  Integration Designer can reach the service — verify `FLOWX_SPEECHTOTEXT_BASEURL` is set in [Integration Designer setup](/5.9/setup-guides/integration-designer-setup)
</Check>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Pod fails to start or crashes">
    **Symptoms:** Pod stays in CrashLoopBackOff or never becomes ready.

    **Solutions:**

    * Check that memory limits are sufficient for the configured Whisper model
    * Verify ffmpeg is available (included in Docker image, may need manual install for local dev)
    * Review pod logs for model download or loading errors
    * Ensure the S3/MinIO endpoint is reachable if Kafka consumer is enabled
  </Accordion>

  <Accordion title="Transcription jobs not being processed">
    **Symptoms:** Integration Designer sends transcription requests but no results are returned.

    **Solutions:**

    * Verify `KAFKA_CONSUMER_ENABLED` is set to `1`
    * Check that `KAFKA_JOB_REQUEST_TOPIC` matches the topic Integration Designer publishes to
    * Ensure the Kafka consumer group (`KAFKA_CONSUMER_GROUPID`) has no conflicting consumers
    * Review pod logs for Kafka connection or authentication errors
  </Accordion>

  <Accordion title="Transcription quality is poor">
    **Symptoms:** Transcription text is inaccurate or contains many errors.

    **Solutions:**

    * Upgrade to a larger Whisper model (`turbo` or `large` instead of `base`)
    * Check audio quality — low bitrate or noisy recordings reduce accuracy
    * For non-English audio, use the `large` model which has better multilingual support
    * Try the OpenAI Whisper API (`provider: openai`) for comparison
  </Accordion>

  <Accordion title="Consumer rebalancing during long transcriptions">
    **Symptoms:** Kafka consumer drops out mid-transcription, causing job failures.

    **Solutions:**

    * Increase `KAFKA_MAX_POLL_INTERVAL_MS` beyond the expected transcription time
    * For very long audio files (30+ minutes), set to `1800000` (30 minutes)
    * Consider splitting long audio files before submitting
  </Accordion>

  <Accordion title="OOM kills with large Whisper models">
    **Symptoms:** Pods restart with OOMKilled status.

    **Solutions:**

    * Reduce `GUNICORN_WORKERS` to `1` — each worker loads its own model copy
    * Increase memory limits (4 Gi minimum for `turbo`, 8 Gi for `large`)
    * Use a smaller model (`small` or `base`) if accuracy requirements allow
    * Scale horizontally with more single-worker pods
  </Accordion>

  <Accordion title="OpenAI API errors">
    **Symptoms:** Cloud transcription or TTS requests fail with API errors.

    **Solutions:**

    * Confirm the request includes `Fx-Organization-Id` (REST) or `organizationId` (Kafka). Without tenant context, the service falls back to `OPENAI_API_KEY`, which may not be set in production deployments.
    * Verify the organization's AUDIO capability is configured under **Organization Settings** → **AI Settings** → **Defaults & Fallbacks**. Pod logs show `no AUDIO capability configured for org, falling back to env` when missing.
    * If falling back to env, verify `OPENAI_API_KEY` is set and valid.
    * Check API rate limits on your OpenAI account.
    * For large files, the service automatically chunks audio into 10-minute segments — ensure network is stable.
    * Review pod logs for specific API error messages.
  </Accordion>

  <Accordion title="AI Providers cache not refreshing">
    **Symptoms:** Organization admin updates the AUDIO model in Defaults & Fallbacks, but the service continues to use the previous model.

    **Solutions:**

    * Verify the Kafka topic `ai.flowx.llm.config.changed.v1` exists in the cluster and the service has consumer permissions.
    * Check pod logs at startup for `LLM config resolver initialized`. If this line is missing or followed by `falling back to env vars`, the resolver did not start — the service is permanently in fallback mode and will not pick up AI Providers changes.
    * Restart the pods to force a cache refresh if the Kafka topic is unreachable.
  </Accordion>
</AccordionGroup>

***

## Related resources

<CardGroup cols={2}>
  <Card title="Integration Designer setup" icon="pencil" href="./integration-designer-setup">
    Configure Integration Designer, which orchestrates Speech to Text jobs
  </Card>

  <Card title="Speech to Text node" icon="microphone" href="../ai-platform/agent-builder/speech-to-text">
    Configure the Speech to Text workflow node in Integration Designer
  </Card>

  <Card title="Kafka Authentication" icon="lock" href="./kafka-authentication-config">
    Configure Kafka security and authentication
  </Card>

  <Card title="Web Crawler setup" icon="spider-web" href="./web-crawler-setup">
    Configure the Web Crawler service for web page extraction
  </Card>
</CardGroup>