Transcription

Transcription converts inbound audio or voice attachments into text before the assistant turn runs. It is part of media understanding, not a general model-visible chat tool.

When transcription succeeds, Anyy appends an [Audio transcript] block to the user text and marks turn metadata with text_source: audio_transcription.

Configuration

Configure audio understanding under media_understanding.audio:

media_understanding:
  enabled: true
  audio:
    enabled: true
    provider: openai
    mode: auto
    on_failure: annotate
    timeout_ms: 60000
    max_bytes: 25000000
    max_chars: 12000
    language: ""
    prompt: ""
    providers:
      openai:
        type: openai
        model: gpt-4o-mini-transcribe
        base_url: https://api.openai.com/v1
        api_key_ref: env:OPENAI_API_KEY

Transcription is disabled when media_understanding.enabled is false, audio.enabled is false, audio.mode is disabled, or audio.provider is empty.

Command Transcription

Custom command providers run a local command:

providers:
  local-whisper:
    type: command
    command: /usr/local/bin/transcribe
    args: ["--json"]
    cwd: /tmp
    env:
      MODEL_PATH: /models/whisper
    max_chars: 12000

Environment values can use secret references. Anyy resolves them before launching the command.

Provider Transcription

OpenAI-compatible providers use:

Field	Meaning
type	openai or openai-compatible.
model	Provider transcription model. Defaults can come from the audio block.
base_url	API base URL.
credential_ref / api_key_ref	Secret reference for authentication.
timeout_ms	Provider call timeout.
max_bytes	Maximum input artifact size.
max_chars	Maximum transcript chars returned to the turn.

The built-in default OpenAI model constant is gpt-4o-mini-transcribe.

Inbound Audio

Anyy attempts transcription only for attachments with kind voice or audio.

The runtime resolves the attachment through the artifact path broker, checks size limits, runs the transcriber, and appends transcript text:

[Audio transcript]
...

For multiple audio attachments, labels include the index:

[Audio transcript #2]
...

If the inbound metadata already indicates a Weixin voice item text source, Anyy does not transcribe the same audio again.

Failure Policy

media_understanding.audio.on_failure controls failed transcription:

Value	Behavior
annotate	Append [Audio attachment could not be transcribed.] and continue. Default.
ignore	Leave the failed attachment without transcript text.
fail	Fail the turn with audio transcription failed.

Attachment metadata records transcription_status, provider/model/language on success, and transcription_error: transcription_failed on failure.

Troubleshooting

Symptom	Check
No transcript appears	Confirm media understanding and audio transcription are enabled and the attachment kind is audio or voice.
Turn fails	Check on_failure; fail turns provider problems into turn errors.
Provider unavailable	Confirm the provider id exists and credentials resolve.
Large audio rejected	Lower file size or raise configured max_bytes carefully.
Transcript truncated	Raise max_chars for the audio block or provider.