diff --git a/ai/generative-ai-service/tts-model-benchmarking/LICENSE b/ai/generative-ai-service/tts-model-benchmarking/LICENSE new file mode 100644 index 0000000000..94dca6c927 --- /dev/null +++ b/ai/generative-ai-service/tts-model-benchmarking/LICENSE @@ -0,0 +1,32 @@ +Copyright (c) 2026 Oracle and/or its affiliates. + +The Universal Permissive License (UPL), Version 1.0 + +Subject to the condition set forth below, permission is hereby granted to any +person obtaining a copy of this software, associated documentation and/or data +(collectively the "Software"), free of charge and under any and all copyright +rights in the Software, and any and all patent rights owned or freely +licensable by each licensor hereunder covering either (i) the unmodified +Software as contributed to or provided by such licensor, or (ii) the Larger +Works (as defined below), to deal in both + +(a) the Software, and + +(b) any piece of software and/or hardware listed in the lrg_coverage.txt file, +if one is included with the Software (each a "Covered Work", collectively +"Covered Works"), that is available under a license listed in the +lrg_coverage.txt file, if one is included with the Software (the "Covered +License"), and + +without limitation, the right to copy, create derivative works of, display, +perform, and distribute the Software and make, use, sell, offer for sale, +import, export, have made, and have sold the Software and the Covered Works, +in each case subject to the conditions and limitations set forth herein. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/ai/generative-ai-service/tts-model-benchmarking/README.md b/ai/generative-ai-service/tts-model-benchmarking/README.md new file mode 100644 index 0000000000..bbf2d2e6f9 --- /dev/null +++ b/ai/generative-ai-service/tts-model-benchmarking/README.md @@ -0,0 +1,75 @@ +# TTS Model Benchmarking on OCI Generative AI + +*A benchmark toolkit for text-to-speech models hosted on OCI Generative AI (imported models such as OmniVoice on a dedicated AI cluster, or on-demand models on the same speech route). It measures time to first audio, end-to-end latency, real-time factor, throughput and error rate under concurrency, with and without streaming, in English and Arabic, and scores intelligibility with Whisper to find the input length at which single-pass synthesis degrades.* + +Author: Brona Nilsson + +Reviewed: 05.10.2026 + +# When to use this asset? + +*Use this asset when you need numbers, not impressions, about a TTS endpoint on OCI: how fast it answers, how much load one hosting unit takes, and whether long inputs stay intelligible.* + +### Who +- Data scientists and ML engineers evaluating a TTS model in an OCI Data Science notebook session +- Solution architects sizing a dedicated AI cluster (A10, A100, H100) for a voice workload +- Teams comparing hosting shapes, model versions or serving-stack upgrades with repeatable runs + +### When +- Choosing a hosting shape: run the same suites on each shape and compare the CSVs +- Designing a voice application: pick streaming vs chunking from measured TTFA, not guesswork +- Validating a fix: OmniVoice served through vLLM-Omni on OCI garbles speech past ~30 s per request; the `long-text` suite shows the breaking point as a WER curve and confirms when an upgrade removes it +- Capacity planning: the concurrency sweep finds the knee where latency climbs and throughput flattens + +# How to use this asset? + +*Point the scripts at an endpoint, run a suite, read the report.* + +1. `pip install -r files/requirements.txt`, copy `files/.env.example` to `files/.env` and set the endpoint and compartment OCIDs. In a Data Science notebook session the resource principal is picked up automatically. +2. `python files/tts_bench.py --suite quick --label ` for a two-minute smoke test, then `--suite latency`, `--suite long-text`, `--suite concurrency` or `--suite full`. +3. `python files/wer.py results/` to score the saved clips for intelligibility, and `python files/plots.py results/` for charts. Or open `files/notebook/tts_benchmark.ipynb` and run the cells. + +See the detailed [README](files/README.md) in the `files/` folder for setup on each platform, the metric definitions, the suites, reference numbers from an H100_X1 cluster and troubleshooting. + +### Key Capabilities +- Industry-standard serving metrics: TTFA, E2E, RTF, p50/p90/p95/p99, error rate, requests per minute and audio seconds per second +- Three request modes: synchronous, streaming, and client-side sentence chunking (the workaround for long inputs) +- Closed-loop concurrency sweep (1, 2, 4, 8, 16 parallel requests) to find the capacity of one hosting unit +- Target durations in seconds (10 to 60 s) with per-language calibration of characters per second, so English and Arabic are compared at equal speech length +- Whisper-based WER/CER scoring with Arabic normalisation; CPU, CUDA or Apple Silicon backends +- Constant voice across all requests through a bundled synthetic reference clip; voice design and preset-voice modes for other models +- Works with API key, resource principal, instance principal, session token or bearer-key auth +- CSV, Markdown report, sample audio and PNG charts per run; runs are labelled by shape for side-by-side comparison + +### File Structure +``` +. +├── README.md # This file +├── LICENSE +└── files/ + ├── README.md # Detailed guide: setup, metrics, suites, reference numbers + ├── tts_bench.py # The benchmark + ├── wer.py # Intelligibility scoring (Whisper WER / CER) + ├── plots.py # Charts from a run folder + ├── tts_client.py # Client for the OCI speech route with OCI signing + ├── requirements.txt + ├── .env.example + ├── notebook/tts_benchmark.ipynb + ├── texts/ # en_science, en_report, ar_coffee, ar_city + ├── voices/ref_voice.wav # synthetic reference voice + transcript + └── results/ # one folder per run (git-ignored) +``` + +# Useful Links + +- [OCI Generative AI documentation](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm) +- [Importing K2-FSA OmniVoice models](https://docs.oracle.com/en-us/iaas/Content/generative-ai/imported-k2-fsa-models.htm) +- [OCI OpenAI-compatible API](https://docs.oracle.com/en-us/iaas/Content/generative-ai/openai-compatible-api.htm) +- [OCI Data Science notebook sessions](https://docs.oracle.com/en-us/iaas/data-science/using/use-notebook-sessions.htm) + +# License + +Copyright (c) 2026 Oracle and/or its affiliates. +Licensed under the Universal Permissive License (UPL), Version 1.0. + +See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE.txt) for more details. diff --git a/ai/generative-ai-service/tts-model-benchmarking/files/.env.example b/ai/generative-ai-service/tts-model-benchmarking/files/.env.example new file mode 100644 index 0000000000..acafbca400 --- /dev/null +++ b/ai/generative-ai-service/tts-model-benchmarking/files/.env.example @@ -0,0 +1,25 @@ +# Copy to .env and fill in. Nothing here is a secret unless you choose the bearer-key route. + +# The TTS endpoint to benchmark: a dedicated endpoint OCID (imported model on a DAC) +# or, for on-demand models that answer on the same speech route, the model name. +TTS_MODEL=ocid1.generativeaiendpoint.oc1.. + +# Compartment the endpoint lives in (sent as the compartment header). +GENAI_COMPARTMENT_ID=ocid1.compartment.oc1.. + +# How to authenticate. One of: +# api_key ~/.oci/config profile (laptops, default) +# resource_principal OCI Data Science notebook sessions and jobs (auto-detected there) +# instance_principal OCI compute VMs with a dynamic group policy +# security_token `oci session authenticate` profiles +# bearer an OCI Generative AI API key, see GENAI_API_KEY +OCI_AUTH=api_key +OCI_CONFIG_PROFILE=DEFAULT + +# Only for OCI_AUTH=bearer. Create it under Generative AI > API keys, in the SAME region as the endpoint. +# GENAI_API_KEY=sk-... + +# Optional: force the inference host. Default is derived from the region in TTS_MODEL, +# or from OCI_REGION when TTS_MODEL is not an OCID. +# GENAI_SERVICE_ENDPOINT=https://inference.generativeai..oci.oraclecloud.com +# OCI_REGION=eu-frankfurt-1 diff --git a/ai/generative-ai-service/tts-model-benchmarking/files/README.md b/ai/generative-ai-service/tts-model-benchmarking/files/README.md new file mode 100644 index 0000000000..3fe4a93cba --- /dev/null +++ b/ai/generative-ai-service/tts-model-benchmarking/files/README.md @@ -0,0 +1,217 @@ +# TTS model benchmarking: detailed guide + +Scripts to measure a text-to-speech endpoint on OCI Generative AI the way a serving team +would: time to first audio, end-to-end latency, real-time factor, throughput under +concurrency, error rate, and intelligibility of the audio as the input gets longer. Built +and verified against `k2-fsa/OmniVoice` imported on a dedicated AI cluster; works for any +model that answers on OCI's OpenAI-compatible speech route. + +## Contents + +| File | Purpose | +|---|---| +| `tts_bench.py` | The benchmark. Suites, calibration, concurrency sweep, CSV + Markdown report, sample audio. | +| `wer.py` | Transcribes the sample audio with Whisper and scores word / character error rate. Finds where long inputs break. | +| `plots.py` | PNG charts from a run folder: latency vs length, RTF, load sweep, WER. | +| `tts_client.py` | Minimal client for `/openai/v1/audio/speech` with OCI signing (API key, resource principal, instance principal, session token) or a bearer key. | +| `notebook/tts_benchmark.ipynb` | The same steps as notebook cells, for OCI Data Science. | +| `texts/` | Benchmark texts, `_.txt`. English (`science`, `report` with many numbers) and Arabic (`coffee`, `city`), about 1,700-3,000 characters each. Add your own with the same naming. | +| `voices/ref_voice.wav` + `.txt` | 7 s synthetic reference clip, generated by OmniVoice itself from a text prompt, with its transcript. Cloned in every request so the voice is constant. Not a real person. | +| `results/` | One folder per run. Git-ignored. | +| `.env.example` | Endpoint, compartment and auth settings. | + +## 1. Setup + +### On a laptop + +```bash +python3 -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt +cp .env.example .env # set TTS_MODEL and GENAI_COMPARTMENT_ID +python tts_bench.py --suite quick --label mylaptop +``` + +Auth uses your `~/.oci/config` profile (`OCI_CONFIG_PROFILE`). The inference region is +derived from the endpoint OCID, so the profile can be in any region. + +### In an OCI Data Science notebook session + +Open `notebook/tts_benchmark.ipynb`, or in a terminal: + +```bash +pip install -r requirements.txt +export TTS_MODEL=ocid1.generativeaiendpoint.oc1.. +export GENAI_COMPARTMENT_ID=ocid1.compartment.oc1.. +python tts_bench.py --suite quick --label H100_X1 +``` + +The resource principal is detected automatically (`OCI_RESOURCE_PRINCIPAL_VERSION` is set in +notebook sessions and jobs). The session's dynamic group needs: + +``` +allow dynamic-group to use generative-ai-inference in compartment +``` + +### Other auth modes + +| `OCI_AUTH` | Use when | +|---|---| +| `api_key` (default) | `~/.oci/config` with an API signing key | +| `resource_principal` | Data Science notebooks and jobs, Functions, Container Instances | +| `instance_principal` | A compute VM in a dynamic group | +| `security_token` | A profile created by `oci session authenticate` | +| `bearer` | An OCI Generative AI API key (`GENAI_API_KEY=sk-...`), created in the same region as the endpoint | + +### Which model + +`TTS_MODEL` is the **endpoint OCID** for an imported model on a DAC (display names return +404). For on-demand models that answer on the same speech route, set the model name and +`OCI_REGION`. Models with built-in voices need `--voice preset --voice-name `; +OmniVoice has none and rejects `voice`, which is why the default is `--voice clone`. + +## 2. Run + +```bash +python tts_bench.py --suite quick --label H100_X1 # 2 min smoke test +python tts_bench.py --suite latency --label H100_X1 # latency vs length, 3 modes +python tts_bench.py --suite long-text --label H100_X1 --score # where does it break? + WER +python tts_bench.py --suite concurrency --label H100_X1 # 1,2,4,8,16 parallel +python tts_bench.py --suite full --label H100_X1 # all of the above +python plots.py results/H100_X1_