Best AI voice cloning in 2026: how to clone your voice

Clone my voice with an open-source TTS model. Work through these steps in
order and verify each one before moving on. Tell me when something fails; do
not silently retry.

=== 0. SAFETY: THIS PROMPT CONTAINS NO SECRETS. DO NOT ADD ANY. ===

Do not paste an API key, token, or password into this prompt or into any chat
window. Everything below authenticates interactively or through a file the
agent never prints.

Agent: never echo, log, or repeat back the contents of a credential file. If
you need to confirm a token loaded, print its LENGTH, not its value.

=== 0b. TOOLING YOU NEED FIRST ===

RunPod (rents the GPU). Connect the MCP server so the agent can create and
destroy pods:

    claude mcp add --transport http runpod -s user https://mcp.getrunpod.io/

This uses a browser OAuth flow, so no API key is stored on disk and nothing
sensitive passes through this prompt. Then add an ACCOUNT-LEVEL SSH key at
runpod.io/console/user/settings. The SSH proxy authenticates against that key,
not the pod's PUBLIC_KEY, and without it every connection fails with
"Permission denied (publickey)".

Hugging Face (hosts the base models, your corpus, and your trained weights).
Create a token at huggingface.co/settings/tokens. Prefer a FINE-GRAINED token
scoped to only the repos you will create, with write access. A global write
token can delete every repo you own, and an agent will be handling it.

Log in INTERACTIVELY so the token never lands in shell history or in a process
argument list:

    huggingface-cli login

To place it on a pod, read it from stdin and write it to a file. Never pass a
token as a command-line flag:

    read -rs HF_TOKEN && printf '%s' "$HF_TOKEN" > /root/.hf_token && unset HF_TOKEN
    chmod 600 /root/.hf_token

=== 1. RECORD ===

30-45 minutes in ONE session. One room, one mic, one sitting. A phone is fine.
Do not pool clips from different days: mixed sessions teach the model the room
differences along with the voice.
Scraped audio does not scale: 14.5 min of clips pulled from old videos tied
2.5 min of the same. One clean session of 30-45 min is what actually worked.
No noise reduction, no enhancement. Enhanced audio scored worse.

=== 2. BUILD THE CORPUS ===

    ffmpeg -i SOURCE.mov -ar 24000 -ac 1 -c:a pcm_s16le audio_24k.wav

Chunk into 4-12s clips: split on . ! ?, merge forward until each clip is >= 4s,
force-split anything over 12s at a comma or a gap > 0.35s.
NEVER discard a segment for being too long.

Print a manifest and check it against the source length:

    clips=360 total_min=45.36 min=4.00 mean=7.56 max=12.00

STOP if total_min is not within ~10% of what I recorded. An early version of my
chunker dropped over-long segments instead of splitting them and silently lost
27 of 52 minutes with no error.

Cut restarts, coughs, off-mic lines and dead air. A bad clip hurts more than a
missing one.

=== 3. TRANSCRIBE ===

openai-whisper, model large-v3, on the GPU. Write train.jsonl with one row per
clip: {"text": ..., "audio": ...}
Correcting the transcripts afterwards is OPTIONAL. I tested it and it did not
improve output quality. Skip unless I ask.

=== 4. PIN THE CORPUS TO HUGGING FACE, BEFORE ANY TRAINING ===

    huggingface-cli repo create my-voice-corpus --type dataset --private
    # upload the clips + train.jsonl

This is not bookkeeping. The corpus must be ONE artifact that every training run
fetches, or you will end up comparing models trained on different data without
realising it. Every training script should fetch the pin and assert the clip
count before starting.

=== 5. START A POD ===

Use the RunPod MCP tools. Ask for a 24GB GPU (RTX 4090 is enough for a 0.5B
model). Community 4090 is about $0.34/hr, secure about $0.74/hr.

CUDA MUST BE >= 12.8. create-pod returns cudaVersion: if it is 12.2 or 12.4,
DELETE the pod and create another. Those hosts crash-loop the container while
the API still reports RUNNING. Each rejected pod costs under a cent.

Then prove the GPU is real before installing anything:

    python -c "import torch; \
      print('GPUCHECK', torch.cuda.is_available(), \
      (torch.randn(999,999,device='cuda') @ torch.randn(999,999,device='cuda')).sum().item())"

If that fails, destroy the pod and get another. Never let training fall back to
CPU: a silent CPU fallback hid a dead GPU from me for an entire run.

=== 6. BUILD THE ENVIRONMENT ===

FIND THE PROJECT'S docker/Dockerfile AND FOLLOW IT. It is the only authoritative
record of the environment. Guessing at it cost me many hours.

For the model I ended up shipping, dots.tts (note: the GitHub org is
studio-dots-ai, while the Hugging Face org is dots-studio, and guessing wrong
makes git prompt for a username instead of reporting a missing repo):

    curl -sL https://raw.githubusercontent.com/studio-dots-ai/dots.tts/main/constraints/recommended.txt \
      | grep -v '^gradio' > constraints.txt      # the package and its own
                                                 # constraints file disagree
    git clone --depth 1 https://github.com/studio-dots-ai/dots.tts.git
    pip install -e dots.tts -c constraints.txt

For CosyVoice 3 specifically:

    # the image may not ship conda at all; install miniconda if `conda` is missing
    conda create -y -n cosyvoice python=3.10 --override-channels -c conda-forge
    conda install -y -n cosyvoice --override-channels -c conda-forge pynini==2.1.5
    pip install "setuptools<81" wheel
    pip install torch==2.3.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
    pip install -r requirements.txt          # hold out openai-whisper
    pip install openai-whisper --no-build-isolation
    pip install huggingface_hub hf_transfer

Four traps, all of which fail SILENTLY or misleadingly:
  - `conda activate` does nothing in a non-interactive shell. Use absolute
    paths: /opt/conda/envs/NAME/bin/python. Otherwise your verification passes
    against the WRONG interpreter.
  - Anaconda default channels now need a ToS accept that fails unattended.
    Use --override-channels -c conda-forge.
  - openai-whisper fails to build inside isolation and aborts the ENTIRE
    requirements install, so nothing else lands.
  - Some images export HF_HUB_ENABLE_HF_TRANSFER=1 without shipping
    hf_transfer, so every Hugging Face download raises until you install it.

Assert every import before training:

    /opt/conda/envs/cosyvoice/bin/python -c \
      "import torch,torchaudio,hyperpyyaml,whisper,pkg_resources; print('DEPS_OK')"

=== 7. TRAIN ===

Save a checkpoint EVERY epoch. Expect ~22 seconds per epoch for a 45-minute
corpus on a 4090, so roughly 11 minutes total.

Checkpoints are ~2GB each, so prune to a sparse ladder as you go or a 150GB
disk fills at about epoch 36.

Do NOT assume the last checkpoint is the best. Audition early ones. Mine peaked at epoch 4 of 31.
Epoch count scales INVERSELY with corpus size: one epoch on 45 min is ~3x the
gradient steps of one epoch on 15 min, so a bigger corpus peaks at a LOWER
epoch number.

=== 8. PUSH EVERY CHECKPOINT TO HUGGING FACE ===

    huggingface-cli repo create my-voice-model --private
    # upload each kept checkpoint, winner first

NON-NEGOTIABLE, and do it BEFORE destroying the pod. Verify by reading the file
sizes back off Hugging Face.

A stopped RunPod pod is NOT a saved pod. /workspace is container overlay
storage, not a network volume. I have seen both failure modes in one hour: a
stopped pod that could not restart ("not enough free GPUs on the host"), and one
that restarted on a different host with /workspace completely empty.

=== 9. RENDER, 3+ TAKES PER CHECKPOINT ===

Generation is stochastic and most of these models take NO seed, so one take is
a lottery draw rather than a measurement.

Render 3 to 5. Then screen them in two passes.

PASS 1, MECHANICAL. Sort the takes by duration and drop any that are materially
shorter than the MEDIAN OF THAT MODEL'S OWN TAKES. Do not use an absolute
words-per-minute threshold: two models I trained read at ~175 and ~150 wpm, so
one cutoff either discards good takes from the fast model or passes bad ones
from the slow one. Five takes of a 67-word script came in at 22.56, 22.88,
22.88, 24.00 and 24.32 seconds; the 22.56 one had silently dropped the closing
sentence and sounded completely fine. A 1.5 second gap against its siblings was
the only evidence.

PASS 2, HUMAN. Listen to what survives and pick by ear.

Do NOT rank takes by Whisper word coverage. I did this for weeks and it is
worse than useless: it scores Whisper's transcription, not the model's speech,
so a robotic take Whisper parses cleanly outranks a natural one it stumbles on.
It put a take I described as sounding like Steve Urkel at the top of a ladder,
reported 24 missing words in a take where the full script was plainly present,
and scored 100% on clips where I could hear words being dropped.

If the inference call returns a GENERATOR, consume ALL of it and concatenate.
CosyVoice yields one segment per sentence group; taking only the first silently
truncated a 29.5s clip to 19.7s for me, with no error.

Duration only catches MISSING SPEECH. It cannot catch a mangled word inside a
full-length take, and it is useless for any model that pads its output to a
constant length: one model I tested returned 25 renders all between 45.85 and
46.18 seconds, some clean and some dropping words. There is no automatic check
for that. A human has to listen.

=== 10. BLIND TEST ===

Normalise every candidate to -16 LUFS, pad to one duration, and force an
identical byte size. Loudness, length and file size all leak which file is
which.

Shuffle the labels, seal the mapping, and do not reveal it until after I have
ranked. Keep sets to 5-6 files.

Balance the arms: do not put 4 checkpoints of one model against 1 of another.
One bad render sinks a lone entry.

=== 11. TEAR DOWN ===

Once checkpoints are pushed AND verified by read-back, TERMINATE the pod. Do
not stop it. Stopping still bills for storage and may lose the disk anyway.

=== REPORT BACK ===

Manifest numbers, per-epoch loss, the checkpoint list, take durations and word
coverage per render. Name what failed.