Who
Label the speakers in a recording and see exactly when each person starts and stops talking, entirely on the device.
Label who said what in audio and video.
Who is made for podcasts, interviews, calls and meetings, and handles recordings of any length. A 50-minute interview takes 3s on a MacBook Pro, entirely on the device. Use the timestamps to label a transcript by speaker, jump to the next time a guest talks, or cut every time the speaker changes.
Who numbers the speakers in the order they first speak, and when two people talk at once, Who marks both of them. Who processes each five-minute section in one pass and matches voices across sections.
On AMI meeting recordings, Who labels speakers with a 12.6% error rate, lower than 13.1% for pyannote on a server GPU and 19.4% for FluidAudio on the same Mac. Who also runs twice as fast as FluidAudio.
An hour of audio in 4s.
Who runs at 950x realtime on a MacBook Pro with the M5 Pro, through the SDK. On AMI meetings, Who's diarization error rate is 12.6%, against 13.1% for pyannote and 19.4% for FluidAudio.
Whole test sets on one MacBook Pro (M5 Pro), one recording at a time. Who's times include loading the model. FluidAudio on AMI takes 73s by its own clock and 79s with its model loads.
| Test set | Who time | Who realtime | FluidAudio realtime |
|---|---|---|---|
| AMI test, 544 minutes | 34s | 950x | 445x |
| VoxConverse test, 43.5 hours | 165s | 948x | 426x |
Diarization error rate on whole recordings, lower is better. The same recordings and scorer in every row
| Model | AMI | VoxConverse, up to 4 speakers | NOTSOFAR-1, up to 4 speakers | Earnings-21 | DiPCo |
|---|---|---|---|---|---|
| Who, on the Neural Engine | 12.6% | 8.4% | 16.2% | 27.7% | 41.1% |
| pyannote community-1, PyTorch | 13.1% | 7.3% | 18.7% | 7.3% | 33.4% |
| FluidAudio, Core ML | 19.4% | 9.6% | 32.7% | 12.0% | 42.7% |
Who leads on the two meeting sets, AMI and NOTSOFAR-1, and trails pyannote on the other three: Earnings-21 calls run up to 20 speakers, and Who scores 24.7% on VoxConverse with nine or more speakers. pyannote ran on server GPUs; Who and FluidAudio ran on the M5 Pro's Neural Engine. The pyannote and FluidAudio scores are our own runs and are not on the model card.
Use cases
Label who said what in audio and video.
Label a transcript by speaker
Pair Who with Voz, and each line of the transcript shows who said it. A reader follows the conversation instead of one block of anonymous text.
Jump to the next time a guest talks
Filter a two-hour interview or meeting to one speaker and go straight to their turns instead of scrubbing the timeline. The recording stays on the device.
Cut every time the speaker changes
Switch camera angles in a two-camera interview or a video podcast wherever the speaker changes. Every turn has a start and an end time, so each change of speaker is a cut point.
Leave out the laughter
Drop the audience laughter and the voice that said one line from across the room. Who leaves out every speaker under the share of the recording you set and renumbers the rest.
What the model does
- Who labels up to four speakers in each five-minute section, and more across a whole recording, by matching voices across sections.
- Each turn is one stretch of one speaker's speech, with a speaker number and start and end times in seconds, to the nearest 10ms. Turns of different speakers overlap when people talk over each other.
- Who numbers speakers in the SDK from zero, in the order they first speak, and doesn't identify anyone by name.
- Two Core ML graphs on the Neural Engine read each section, an encoder and a query graph. A third graph makes a voice print of each speaker from the moments they speak alone, and Who matches every speaker across the whole recording at once.
- A speaker with under a second of solo speech has no voice print of their own and joins the nearest speaker in time. A second of laughter inside someone else's turn goes to that speaker.
- 950x realtime on a MacBook Pro with the M5 Pro, through the SDK: 544 minutes of AMI meetings in 34s. The 8-bit build runs at 953x.
- Memory through the SDK: 275MB with the model loaded, and a peak of 336MB on an 8-minute recording, 437MB on a 34-minute recording, and 927MB on a 2.65-hour recording.
- The Core ML download is 81MB as float16 or 42MB with 8-bit weights, and both builds have the same error rate on AMI meetings, 12.6%. The 8-bit build saves no memory, since the Neural Engine decompresses the weights at load.
- Who also comes as a 49MB LiteRT build, which the SDK runs on Android, Linux and Windows, and a 99MB ONNX build, which the SDKs don't run yet. On CPU alone, the LiteRT build runs 544 minutes of audio in 87s on the M5 Pro and in 191s on a 24-core AMD EPYC server, measured outside the SDK.
keepingSpeakers(minimumShare:)drops every speaker under a share of the recording and renumbers the rest, so laughter, applause, and jingles drop out. At 5%, the filter cleared the laughter and audience on the podcasts we tried. For a news program with many brief speakers, keep the share low or off.- Diarize an audio file by URL or path, or pass mono samples at any sample rate. The SDK converts the sample rate for you.
- The SDK downloads the weights from Hugging Face on first use and caches them.
download(progress:)fetches them ahead of time, andWho(directory:)runs offline from a folder you manage.
Getting started
Add who said what to your iOS or macOS or Android app in a few lines of code. Who docs.
// Swift Package Manager
.package(url: "https://github.com/Desert-Ant-Labs/desert-ant-core.git", from: "4.0.0")
// target dependency
.product(name: "Who", package: "desert-ant-core")import Who
let who = Who() // downloads on first use
let result = try await who.diarize(contentsOf: url)
result.speakerCount // 3
for turn in result.turns {
print(turn.speaker, turn.start, turn.end) // 0 12.4 15.9
}Add Who from Desert Ant Labs to this Swift project (iOS, macOS). What it does: On-device Speaker Labeling for Audio and Video. SDK: Swift (iOS, macOS) Repo: https://github.com/Desert-Ant-Labs/desert-ant-core#readme // Swift Package Manager .package(url: "https://github.com/Desert-Ant-Labs/desert-ant-core.git", from: "4.0.0") // target dependency .product(name: "Who", package: "desert-ant-core") Reference: - Model page: https://desertant.com/models/who/ - Full catalog and other models: https://desertant.com/llms.txt Add the SDK, then follow its README for the exact API and current version. Do not invent API names or method signatures; confirm them against the README.
// build.gradle.kts (Maven Central)
implementation("ai.desertant:who:4.0.0")import ai.desertant.who.Who
val who = Who(context) // downloads on first use
val result = who.diarize(samples, 16_000.0)
result.speakerCount // 3
for (turn in result.turns) {
println("${turn.speaker} ${turn.start} ${turn.end}") // 0 12.4 15.9
}
who.close()Add Who from Desert Ant Labs to this Kotlin project (Android).
What it does: On-device Speaker Labeling for Audio and Video.
SDK:
Kotlin (Android)
Repo: https://github.com/Desert-Ant-Labs/desert-ant-core#readme
// build.gradle.kts (Maven Central)
implementation("ai.desertant:who:4.0.0")
Reference:
- Model page: https://desertant.com/models/who/
- Full catalog and other models: https://desertant.com/llms.txt
Add the SDK, then follow its README for the exact API and current version. Do not invent API names or method signatures; confirm them against the README.
Specs
- Speed
- 950x realtime on a MacBook Pro (M5 Pro), a 50-minute interview in 3s
- Accuracy
- 12.6% diarization error rate on AMI meetings; 8.4% on VoxConverse with up to 4 speakers
- Speakers
- Up to 4 in each five-minute section, more across a recording of any length
- Output
- Per-speaker turns with start and end times in seconds, to the nearest 10ms; overlapping speech kept
- On-device size
- 81MB Core ML (float16, default); 42MB with 8-bit weights
- Memory
- 275MB with the model loaded; 927MB peak on a 2.65-hour recording
- Platforms
- iOS 18+, macOS 15+, tvOS 18+, visionOS 2+ (Core ML); Android, Linux and Windows (LiteRT, on the CPU)
Who does best on close-miked audio, such as podcasts, interviews, and remote calls where each person has their own microphone. Far from the microphone Who misses more speech: 19-21% in group conversations recorded across the room, against 8% in meetings with headsets.
Who misses the most speech when two people talk at once. On VoxConverse the error rate is 8% with one or two speakers, 17% with five to eight, and 25% with nine or more.
On Earnings-21 calls, with up to 20 speakers making short remarks, Who assigns 16% of the speech to the wrong speaker. Who labels speakers by number, not by name, and doesn't transcribe; for a transcript labeled by speaker, pair Who with Voz.
On Android, Linux and Windows, Who runs on the CPU, three to seven times slower than on the Neural Engine. Who doesn't run in the browser or in Node yet.
FAQ
What is Who?
Label the speakers in a recording and see exactly when each person starts and stops talking, entirely on the device.
Does Who run on device?
Yes. Who runs on the device, with no server call, so the data stays with the user.
Which platforms does Who support?
Who ships as a native on-device SDK for Swift, Kotlin.
How much does Who cost?
Each model is free up to 100k monthly active devices. Inference is unlimited. Contact us for custom licenses.
How accurate or fast is Who?
Who runs at 950x realtime on a MacBook Pro with the M5 Pro, through the SDK. On AMI meetings, Who's diarization error rate is 12.6%, against 13.1% for pyannote and 19.4% for FluidAudio.