Desert Ant Labs

Who

On-device speaker labeling: per-person turns with timestamps.

Who model page
Platforms
iOS, macOS, tvOS, visionOS, Android, Linux, Windows, Browser, Node
Weights
v0.3.4
Status
Beta

Label the speakers in a recording and see exactly when each person starts and stops talking. A 50-minute interview takes 3s on a MacBook Pro, entirely on the device.

Use the timestamps to label a transcript by speaker, jump to the next time a guest talks, or cut every time the speaker changes. Who is a speaker diarization model made for podcasts, interviews, calls and meetings, and handles recordings of any length.

Who numbers the speakers in the order they first speak, and when two people talk at once, Who marks both of them. Who processes each five-minute section in one pass and matches voices across sections. Who runs on the Neural Engine on Apple platforms, through LiteRT on Linux, Windows and Android, and through WebAssembly in the browser and Node. There's also an ONNX build. The Core ML download is 81MB as float16 or 42MB with 8-bit weights, and both builds have the same error rate on AMI meetings, 12.6%.

Install

requirements

Swift
.package(url: "https://github.com/Desert-Ant-Labs/desert-ant-core.git", from: "4.3.0")

Then add the Who product to your target.

requirements

Kotlin
implementation("ai.desertant:who:4.3.0")

requirements

Terminal
npm i @desert-ant-labs/who @litertjs/core   # browser
npm i @desert-ant-labs/who                  # Node, prebuilt native core

Usage

Swift
import Who

let who = Who()                                        // downloads on first use
let result = try await who.diarize(contentsOf: url)

result.speakerCount        // 3
for turn in result.turns {
    turn.speaker           // 0, 1, 2: numbered in order of first appearance
    turn.start             // 12.4, seconds from the start
    turn.end               // 15.9
}

Create one Who instance and reuse it. Creating the instance does no work and starts no download. The SDK loads the model on the first call to diarize or download(progress:), off your calling thread.

You can also pass decoded mono samples at any sample rate:

Swift
let result = try await who.diarize(samples: samples, sampleRate: 48000)
Kotlin
val who = Who(context)                       // downloads on first use
val result = who.diarize(samples, 16_000.0)

result.speakerCount                          // 3
for (turn in result.turns) {
    turn.speaker                             // 0, 1, 2: numbered in order of first appearance
    turn.start                               // 12.4, seconds from the start
    turn.end                                 // 15.9
}
who.close()

On Android the graphs run through LiteRT on the CPU, the 8-bit build, 32 MB. Create one Who and reuse it; diarize suspends on a background dispatcher. result.keepingSpeakers(minimumShare = 0.05) drops the speakers under a share of the recording, a laugh or a jingle that got a number of its own, as the Swift SDK's keepingSpeakers(minimumShare:) does.

TypeScript
import { Who } from "@desert-ant-labs/who";           // browser
// import { Who } from "@desert-ant-labs/who/native"; // server-side Node

const who = await Who.load();                          // downloads on first use
const result = await who.diarize(file);                // a File or Blob, the bytes of an audio file, or samples

result.speakerCount;                                   // 3
for (const turn of result.turns) {
  turn.speaker;                                        // 0, 1, 2: numbered in order of first appearance
  turn.start;                                          // 12.4, seconds from the start
  turn.end;                                            // 15.9
}
who.dispose();

diarize takes the bytes of an audio file (File, Blob, ArrayBuffer, Uint8Array), mono Float32Array samples at 16 kHz, or { samples, sampleRate } at any rate. In the browser a file is decoded with Web Audio, so any format the browser plays works; in Node the portable decoder reads WAV, so decode other formats first. The browser build runs the graphs on the CPU through WebAssembly; pass accelerator: "auto" to Who.load to try WebGPU. result.keepingSpeakers({ minimumShare: 0.05 }) drops the speakers under a share of the recording, as on the other platforms. Create one Who and reuse it; Who.load downloads the 8-bit build, 32 MB, on first use.

Long recordings

A recording of up to five minutes is read in one pass, because the part of the model that decides who the speakers are sees it whole. A longer recording is cut into five-minute pieces, each read in one pass, and the speakers are joined across the pieces by voice: a speaker-verification network gives each lane a voice from the moments it speaks alone, and the lanes of the whole recording are clustered at once, so the numbering holds from the first minute to the last. There is no length limit, and a recording that fits one pass is the model at its most accurate.

Linux and Windows

The same package and the same calls. Add the package as on Apple platforms, add the Who product, and write the code above unchanged:

Swift
import Who

let who = Who()                                        // downloads the 8-bit build on first use
let result = try await who.diarize(contentsOf: url)    // or diarize(samples:sampleRate:)

What differs underneath: the model runs through LiteRT on the CPU, and the download is the 8-bit build, 32 MB, the one precision published for these platforms. Who(variant: .fp16) is an error here rather than a silent switch, since there is no float16 build to load. The turns match the Apple platforms' within the arithmetic of the runtimes.

Threads: LiteRT takes half the machine's logical cores, at most eight; set DAL_LITERT_THREADS to choose.

Choosing a precision

Two precisions are published. Both compute at float16 on the Neural Engine and run at the same speed; .int8 stores its weights as 8-bit integers and halves the download, at a cost of about half a point of diarization error and with no saving in memory, since the Engine decompresses the weights at load. .fp16 is the default and the one to ship unless the download size decides.

Swift
let who = Who(variant: .int8)                          // about half the download

.fp16 is the default.

Downloading ahead of time

Swift
let who = Who()
if !who.isDownloaded() {
    try await who.download { fraction in print("\(Int(fraction * 100))%") }
}

Who(directory:) points the SDK at a directory you manage. If it already holds the model files the SDK uses them offline; otherwise it downloads them there.

What comes back

Diarization.turns is sorted by start time. Turns of different speakers can overlap, because people do; a turn is one stretch of one speaker's speech, split where that speaker pauses for longer than a frame. speakerCount is the number of distinct speakers heard. Speakers are numbered from zero in the order they are first heard, so the labels mean the same thing from one run to the next on the same recording, and nothing else: the model does not know who the people are.

Files

Each folder holds the model for one runtime, plus its own copy of the four supporting files.

DirectoryRuntimeWeightsSize
coreml-fp16/Core ML, compiledfloat1681MB
coreml-int8/Core ML, compiled8-bit42MB
litert-int8/LiteRT8-bit32MB
onnx-int8/ONNX Runtime8-bit matrix products, the rest float3299MB

The Core ML and ONNX folders hold five graphs each. The LiteRT folder holds the same five as signatures of one file, who.tflite.

FileContents
encoder_6000From 60s of audio to voice frames
queries_6000The speakers in those 60s and when each one speaks
encoder_30000From 300s of audio to voice frames
queries_30000The speakers in those 300s and when each one speaks
voice_3256A voice print of a 30s stretch, used to match speakers across sections
who_meta.jsonThe front end's geometry, each graph's input and output names, the decision thresholds
who_mel_banks.f32The mel filterbank the encoders read, 80 by 257, little-endian
who_voice_banks.f32The mel filterbank the voice graph reads
who_embed.f16The voice graph's last layer, as a table

Inputs and outputs

  • In: 16kHz mono audio of any length, from a few seconds up. The SDK converts other sample rates for you.
  • Out: each speaker, numbered in the order they first speak, and when they start and stop talking. Times are in seconds, to the nearest 10ms. When two people talk at once, both are marked. Who labels up to four speakers in each five-minute section, and more across a whole recording.

Languages

Who works from the sound of each voice and has no language setting. Every score on this page was measured on English recordings.

Results

Measured through the SDK on whole recordings.

CorpusSamplesDiarization error rateMissed speechFalse alarmSpeaker confusion
AMI meetings, headset mix160.1260.0790.0260.021
VoxConverse, up to 4 speakers860.0840.0270.0400.017
VoxConverse, 2-21 speakers2320.1750.0260.0730.076
NOTSOFAR-1 meetings, far-field, up to 4 speakers650.1620.0600.0520.050
NOTSOFAR-1 meetings, far-field, 3-8 speakers1290.2430.0870.0570.098
Earnings-21 calls, 2-20 speakers440.2770.0590.0550.163
DiPCo dinner parties, far-field100.4110.2070.0640.140
CHiME-6 dinner parties, far-field20.3990.1900.0810.128

On VoxConverse, the error rate rises with the number of speakers in a recording.

SpeakersSamplesDiarization error rate
1 to 2460.077
3180.116
4220.078
5 to 8850.170
9 or more610.247

Performance

Through the SDK.

DeviceSDKWorkloadResult
MacBook Pro, M5 Pro, macOS 26.6desert-ant-core 4.2.0, coremlfloat16 build, 16 meetings, 544 minutes953x faster than real time
MacBook Pro, M5 Pro, macOS 26.6desert-ant-core 4.2.0, coreml8-bit build, 16 meetings, 544 minutes965x faster than real time
MacBook Pro, M5 Pro, macOS 26.6desert-ant-core 4.2.0, coremlfloat16 build, model loaded241MB memory
MacBook Pro, M5 Pro, macOS 26.6desert-ant-core 4.2.0, coremlfloat16 build, an 8-minute recording279MB peak memory
MacBook Pro, M5 Pro, macOS 26.6desert-ant-core 4.2.0, coremlfloat16 build, a 34-minute recording377MB peak memory
MacBook Pro, M5 Pro, macOS 26.6desert-ant-core 4.2.0, coremlfloat16 build, a 2.65-hour recording867MB peak memory

On CPU alone, the 8-bit LiteRT build runs the same 544 minutes in 89s on the M5 Pro, measured outside the SDK.

Limitations

  • More than four speakers in five minutes. Who splits a recording into five-minute sections and can label up to four speakers per section. Who matches voices across sections, so it can identify more than four speakers in a longer recording. On VoxConverse, the error rate is 8% with one or two speakers, versus 17% with five to eight and 25% with nine or more.
  • Far-field recordings. Who misses more speech when the speakers are far away from the microphone. In group conversations recorded across the room, Who misses 19-21% of the speech, versus 8% in meetings where everyone wears a headset. Who does best on close-miked audio, such as podcasts, interviews and calls. On room recordings, expect stretches of speech with no speaker label.
  • Overlapping speech. Who misses the most speech when two people talk at once. On AMI meetings, Who misses 8% of all speech, and about 65% of that is overlapping talk. Expect more misses in lively discussions with lots of interruptions.
  • Short stretches of speech. Who mixes up speakers more often when many people each speak for only a few seconds. On Earnings-21 calls, with up to 20 speakers, Who assigns 16% of the speech to the wrong speaker. Expect more mix-ups on calls with lots of short remarks.
  • Laughter, applause and jingles. Who can give a laugh, applause or a jingle its own speaker number. To hide these, set the SDK to drop any speaker with less than a chosen share of the recording's speech.
  • Who labels speakers as 1, 2, 3 and so on, not by name, and doesn't transcribe. For a transcript labeled by speaker, pair Who with Voz.

Built on

Open models and tools this work builds on, with their licenses:

Dataset attributions are in the weights' third-party notices, linked under License.