Research
The benchmark came first, and so did the hard case
We published an Amharic ASR benchmark and scored six open models against the same test set before putting our own number next to theirs. A benchmark released alongside the model that wins it is marketing.
hohe-asr/16.1% WER/open weights
Where we stand
One number, and the conditions it was measured under
16.1% word error for Hohe ASR on the published Amharic benchmark, which is read speech. That is a claim about that test set, not about every recording you might have.
16.1%
word error, read speech
- 16.1%, measured
- Hohe ASR on the published Amharic benchmark. Read speech, one test set, six open models scored against the same one.
- telephone audio, not measured
- Substantially worse. Test on your own recordings before planning around any figure on this page.
6
open models scored
Against the same test set as ours.
1
published test set
open
weights
Where it is worse
On real telephone audio, the model is substantially worse
This is the most important sentence on this page, so it is not buried.
Telephone recordings are narrowband, noisy and full of overlapping speech, and the same model that scores 16.1% on read speech is far weaker on them. Narrowband, noisy, overlapping speech is the largest unsolved problem in our stack and it is our current priority.
If your audio is a call centre, a voicemail or a phone interview, do not plan around 16.1%. Test on your own recordings first. Our two free hours a month exist so that you can.
- 300 Hz to 3.4 kHz
- What a call carries, and all the model gets to work with.
- everything else
- Present in the benchmark recordings, gone by the time a phone call arrives.
The longer account of what the headline number hides is in 16.1% word error, and what that number hides.
The benchmark
Published first, so the number can be disputed
Six open models were scored against one Amharic test set, and ours was scored against the same one. The point of publishing the test set is that anyone can repeat the comparison.
what it compares
Word error rate on Amharic read speech, for six open models and for Hohe ASR, on the same test set. One test set means the figures are comparable to each other and to anything you score yourself.
what it does not cover
Telephone audio, and any other recording condition the test set does not contain. A model can lead on read speech and still fail on a call, which is exactly the case above.
The per-model scores and the methodology write-up are not on this page yet, and we would rather leave them off than paraphrase them. If you are evaluating us, ask for the test set and the scores and we will send them. [email protected]
The model
Open weights, small enough to run in a browser
Hohe ASR runs inside a browser tab with no GPU, and its weights are open, so the number above can be reproduced on your own hardware.
0
GPUs required
Runs in a browser tab on a mid-range Android phone.
open
weights
am
language served by the API
Amharic. The API lists the languages each model serves.
hohe-asr
model id
The hosted API serves hohe-asr. Its request and response shapes are in the models reference, and what it costs is on the pricing page.
Run the model yourself and check the number
The weights are open. If your figure differs from ours, we would rather hear it from you than read it somewhere else.