---
title: "Speech recognition:"
subtitle: "Art of the possible"
author: "Dominik Lukeš, @techczech"
outline_version: 2
auto_title_slide: false
auto_thanks_slide: false
---

## Speech recognition: Art of the possible
{id=pptx-0983bd1a80c7-001} {title}

Dominik.Lukes@ctl.ox.ac.uk @techczech

## Dominik’s journey
{id=pptx-0983bd1a80c7-002}

**1990–1995**

- Computational linguistics
- Cognitive linguistics
- Language teaching

**1995–2008**

- Language teacher training
- Translation
- Metaphor / discourse studies

**2009 – present**

- Readability
- Learning / Assistive technology
- Dyslexia teacher training

## Bill Gates in 2011
{id=pptx-0983bd1a80c7-003}

> “The next big thing is definitely speech and voice recognition.”

![Photograph of Bill Gates speaking, beside a large quotation mark.](assets/media/img-5dfa3365e4c5cea2.png)

## What do we want to know?
{id=pptx-0983bd1a80c7-004}

- What is the current state of the art?
- How we got here?
- Where are going?

## Are we asking the right questions?
{id=pptx-0983bd1a80c7-005}

## Tasks for speech recognition by difficulty
{id=pptx-0983bd1a80c7-006}

1. Select word from list
2. Interpret command
3. Type dictation
4. Transcribe presentation
5. Transcribe conversation

![A rising curved arrow with five points from easiest to hardest: Select word from list, Interpret command, Type dictation, Transcribe presentation, Transcribe conversation, each with a small icon.](assets/pages/page-06.png)

## How we think of it vs how it is
{id=pptx-0983bd1a80c7-007}

- Select word from list
- Interpret command
- Type dictation
- Transcribe presentation
- Transcribe conversation

![Two diagrams of the same five tasks. On the left, “How we think of it”: nested circles, with Select word from list innermost, then Interpret command, Type dictation, Transcribe presentation and Transcribe conversation outermost. On the right, “how it is”: five separate, overlapping circles of different sizes with the same labels and a few small empty circles around them.](assets/pages/page-07.png)

## Speech recognition approximate timeline
{id=pptx-0983bd1a80c7-008}

- 1950s: Select digit
- 1970s: Select from 1000 words
- 1980s: Select from large vocabulary
- 1990s: Dictate word by word
- 1997: Dictate whole sentences
- 2012: Transcribe YouTube video
- 2019: Transcribe conversation

## What is the actual job of speech recognition?
{id=pptx-0983bd1a80c7-009}

## What is this word?
{id=pptx-0983bd1a80c7-010}

![A narrow phonetic transcription of the word, with aspiration, a devoiced and labialised r, a glottal stop and a syllabic dark l.](assets/media/img-10eea7b77cd7c477.png)

`[pʰɹɛtsɫ̩]`

`[pɹɛtsl]`

`/pretsəl/`

`<pretzel>`

## What’s the problem
{id=pptx-0983bd1a80c7-011}

- aspirated /p/ at start of a stressed syllable
- devoiced /r/ following /p/
- labialised /r/ following /p/
- dark /l/
- syllabic consonant
- glottal stop

![The narrow phonetic transcription of “pretzel” with six red arrows pointing at its parts, labelled aspirated /p/ at start of a stressed syllable, devoiced /r/ following /p/, labialised /r/ following /p/, dark /l/, syllabic consonant and glottal stop.](assets/pages/page-11.png)

## It gets worse: find the missing sounds
{id=pptx-0983bd1a80c7-012}

![A waveform above a spectrogram of a spoken phrase, with a red cursor line. Beside each is a small loudspeaker icon for an audio clip; the audio is not part of this package.](assets/media/img-3364a4f07219785a.png)

## Course on speech recognition 1993
{id=pptx-0983bd1a80c7-013}

> Faster computers won’t help improve speech recognition. We need a new approach.

## Dragon Naturally Speaking released in 1997. Can recognise whole sentences. What happened?
{id=pptx-0983bd1a80c7-014}

## How speech recognition does not work?
{id=pptx-0983bd1a80c7-015}

Finding individual sounds (phonemes) in the speech and matching them to letters.

## How speech recognition actually works?
{id=pptx-0983bd1a80c7-016}

P(W|C)

What is the likelihood that the next word is X given what came before?

## Actually, it is quite a bit more complicated (Huang and Deng 2009)
{id=pptx-0983bd1a80c7-017}

![Equation: W-hat equals the arg max over W of P(W given A), which equals the arg max over W of P(W) times P(A given W), divided by P(A).](assets/media/img-5b291982e5d2f7a4.png)

![Block diagram of a speech recogniser: Voice feeds Signal processing, which feeds a Decoder; the Decoder connects to an Adaptation block, to Acoustic models and Language models, and to the Application.](assets/media/img-eae855b469738282.png)

## Probabilistic (stochastic) ASR enabled the change. Linguistics took the back seat.
{id=pptx-0983bd1a80c7-018}

## Fred Jelinek (ASR Pioneer - 1988?)
{id=pptx-0983bd1a80c7-019}

> "Every time I fire a linguist, the performance of the speech recognizer goes up"

## Consequence of probabilistic approach: Worse on words not predictable from context
{id=pptx-0983bd1a80c7-020}

- Names
- Acronyms
- Specialist Terms

## Question in 2011
{id=pptx-0983bd1a80c7-021}

> I recorded a lecture, can I use Dragon to transcribe it?

## “Caption fails” in 2014 provided source for comedy
{id=pptx-0983bd1a80c7-022}

![Still frame from a comedy video of a man in a teal T-shirt, with a poorly recognised caption beneath him.](assets/media/img-e2ee75a7b739a36a.png)

![Second frame of the same video: the man holding a phone to his ear, with a caption that misrecognises his words.](assets/media/img-5f203bea05d50b92.png)

![Third frame of the same video with a mis-captioned line.](assets/media/img-92558570929adc89.png)

![Fourth frame of the same video, partly overlapped, with another mis-captioned line.](assets/media/img-e784cf6621bc5d1b.png)

## YouTube Captions today are usable and useful
{id=pptx-0983bd1a80c7-023}

![Original slide 23 as rendered: the poster frame of a screen recording of a YouTube video with automatic captions, in front of a news article headlined “Dua Lipa sued by reggae band claiming ‘Levitating’ ripped off their song”; the caption reads “alipa”. The video itself is not part of this package.](assets/pages/page-23.png)

## So what happened between 2014 and 2022?
{id=pptx-0983bd1a80c7-024}

## Ingredients of success
{id=pptx-0983bd1a80c7-025}

- Larger data sets
- More computing power
- Neural networks

## Patrick Winston (2015) MIT Lecture 12a in AI course
{id=pptx-0983bd1a80c7-026}

> It was in **2010**, yes, that's right. It was in **2010**. We were having our annual discussion about what we would dump from 6034 in order to make room for some other stuff. And **we almost killed off neural nets.** That might seem strange because our heads are stuffed with neurons. … But many of us felt that the **neural models of the day weren't much in the way of faithful models** of what actually goes on inside our heads. And besides that, **nobody had ever made a neural net that was worth a darn for doing anything**.

## 2012 – ImageNet showed that Neural Networks are much better at computing the probabilities for complex data.
{id=pptx-0983bd1a80c7-027}

## Ok, we have neural nets, what does that mean?
{id=pptx-0983bd1a80c7-028}

## Things to know about Neural Nets
{id=pptx-0983bd1a80c7-029}

- Everything has a probability
- Same input does not produce same output
- They have no ‘sanity check’ or ‘common sense’

## What do probabilities look like?
{id=pptx-0983bd1a80c7-030}

![Diagram of an image classifier: a photograph of a white car feeds columns of convolution (CONV), RELU and POOL layers, then a fully connected layer (FC) with scores for car (highest), truck, airplane, ship and horse.](assets/media/img-8c76f5c5170c8399.jpeg)

## BERT Examples
{id=pptx-0983bd1a80c7-031}

![A table of fill-in-the-blank contexts and the words BERT-large predicts: “A robin is a ___” gives bird, robin, person, hunter, pigeon; “A daisy is a ___” gives daisy, rose, flower, berry, tree; “A hammer is a ___” gives hammer, tool, weapon, nail, device; “A hammer is an ___” gives object, instrument, axe, implement, explosive. For the negated contexts, “A robin is not a ___” gives robin, bird, penguin, man, fly; “A daisy is not a ___” gives daisy, rose, flower, lily, cherry; “A hammer is not a ___” gives hammer, tool, weapon, gun, rock; “A hammer is not an ___” gives object, instrument, axe, animal, artifact.](assets/media/img-c0d7642cf2c441ad.png)

**What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models**

[Allyson Ettinger](https://arxiv.org/search/cs?searchtype=author&query=Ettinger%2C+A) 2019

## XKCD on prediction
{id=pptx-0983bd1a80c7-032}

![A comic strip: a sign reads “OH MY GOD, THE VOLCANO IS ERU_”, and three people look at it; one says “ERUDITE?”.](assets/media/img-32fbf9366b4bb19e.png)

[https://what-if.xkcd.com/34](https://what-if.xkcd.com/34)

## Output changes as more information is made available. (Not always for the better)
{id=pptx-0983bd1a80c7-033}

## Examples from today’s captions
{id=pptx-0983bd1a80c7-034}

- Crystal > Chris is
- Am > and

![Caption screenshot in white on black: “conversation with some incredible experience galaxy brained individuals working at the forefront of SPeech”.](assets/media/img-26208484bf781852.png)

![Caption screenshot with the word “experts” highlighted: “with some incredible experts galaxy brained individuals working at the forefront of speech recognition, research, and that will be very much a discussion again lots of opportunity for questions.”](assets/media/img-7b23d377dd568e8f.png)

- experts > experience
- AR > a our

![Caption screenshot with the words “a our” highlighted: “On this slide is an image of the the holoLens so microsoft’s a our products that commercial device.”](assets/media/img-a1fa0e96555cfb93.png)

## Different ways of transcribing Dua Lipa
{id=pptx-0983bd1a80c7-035}

- alipa
- dualipa
- dua lipa
- lipa
- duda lipa

## Rise and mostly fall of Google’s new spell Czech
{id=pptx-0983bd1a80c7-036}

![Screenshot of Czech text “Před nějakým … Merleau-Pontiho o tom, že pro” with the name Merleau-Pontiho underlined in red and a pop-up suggesting “Pontyho”.](assets/media/img-3db53ac13da4435d.png)

![Screenshot of an English sentence, “what you can do with the language for the meet your communicative goals.”, with a pop-up suggesting “to”.](assets/media/img-7c8e2110f723047f.png)

![Screenshot of the Czech text “-Pontyho doporučoval i” with “Pontyho” underlined in red.](assets/media/img-1ad0f4a8be8d4274.png)

## Tracking faces at the tips of the shoes
{id=pptx-0983bd1a80c7-037}

![Photograph of a seated man seen from behind, looking at a wall of video-call participants, with a tracking camera in the foreground.](assets/media/img-8635b6c76e207454.jpeg)

![Close-up of a conference camera on a stand.](assets/media/img-24f7d8e43d10ba95.png)

## Hallucination is a big problem
{id=pptx-0983bd1a80c7-038}

![A Wikipedia infobox for Constant Vanden Stock (birth date October 30, 1914; death date April 19, 2008, aged 93) beside the neural baseline output “Constant Vanden Stock (June 13, 1914 – April 19, 2008) was an American Figure Skater”, with “American Figure Skater” highlighted in red.](assets/media/img-de506c74f8fd753c.png)

## Question asked by faculty member in 2021
{id=pptx-0983bd1a80c7-039}

> We correct the transcripts, why doesn’t the system learn the correct spelling?

## Adding your own word list just tweaks the probabilities.
{id=pptx-0983bd1a80c7-040}

## Setting a genre setting tweaks the probabilities.
{id=pptx-0983bd1a80c7-041}

## Another thing to know about NN
{id=pptx-0983bd1a80c7-042}

Neural Nets use very large data sets and can take days or weeks to train.

## Consequences of NN size
{id=pptx-0983bd1a80c7-043}

- Speech recognition is often not done on device.
- Individual input often cannot adjust the quality (except in pre-training)
- Most applications use APIs from the big players
- Few open source/free options

## Big players in the field
{id=pptx-0983bd1a80c7-044}

![Google logo.](assets/media/img-4c00ded3407890ba.png)

![Microsoft logo.](assets/media/img-c7641587e0950641.png)

![Amazon logo.](assets/media/img-a01dfcfe753069fa.png)

- Google
- Microsoft (now also Nuance)
- Amazon

## Interesting smaller companies
{id=pptx-0983bd1a80c7-045}

![Verbit logo.](assets/media/img-f15678cb425ed77e.png)

![Carescribe logo: a teal speech bubble with three dots beside a teal page of text lines.](assets/media/img-0a00406d9cb3cd06.png)

![Otter.ai logo: a blue circle and three blue bars above the word Otter.ai.](assets/media/img-cd0be0c4b7e0d392.png)

![Rev.ai logo.](assets/media/img-a167d4696c91a6df.png)

- Verbit.ai
- Carescribe.io (Caption.Ed)
- Otter.ai
- Rev.ai

## Interesting applications
{id=pptx-0983bd1a80c7-046}

- Descript
- Microsoft Reading Progress
- Microsoft Presentation Coach

## What can we expect in the future
{id=pptx-0983bd1a80c7-047}

## Cautionary tale by SMBC
{id=pptx-0983bd1a80c7-048}

![A comic by SMBC, “How AI reporting works”: a man says a computer using scanned human literature can make a novel statement about life, a grey creature answers with a speech bubble full of scribbles, six repeated panels show it scribbling, and the last panel has a newspaper headline “MACHINE INTELLIGENCE COMPREHENDS HUMAN DESIRES!”](assets/pages/page-48.png)

## Which Roomba is which? Video comparing 1st and latest.
{id=pptx-0983bd1a80c7-049}

![Original slide 49 as rendered: a frame of a video from above of two robot vacuum cleaners working on a table scattered with crumbs. The video itself is not part of this package.](assets/pages/page-49.png)

## The Original Roomba (2002) vs Roomba S9+ (2019) - Wow!
{id=pptx-0983bd1a80c7-050}

![A video thumbnail with the words “2002 VS 2019” above two robot vacuum cleaners, an old pale one marked Roomba and a newer dark one, with curved blue arrows pointing at each.](assets/media/img-ea1b1671c8bad14d.jpeg)

## What happens in speeches
{id=pptx-0983bd1a80c7-051}

**Fillers**

**Repetition**

![Two transcript screenshots. On the left, headed Fillers, a transcript search for “you know” with matches highlighted and an enlarged line “Kind of like, yeah okay okay so”. On the right, headed Repetition, a transcript search for “to to” with matches highlighted.](assets/pages/page-51.png)

## What does conversation actually look like?
{id=pptx-0983bd1a80c7-052}

![A conversation-analysis transcript headed “Excerpt 1.1, from Heritage, 1984a: 236, NB:VII:2”: turns marked E and M, with square brackets for overlapping speech, colons for stretched sounds, equals signs for latched speech, parentheses for pauses and unclear talk, and underlining for stress. The full text of the turns is given below.](assets/media/img-17830e69efda6434.jpeg)

**EXCERPT 1.1, FROM HERITAGE, 1984A: 236 [NB:VII:2]**

**E:** `Oh honey that was a lovely luncheon I shoulda ca:lled you s:soo[:ner but I:]l:[lo:ved it.`

**M:** `[((f)) Oh:::]  [(    )`

**E:** `It w’s just deli:ghtfu[:l. ]`

**M:** `[Well]=`

**M:** `I w’s gla[d you] (came).]`

**E:** `[‘nd yer f:] friends] ’re so da:rli:ng,=`

**M:** `=Oh::: [: it w’z ]`

**E:** `[e-that P]a:t isn’she a do:[ :ll?,]`

**M:** `[iYe]h isn’t she pretty,`

`(.)`

**E:** `Oh: she’s a beautiful girl.=`

**M:** `=Yeh I think she’s a pretty gir[l.=`

**E:** `[En’ that Reinam’n::`

`(.)`

**E:** `She SCA:RES me.`

The image prints the excerpt on 16 lines, and each turn starts on the left under its speaker label. The overlapping turns are indented so that the overlapping words stand under each other, and some letters are underlined to mark stress. The turns above keep every character and symbol as printed. The column positions and the underlining are recorded in the two tables below, which count the printed lines from the top: line 2 is the second printed line of the first E turn, and lines 11 and 15 hold only a pause.

Underlined letters. Only the letters named are underlined; the colons that follow them are not.

| Line | Speaker | Word | Underlined letters |
| ---: | --- | --- | --- |
| 1 | E | honey | h, the first letter |
| 1 | E | luncheon | l, the first letter |
| 1 | E | ca:lled | c, the first letter |
| 2 | E | the I before the colon in `I:]` | I |
| 7 | E | friends (the f of `f:]` is not underlined) | f, the first letter |
| 7 | E | da:rli:ng | i, the letter after `rl` |
| 8 | M | Oh, after the `=` | O and h |
| 9 | E | Pa:t | P and a, as two separate underlines with the `]` between them |
| 10 | M | isn’t | i, the first letter |
| 12 | E | Oh: | O and h |
| 13 | M | I think | I |
| 16 | E | She | S, h and e |

Overlap brackets. Each position is the distance in pixels from the left edge of the image, which is 728 pixels wide; two brackets are described as lining up when they are within 6 pixels of each other.

| Lines | Brackets and what lines up |
| --- | --- |
| 2 and 3 | E `[:ner but I:]` and M `[((f)) Oh:::]` open at the same place (positions 122 and 119). M closes at 234, just after E’s underlined I and before E’s colon; E closes at 247, after that colon. |
| 2 and 3 | E `[lo:ved it.` and M `[(    )` open at the same place (266 and 265). Neither has a closing bracket in the image, so this overlap is left open. |
| 4 and 5 | E `[:l. ]` and M `[Well]=` open at the same place (288 and 289). E closes at 330 and M closes at 348, about one and a half characters to the right of E. |
| 4 and 6 | M `you]` on line 6 closes at 331, level with E’s closing bracket on line 4 (330); M’s closing bracket on line 5 (348) falls between them. |
| 6 to 10 | Five opening brackets stand in one column (154 to 157): M `[d` on line 6, E `[‘nd` on line 7, M `[:` on line 8, E `[e-that` on line 9 and M `[iYe]` on line 10. |
| 7 to 10 | The closing brackets on these lines are at 268 and 364 (E `f:]` and `friends]` on line 7), 252 (M `w’z ]` on line 8), 254 (E `P]` on line 9) and 198 (M `[iYe]` on line 10). Only the closers on lines 8 and 9 line up; line 7’s first closer is 14 pixels to their right and line 10’s is 54 pixels to their left. |
| 6 | M `(came).]` closes at 434; no bracket on the lines next to it lines up with it. |
| 9 | E `do:[ :ll?,]` opens at 451 and closes at 513; no bracket on lines 8 or 10 lines up with either. |
| 13 and 14 | M `gir[l.=` and E `[En’ that Reinam’n::` open at the same place (402 and 407). Neither has a closing bracket in the image, so this last overlap is also left open. |

## Possible futures?
{id=pptx-0983bd1a80c7-053}

**Incremental improvement**, similar to Roomba in 17 years

- Accurate lecture transcripts
- Fluent dictation with pauses
- Better meeting transcription

**Revolutionary change**, similar to change in speech recognition in 6 years

- Informal conversation transcription
- Interactive dictation
- Multilingual speech transcription

![Two large arrows pointing outwards from the middle: on the left Incremental improvement, with Fluent dictation with pauses, Accurate lecture transcripts and Better meeting transcription; on the right Revolutionary change, with Interactive dictation, Multilingual speech transcription and Informal conversation transcription.](assets/pages/page-53.png)

## How should we think about accuracy?
{id=pptx-0983bd1a80c7-054}

- We speak 120-180 words per minute
- 99% accurate = 2 errors per minute

## From Sept 2014 xkcd.com/1425
{id=pptx-0983bd1a80c7-055}

Sometimes it is hard to judge how much effort will be needed to solve a seemingly easy problem.

![An xkcd comic: “When a user takes a photo, the app should check whether they’re in a national park…” “Sure, easy GIS lookup. Gimme a few hours.” “… and check whether the photo is of a bird.” “I’ll need a research team and five years.” Caption: “In CS, it can be hard to explain the difference between the easy and the virtually impossible.”](assets/media/img-db72921f1a38661a.png)

## Wishlist (a few hours of coding)
{id=pptx-0983bd1a80c7-056}

- Transcripts indicate level of confidence
- Benchmarks for lecture transcripts
- Better manual control of transcripts (like Descript)

## Dreamlist (5 years and a research team)
{id=pptx-0983bd1a80c7-057}

- Multilingual transcription (identify change in language)
- Multimodal transcription (use information from video)
- Raw to readable transcript

## Welcome to the panel
{id=pptx-0983bd1a80c7-058}

## Panel
{id=pptx-0983bd1a80c7-059}

- Kate Knill, Machine Intelligence Lab, University of Cambridge
- Richard Cave, MND Association (and formerly Google project Euphonia)
- Richard Purcell, Caption.Ed
- Irit Opher, Head of Research at Verbit.ai

![Four portrait photographs in a row, each above a panellist’s name and affiliation: Kate Knill, Richard Cave, Richard Purcell and Irit Opher.](assets/pages/page-59.png)

## Questions
{id=pptx-0983bd1a80c7-060}

- What is the current state of the art of speech recognition in general and in the transcription of recorded speech in particular?
- What are the current quality metrics and how much do they tell us about suitability of models? Do we need better ones?
- After the big recent jump in performance, are we seeing a plateau with incremental growth or can we expect another step change in quality?
- Where can we see the most innovation? What are the research and development blind spots where more effort is needed?
- What are the currently unsolved problems for which we do not have a solution?
- What is the space for smaller players to innovate in this space? How much do they have to rely on pre-trained models from big providers? Is there space for open source?

## Creative Commons License – CC BY
{id=pptx-0983bd1a80c7-061}

This presentation is licensed under Creative Commons By Attribution license except where otherwise noted.

Icons and stock images from Microsoft Office 365 creative premium. They cannot be distributed separately from this document.