# Speech recognition: Art of the possible Canonical page: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803 - SlideShare title: Speech Recognition: Art of the possible - DigiFest 2022 - Talk date: Not established - SlideShare upload: 2022-03-17 (upload date, not the delivery date) - Event: Not established - Place: Not established - Topics: Access & reading - Language: en - Slides in the original listing: 61 - Coverage: Full text of all 61 slides, each paired with its original slide image (slide N is section N). - Original listing: https://www.slideshare.net/slideshow/speech-recognition-art-of-the-possible-digifest-2022-251369803/251369803 - Cover image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-01.png - Reader data (JSON): https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/presentation.json - TalkWeaver package (outline and images, ZIP): https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803/speech-recognition-art-of-the-possible-digifest-2022-251369803-talkweaver.zip - TalkWeaver outline: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803/talkweaver/speech-recognition-art-of-the-possible-digifest-2022-251369803-outline.md ## Slide 1: Speech recognition: Art of the possible Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-1 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-01.png Dominik.Lukes@ctl.ox.ac.uk @techczech ## Slide 2: Dominik’s journey Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-2 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-02.png **1990–1995** - Computational linguistics - Cognitive linguistics - Language teaching **1995–2008** - Language teacher training - Translation - Metaphor / discourse studies **2009 – present** - Readability - Learning / Assistive technology - Dyslexia teacher training ## Slide 3: Bill Gates in 2011 Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-3 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-03.png > “The next big thing is definitely speech and voice recognition.” ![Photograph of Bill Gates speaking, beside a large quotation mark.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-5dfa3365e4c5cea2.png) ## Slide 4: What do we want to know? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-4 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-04.png - What is the current state of the art? - How we got here? - Where are going? ## Slide 5: Are we asking the right questions? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-5 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-05.png ## Slide 6: Tasks for speech recognition by difficulty Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-6 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-06.png 1. Select word from list 2. Interpret command 3. Type dictation 4. Transcribe presentation 5. Transcribe conversation ![A rising curved arrow with five points from easiest to hardest: Select word from list, Interpret command, Type dictation, Transcribe presentation, Transcribe conversation, each with a small icon.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-06.png) ## Slide 7: How we think of it vs how it is Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-7 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-07.png - Select word from list - Interpret command - Type dictation - Transcribe presentation - Transcribe conversation ![Two diagrams of the same five tasks. On the left, “How we think of it”: nested circles, with Select word from list innermost, then Interpret command, Type dictation, Transcribe presentation and Transcribe conversation outermost. On the right, “how it is”: five separate, overlapping circles of different sizes with the same labels and a few small empty circles around them.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-07.png) ## Slide 8: Speech recognition approximate timeline Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-8 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-08.png - 1950s: Select digit - 1970s: Select from 1000 words - 1980s: Select from large vocabulary - 1990s: Dictate word by word - 1997: Dictate whole sentences - 2012: Transcribe YouTube video - 2019: Transcribe conversation ## Slide 9: What is the actual job of speech recognition? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-9 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-09.png ## Slide 10: What is this word? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-10 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-10.png ![A narrow phonetic transcription of the word, with aspiration, a devoiced and labialised r, a glottal stop and a syllabic dark l.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-10eea7b77cd7c477.png) `[pʰɹɛtsɫ̩]` `[pɹɛtsl]` `/pretsəl/` `` ## Slide 11: What’s the problem Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-11 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-11.png - aspirated /p/ at start of a stressed syllable - devoiced /r/ following /p/ - labialised /r/ following /p/ - dark /l/ - syllabic consonant - glottal stop ![The narrow phonetic transcription of “pretzel” with six red arrows pointing at its parts, labelled aspirated /p/ at start of a stressed syllable, devoiced /r/ following /p/, labialised /r/ following /p/, dark /l/, syllabic consonant and glottal stop.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-11.png) ## Slide 12: It gets worse: find the missing sounds Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-12 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-12.png ![A waveform above a spectrogram of a spoken phrase, with a red cursor line. Beside each is a small loudspeaker icon for an audio clip; the audio is not part of this package.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-3364a4f07219785a.png) ## Slide 13: Course on speech recognition 1993 Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-13 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-13.png > Faster computers won’t help improve speech recognition. We need a new approach. ## Slide 14: Dragon Naturally Speaking released in 1997. Can recognise whole sentences. What happened? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-14 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-14.png ## Slide 15: How speech recognition does not work? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-15 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-15.png Finding individual sounds (phonemes) in the speech and matching them to letters. ## Slide 16: How speech recognition actually works? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-16 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-16.png P(W\|C) What is the likelihood that the next word is X given what came before? ## Slide 17: Actually, it is quite a bit more complicated (Huang and Deng 2009) Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-17 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-17.png ![Equation: W-hat equals the arg max over W of P(W given A), which equals the arg max over W of P(W) times P(A given W), divided by P(A).](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-5b291982e5d2f7a4.png) ![Block diagram of a speech recogniser: Voice feeds Signal processing, which feeds a Decoder; the Decoder connects to an Adaptation block, to Acoustic models and Language models, and to the Application.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-eae855b469738282.png) ## Slide 18: Probabilistic (stochastic) ASR enabled the change. Linguistics took the back seat. Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-18 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-18.png ## Slide 19: Fred Jelinek (ASR Pioneer - 1988?) Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-19 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-19.png > "Every time I fire a linguist, the performance of the speech recognizer goes up" ## Slide 20: Consequence of probabilistic approach: Worse on words not predictable from context Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-20 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-20.png - Names - Acronyms - Specialist Terms ## Slide 21: Question in 2011 Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-21 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-21.png > I recorded a lecture, can I use Dragon to transcribe it? ## Slide 22: “Caption fails” in 2014 provided source for comedy Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-22 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-22.png ![Still frame from a comedy video of a man in a teal T-shirt, with a poorly recognised caption beneath him.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-e2ee75a7b739a36a.png) ![Second frame of the same video: the man holding a phone to his ear, with a caption that misrecognises his words.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-5f203bea05d50b92.png) ![Third frame of the same video with a mis-captioned line.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-92558570929adc89.png) ![Fourth frame of the same video, partly overlapped, with another mis-captioned line.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-e784cf6621bc5d1b.png) ## Slide 23: YouTube Captions today are usable and useful Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-23 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-23.png ![Original slide 23 as rendered: the poster frame of a screen recording of a YouTube video with automatic captions, in front of a news article headlined “Dua Lipa sued by reggae band claiming ‘Levitating’ ripped off their song”; the caption reads “alipa”. The video itself is not part of this package.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-23.png) ## Slide 24: So what happened between 2014 and 2022? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-24 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-24.png ## Slide 25: Ingredients of success Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-25 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-25.png - Larger data sets - More computing power - Neural networks ## Slide 26: Patrick Winston (2015) MIT Lecture 12a in AI course Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-26 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-26.png > It was in **2010**, yes, that's right. It was in **2010**. We were having our annual discussion about what we would dump from 6034 in order to make room for some other stuff. And **we almost killed off neural nets.** That might seem strange because our heads are stuffed with neurons. … But many of us felt that the **neural models of the day weren't much in the way of faithful models** of what actually goes on inside our heads. And besides that, **nobody had ever made a neural net that was worth a darn for doing anything**. ## Slide 27: 2012 – ImageNet showed that Neural Networks are much better at computing the probabilities for complex data. Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-27 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-27.png ## Slide 28: Ok, we have neural nets, what does that mean? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-28 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-28.png ## Slide 29: Things to know about Neural Nets Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-29 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-29.png - Everything has a probability - Same input does not produce same output - They have no ‘sanity check’ or ‘common sense’ ## Slide 30: What do probabilities look like? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-30 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-30.png ![Diagram of an image classifier: a photograph of a white car feeds columns of convolution (CONV), RELU and POOL layers, then a fully connected layer (FC) with scores for car (highest), truck, airplane, ship and horse.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-8c76f5c5170c8399.jpeg) ## Slide 31: BERT Examples Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-31 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-31.png ![A table of fill-in-the-blank contexts and the words BERT-large predicts: “A robin is a ***” gives bird, robin, person, hunter, pigeon; “A daisy is a*** ” gives daisy, rose, flower, berry, tree; “A hammer is a ***” gives hammer, tool, weapon, nail, device; “A hammer is an*** ” gives object, instrument, axe, implement, explosive. For the negated contexts, “A robin is not a ***” gives robin, bird, penguin, man, fly; “A daisy is not a*** ” gives daisy, rose, flower, lily, cherry; “A hammer is not a ***” gives hammer, tool, weapon, gun, rock; “A hammer is not an*** ” gives object, instrument, axe, animal, artifact.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-c0d7642cf2c441ad.png) **What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models** [Allyson Ettinger](https://arxiv.org/search/cs?searchtype=author&query=Ettinger%2C+A) 2019 ## Slide 32: XKCD on prediction Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-32 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-32.png ![A comic strip: a sign reads “OH MY GOD, THE VOLCANO IS ERU\_”, and three people look at it; one says “ERUDITE?”.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-32fbf9366b4bb19e.png) ## Slide 33: Output changes as more information is made available. (Not always for the better) Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-33 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-33.png ## Slide 34: Examples from today’s captions Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-34 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-34.png - Crystal \> Chris is - Am \> and ![Caption screenshot in white on black: “conversation with some incredible experience galaxy brained individuals working at the forefront of SPeech”.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-26208484bf781852.png) ![Caption screenshot with the word “experts” highlighted: “with some incredible experts galaxy brained individuals working at the forefront of speech recognition, research, and that will be very much a discussion again lots of opportunity for questions.”](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-7b23d377dd568e8f.png) - experts \> experience - AR \> a our ![Caption screenshot with the words “a our” highlighted: “On this slide is an image of the the holoLens so microsoft’s a our products that commercial device.”](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-a1fa0e96555cfb93.png) ## Slide 35: Different ways of transcribing Dua Lipa Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-35 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-35.png - alipa - dualipa - dua lipa - lipa - duda lipa ## Slide 36: Rise and mostly fall of Google’s new spell Czech Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-36 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-36.png ![Screenshot of Czech text “Před nějakým … Merleau-Pontiho o tom, že pro” with the name Merleau-Pontiho underlined in red and a pop-up suggesting “Pontyho”.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-3db53ac13da4435d.png) ![Screenshot of an English sentence, “what you can do with the language for the meet your communicative goals.”, with a pop-up suggesting “to”.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-7c8e2110f723047f.png) ![Screenshot of the Czech text “-Pontyho doporučoval i” with “Pontyho” underlined in red.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-1ad0f4a8be8d4274.png) ## Slide 37: Tracking faces at the tips of the shoes Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-37 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-37.png ![Photograph of a seated man seen from behind, looking at a wall of video-call participants, with a tracking camera in the foreground.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-8635b6c76e207454.jpeg) ![Close-up of a conference camera on a stand.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-24f7d8e43d10ba95.png) ## Slide 38: Hallucination is a big problem Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-38 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-38.png ![A Wikipedia infobox for Constant Vanden Stock (birth date October 30, 1914; death date April 19, 2008, aged 93) beside the neural baseline output “Constant Vanden Stock (June 13, 1914 – April 19, 2008) was an American Figure Skater”, with “American Figure Skater” highlighted in red.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-de506c74f8fd753c.png) ## Slide 39: Question asked by faculty member in 2021 Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-39 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-39.png > We correct the transcripts, why doesn’t the system learn the correct spelling? ## Slide 40: Adding your own word list just tweaks the probabilities. Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-40 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-40.png ## Slide 41: Setting a genre setting tweaks the probabilities. Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-41 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-41.png ## Slide 42: Another thing to know about NN Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-42 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-42.png Neural Nets use very large data sets and can take days or weeks to train. ## Slide 43: Consequences of NN size Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-43 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-43.png - Speech recognition is often not done on device. - Individual input often cannot adjust the quality (except in pre-training) - Most applications use APIs from the big players - Few open source/free options ## Slide 44: Big players in the field Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-44 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-44.png ![Google logo.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-4c00ded3407890ba.png) ![Microsoft logo.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-c7641587e0950641.png) ![Amazon logo.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-a01dfcfe753069fa.png) - Google - Microsoft (now also Nuance) - Amazon ## Slide 45: Interesting smaller companies Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-45 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-45.png ![Verbit logo.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-f15678cb425ed77e.png) ![Carescribe logo: a teal speech bubble with three dots beside a teal page of text lines.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-0a00406d9cb3cd06.png) ![Otter.ai logo: a blue circle and three blue bars above the word Otter.ai.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-cd0be0c4b7e0d392.png) ![Rev.ai logo.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-a167d4696c91a6df.png) - Verbit.ai - Carescribe.io (Caption.Ed) - Otter.ai - Rev.ai ## Slide 46: Interesting applications Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-46 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-46.png - Descript - Microsoft Reading Progress - Microsoft Presentation Coach ## Slide 47: What can we expect in the future Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-47 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-47.png ## Slide 48: Cautionary tale by SMBC Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-48 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-48.png ![A comic by SMBC, “How AI reporting works”: a man says a computer using scanned human literature can make a novel statement about life, a grey creature answers with a speech bubble full of scribbles, six repeated panels show it scribbling, and the last panel has a newspaper headline “MACHINE INTELLIGENCE COMPREHENDS HUMAN DESIRES!”](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-48.png) ## Slide 49: Which Roomba is which? Video comparing 1st and latest. Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-49 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-49.png ![Original slide 49 as rendered: a frame of a video from above of two robot vacuum cleaners working on a table scattered with crumbs. The video itself is not part of this package.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-49.png) ## Slide 50: The Original Roomba (2002) vs Roomba S9+ (2019) - Wow! Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-50 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-50.png ![A video thumbnail with the words “2002 VS 2019” above two robot vacuum cleaners, an old pale one marked Roomba and a newer dark one, with curved blue arrows pointing at each.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-ea1b1671c8bad14d.jpeg) ## Slide 51: What happens in speeches Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-51 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-51.png **Fillers** **Repetition** ![Two transcript screenshots. On the left, headed Fillers, a transcript search for “you know” with matches highlighted and an enlarged line “Kind of like, yeah okay okay so”. On the right, headed Repetition, a transcript search for “to to” with matches highlighted.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-51.png) ## Slide 52: What does conversation actually look like? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-52 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-52.png ![A conversation-analysis transcript headed “Excerpt 1.1, from Heritage, 1984a: 236, NB:VII:2”: turns marked E and M, with square brackets for overlapping speech, colons for stretched sounds, equals signs for latched speech, parentheses for pauses and unclear talk, and underlining for stress. The full text of the turns is given below.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-17830e69efda6434.jpeg) **EXCERPT 1.1, FROM HERITAGE, 1984A: 236 \[NB:VII:2\]** **E:** `Oh honey that was a lovely luncheon I shoulda ca:lled you s:soo[:ner but I:]l:[lo:ved it.` **M:** `[((f)) Oh:::] [( )` **E:** `It w’s just deli:ghtfu[:l. ]` **M:** `[Well]=` **M:** `I w’s gla[d you] (came).]` **E:** `[‘nd yer f:] friends] ’re so da:rli:ng,=` **M:** `=Oh::: [: it w’z ]` **E:** `[e-that P]a:t isn’she a do:[ :ll?,]` **M:** `[iYe]h isn’t she pretty,` `(.)` **E:** `Oh: she’s a beautiful girl.=` **M:** `=Yeh I think she’s a pretty gir[l.=` **E:** `[En’ that Reinam’n::` `(.)` **E:** `She SCA:RES me.` The image prints the excerpt on 16 lines, and each turn starts on the left under its speaker label. The overlapping turns are indented so that the overlapping words stand under each other, and some letters are underlined to mark stress. The turns above keep every character and symbol as printed. The column positions and the underlining are recorded in the two tables below, which count the printed lines from the top: line 2 is the second printed line of the first E turn, and lines 11 and 15 hold only a pause. Underlined letters. Only the letters named are underlined; the colons that follow them are not. | Line | Speaker | Word | Underlined letters | |---:|----|----|----| | 1 | E | honey | h, the first letter | | 1 | E | luncheon | l, the first letter | | 1 | E | ca:lled | c, the first letter | | 2 | E | the I before the colon in `I:]` | I | | 7 | E | friends (the f of `f:]` is not underlined) | f, the first letter | | 7 | E | da:rli:ng | i, the letter after `rl` | | 8 | M | Oh, after the `=` | O and h | | 9 | E | Pa:t | P and a, as two separate underlines with the `]` between them | | 10 | M | isn’t | i, the first letter | | 12 | E | Oh: | O and h | | 13 | M | I think | I | | 16 | E | She | S, h and e | Overlap brackets. Each position is the distance in pixels from the left edge of the image, which is 728 pixels wide; two brackets are described as lining up when they are within 6 pixels of each other. | Lines | Brackets and what lines up | |----|----| | 2 and 3 | E `[:ner but I:]` and M `[((f)) Oh:::]` open at the same place (positions 122 and 119). M closes at 234, just after E’s underlined I and before E’s colon; E closes at 247, after that colon. | | 2 and 3 | E `[lo:ved it.` and M `[( )` open at the same place (266 and 265). Neither has a closing bracket in the image, so this overlap is left open. | | 4 and 5 | E `[:l. ]` and M `[Well]=` open at the same place (288 and 289). E closes at 330 and M closes at 348, about one and a half characters to the right of E. | | 4 and 6 | M `you]` on line 6 closes at 331, level with E’s closing bracket on line 4 (330); M’s closing bracket on line 5 (348) falls between them. | | 6 to 10 | Five opening brackets stand in one column (154 to 157): M `[d` on line 6, E `[‘nd` on line 7, M `[:` on line 8, E `[e-that` on line 9 and M `[iYe]` on line 10. | | 7 to 10 | The closing brackets on these lines are at 268 and 364 (E `f:]` and `friends]` on line 7), 252 (M `w’z ]` on line 8), 254 (E `P]` on line 9) and 198 (M `[iYe]` on line 10). Only the closers on lines 8 and 9 line up; line 7’s first closer is 14 pixels to their right and line 10’s is 54 pixels to their left. | | 6 | M `(came).]` closes at 434; no bracket on the lines next to it lines up with it. | | 9 | E `do:[ :ll?,]` opens at 451 and closes at 513; no bracket on lines 8 or 10 lines up with either. | | 13 and 14 | M `gir[l.=` and E `[En’ that Reinam’n::` open at the same place (402 and 407). Neither has a closing bracket in the image, so this last overlap is also left open. | ## Slide 53: Possible futures? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-53 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-53.png **Incremental improvement**, similar to Roomba in 17 years - Accurate lecture transcripts - Fluent dictation with pauses - Better meeting transcription **Revolutionary change**, similar to change in speech recognition in 6 years - Informal conversation transcription - Interactive dictation - Multilingual speech transcription ![Two large arrows pointing outwards from the middle: on the left Incremental improvement, with Fluent dictation with pauses, Accurate lecture transcripts and Better meeting transcription; on the right Revolutionary change, with Interactive dictation, Multilingual speech transcription and Informal conversation transcription.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-53.png) ## Slide 54: How should we think about accuracy? Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-54 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-54.png - We speak 120-180 words per minute - 99% accurate = 2 errors per minute ## Slide 55: From Sept 2014 xkcd.com/1425 Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-55 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-55.png Sometimes it is hard to judge how much effort will be needed to solve a seemingly easy problem. ![An xkcd comic: “When a user takes a photo, the app should check whether they’re in a national park…” “Sure, easy GIS lookup. Gimme a few hours.” “… and check whether the photo is of a bird.” “I’ll need a research team and five years.” Caption: “In CS, it can be hard to explain the difference between the easy and the virtually impossible.”](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/img-db72921f1a38661a.png) ## Slide 56: Wishlist (a few hours of coding) Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-56 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-56.png - Transcripts indicate level of confidence - Benchmarks for lecture transcripts - Better manual control of transcripts (like Descript) ## Slide 57: Dreamlist (5 years and a research team) Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-57 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-57.png - Multilingual transcription (identify change in language) - Multimodal transcription (use information from video) - Raw to readable transcript ## Slide 58: Welcome to the panel Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-58 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-58.png ## Slide 59: Panel Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-59 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-59.png - Kate Knill, Machine Intelligence Lab, University of Cambridge - Richard Cave, MND Association (and formerly Google project Euphonia) - Richard Purcell, Caption.Ed - Irit Opher, Head of Research at Verbit.ai ![Four portrait photographs in a row, each above a panellist’s name and affiliation: Kate Knill, Richard Cave, Richard Purcell and Irit Opher.](https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/media/page-59.png) ## Slide 60: Questions Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-60 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-60.png - What is the current state of the art of speech recognition in general and in the transcription of recorded speech in particular? - What are the current quality metrics and how much do they tell us about suitability of models? Do we need better ones? - After the big recent jump in performance, are we seeing a plateau with incremental growth or can we expect another step change in quality? - Where can we see the most innovation? What are the research and development blind spots where more effort is needed? - What are the currently unsolved problems for which we do not have a solution? - What is the space for smaller players to innovate in this space? How much do they have to rely on pre-trained models from big providers? Is there space for open source? ## Slide 61: Creative Commons License – CC BY Cite: https://slides.dominiklukes.net/speech-recognition-art-of-the-possible-digifest-2022-251369803#section-61 · Original slide image: https://slides.dominiklukes.net/readers/speech-recognition-art-of-the-possible-digifest-2022-251369803/pages/page-61.png This presentation is licensed under Creative Commons By Attribution license except where otherwise noted. Icons and stock images from Microsoft Office 365 creative premium. They cannot be distributed separately from this document. Collections: [Artificial intelligence](https://slides.dominiklukes.net/?collection=ai), [Reading, writing and accessibility](https://slides.dominiklukes.net/?collection=access)