RNN vs Transformer for Streaming Speech Recognition on a Phone

A team builds on-device live captioning for phone calls. Requirements: words must appear within about 300 ms of being spoken, the model must run for hours of continuous audio, memory is limited, and accuracy on long utterances must stay high. They have a strong offline Transformer encoder-decoder ASR model (bidirectional-style full attention over the whole recording) and ask whether to ship it unchanged or to build an LSTM-based streaming model.

1. Explain, with numbers, why the offline Transformer cannot be shipped unchanged. Audio frames arrive at 100 per second and one hour of audio is 360,000 frames.
2. Describe how a recurrent streaming model handles the same input. What are its time and memory per new frame and what are its weaknesses?
3. Describe the middle path: how can an attention-based encoder be made streamable, and what accuracy and latency trade-offs does that introduce?
4. Give your recommendation, the training loss family you would look at (alignment-free), and what you would measure before deciding.