EarPrompt

EarPrompt is a small device worn during a conversation. When the other person finishes speaking, it whispers one short coaching tip into the wearer’s earbud: a nudge on how to respond, never a script. It was my team’s undergraduate thesis at the Polytechnic University of the Philippines, EarPrompt: A Real-Time Conversational AI Assistant for Enhancing Verbal Communication, with four named researchers. This page covers the software, most of which I wrote; the research, evaluation and manuscript belong to the whole team.

The problem

Speech anxiety and cognitive overload can make a person go blank in the middle of a conversation. For someone working with a speech-language pathologist, that moment is exactly when help is most useful and least available: practice apps give feedback after the fact, and phone assistants need you to look away and ask.

We wanted a cue that arrives during the conversation, privately, in the ear. That set the constraints:

• It must not speak for the user. The tip says how to respond, never the words to say — the prompt forbids writing a literal reply.
• It has to know who is talking. A tip triggered by the wearer’s own voice is noise.
• It must be short and in the user’s language — natural Taglish, one sentence.
• It runs on a battery in a small enclosure, so the device can’t host large models.

How it works

1. 01

   Listen

   An I²S microphone streams audio. A neural voice-activity detector, checked against a room-noise calibration, decides when an utterance has ended.

   Silero VAD · energy fallback

2. 02

   Who spoke?

   A speaker embedding of the utterance is compared with the wearer’s registered voiceprint. Only the other person’s speech triggers a tip.

   SpeechBrain ECAPA-TDNN

3. 03

   Transcribe

   Silence is trimmed before upload, then the utterance is transcribed.

   ElevenLabs Scribe

4. 04

   Coach

   An LLM writes one Taglish tip of 12 words or fewer, in the chosen tone, using a compact memory of recent turns.

   OpenAI GPT-4.1 mini

5. 05

   Speak

   LLM tokens are streamed into text-to-speech as they arrive, and the audio plays in a Bluetooth earbud.

   ElevenLabs Flash v2.5

Here is that loop in a real session. The device sits on the table, and each tip it whispered is captioned on screen as it arrived.

Around that loop, the device runs a small Flask API that the companion app uses to start and stop listening, switch between automatic and push-button turn detection, register a voiceprint, calibrate room noise, change the voice and tone, and read battery level from an INA219 sensor on the power board. The wearer’s own speech isn’t wasted: it goes to a voice-analysis module that measures pitch, intensity, jitter, shimmer and harmonics-to-noise ratio with Praat (via Parselmouth) and compares them with the user’s baseline.

The Flutter app has two roles. Users register their voice, choose automatic or manual mode, calibrate the room and review past sessions; therapists see their linked patients’ session history. Accounts and session data live in Firebase.

Every turn is timed stage by stage, and the app shows it next to the voice analysis, so a slow session can be traced to transcription, the LLM or speech synthesis instead of guessed at.

My contribution

• The audio and AI pipeline — voice-activity detection, speaker verification, transcription, the LLM prompt and the streaming text-to-speech path. I wrote 2,465 of the 2,663 lines in the pipeline module (backend/assistant.py).
• The device API and hardware control — the Flask endpoints the app calls, the button and LED handling, and battery monitoring over I²C.
• Voice analysis — the acoustic measurements and the baseline comparison that raises alerts such as “pitch too high” or “unstable pitch”.
• Most of the Flutter app — the therapist’s patient list and session history, voice registration, device Wi-Fi setup and the dashboard — about 90% of the app’s Dart code.

Key technical challenges

Telling the wearer apart from the other person

A tip should follow the other person’s turn. Each utterance is embedded with an ECAPA-TDNN model and compared, by cosine similarity, with the centroid of the wearer’s enrollment embeddings (outliers are filtered out at registration). Scripted enrollment and natural conversation produce noticeably lower same-speaker scores than a model card would suggest: in real conversations the wearer typically scored 0.40–0.65 against their centroid and conversation partners 0.05–0.25, so the threshold was set at 0.35.

Short replies were the hard case. With a 0.8 s minimum, a quick “yes” or “okay” from the wearer was too short to embed, got filed as the other person, and set off a tip at exactly the wrong moment. I lowered the minimum to 0.4 s of speech after silence trimming, which covers those replies.

Trade-off: anything still too short to embed reliably is treated as the wearer’s. The system would rather stay quiet than whisper a tip at the wrong moment.

Knowing when someone has finished speaking

Energy-based silence detection alone kept “hearing” speech whenever background noise kept the level up, so turns didn’t end. I added Silero’s neural voice-activity detector on 32 ms windows, with gain for soft or distant speech and a room-noise calibration step in the app. Energy is still the fallback when the detector is unavailable or undecided.

Latency with three cloud services in a row

Speech-to-text, the LLM and text-to-speech run one after another, so every hop counts. Tips are capped at one sentence (12 words, 30 tokens), and instead of waiting for the full LLM reply, the tokens are forwarded to the text-to-speech WebSocket as they arrive, so audio generation starts before the LLM has finished.

Memory needed a bound too. A context-window overflow found in testing was fixed by updating the prompt-truncation logic, and the current code sends a compact summary rather than the transcript: at most ten keywords and two short snippets from the last five turns. The thesis’s cost analysis (Fig. 7) shows the effect of keeping more turns: tokens grow linearly rather than running away, and even 25 turns of context cost under seven US cents per 30-minute session.

Results

All figures are from the thesis evaluation. Latency is the Prompt Delivery Window: the time from the end of the other person’s speech to the tip being delivered.

• Average 3.8 s
• P95 4.2 s

0 s3 s6 s9 s12 s15 s

1. under 1 s Theoretical ideal
2. 1.4–1.7 s Industry median
3. 3–5 s Common experience (P90–P95)
4. 8–15 s Worst case: conversation breaks down

• 5 turns 399
• 10 turns 689
• 15 turns 978
• 20 turns 1,267
• 25 turns 1,556

Two more measurements from the thesis:

• Power and runtime (estimated, not measured on a discharge test): 8.88 W average and 13.64 W peak from component ratings, giving about 3 hours at average load on the two 18650 cells.
• Maintainability: five logged fixes took an average of 1.6 hours to diagnose, patch and validate, the longest being the context-window overflow (2.5 h). The acceptability survey used a 4-point scale. The overall mean was 2.81 (“High”). Accuracy of the tips scored highest (3.00 from both groups). Reliability scored lowest (2.64): the patient rated it “Low” (2.40), citing glitches, unclear malfunction alerts and weaker performance in long sessions.

Limitations and what I’d do next

• The evaluation is small and measured acceptability, not therapeutic effect. One patient and three speech-language pathologists took part in a single post-use evaluation. The thesis itself says the results show technical viability, not clinical effectiveness.
• 3.8 seconds is slow for conversation. It avoids the breakdown range, but tips can arrive after the moment has passed. The next step is an on-device or edge language model and cached voiceprints, which would also remove the dependency on internet access.
• Speaker verification was tuned, not formally measured. I’d record a labelled set of real sessions and report false-accept and false-reject rates instead of relying on observed score ranges.
• Hardware: a directional microphone array with noise reduction, and a smaller, cooler enclosure — the reliability complaints came from exactly these areas.

The full thesis, including methodology and survey instruments, and the source code are linked at the top of this page.