Improve Voice Live Audio Input Quality — How to Choose Noise Suppression, Echo Cancellation, and Live-Reference AEC
- Azure
- MicrosoftFoundry
- VoiceLiveAPI
- dotnet
- AI
Introduction
In voice agents, the quality of the sound entering the microphone matters more than you might expect. Noise and audio bleeding in from the speaker can throw off not only speech recognition accuracy, but also speech-segment detection (VAD), interruption detection, and end-of-utterance (EOU) detection. A typical failure is when the agent mistakes its own voice for “user speech” and cuts off its own speech.
Voice Live has a three-stage mechanism for conditioning input audio.
- Noise suppression (azure_deep_noise_suppression)
- Server-side echo cancellation (server_echo_cancellation)
- Live-Reference AEC(uses the audio actually played by the client as the reference for echo cancellation)
This article explains the decision-making process for “how much to enable” and the key implementation considerations for No. 3, which requires the most effort.
This is the second installment in the Voice Live feature guide series.
List of articles in the series (SDK 1.2.0 / API 2026-07-15)
-
Changes in Voice Live SDK 1.2.0 and GA for API 2026-07-15
Categorizing SDK features that have no types -
Handling conversational “pauses” with the Voice Live API
Speak first, wait for a hesitation, interrupt, or speak during a pause - Improving Voice Live Audio Input Quality (this article)
How to choose noise suppression, echo cancellation, and Live-Reference AEC -
Using MCP server tools with Voice Live
Differences from function calling and parallel tool invocation -
Calling a Foundry agent as a tool in Voice Live
How to use the chat-supervisor pattern and choose the upstream model -
Create “your own voice” from 30 seconds of audio and make a Microsoft Foundry agent speak
From applying for Personal Voice to implementation - Choosing between Voice Live API avatars
Configuring and implementing standard, custom-photo, and custom-video avatars -
Receiving Voice Live avatar video over WebSocket
Differences from WebRTC and the video format
Bonus: Connecting to Voice Live's WebRTC audio connection using SIPSorcery for .NET
A record of comparing the SDP line by line
Here is the code used for testing.
This article reflects information as of October 2026. It covers Voice Live API 2026-07-15 (GA), while real-device testing of Live-Reference AEC was conducted using 2026-06-01-preview (the reason is explained later).
An overview of the three stages
| Stage | Configuration | What it removes | Implementation effort |
|---|---|---|---|
| 1. Noise suppression | input_audio_noise_reduction | Ambient noise | Configuration only |
| 2. Server-side echo cancellation | input_audio_echo_cancellation (reference is server-side) | Echo of the agent's own voice | Configuration only |
| 3. Live-Reference AEC | Same as above (reference is client-side) | the sound actually output through the speakers echoing back | Large (stereo reference generation) |
For 1 and 2, you only need to add one line each to session.update.
In the sample, 1 and 2 are always enabled, but no tests have been conducted to compare recognition accuracy and the like with them enabled and disabled. Only 3 involves actual measurements in this article.
Noise suppression offers three modes
Initially, noise suppression had only one option, azure_deep_noise_suppression,but you can now choose from three modes.It depends on "which voice you want to keep."
| type | Behavior | Use case |
|---|---|---|
| azure_deep_noise_suppression | Suppresses background noise and is optimized for the speaker closest to the microphone | Conventional general-purpose noise suppression |
| azure_deep_noise_suppression_headset | Preserves the voices of nearby speakers while suppressing background "voices" and noise | Use when you want to let through only the speaker directly in front of you in an environment where people are talking nearby |
| azure_personal_noise_suppression | Use information about the target speaker to keep only that person's voice (also remove the voices of other speakers in front of you) | Simply separating foreground from background is not enough; you want to narrow it down on a per-speaker basis |
The difference is whether you separate based on "background or foreground" or based on "whose voice it is." If you want to eliminate conversations at the next table in a café, use headset; if multiple people are in the same place and you want to pick up only one specific person, use personal
Both headset and personal also have a detection_input setting
| detection_input | Expected environment |
|---|---|
| raw_ec (default) | General environments such as phone calls|
| mas_dns | Noisy, acoustically challenging environments
There are three points to keep in mind before using it.
- Choosing the noise suppression mode is independent of enabling or disabling echo cancellation. input_audio_echo_cancellation is configured separately
- session.updated only returns the "requested settings" and does not indicate whether personalization is actually working (this "accepted, but unclear whether it works" pattern appears throughout Voice Live)
- Speaker targeting is not an identity verification mechanism. Also, noise suppression does not guarantee that other speakers will not appear in the transcription
Note that, since the author has not yet compared and tested the three modes, Live-Reference AEC remains the one accompanied by empirical measurements in this article.
Another choice: transcription model
From the perspective of input audio quality,which transcription model you usealso makes a difference. There are now more options here as well. It is specified with input_audio_transcription.model.
| Transcription model | Compatible chat model |
|---|---|
| azure-speech | All non-multimodal models and agents (enabled by default) |
| mai-transcribe / mai-transcribe-2 (preview) | gpt-realtime, gpt-realtime-mini, all non-multimodal models, agents |
| whisper-1 / gpt-4o-transcribe / gpt-4o-mini-transcribe / gpt-4o-transcribe-diarize | gpt-realtime and gpt-realtime-mini only |
If the combination is incorrect, the session will be rejected. OpenAI transcription models can only be combined with realtime models, and specifying a cascade model such as gpt-4o will cause them to be rejected. The author previously misinterpreted this as “realtime models cannot be used in this region.”
The mai-transcribe family supports phrase_list (listing terms you want to recognize in order to improve accuracy). There is also the lower-latency mai-transcribe-2-streaming (preview), but it is available in different regions from mai-transcribe-2, so check the region table before choosing.
Deciding how much to include
First, determine whether server-side echo cancellation is sufficient
Server-side echo cancellation uses the TTS audio sent by the server as a reference signal to subtract the agent's own voice from the microphone input. In other words, it assumes that “audio sent by the server = audio actually played through the speaker.”
The official documentation also explains the limitations of this assumption.
The service assumes the client plays response audio as soon as it receives it. If playback is delayed for more than two seconds, echo cancellation quality is impacted.
― How to use the Voice Live API
Conversely, in an environment where this assumption holds, server-side processing is sufficient.
| Environments where server-side processing is sufficient | Reason |
|---|---|
| Headsets/headphones | Sound leakage is inherently minimal |
| Just play the received TTS as-is | The reference matches the actual playback audio |
| Keep the client simple | It isn't worth the implementation cost of 3 |
Environments where Live-Reference AEC is needed
The premise breaks down whenthe sound coming from the speaker does not match the server's TTS.
| Environment | Why a server-side reference alone is insufficient |
|---|---|
| Volume adjustment, EQ, normalization, and device DSP (loudness correction, spatial audio, etc.) | The subtraction no longer works if the amplitude or frequency response changes|
| Re-encoding and resampling over Bluetooth, etc. | The waveform does not match the original TTS |
| Simultaneously playing sound effects, BGM, notification sounds, and video or game audio | App-specific sounds are not included in the server's reference |
| Multiple agents or avatars speak simultaneously | What comes out of the speaker is the post-mix audio |
| Playback latency, jitter, and interruptions during playback (barging-in, etc.) | The guarantee that it "plays back exactly as sent" breaks down |
| The speaker and microphone are close together, or the volume is high (kiosks, in-vehicle systems, robots, smart speakers, XR headsets) | There is substantial sound bleed, so the accuracy of the reference directly affects the result |
A guideline is: "The more the speaker's actual output differs from the server's TTS, the more valuable 3 becomes." However, as described below, if the reference cannot be correctly time-aligned, it can have the opposite effect. If it can be built, 3; if not, 2 is a practical dividing line.
Live-Reference AEC
How it works
Instead of sending the microphone audio alone,send stereo PCM16 data with the microphone (channel 0) and the actual playback audio (channel 1) interleaved.
Instead of the internal TTS loopback, the server uses channel 1 as the reference for echo cancellation. The echo cancellation processing itself is performed on the server, and the client only passes the reference.
Settings
Only two combinations of reference_source and channels are permitted.
| reference_source | channels | Result |
|---|---|---|
| server (default) | 1 (default) | OK. Traditional server-side echo cancellation |
| server | 2 | Error invalid_ec_channels_requires_client |
| client | 1 | Error invalid_ec_reference_channels |
| client | 2 | OK. Live-Reference AEC |
Also, reference_source, channels, input_audio_format, and input_audio_sampling_rate cannot be changed during a session. This feature was added in API 2026-06-01-preview and is also available in 2026-07-15 (GA). .
Bandwidth nearly doubles because it becomes stereo. According to the official estimates, at 24 kHz PCM16, the raw data is approximately 94 KB/s, and approximately 125 KB/s after base64 encoding. Since reference channels are removed before billing is measured, audio charges do not increase.
In SDK 1.2.0, ReferenceSource and Channels were added to AudioEchoCancellation, so the configuration can be written in a type-safe manner.
Connect with a feature flag
To use reference_source and channels, you must add &features=client_ec_reference:true to the WebSocket URL (as documented in the API reference).
The live testing in this article was conducted with 2026-06-01-preview ＋ this flag. Since it has not been confirmed whether it works without the flag, it is safest to try it with the flag first.
An official browser sample (Live-Reference AEC sample) is also available. If you want to try it in a web app, this is the quickest route.
https://github.com/microsoft-foundry/voicelive-samples/tree/main/javascript/live-reference-aec
Implementation key point: aligning the timing of the reference signal
The configuration is only a few lines, but it is difficult to get it working
. The server uses the supplied reference as-is for subtraction, so if the reference is out of sync with the sound actually coming from the speaker, the echo will not be removed. The sample’s initial naïve implementation (enqueueing the played audio into a FIFO and dequeuing it for each microphone frame) gradually drifted and never returned to alignment after a while.The following four points ultimately helped stabilize it.
1. Do not discard audio in the playback buffer
The response audio arrives in batches faster than real time. The sample’s playback buffer (NAudio’s BufferedWaveProvider) was initially 10 seconds and was configured to discard overflowing audio (DiscardOnBufferOverflow). This caused audio to be silently dropped during longer responses,so the audio enqueued as the reference no longer matched the audio actually played.
We expanded it to 120 seconds and disabled discarding.
This was not just an echo-removal issue; it was a bug affecting all modes. If responses longer than 10 seconds are intermittently cut off midway, suspect an overflow of the playback buffer.
2. Make the reference only read at the “Latest − Pending Playback” position
What the microphone is currently picking up is the sound currently coming from the speaker. It is only behind the “latest audio the client has enqueued in the playback queue” by the amount of audio “still remaining in the playback buffer (pending playback).”
The key is to recalculate this position for every microphone frame and only read from it. If you dequeue and consume it, the read position advances at the microphone’s rate. When playback lags, that read position gets ahead of the actual playback, and once it gets ahead, it never returns. If you recalculate it every time, it naturally returns to alignment on the next frame even if it drifts.
The “playback backlog” is obtained from the amount remaining in the playback buffer (BufferedWaveProvider.BufferedBytes). The reference history must be retained for longer than the playback backlog, so the sample allocates 30 seconds’ worth.
3. Do not lock the playback buffer in the microphone callback
If you try to read the playback backlog accurately and lock the playback buffer inside the microphone callback, playback will be interrupted. Read the remaining amount without locking, and absorb any slight error through “recalculating each time” in section 2.
4. Reset the reference as well at the beginning
When the user interrupts and the playback buffer is discarded, discard the reference history as well. Otherwise, sounds that will no longer be played remain in the reference, and subsequent position calculations become inaccurate.
How to check whether it is working
Because the effect of echo cancellation is difficult to judge by ear, the sample outputs statistics every five seconds.
Check the following two points.
- playback backlog (playback backlog) and ref buffered (reference history) are tracking each other within approximately ±100 ms
- The ending remains aligned (there are no frames where the reference runs ahead of playback because there is insufficient history)
On the actual device, it remained aligned for 120 consecutive seconds, and the reference tracked the playback latency within ±100 ms. Self-triggering caused by the microphone picking up the agent’s voice and masking its own utterance no longer occurs.
This implementation does not model latency beyond the playback buffer (device or driver latency). It is a best effort to verify the mechanism over the wire, not full acoustic alignment. Even so, it was sufficient to eliminate self-triggering.
Summary
| Situation | What to choose |
|---|---|
| There are people talking nearby | Set noise suppression to headset mode |
| Want to pick up only one specific person among multiple people | Set noise suppression to personal mode |
| Just play the headset and TTS as-is | Noise suppression + server-side echo cancellation |
| The speaker output does not match the TTS (DSP, BGM mixing, Bluetooth, high volume) | Live-Reference AEC |
| Unable to implement timing alignment for Live-Reference AEC | Leave server-side echo cancellation as is |
- Noise suppression and server-side echo cancellation can be enabled only through the settings. Choose from three noise suppression modes based on “which voices to keep”
- There are constraints on combinations of transcription models. OpenAI-family models can only be combined with realtime-family models
- For Live-Reference AEC, the configuration is only a few lines; most of the implementation is aligning the reference in time
- Do not discard audio in the playback buffer
- For the reference, simply recalculate “latest − playback latency” each time and read it
- Do not lock the playback buffer in the microphone callback
- Also reset the reference on barge-in
- Append &features=client_ec_reference:true to the connection URL
Next time, we will cover MCP server tools and parallel tool calls.
Reference links
Go to list of users who liked
Go to list of comments
Register as a new user and use Qiita more conveniently
- You get articles that match your needs
- You can efficiently read back useful information
- You can use dark theme
