Table of Contents

Class MultiModalLiveClient

Namespace
GenerativeAI.Live
Assembly
GenerativeAI.Live.dll

A client for interacting with the Gemini Multimodal Live API using WebSockets.

public class MultiModalLiveClient : IDisposable
Inheritance
MultiModalLiveClient
Implements
Inherited Members
Extension Methods

Constructors

MultiModalLiveClient(IPlatformAdapter, string, GenerationConfig?, ICollection<SafetySetting>?, string?, bool, bool, ILogger?)

Represents a client for managing multi-modal interactions with generative models.

public MultiModalLiveClient(IPlatformAdapter platformAdapter, string modelName, GenerationConfig? config = null, ICollection<SafetySetting>? safetySettings = null, string? systemInstruction = null, bool inputAudioTranscriptionEnabled = false, bool outputAudioTranscriptionEnabled = false, ILogger? logger = null)

Parameters

platformAdapter IPlatformAdapter
modelName string
config GenerationConfig
safetySettings ICollection<SafetySetting>
systemInstruction string
inputAudioTranscriptionEnabled bool
outputAudioTranscriptionEnabled bool
logger ILogger

Properties

Client

Gets the WebSocket client used to communicate with the Gemini Multimodal Live API.

public IWebsocketClient? Client { get; }

Property Value

IWebsocketClient

Config

Gets the configuration settings for content generation.

public GenerationConfig? Config { get; }

Property Value

GenerationConfig

ConnectionId

Gets the unique identifier for this WebSocket connection.

public Guid ConnectionId { get; }

Property Value

Guid

ContextWindowCompression

Context window compression settings sent in the setup message, allowing sessions longer than the context window.

public ContextWindowCompressionConfig? ContextWindowCompression { get; set; }

Property Value

ContextWindowCompressionConfig

FunctionTools

Gets or sets the list of function tools available for the session.

public List<IFunctionTool>? FunctionTools { get; set; }

Property Value

List<IFunctionTool>

HistoryConfig

History settings sent in the setup message.

public HistoryConfig? HistoryConfig { get; set; }

Property Value

HistoryConfig

InputAudioTranscriptionConfig

Options for transcribing input audio (languages, vocabulary, diarization and so on). When set, input transcription is enabled with these options regardless of InputAudioTranscriptionEnabled.

public AudioTranscriptionConfig? InputAudioTranscriptionConfig { get; set; }

Property Value

AudioTranscriptionConfig

InputAudioTranscriptionEnabled

Gets or sets a value indicating whether input audio transcription is enabled. When enabled, audio inputs will be transcribed into text for further processing.

public bool InputAudioTranscriptionEnabled { get; set; }

Property Value

bool

LastSessionResumptionHandle

The most recent resumable session handle received from the server, or null if none has been received. Only populated when SessionResumption is set.

public string? LastSessionResumptionHandle { get; }

Property Value

string

ModelName

Gets or sets the name of the model being used.

public string ModelName { get; set; }

Property Value

string

OutputAudioTranscriptionConfig

Options for transcribing output audio. When set, output transcription is enabled with these options regardless of OutputAudioTranscriptionEnabled.

public AudioTranscriptionConfig? OutputAudioTranscriptionConfig { get; set; }

Property Value

AudioTranscriptionConfig

OutputAudioTranscriptionEnabled

Indicates whether transcription of output audio is enabled.

public bool OutputAudioTranscriptionEnabled { get; set; }

Property Value

bool

Proactivity

Proactivity settings sent in the setup message (for example proactive audio).

public ProactivityConfig? Proactivity { get; set; }

Property Value

ProactivityConfig

RealtimeInputConfig

Voice activity detection and turn-taking settings sent in the setup message. Set Disabled to true to signal turns yourself with SendActivityStartAsync(CancellationToken) and SendActivityEndAsync(CancellationToken).

public RealtimeInputConfig? RealtimeInputConfig { get; set; }

Property Value

RealtimeInputConfig

ResendSetupOnReconnect

Whether to re-send the last setup message automatically after the websocket reconnects (default true). The Live API requires a setup message at the start of every connection. If SessionResumption is set and a resumable handle has been received, the session is resumed with that handle.

public bool ResendSetupOnReconnect { get; set; }

Property Value

bool

SafetySettings

Gets the collection of safety settings applied to the generation process.

public ICollection<SafetySetting>? SafetySettings { get; }

Property Value

ICollection<SafetySetting>

SessionResumption

Session resumption settings sent in the setup message. To resume a previous session, set Handle to LastSessionResumptionHandle from that session.

public SessionResumptionConfig? SessionResumption { get; set; }

Property Value

SessionResumptionConfig

SystemInstruction

Gets the system instruction to guide model behavior during the session.

public string? SystemInstruction { get; }

Property Value

string

ToolConfig

Gets or sets the configuration settings for the enabled tools.

public ToolConfig? ToolConfig { get; set; }

Property Value

ToolConfig

UseCodeExecutor

Gets or sets a value indicating whether the code executor is enabled for the session.

public bool UseCodeExecutor { get; set; }

Property Value

bool

UseGoogleSearch

Gets or sets a value indicating whether Google Search is enabled for the session.

public bool UseGoogleSearch { get; set; }

Property Value

bool

WebsocketClientFactory

Optional factory that ConnectAsync(bool, CancellationToken) uses to create the WebSocket client instead of the default reconnecting Websocket.Client.WebsocketClient. It receives the Live API endpoint URL from the platform adapter (which, for the Gemini API, already contains the API key) and must return a client that has not been started; ConnectAsync(bool, CancellationToken) subscribes to it and then calls Websocket.Client.IWebsocketClient.Start(). When null (the default), the client is created as before.

public Func<string, IWebsocketClient>? WebsocketClientFactory { get; set; }

Property Value

Func<string, IWebsocketClient>

Remarks

The factory is responsible for any request headers the endpoint needs. In particular, the default client adds an Authorization: Bearer header from GetAccessTokenAsync(CancellationToken) (required by Vertex AI); a custom factory must add it itself. With an ephemeral token (auth_tokens/...) as the API key, the URL is the constrained endpoint and the factory must send Authorization: Token {token} instead. This is useful for custom transports, proxies, and unit testing.

Methods

AddFunctionTools(List<IFunctionTool>, ToolConfig?)

Assigns a collection of function tools and the associated configuration to the client.

public void AddFunctionTools(List<IFunctionTool> functionTools, ToolConfig? toolConfig)

Parameters

functionTools List<IFunctionTool>

The list of function tools to be added.

toolConfig ToolConfig

The configuration settings for the added tools.

ConnectAsync(bool, CancellationToken)

Connects to the MultiModal Live API WebSocket endpoint.

public Task ConnectAsync(bool autoSendSetup = true, CancellationToken cancellationToken = default)

Parameters

autoSendSetup bool
cancellationToken CancellationToken

Returns

Task

A task representing the asynchronous operation.

DisconnectAsync(CancellationToken)

Disconnects the client from the MultiModal Live API and releases related resources.

public Task DisconnectAsync(CancellationToken cancellationToken = default)

Parameters

cancellationToken CancellationToken

A token to monitor for cancellation requests.

Returns

Task

A task representing the asynchronous operation.

Dispose()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.

public void Dispose()

Dispose(bool)

Releases the unmanaged resources used by the MultiModalLiveClient and optionally releases the managed resources.

protected virtual void Dispose(bool disposing)

Parameters

disposing bool

true to release both managed and unmanaged resources; false to release only unmanaged resources.

SendActivityEndAsync(CancellationToken)

Signals that the user has stopped speaking. Only use this when automatic activity detection is disabled through RealtimeInputConfig.

public Task SendActivityEndAsync(CancellationToken cancellationToken = default)

Parameters

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SendActivityStartAsync(CancellationToken)

Signals that the user has started speaking. Only use this when automatic activity detection is disabled through RealtimeInputConfig.

public Task SendActivityStartAsync(CancellationToken cancellationToken = default)

Parameters

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

Remarks

Activity markers delimit audio input. The Gemini API rejects realtime text sent between SendActivityStartAsync(CancellationToken) and SendActivityEndAsync(CancellationToken) ("Precondition check failed"); send text with SendRealtimeTextAsync(string, CancellationToken) outside the markers instead.

SendAudioAsync(byte[], string, CancellationToken)

Asynchronously sends audio data to the multi-modal client for processing.

public Task SendAudioAsync(byte[] audioData, string mimeType = "audio/pcm;rate=16000", CancellationToken cancellationToken = default)

Parameters

audioData byte[]

The audio data in byte array format to be sent.

mimeType string

The MIME type of the audio data. Defaults to "audio/pcm; rate=16000;".

cancellationToken CancellationToken

A token to cancel the asynchronous operation if needed.

Returns

Task

A task that represents the asynchronous operation.

SendAudioStreamEndAsync(CancellationToken)

Signals that the audio stream has ended, for example because the microphone was turned off, so the server can flush any buffered audio. Gemini API only.

public Task SendAudioStreamEndAsync(CancellationToken cancellationToken = default)

Parameters

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SendClientContentAsync(LiveClientContent, CancellationToken)

Sends a client content message to the connected generative AI service.

public Task SendClientContentAsync(LiveClientContent clientContent, CancellationToken cancellationToken = default)

Parameters

clientContent LiveClientContent

The content to be sent, encapsulated in a LiveClientContent object.

cancellationToken CancellationToken

A token to observe for cancellation of the send operation.

Returns

Task

A task that represents the asynchronous operation.

SendContextUpdateAsync(LiveClientContextUpdate, CancellationToken)

Updates the system instruction and/or tools of the running session without reconnecting.

public Task SendContextUpdateAsync(LiveClientContextUpdate contextUpdate, CancellationToken cancellationToken = default)

Parameters

contextUpdate LiveClientContextUpdate

The context update to send.

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SendContextUpdateAsync(string?, IEnumerable<Tool>?, CancellationToken)

Updates the system instruction and/or tools of the running session without reconnecting.

public Task SendContextUpdateAsync(string? systemInstruction, IEnumerable<Tool>? tools = null, CancellationToken cancellationToken = default)

Parameters

systemInstruction string

The new system instruction, or null to keep the current one.

tools IEnumerable<Tool>

The new set of tools, or null to keep the current ones. An empty collection removes all tools. This does not change FunctionTools, which the client uses to execute function calls; update it as well if you add or remove function declarations.

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SendRealtimeInputAsync(LiveClientRealtimeInput, CancellationToken)

Sends a realtime input message (audio, video, text or activity signals).

public Task SendRealtimeInputAsync(LiveClientRealtimeInput realtimeInput, CancellationToken cancellationToken = default)

Parameters

realtimeInput LiveClientRealtimeInput

The realtime input to send.

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SendRealtimeTextAsync(string, CancellationToken)

Sends text as realtime input. Unlike SentTextAsync(string, CancellationToken), which sends a complete user turn, realtime text is processed like streamed audio and can interrupt the model.

public Task SendRealtimeTextAsync(string text, CancellationToken cancellationToken = default)

Parameters

text string

The text to send.

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SendSetupAsync(LiveClientSetup, CancellationToken)

Configures the multi-modal live client by sending setup details including generation settings, tools, and model configurations asynchronously.

public Task SendSetupAsync(LiveClientSetup setup, CancellationToken cancellationToken = default)

Parameters

setup LiveClientSetup

The setup configuration for the bidirectional generate content session.

cancellationToken CancellationToken

A cancellation token that may be used to cancel the asynchronous operation prior to its completion.

Returns

Task

A task that represents the asynchronous operation.

Remarks

If Tools is null or empty, this method will automatically include tools from the client's FunctionTools property, as well as code execution and Google Search tools if UseCodeExecutor or UseGoogleSearch are enabled.

SendSetupAsync(CancellationToken)

Sends a setup configuration that initializes the generative model with appropriate tools, system instructions, and generation settings.

public Task SendSetupAsync(CancellationToken cancellationToken = default)

Parameters

cancellationToken CancellationToken

A token to observe for cancellation requests during the setup process.

Returns

Task

A task that represents the asynchronous operation of sending the setup configuration.

SendToolResponseAsync(LiveClientToolResponse, CancellationToken)

Sends a tool response message of type LiveClientToolResponse through the WebSocket connection.

public Task SendToolResponseAsync(LiveClientToolResponse toolResponse, CancellationToken cancellationToken = default)

Parameters

toolResponse LiveClientToolResponse

The tool response to be sent.

cancellationToken CancellationToken

A cancellation token to observe while waiting for the operation to complete.

Returns

Task

A task that represents the asynchronous send operation.

SendVideoFrameAsync(byte[], string, CancellationToken)

Sends a video frame (for example a camera or screen capture image) as realtime input.

public Task SendVideoFrameAsync(byte[] frameData, string mimeType = "image/jpeg", CancellationToken cancellationToken = default)

Parameters

frameData byte[]

The encoded frame.

mimeType string

The MIME type of the frame. Defaults to "image/jpeg".

cancellationToken CancellationToken

A token to cancel the operation.

Returns

Task

A task that represents the asynchronous operation.

SentTextAsync(string, CancellationToken)

Sends a text prompt as a user input to the model for processing.

public Task SentTextAsync(string prompt, CancellationToken cancellationToken = default)

Parameters

prompt string

The text input provided by the user to the model.

cancellationToken CancellationToken

A token to observe while waiting for the task to complete, allowing cancellation if needed.

Returns

Task

A task that represents the asynchronous operation.

Events

AudioChunkReceived

Event triggered when an audio chunk is received.

public event EventHandler<AudioBufferReceivedEventArgs>? AudioChunkReceived

Event Type

EventHandler<AudioBufferReceivedEventArgs>

AudioReceiveCompleted

Event triggered when the audio reception is completed.

public event EventHandler<AudioBufferReceivedEventArgs>? AudioReceiveCompleted

Event Type

EventHandler<AudioBufferReceivedEventArgs>

ClientCreated

Event triggered after the client is created, but before a connection is attempted.

public event EventHandler<ClientCreatedEventArgs>? ClientCreated

Event Type

EventHandler<ClientCreatedEventArgs>

Connected

Event triggered when the WebSocket client is successfully connected.

public event EventHandler? Connected

Event Type

EventHandler

Disconnected

Event triggered when the WebSocket client is disconnected.

public event EventHandler? Disconnected

Event Type

EventHandler

ErrorOccurred

Event triggered when an error occurs.

public event EventHandler<ErrorEventArgs>? ErrorOccurred

Event Type

EventHandler<ErrorEventArgs>

GenerationInterrupted

Event triggered when generation is interrupted.

public event EventHandler? GenerationInterrupted

Event Type

EventHandler

GoAwayReceived

Message sent by the server to indicate that the current connection should be terminated and the client should cease sending further requests on this stream. This is often used for graceful shutdown or when the server is no longer able to process requests on the current stream.

public event EventHandler<LiveServerGoAway>? GoAwayReceived

Event Type

EventHandler<LiveServerGoAway>

InputTranscriptionReceived

Event triggered upon receiving input transcription data.

public event EventHandler<Transcription>? InputTranscriptionReceived

Event Type

EventHandler<Transcription>

MessageReceived

Event triggered when a message is received from the server.

public event EventHandler<MessageReceivedEventArgs>? MessageReceived

Event Type

EventHandler<MessageReceivedEventArgs>

OutputTranscriptionReceived

An event triggered when an output transcription is received from the system.

public event EventHandler<Transcription>? OutputTranscriptionReceived

Event Type

EventHandler<Transcription>

SessionResumableUpdateReceived

Occurs when the server sends an update that allows the current session to be resumed. This event provides information related to session resumption, enabling the client to continue an existing session without starting over.

public event EventHandler<LiveServerSessionResumptionUpdate>? SessionResumableUpdateReceived

Event Type

EventHandler<LiveServerSessionResumptionUpdate>

TextChunkReceived

Event triggered when a chunk of text is received from the server during the live API session.

public event EventHandler<TextChunkReceivedArgs>? TextChunkReceived

Event Type

EventHandler<TextChunkReceivedArgs>

TurnCompleted

Occurs when the server signals that the model's turn is complete (for both text and audio responses).

public event EventHandler? TurnCompleted

Event Type

EventHandler

See Also