Class MultiModalLiveClient
- Namespace
- GenerativeAI.Live
- Assembly
- GenerativeAI.Live.dll
A client for interacting with the Gemini Multimodal Live API using WebSockets.
public class MultiModalLiveClient : IDisposable
- Inheritance
-
MultiModalLiveClient
- Implements
- Inherited Members
- Extension Methods
Constructors
MultiModalLiveClient(IPlatformAdapter, string, GenerationConfig?, ICollection<SafetySetting>?, string?, bool, bool, ILogger?)
Represents a client for managing multi-modal interactions with generative models.
public MultiModalLiveClient(IPlatformAdapter platformAdapter, string modelName, GenerationConfig? config = null, ICollection<SafetySetting>? safetySettings = null, string? systemInstruction = null, bool inputAudioTranscriptionEnabled = false, bool outputAudioTranscriptionEnabled = false, ILogger? logger = null)
Parameters
platformAdapterIPlatformAdaptermodelNamestringconfigGenerationConfigsafetySettingsICollection<SafetySetting>systemInstructionstringinputAudioTranscriptionEnabledbooloutputAudioTranscriptionEnabledboolloggerILogger
Properties
Client
Gets the WebSocket client used to communicate with the Gemini Multimodal Live API.
public IWebsocketClient? Client { get; }
Property Value
- IWebsocketClient
Config
Gets the configuration settings for content generation.
public GenerationConfig? Config { get; }
Property Value
ConnectionId
Gets the unique identifier for this WebSocket connection.
public Guid ConnectionId { get; }
Property Value
ContextWindowCompression
Context window compression settings sent in the setup message, allowing sessions longer than the context window.
public ContextWindowCompressionConfig? ContextWindowCompression { get; set; }
Property Value
FunctionTools
Gets or sets the list of function tools available for the session.
public List<IFunctionTool>? FunctionTools { get; set; }
Property Value
HistoryConfig
History settings sent in the setup message.
public HistoryConfig? HistoryConfig { get; set; }
Property Value
InputAudioTranscriptionConfig
Options for transcribing input audio (languages, vocabulary, diarization and so on). When set, input transcription is enabled with these options regardless of InputAudioTranscriptionEnabled.
public AudioTranscriptionConfig? InputAudioTranscriptionConfig { get; set; }
Property Value
InputAudioTranscriptionEnabled
Gets or sets a value indicating whether input audio transcription is enabled. When enabled, audio inputs will be transcribed into text for further processing.
public bool InputAudioTranscriptionEnabled { get; set; }
Property Value
LastSessionResumptionHandle
The most recent resumable session handle received from the server, or null if none has been received. Only populated when SessionResumption is set.
public string? LastSessionResumptionHandle { get; }
Property Value
ModelName
Gets or sets the name of the model being used.
public string ModelName { get; set; }
Property Value
OutputAudioTranscriptionConfig
Options for transcribing output audio. When set, output transcription is enabled with these options regardless of OutputAudioTranscriptionEnabled.
public AudioTranscriptionConfig? OutputAudioTranscriptionConfig { get; set; }
Property Value
OutputAudioTranscriptionEnabled
Indicates whether transcription of output audio is enabled.
public bool OutputAudioTranscriptionEnabled { get; set; }
Property Value
Proactivity
Proactivity settings sent in the setup message (for example proactive audio).
public ProactivityConfig? Proactivity { get; set; }
Property Value
RealtimeInputConfig
Voice activity detection and turn-taking settings sent in the setup message. Set Disabled to true to signal turns yourself with SendActivityStartAsync(CancellationToken) and SendActivityEndAsync(CancellationToken).
public RealtimeInputConfig? RealtimeInputConfig { get; set; }
Property Value
ResendSetupOnReconnect
Whether to re-send the last setup message automatically after the websocket reconnects (default true). The Live API requires a setup message at the start of every connection. If SessionResumption is set and a resumable handle has been received, the session is resumed with that handle.
public bool ResendSetupOnReconnect { get; set; }
Property Value
SafetySettings
Gets the collection of safety settings applied to the generation process.
public ICollection<SafetySetting>? SafetySettings { get; }
Property Value
SessionResumption
Session resumption settings sent in the setup message. To resume a previous session, set Handle to LastSessionResumptionHandle from that session.
public SessionResumptionConfig? SessionResumption { get; set; }
Property Value
SystemInstruction
Gets the system instruction to guide model behavior during the session.
public string? SystemInstruction { get; }
Property Value
ToolConfig
Gets or sets the configuration settings for the enabled tools.
public ToolConfig? ToolConfig { get; set; }
Property Value
UseCodeExecutor
Gets or sets a value indicating whether the code executor is enabled for the session.
public bool UseCodeExecutor { get; set; }
Property Value
UseGoogleSearch
Gets or sets a value indicating whether Google Search is enabled for the session.
public bool UseGoogleSearch { get; set; }
Property Value
WebsocketClientFactory
Optional factory that ConnectAsync(bool, CancellationToken) uses to create the WebSocket client instead of the default reconnecting Websocket.Client.WebsocketClient. It receives the Live API endpoint URL from the platform adapter (which, for the Gemini API, already contains the API key) and must return a client that has not been started; ConnectAsync(bool, CancellationToken) subscribes to it and then calls Websocket.Client.IWebsocketClient.Start(). When null (the default), the client is created as before.
public Func<string, IWebsocketClient>? WebsocketClientFactory { get; set; }
Property Value
Remarks
The factory is responsible for any request headers the endpoint needs. In particular, the default client adds
an Authorization: Bearer header from GetAccessTokenAsync(CancellationToken) (required by Vertex AI);
a custom factory must add it itself. With an ephemeral token (auth_tokens/...) as the API key, the URL is the
constrained endpoint and the factory must send Authorization: Token {token} instead.
This is useful for custom transports, proxies, and unit testing.
Methods
AddFunctionTools(List<IFunctionTool>, ToolConfig?)
Assigns a collection of function tools and the associated configuration to the client.
public void AddFunctionTools(List<IFunctionTool> functionTools, ToolConfig? toolConfig)
Parameters
functionToolsList<IFunctionTool>The list of function tools to be added.
toolConfigToolConfigThe configuration settings for the added tools.
ConnectAsync(bool, CancellationToken)
Connects to the MultiModal Live API WebSocket endpoint.
public Task ConnectAsync(bool autoSendSetup = true, CancellationToken cancellationToken = default)
Parameters
autoSendSetupboolcancellationTokenCancellationToken
Returns
- Task
A task representing the asynchronous operation.
DisconnectAsync(CancellationToken)
Disconnects the client from the MultiModal Live API and releases related resources.
public Task DisconnectAsync(CancellationToken cancellationToken = default)
Parameters
cancellationTokenCancellationTokenA token to monitor for cancellation requests.
Returns
- Task
A task representing the asynchronous operation.
Dispose()
Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
public void Dispose()
Dispose(bool)
Releases the unmanaged resources used by the MultiModalLiveClient and optionally releases the managed resources.
protected virtual void Dispose(bool disposing)
Parameters
disposingbooltrue to release both managed and unmanaged resources; false to release only unmanaged resources.
SendActivityEndAsync(CancellationToken)
Signals that the user has stopped speaking. Only use this when automatic activity detection is disabled through RealtimeInputConfig.
public Task SendActivityEndAsync(CancellationToken cancellationToken = default)
Parameters
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SendActivityStartAsync(CancellationToken)
Signals that the user has started speaking. Only use this when automatic activity detection is disabled through RealtimeInputConfig.
public Task SendActivityStartAsync(CancellationToken cancellationToken = default)
Parameters
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
Remarks
Activity markers delimit audio input. The Gemini API rejects realtime text sent between SendActivityStartAsync(CancellationToken) and SendActivityEndAsync(CancellationToken) ("Precondition check failed"); send text with SendRealtimeTextAsync(string, CancellationToken) outside the markers instead.
SendAudioAsync(byte[], string, CancellationToken)
Asynchronously sends audio data to the multi-modal client for processing.
public Task SendAudioAsync(byte[] audioData, string mimeType = "audio/pcm;rate=16000", CancellationToken cancellationToken = default)
Parameters
audioDatabyte[]The audio data in byte array format to be sent.
mimeTypestringThe MIME type of the audio data. Defaults to "audio/pcm; rate=16000;".
cancellationTokenCancellationTokenA token to cancel the asynchronous operation if needed.
Returns
- Task
A task that represents the asynchronous operation.
SendAudioStreamEndAsync(CancellationToken)
Signals that the audio stream has ended, for example because the microphone was turned off, so the server can flush any buffered audio. Gemini API only.
public Task SendAudioStreamEndAsync(CancellationToken cancellationToken = default)
Parameters
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SendClientContentAsync(LiveClientContent, CancellationToken)
Sends a client content message to the connected generative AI service.
public Task SendClientContentAsync(LiveClientContent clientContent, CancellationToken cancellationToken = default)
Parameters
clientContentLiveClientContentThe content to be sent, encapsulated in a LiveClientContent object.
cancellationTokenCancellationTokenA token to observe for cancellation of the send operation.
Returns
- Task
A task that represents the asynchronous operation.
SendContextUpdateAsync(LiveClientContextUpdate, CancellationToken)
Updates the system instruction and/or tools of the running session without reconnecting.
public Task SendContextUpdateAsync(LiveClientContextUpdate contextUpdate, CancellationToken cancellationToken = default)
Parameters
contextUpdateLiveClientContextUpdateThe context update to send.
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SendContextUpdateAsync(string?, IEnumerable<Tool>?, CancellationToken)
Updates the system instruction and/or tools of the running session without reconnecting.
public Task SendContextUpdateAsync(string? systemInstruction, IEnumerable<Tool>? tools = null, CancellationToken cancellationToken = default)
Parameters
systemInstructionstringThe new system instruction, or null to keep the current one.
toolsIEnumerable<Tool>The new set of tools, or null to keep the current ones. An empty collection removes all tools. This does not change FunctionTools, which the client uses to execute function calls; update it as well if you add or remove function declarations.
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SendRealtimeInputAsync(LiveClientRealtimeInput, CancellationToken)
Sends a realtime input message (audio, video, text or activity signals).
public Task SendRealtimeInputAsync(LiveClientRealtimeInput realtimeInput, CancellationToken cancellationToken = default)
Parameters
realtimeInputLiveClientRealtimeInputThe realtime input to send.
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SendRealtimeTextAsync(string, CancellationToken)
Sends text as realtime input. Unlike SentTextAsync(string, CancellationToken), which sends a complete user turn, realtime text is processed like streamed audio and can interrupt the model.
public Task SendRealtimeTextAsync(string text, CancellationToken cancellationToken = default)
Parameters
textstringThe text to send.
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SendSetupAsync(LiveClientSetup, CancellationToken)
Configures the multi-modal live client by sending setup details including generation settings, tools, and model configurations asynchronously.
public Task SendSetupAsync(LiveClientSetup setup, CancellationToken cancellationToken = default)
Parameters
setupLiveClientSetupThe setup configuration for the bidirectional generate content session.
cancellationTokenCancellationTokenA cancellation token that may be used to cancel the asynchronous operation prior to its completion.
Returns
- Task
A task that represents the asynchronous operation.
Remarks
If Tools is null or empty, this method will automatically include tools from the client's FunctionTools property, as well as code execution and Google Search tools if UseCodeExecutor or UseGoogleSearch are enabled.
SendSetupAsync(CancellationToken)
Sends a setup configuration that initializes the generative model with appropriate tools, system instructions, and generation settings.
public Task SendSetupAsync(CancellationToken cancellationToken = default)
Parameters
cancellationTokenCancellationTokenA token to observe for cancellation requests during the setup process.
Returns
- Task
A task that represents the asynchronous operation of sending the setup configuration.
SendToolResponseAsync(LiveClientToolResponse, CancellationToken)
Sends a tool response message of type LiveClientToolResponse through the WebSocket connection.
public Task SendToolResponseAsync(LiveClientToolResponse toolResponse, CancellationToken cancellationToken = default)
Parameters
toolResponseLiveClientToolResponseThe tool response to be sent.
cancellationTokenCancellationTokenA cancellation token to observe while waiting for the operation to complete.
Returns
- Task
A task that represents the asynchronous send operation.
SendVideoFrameAsync(byte[], string, CancellationToken)
Sends a video frame (for example a camera or screen capture image) as realtime input.
public Task SendVideoFrameAsync(byte[] frameData, string mimeType = "image/jpeg", CancellationToken cancellationToken = default)
Parameters
frameDatabyte[]The encoded frame.
mimeTypestringThe MIME type of the frame. Defaults to "image/jpeg".
cancellationTokenCancellationTokenA token to cancel the operation.
Returns
- Task
A task that represents the asynchronous operation.
SentTextAsync(string, CancellationToken)
Sends a text prompt as a user input to the model for processing.
public Task SentTextAsync(string prompt, CancellationToken cancellationToken = default)
Parameters
promptstringThe text input provided by the user to the model.
cancellationTokenCancellationTokenA token to observe while waiting for the task to complete, allowing cancellation if needed.
Returns
- Task
A task that represents the asynchronous operation.
Events
AudioChunkReceived
Event triggered when an audio chunk is received.
public event EventHandler<AudioBufferReceivedEventArgs>? AudioChunkReceived
Event Type
AudioReceiveCompleted
Event triggered when the audio reception is completed.
public event EventHandler<AudioBufferReceivedEventArgs>? AudioReceiveCompleted
Event Type
ClientCreated
Event triggered after the client is created, but before a connection is attempted.
public event EventHandler<ClientCreatedEventArgs>? ClientCreated
Event Type
Connected
Event triggered when the WebSocket client is successfully connected.
public event EventHandler? Connected
Event Type
Disconnected
Event triggered when the WebSocket client is disconnected.
public event EventHandler? Disconnected
Event Type
ErrorOccurred
Event triggered when an error occurs.
public event EventHandler<ErrorEventArgs>? ErrorOccurred
Event Type
GenerationInterrupted
Event triggered when generation is interrupted.
public event EventHandler? GenerationInterrupted
Event Type
GoAwayReceived
Message sent by the server to indicate that the current connection should be terminated and the client should cease sending further requests on this stream. This is often used for graceful shutdown or when the server is no longer able to process requests on the current stream.
public event EventHandler<LiveServerGoAway>? GoAwayReceived
Event Type
InputTranscriptionReceived
Event triggered upon receiving input transcription data.
public event EventHandler<Transcription>? InputTranscriptionReceived
Event Type
MessageReceived
Event triggered when a message is received from the server.
public event EventHandler<MessageReceivedEventArgs>? MessageReceived
Event Type
OutputTranscriptionReceived
An event triggered when an output transcription is received from the system.
public event EventHandler<Transcription>? OutputTranscriptionReceived
Event Type
SessionResumableUpdateReceived
Occurs when the server sends an update that allows the current session to be resumed. This event provides information related to session resumption, enabling the client to continue an existing session without starting over.
public event EventHandler<LiveServerSessionResumptionUpdate>? SessionResumableUpdateReceived
Event Type
TextChunkReceived
Event triggered when a chunk of text is received from the server during the live API session.
public event EventHandler<TextChunkReceivedArgs>? TextChunkReceived
Event Type
TurnCompleted
Occurs when the server signals that the model's turn is complete (for both text and audio responses).
public event EventHandler? TurnCompleted