Add Gemini 2.0 multi-modal support via realtime plugin

[rjp44]

Is this something that is already being looked at?
If not, would a PR be welcome.

[nbsp]

it's on our roadmap, but not being looked into at the moment. this feature is already in Agents for Python, if you need it!
a PR would be greatly appreciated, you can have a look at livekit/agents#1084 and https://github.com/livekit/agents/pull/1214/files for an idea of how it should look

[rjp44]

Thanks, that PR and files look like they implement text modality only (pipeline agent) Gemini 2.0 via the Google OpenAI compat API and your existing OpenAI plugin.

What I'm looking to implement is the realtime interface.

As far as I can tell, this has no OpenAI compatibility API from Google so would need to be implemented from scratch targeting their WebSocket interface documented at: https://ai.google.dev/api/multimodal-live

You do have a branch that is being worked on by @jayeshp19 in agents: https://github.com/livekit/agents/tree/gemini-live-api that looks like it is taking this approach.

I'm not sure whether to wait for this to land and be ported here, or write a new plugin from scratch with just the realtime directory here.

[davidzhao]

Gemini live example can be found here on Python. As mentioned, we do plan to port this over to Node