How to use Azure Openai Realtime API in this

[ihuzaifashoukat]

Hey,

I am using azure openai services and i tried adding those details but failed to do it was getting 400 error Below is the api route example.

`import { NextResponse } from 'next/server';
import { WebSocket } from 'ws';

export async function POST(request) {
try {
const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) {
throw new Error('OPENAI_API_KEY is not set');
}

    const apiVersion = '2024-10-01-preview';
    const url = `azure URL`;

    return new Promise((resolve, reject) => {
        const ws = new WebSocket(url, {
            headers: {
                'api-key': apiKey,
            }
        });

        ws.on('open', () => {
            console.log('WebSocket connection established');
            const message = JSON.stringify({
                voice: 'alloy',
                modalities: ['audio', 'text'],
                instructions: "Start conversation with the user by saying 'Hello, how can I help you today?'",
                tools: [
                    {
                        type: 'function',
                        name: 'getPageHTML',
                        description: 'Gets the HTML for the current page',
                        parameters: {
                            type: 'object',
                            properties: {}
                        }
                    },
                    {
                        type: 'function',
                        name: 'getWeather',
                        description: 'Gets the current weather',
                        parameters: {
                            type: 'object',
                            properties: {}
                        }
                    },
                    {
                        type: 'function',
                        name: 'getCurrentTime',
                        description: 'Gets the current time',
                        parameters: {
                            type: 'object',
                            properties: {}
                        }
                    },
                ],
                tool_choice: 'auto',
            });
            ws.send(message);
        });

        ws.on('message', (data) => {
            console.log('Received:', data);
            const parsedData = JSON.parse(data);
            ws.close();
            resolve(NextResponse.json(parsedData));
        });

        ws.on('error', (error) => {
            console.error('WebSocket error:', error);
            reject(error);
        });

        ws.on('close', (code, reason) => {
            console.log(`WebSocket closed with code ${code} and reason: ${reason}`);
        });
    });
} catch (error) {
    console.error('Error in session API:', error);
    return NextResponse.json({ error: error.message || 'An unexpected error occurred' }, { status: 500 });
}

}import { NextResponse } from 'next/server';
import { WebSocket } from 'ws';

export async function POST(request) {
try {
const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) {
throw new Error('OPENAI_API_KEY is not set');
}

    const apiVersion = '2024-10-01-preview';
    const url = `azure URL`;

    return new Promise((resolve, reject) => {
        const ws = new WebSocket(url, {
            headers: {
                'api-key': apiKey,
            }
        });

        ws.on('open', () => {
            console.log('WebSocket connection established');
            const message = JSON.stringify({
                voice: 'alloy',
                modalities: ['audio', 'text'],
                instructions: "Start conversation with the user by saying 'Hello, how can I help you today?'",
                tools: [
                    {
                        type: 'function',
                        name: 'getPageHTML',
                        description: 'Gets the HTML for the current page',
                        parameters: {
                            type: 'object',
                            properties: {}
                        }
                    },
                    {
                        type: 'function',
                        name: 'getWeather',
                        description: 'Gets the current weather',
                        parameters: {
                            type: 'object',
                            properties: {}
                        }
                    },
                    {
                        type: 'function',
                        name: 'getCurrentTime',
                        description: 'Gets the current time',
                        parameters: {
                            type: 'object',
                            properties: {}
                        }
                    },
                ],
                tool_choice: 'auto',
            });
            ws.send(message);
        });

        ws.on('message', (data) => {
            console.log('Received:', data);
            const parsedData = JSON.parse(data);
            ws.close();
            resolve(NextResponse.json(parsedData));
        });

        ws.on('error', (error) => {
            console.error('WebSocket error:', error);
            reject(error);
        });

        ws.on('close', (code, reason) => {
            console.log(`WebSocket closed with code ${code} and reason: ${reason}`);
        });
    });
} catch (error) {
    console.error('Error in session API:', error);
    return NextResponse.json({ error: error.message || 'An unexpected error occurred' }, { status: 500 });
}

}`

[cameronking4]

on it!

[cameronking4]

Looks like Azure OpenAI doesn't support WebRTC yet

[ihuzaifashoukat]

Thanks, Can you try it with Websocket?

…

On Tue, 24 Dec 2024, 7:16 am Cameron King, ***@***.***> wrote: Looks like Azure OpenAI doesn't support WebRTC yet — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A56HLWSBH7TWVSN4K2YRLSL2HC7YRAVCNFSM6AAAAABUDBHD72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRQGU2DINRXGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cameronking4]

@ihuzaifashoukat Yeah, I can try to do a clone using websocket. In meantime I found this one https://github.com/Azure-Samples/aoai-realtime-audio-sdk

[ihuzaifashoukat]

Thanks, I need nextjs example I have tried these but doesn't work In nextjs architecture.

…

On Fri, 27 Dec 2024, 12:54 am Cameron King, ***@***.***> wrote: @ihuzaifashoukat <https://github.com/ihuzaifashoukat> Yeah, I can try to do a clone using websocket. In meantime I found this one https://github.com/Azure-Samples/aoai-realtime-audio-sdk — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A56HLWTDMUUAX7MNPYYWV632HRNIPAVCNFSM6AAAAABUDBHD72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRTGA3DCNJWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[manoharant]

Does anyone know, when the Azure Openai with WebRTC support available?

[LI-Jialu]

I'm also looking for it, I also hope to use Azure api for voice chat webiste. I am wondering if it's possible to use https://github.com/Azure-Samples/aoai-realtime-audio-sdk as the backend? And deploy frontend like this repo?