Server SDK quickstart

This page shows you how to build a server that starts and stops a voice AI agent using the Agora Conversational AI SDKs. The SDKs wrap the Conversational AI REST API with typed vendor configuration, automatic token generation, and a session lifecycle API.

Prerequisites

Before you begin, make sure you have:

Enabled Agora Conversational AI for your project.
Completed the Voice Calling or Video Calling quickstart.
Your App ID and App Certificate from Agora Console.
API keys for your chosen STT, LLM, and TTS providers. This guide uses Deepgram, OpenAI, and ElevenLabs as examples.

Node
Python
Go

Node.js 18 or later.

Install the SDK

Node
Python
Go

npm install agora-agent-server-sdk

The package name on npm is agora-agent-server-sdk. Imports use agora-agent-sdk.

pip install agora-agent-sdk

go get github.com/AgoraIO-Conversational-AI/agent-server-sdk-go

Set up environment variables

Store credentials in a .env file. Never commit this file or expose its contents client-side; your App Certificate must stay on the server.

Sync with Console to see your credentials

AGORA_APP_ID=<your_agora_app_id>
AGORA_APP_CERT=<your_agora_app_cert>

# ASR,LLM and TTS vendor keys
DEEPGRAM_API_KEY=<your_deepgram_key>
OPENAI_API_KEY=<your_openai_key>
ELEVENLABS_API_KEY=<your_elevenlabs_key>
ELEVENLABS_VOICE_ID=<your_voice_id>

Implementation

The following steps set up a client, configure an agent, and manage session lifecycle.

Create a client

Node
Python
Go

AgoraClient holds your credentials and handles authentication. Pass your App ID and App Certificate to use app-credentials mode. The SDK auto-generates a temporary token scoped to each channel before every request.

import { AgoraClient, Area } from 'agora-agent-sdk';
const client = new AgoraClient({
  area: Area.US, // Sets the regional API endpoint
  appId: process.env.AGORA_APP_ID!,
  appCertificate: process.env.AGORA_APP_CERT!,
});

Use the region closest to your users: Area.US, Area.EU, Area.AP, or Area.CN.

The Python SDK supports both synchronous and asynchronous usage. See Sync vs. Async to choose the right approach for your application.

Agora (sync) and AsyncAgora (async) hold your credentials and handle authentication. Pass your App ID and App Certificate to use app-credentials mode. The SDK auto-generates a temporary token scoped to each channel before every request.

Sync

_8import os _8from agora_agent import Agora, Area _8 _8client = Agora( _8 area=Area.US, _8 app_id=os.environ['AGORA_APP_ID'], _8 app_certificate=os.environ['AGORA_APP_CERT'], _8)
Async

_8import os _8from agora_agent import AsyncAgora, Area _8 _8client = AsyncAgora( _8 area=Area.US, _8 app_id=os.environ['AGORA_APP_ID'], _8 app_certificate=os.environ['AGORA_APP_CERT'], _8)

Use the region closest to your users: Area.US, Area.EU, Area.AP, or Area.CN.

In Go, the client handles authentication for REST API calls, while your App ID and App Certificate are passed separately when creating a session. The session uses them to auto-generate a temporary token for the Agora channel.

import (
    "os"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/client"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/option"
)
c := client.NewClient(
    option.WithArea(option.AreaUS),
)

Use the region closest to your users: option.AreaUS, option.AreaEU, option.AreaAP, or option.AreaCN.

Define an agent

Agent is a reusable configuration object. Define it once at startup and call createSession() on it for each user conversation.

Node
Python
Go

import { Agent, OpenAI, ElevenLabsTTS, DeepgramSTT } from 'agora-agent-sdk';
const agent = new Agent({
  name: 'support-assistant',
  instructions: 'You are a helpful voice assistant. Keep responses concise.', // LLM system prompt
  greeting: 'Hello! How can I help you today?', // the first thing the agent says
  maxHistory: 20,
})
  .withLlm(new OpenAI({
    apiKey: process.env.OPENAI_API_KEY!,
    model: 'gpt-4o-mini',
  }))
  .withTts(new ElevenLabsTTS({
    key: process.env.ELEVENLABS_API_KEY!,
    modelId: 'eleven_flash_v2_5',
    voiceId: process.env.ELEVENLABS_VOICE_ID!,
    sampleRate: 24000,
  }))
  .withStt(new DeepgramSTT({
    apiKey: process.env.DEEPGRAM_API_KEY!,
    model: 'nova-2',
    language: 'en-US',
  }));

Vendor classes are imported from agora_agent.agentkit.vendors. Builder methods use snake_case.

import os
from agora_agent.agentkit import Agent
from agora_agent.agentkit.vendors import OpenAI, ElevenLabsTTS, DeepgramSTT
agent = (
    Agent(
      name='support-assistant', 
      instructions='You are a helpful voice assistant.', # LLM system prompt
      greeting='Hello! How can I help you today?', # first words spoken on session start
    )
    .with_llm(OpenAI(api_key=os.environ['OPENAI_API_KEY'], model='gpt-4o-mini'))
    .with_tts(ElevenLabsTTS(key=os.environ['ELEVENLABS_API_KEY'], model_id='eleven_flash_v2_5', voice_id=os.environ['ELEVENLABS_VOICE_ID']))
    .with_stt(DeepgramSTT(api_key=os.environ['DEEPGRAM_API_KEY'], language='en-US'))
)

Go uses functional options to configure the agent. Vendor classes are imported from agentkit/vendors and constructed with options structs.

import (
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/agentkit"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/agentkit/vendors"
)
agent := agentkit.NewAgent(
    agentkit.WithName("support-assistant"),
    agentkit.WithInstructions("You are a helpful voice assistant. Keep responses concise."), // LLM system prompt
    agentkit.WithGreeting("Hello! How can I help you today?"), // the first thing the agent says
    agentkit.WithMaxHistory(10),
).WithLlm(
    vendors.NewOpenAI(vendors.OpenAIOptions{
        APIKey: os.Getenv("OPENAI_API_KEY"),
        Model:  "gpt-4o-mini",
    }),
).WithTts(
    vendors.NewElevenLabsTTS(vendors.ElevenLabsTTSOptions{
        Key:     os.Getenv("ELEVENLABS_API_KEY"),
        ModelID: "eleven_flash_v2_5",
        VoiceID: os.Getenv("ELEVENLABS_VOICE_ID"),
    }),
).WithStt(
    vendors.NewDeepgramSTT(vendors.DeepgramSTTOptions{
        APIKey: os.Getenv("DEEPGRAM_API_KEY"),
    }),
)

For the full list of Agent options and vendor classes, see the SDK reference.

Create and start a session

Bind the agent to a specific RTC channel, register event listeners, then start the session.

Node
Python
Go

import { ExpiresIn } from 'agora-agent-sdk';
const session = agent.createSession(client, {
  channel: 'your-channel-name',
  agentUid: '1',
  remoteUids: ['100'],
  idleTimeout: 120,
  expiresIn: ExpiresIn.hours(1),
});
// Register before start()
session.on('started', ({ agentId }) => {
  console.log('Agent is live:', agentId);
});
session.on('stopped', ({ agentId }) => {
  console.log('Agent has left:', agentId);
});
session.on('error', (err) => {
  console.error('Session error:', err);
});
const agentId = await session.start();

start() transitions the session through idle → starting → running states and returns the agentId. The agentId is the same value as the agent_id returned by the REST API.

You can check the session state at any time with session.status.

Event handlers receive a dict payload. Register them before calling start().

Sync

_17from agora_agent.agentkit import expires_in_hours _17 _17session = agent.create_session( _17 client, _17 channel='your-channel-name', _17 agent_uid='1', _17 remote_uids=['100'], _17 idle_timeout=120, _17 expires_in=expires_in_hours(1), _17) _17 _17# Register before start() _17session.on('started', lambda data: print('Agent is live:', data['agent_id'])) _17session.on('stopped', lambda data: print('Agent has left:', data['agent_id'])) _17session.on('error', lambda err: print('Session error:', err)) _17 _17agent_id = session.start()
Async

_1agent_id = await session.start()

start() transitions the session through idle → starting → running states and returns the agentId. The agentId is the same value as the agent_id returned by the REST API.

You can check the session state at any time with session.status.

import (
    "context"
    "fmt"
    "log"
    "os"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/agentkit"
)
session := agentkit.NewAgentSession(agentkit.AgentSessionOptions{
    Client:         c.Agents,
    Agent:          agent,
    AppID:          os.Getenv("AGORA_APP_ID"),
    AppCertificate: os.Getenv("AGORA_APP_CERT"),
    Channel:        "your-channel-name",
    AgentUID:       "1",
    RemoteUIDs:     []string{"100"},
})
// Register before Start()
session.On("started", func(data interface{}) {
    info := data.(map[string]string)
    fmt.Println("Agent is live:", info["agent_id"])
})
session.On("stopped", func(data interface{}) {
    fmt.Println("Agent has left")
})
session.On("error", func(data interface{}) {
    log.Println("Session error:", data)
})
ctx := context.Background()
agentId, err := session.Start(ctx)
if err != nil {
    log.Fatalf("Failed to start session: %v", err)
}

Start() transitions the session through idle → starting → running states and returns the agent ID. The agent ID is the same value as the agent_id returned by the REST API.

You can check the current session state at any time with session.Status().

Interact with a running session

Once a session is in the running state, you can instruct the agent or update its configuration mid-session.

Node
Python
Go

// Make the agent speak a specific message
await session.say('One moment while I look that up for you.');
// Interrupt the agent mid-speech
await session.interrupt();
// Update agent configuration without restarting the session.
// Accepts a partial properties object in REST API format.
await session.update({
  llm: {
    system_messages: [{ role: 'system', content: 'You are now in concise mode.' }],
  },
});

Sync

_13# Make the agent speak a specific message _13session.say('One moment while I look that up for you.') _13 _13# Interrupt the agent mid-speech _13session.interrupt() _13 _13# Update agent configuration without restarting the session. _13# Accepts a partial properties object in REST API format. _13session.update({ _13 'llm': { _13 'system_messages': [{'role': 'system', 'content': 'You are now in concise mode.'}], _13 }, _13})
Async

_7await session.say('One moment while I look that up for you.') _7await session.interrupt() _7await session.update({ _7 'llm': { _7 'system_messages': [{'role': 'system', 'content': 'You are now in concise mode.'}], _7 }, _7})

// Make the agent speak a specific message
err := session.Say(ctx, "One moment while I look that up for you.", nil, nil)
if err != nil {
    log.Printf("Say failed: %v", err)
}
// Interrupt the agent mid-speech
err = session.Interrupt(ctx)
if err != nil {
    log.Printf("Interrupt failed: %v", err)
}
// Update agent configuration without restarting the session.
// Accepts a partial UpdateAgentsRequestProperties struct in REST API format.
err = session.Update(ctx, &Agora.UpdateAgentsRequestProperties{
    Llm: &Agora.UpdateAgentsRequestPropertiesLlm{
        SystemMessages: []map[string]any{
            {"role": "system", "content": "You are now in concise mode."},
        },
    },
})
if err != nil {
    log.Printf("Update failed: %v", err)
}

Stop the session

Call the session stop method when the user ends the conversation. This transitions the session through running → stopping → stopped states and removes the agent from the channel.

Node
Python
Go

await session.stop();

Sync

_1session.stop()
Async

_1await session.stop()

err := session.Stop(ctx)
if err != nil {
    log.Fatalf("Failed to stop session: %v", err)
}

Complete example

The following is a minimal server your client app can call to manage agent sessions.

Node
Python
Go

Express server example

import 'dotenv/config';
import express from 'express';
import {
  AgoraClient,
  Area,
  Agent,
  ExpiresIn,
  OpenAI,
  ElevenLabsTTS,
  DeepgramSTT,
  AgoraError,
} from 'agora-agent-sdk';
const app = express();
app.use(express.json());
// Initialized once at startup; shared across all requests
const client = new AgoraClient({
  area: Area.US,
  appId: process.env.AGORA_APP_ID!,
  appCertificate: process.env.AGORA_APP_CERT!,
});
const agent = new Agent({
  name: 'support-assistant',
  instructions: 'You are a helpful voice assistant. Keep responses concise.',
  greeting: 'Hello! How can I help you today?',
  maxHistory: 20,
})
  .withLlm(new OpenAI({ apiKey: process.env.OPENAI_API_KEY!, model: 'gpt-4o-mini' }))
  .withTts(new ElevenLabsTTS({
    key: process.env.ELEVENLABS_API_KEY!,
    modelId: 'eleven_flash_v2_5',
    voiceId: process.env.ELEVENLABS_VOICE_ID!,
    sampleRate: 24000,
  }))
  .withStt(new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY!, model: 'nova-2', language: 'en-US' }));
// Maps agentId → session object.
// In production, use a database or Redis so any server process can stop any session.
const sessions = new Map<string, ReturnType<typeof agent.createSession>>();
// POST /sessions/start
// Called by your client app after the user joins the Agora channel.
// Body: { channel: string, userId: string }
// Returns: { agentId: string }
app.post('/sessions/start', async (req, res) => {
  const { channel, userId } = req.body;
  const session = agent.createSession(client, {
    channel,
    agentUid: '1',
    remoteUids: [userId],
    idleTimeout: 120,
    expiresIn: ExpiresIn.hours(1),
  });
  session.on('error', (err) => console.error(`Session error [${channel}]:`, err));
  try {
    const agentId = await session.start();
    sessions.set(agentId, session);
    res.json({ agentId });
  } catch (err) {
    if (err instanceof AgoraError) {
      res.status(err.statusCode).json({ error: err.message });
    } else {
      res.status(500).json({ error: 'Failed to start agent' });
    }
  }
});
// POST /sessions/stop
// Called by your client app when the user ends the conversation.
// Body: { agentId: string }
app.post('/sessions/stop', async (req, res) => {
  const { agentId } = req.body;
  const session = sessions.get(agentId);
  if (!session) {
    return res.status(404).json({ error: 'Session not found' });
  }
  try {
    await session.stop();
    sessions.delete(agentId);
    res.json({ status: 'stopped' });
  } catch (err) {
    if (err instanceof AgoraError) {
      res.status(err.statusCode).json({ error: err.message });
    } else {
      res.status(500).json({ error: 'Failed to stop agent' });
    }
  }
});
app.listen(3000, () => console.log('Server listening on port 3000'));

Python example

Sync

_36import os _36from agora_agent import Agora, Area _36from agora_agent.agentkit import Agent _36from agora_agent.agentkit.vendors import OpenAI, ElevenLabsTTS, DeepgramSTT _36 _36client = Agora( _36 area=Area.US, _36 app_id=os.environ['AGORA_APP_ID'], _36 app_certificate=os.environ['AGORA_APP_CERT'], _36) _36 _36agent = ( _36 Agent( _36 name='support-assistant', _36 instructions='You are a helpful voice assistant.', # LLM system prompt _36 greeting='Hello! How can I help you today?', # first words spoken on session start _36 ) _36 .with_llm(OpenAI(api_key=os.environ['OPENAI_API_KEY'], model='gpt-4o-mini')) _36 .with_tts(ElevenLabsTTS( _36 key=os.environ['ELEVENLABS_API_KEY'], _36 model_id='eleven_flash_v2_5', _36 voice_id=os.environ['ELEVENLABS_VOICE_ID'], _36 )) _36 .with_stt(DeepgramSTT(api_key=os.environ['DEEPGRAM_API_KEY'], language='en-US')) _36) _36 _36session = agent.create_session( _36 client, _36 channel='support-room-123', _36 agent_uid='1', _36 remote_uids=['100'], _36) _36agent_id = session.start() _36 _36session.say('Hello! How can I help you today?') _36session.stop()
Async

_40import asyncio _40import os _40from agora_agent import AsyncAgora, Area _40from agora_agent.agentkit import Agent _40from agora_agent.agentkit.vendors import OpenAI, ElevenLabsTTS, DeepgramSTT _40 _40async def main(): _40 client = AsyncAgora( _40 area=Area.US, _40 app_id=os.environ['AGORA_APP_ID'], _40 app_certificate=os.environ['AGORA_APP_CERT'], _40 ) _40 _40 agent = ( _40 Agent( _40 name='support-assistant', _40 instructions='You are a helpful voice assistant.', # LLM system prompt _40 greeting='Hello! How can I help you today?', # first words spoken on session start _40 ) _40 .with_llm(OpenAI(api_key=os.environ['OPENAI_API_KEY'], model='gpt-4o-mini')) _40 .with_tts(ElevenLabsTTS( _40 key=os.environ['ELEVENLABS_API_KEY'], _40 model_id='eleven_flash_v2_5', _40 voice_id=os.environ['ELEVENLABS_VOICE_ID'], _40 )) _40 .with_stt(DeepgramSTT(api_key=os.environ['DEEPGRAM_API_KEY'], language='en-US')) _40 ) _40 _40 session = agent.create_session( _40 client, _40 channel='support-room-123', _40 agent_uid='1', _40 remote_uids=['100'], _40 ) _40 agent_id = await session.start() _40 _40 await session.say('Hello! How can I help you today?') _40 await session.stop() _40 _40asyncio.run(main())

Go example

package main
import (
    "context"
    "fmt"
    "log"
    "os"
    // uncomment for typed structs and pointer helpers
    // Agora "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/agentkit"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/agentkit/vendors"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/client"
    "github.com/AgoraIO-Conversational-AI/agent-server-sdk-go/option"
)
func main() {
    // Initialized once at startup; shared across all requests
    c := client.NewClient(
        option.WithArea(option.AreaUS),
    )
    agent := agentkit.NewAgent(
        agentkit.WithName("support-assistant"),
        agentkit.WithInstructions("You are a helpful voice assistant. Keep responses concise."), // LLM system prompt
        agentkit.WithGreeting("Hello! How can I help you today?"),                               // first words spoken on session start
        agentkit.WithMaxHistory(10),
    ).WithLlm(
        vendors.NewOpenAI(vendors.OpenAIOptions{
            APIKey: os.Getenv("OPENAI_API_KEY"),
            Model:  "gpt-4o-mini",
        }),
    ).WithTts(
        vendors.NewElevenLabsTTS(vendors.ElevenLabsTTSOptions{
            Key:     os.Getenv("ELEVENLABS_API_KEY"),
            ModelID: "eleven_flash_v2_5",
            VoiceID: os.Getenv("ELEVENLABS_VOICE_ID"),
        }),
    ).WithStt(
        vendors.NewDeepgramSTT(vendors.DeepgramSTTOptions{
            APIKey: os.Getenv("DEEPGRAM_API_KEY"),
        }),
    )
    session := agentkit.NewAgentSession(agentkit.AgentSessionOptions{
        Client:         c.Agents,
        Agent:          agent,
        AppID:          os.Getenv("AGORA_APP_ID"),
        AppCertificate: os.Getenv("AGORA_APP_CERT"),
        Channel:        "support-room-123",
        AgentUID:       "1",
        RemoteUIDs:     []string{"100"},
    })
    session.On("started", func(data interface{}) {
        info := data.(map[string]string)
        fmt.Println("Agent is live:", info["agent_id"])
    })
    session.On("stopped", func(data interface{}) {
        fmt.Println("Agent has left")
    })
    session.On("error", func(data interface{}) {
        log.Println("Session error:", data)
    })
    ctx := context.Background()
    agentId, err := session.Start(ctx)
    if err != nil {
        log.Fatalf("Failed to start session: %v", err)
    }
    fmt.Println("Agent running with ID:", agentId)
    err = session.Say(ctx, "Hello! How can I help you today?", nil, nil)
    if err != nil {
        log.Printf("Say failed: %v", err)
    }
    err = session.Stop(ctx)
    if err != nil {
        log.Fatalf("Failed to stop session: %v", err)
    }
}