Client Commands

Your client sends commands to the server during an active WebRTC or WebSocket session: send_message to send information to the agent, set_settings to update session configuration on the fly, send_mcp_approval to answer an MCP tool approval, and talk to make the agent speak a line in puppeteer mode.

Sending commands

WebRTC (Web SDK)

Call sendCommand on the SDK instance. The Web SDK also ships typed command definitions and usage patterns for this — see Web SDK → Messaging.

instance.sendCommand({
  type: "send_message",
  data: {
    role: "user",
    text: "Hello",
    trigger_response: true,
    delay: false
  }
});

WebSocket

Send the command as JSON over the WebSocket connection:

ws.send(JSON.stringify({
  type: "send_message",
  data: {
    role: "user",
    text: "Hello",
    trigger_response: true,
    delay: false
  }
}));

send_message

Send text to the agent as a user message or a system context update.

{
  "type": "send_message",
  "data": {
    "role": "user",
    "text": "What is the status of my order?",
    "trigger_response": true,
    "delay": false
  }
}

Parameters

User messages

When role is user, the agent treats the text as if the user typed it instead of speaking. This is useful for text-based input alongside audio, or for injecting user intent programmatically.

{
  "type": "send_message",
  "data": {
    "role": "user",
    "text": "Tell me more about the premium plan",
    "trigger_response": true
  }
}

System messages

When role is system, the text is treated as a context update. The agent incorporates the information into its context without displaying it to the user. Use this when something changes in your application that the agent needs to know about.

{
  "type": "send_message",
  "data": {
    "role": "system",
    "text": "The user just added 'Wireless Headphones' to their shopping cart.",
    "trigger_response": false
  }
}

Triggering a response

When trigger_response is true, the agent generates a response immediately. When false, the agent absorbs the information silently — it does not respond right away, but may incorporate it into future replies based on the conversation context.

The delay parameter controls when the message is delivered. When delay is false (the default), the message is delivered immediately, interrupting the agent if it is currently speaking. When delay is true, the message is held until the agent finishes its current speech, then delivered — and if trigger_response is true, the agent responds after that.

{
  "type": "send_message",
  "data": {
    "role": "system",
    "text": "The user's subscription has been upgraded to premium.",
    "trigger_response": true,
    "delay": true
  }
}

set_settings

Update session parameters in real time without creating a new connection. Change the agent's instructions, temperature, turn detection, modality, or tools, or refresh MCP tokens.

{
  "type": "set_settings",
  "data": {
    "instructions": "You are a customer support agent. Be concise and helpful.",
    "temperature": 0.7,
    "turn_detection": {
      "threshold": 0.5,
      "prefix_padding_ms": 300,
      "silence_duration_ms": 500
    }
  }
}

Parameters

set_settings isn't supported on Microsoft Foundry.

turn_detection

Common patterns

Replacing instructions mid-conversation — instructions fully replaces the agent's system prompt. Use this when the agent's entire role or behavior needs to change, for example when the user moves to a different part of your application:

{
  "type": "set_settings",
  "data": {
    "instructions": "You are a checkout assistant for an e-commerce store. Guide the user through completing their purchase. Confirm the shipping address, suggest available shipping options, and answer questions about return policies. Do not suggest additional products."
  }
}

Providing context updates — use send_message with role: system to give the agent situational context without replacing its instructions. For example, notify the agent when the user navigates to a different page:

{
  "type": "send_message",
  "data": {
    "role": "system",
    "text": "The user just navigated to the checkout page.",
    "trigger_response": false
  }
}

Registering tools mid-session — add tools the agent can call for the rest of the session, without creating them with the API first. Each item uses the same shape as a tool definition:

{
  "type": "set_settings",
  "data": {
    "inline_functions": [
      {
        "id": "get_cart",
        "flow": "implicit",
        "data": {
          "name": "get_cart",
          "description": "Get the items currently in the user's shopping cart.",
          "parameters": { "type": "object", "properties": {} }
        }
      }
    ]
  }
}

Only implicit tools, and explicit tools served over an http or https URL, can be registered this way. Names must be unique and can't collide with the agent's own tools. Each set_settings with inline_functions replaces the tools registered this way before.

Refreshing an MCP token — when a per-user MCP server's access token is about to expire, send the fresh token. The list replaces all tokens, so include every server that needs one:

{
  "type": "set_settings",
  "data": {
    "mcp": {
      "authorizations": [
        {
          "mcp_server_id": "crm",
          "token": "ya29.a0AfH6_refreshed..."
        }
      ]
    }
  }
}

Adjusting turn detection for noisy environments — increase silence_duration_ms so the agent waits longer before treating silence as the end of the user's turn:

{
  "type": "set_settings",
  "data": {
    "turn_detection": {
      "silence_duration_ms": 1000
    }
  }
}

send_mcp_approval

Answer a pending mcp_approval_request. Only relevant when an attached MCP server is registered with requireApproval: "always".

{
  "type": "send_mcp_approval",
  "data": {
    "approval_request_id": "mcpr_x9y8z7",
    "approve": true
  }
}

Parameters

Send exactly one decision per request — later or duplicate decisions for the same approval_request_id are ignored. If nothing arrives within 15 seconds, the request is rejected automatically and you receive a cancelled event.

talk

In puppeteer mode, send talk to make the agent speak a line of text exactly as written.

{
  "type": "talk",
  "data": {
    "talkId": "intro-1",
    "text": "Welcome to Acme. Let me show you around the new dashboard."
  }
}

Parameters

Track each line with talk_state_changed: preparing, then started and ended. If the user starts speaking while the agent is talking, the current line stops and you receive canceled.

Next steps