Newera Core · API
API Documentation
A single sovereign API for every model your organisation runs, deployed inside the Kingdom with no data egress.
Newera Core is OpenAI-compatible wherever the response shape allows, so you can integrate with the OpenAI SDK using a Newera key. Prompts and completions are never stored. Every request is metered, and every response carries an x-request-id for billing and support.
- Base URL
- https://api.newera.ai/v1
- Auth
- Bearer <API key>
- Format
- JSON · multipart
Get started
Quickstart
Create a key in the portal (API keys in the Newera portal). A key is shown once, at creation. Store it, and issue another if you lose it.
export NEWERA_API_KEY="nwc_live_..."Your first request:
curl https://api.newera.ai/v1/chat/completions \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-chat-large",
"messages": [{"role": "user", "content": "Say hello in one sentence."}]
}'
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.newera.ai/v1",
api_key=os.environ["NEWERA_API_KEY"],
)
resp = client.chat.completions.create(
model="newera-chat-large",
messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(resp.choices[0].message.content)
print(resp.usage) # what you are billed on
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.newera.ai/v1",
apiKey: process.env.NEWERA_API_KEY,
});
const resp = await client.chat.completions.create({
model: "newera-chat-large",
messages: [{ role: "user", content: "Say hello in one sentence." }],
});
console.log(resp.choices[0].message.content);
console.log(resp.usage); // what you are billed on
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.newera.ai/v1",
apiKey: process.env.NEWERA_API_KEY!,
});
const resp: OpenAI.Chat.ChatCompletion = await client.chat.completions.create({
model: "newera-chat-large",
messages: [{ role: "user", content: "Say hello in one sentence." }],
});
console.log(resp.choices[0].message.content);
console.log(resp.usage); // what you are billed on
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-chat-large",
"messages": []map[string]string{
{"role": "user", "content": "Say hello in one sentence."},
},
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
Usage map[string]int `json:"usage"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
fmt.Println(out.Choices[0].Message.Content)
fmt.Println(out.Usage)
}
Point base_url at Newera and nothing else about your OpenAI client changes.
Get started
Authentication
Send your key as a bearer token. x-api-key is accepted too, for clients that cannot set an Authorization header.
Authorization: Bearer nwc_live_...Keys look like nwc_live_<prefix>_<secret>. Only a hash is stored, so the full secret exists only in your copy.
- One tenant per key. A key sees only the models on your organisation's contract, and may be limited further to some kinds of model (chat, embeddings, voice calls and so on), chosen when you create it.
- Shown once. Treat a key like a password; rotate it from the portal.
- Revocation is immediate. A revoked or expired key stops working everywhere within about a second.
A request with no key, or a malformed one, returns 401. A well-formed key that is not entitled to the model you asked for returns 403, not 404: the model exists, and access is a contract question.
Get started
Keeping keys safe
Anyone holding a key can spend your credit, so treat it like a password. Most leaked keys leak from the client side: committed to a repository, built into an app, or pasted into a chat.
- Keep keys on your server. Read them from an environment variable or a secret manager. Never write a key into source code or a config file you commit.
- Never put a key in a browser page or a mobile app. Anything shipped to a device can be read from it. Call Newera Core from your server; for voice calls from a browser, trade the key for a 60-second client secret on your server (see newera-s2s).
- One key per application and environment. Label each one in the portal, so a leak can be revoked without stopping everything else, and usage per key shows which application spent what.
- Give a key only what it needs. Limit it to the kinds of model it calls, set an expiry date, and cap what it may spend in a month.
- Rotate keys. Replace keys every 90 days or so, and whenever someone who had one leaves. Issue the new key, deploy it, then revoke the old one; we email a reminder when a key is 90 days old.
- Revoke at once if a key may have leaked. Revoke it from API keys in the portal; it stops working everywhere within about a second.
If a key shows use you do not recognise, revoke it and contact us from the Support button in the portal.
Get started
Making requests
| Base URL | https://api.newera.ai/v1 |
| Chat · embeddings · rerank · speech synthesis | Content-Type: application/json |
| OCR · segmentation · transcription | multipart/form-data (file upload) |
| Every response | carries x-request-id, always generated by Newera |
Pass a model by its alias (newera-chat-large) or its pinned id (newera-chat-large-v1). The alias always points at the current best version of a capability; the pinned id is frozen. See Model lifecycle.
The OpenAI-shaped endpoints (chat, embeddings, speech) work with the OpenAI SDK unchanged. The Newera-native endpoints (rerank, OCR, segmentation) are plain HTTP; call them with httpx or curl. Both styles appear in each model's examples.
Core concepts
Models
The models, grouped by what they do. Each has a stable public id beginning newera-. Call GET /v1/models to see the ones your key is entitled to.
| Model | Does | Endpoint | Counted per |
|---|---|---|---|
newera-chat-large | Chat + vision, longest context, deepest reasoning | /chat/completions | token |
newera-chat-medium | Long, complex tasks and harder reasoning, 128,000-token context | /chat/completions | token |
newera-chat-small | Fast everyday chat with image and audio input, 64,000-token context | /chat/completions | token |
newera-embedding | Text embeddings for search and RAG | /embeddings | input token |
newera-rerank | Re-order retrieval candidates by relevance | /rerank | document scored |
newera-ocr | Extract text from PDFs and images | /documents/extract | page |
newera-segment | Segment named concepts in an image | /images/segment | image |
newera-tts | Natural Saudi Arabic text to speech | /audio/speech | second of audio |
newera-stt | Arabic speech to text | /audio/transcriptions | second of audio |
newera-s2s | Realtime voice conversations in Saudi Arabic | wss://…/realtime | per connected second |
newera-s2s-mini | The same voice conversations on a lighter model, for high-volume, well-defined calls | wss://…/realtime?model=newera-s2s-mini | per connected second |
For images, use any of the three chat models: newera-chat-small, newera-chat-medium or newera-chat-large; all accept image input. For audio clips of up to 30 seconds, use newera-chat-small.
For rates, open a model on the Nawah page in the portal.
GET /v1/models
Lists the models on your contract, and nothing more. It is not a catalogue browser.
curl https://api.newera.ai/v1/models \
-H "Authorization: Bearer $NEWERA_API_KEY"{
"object": "list",
"data": [
{
"id": "newera-chat-large",
"object": "model",
"owned_by": "newera",
"newera": {
"class": "chat",
"modality": "text+image",
"unit_type": "tokens",
"context_length": 262144,
"status": "hot",
"pinned_id": "newera-chat-large-v1",
"sunset_at": null
}
}
]
}Model lifecycle
Two identifiers, so a model can improve without your code changing:
- Alias (
newera-chat-large). The permanent name for a capability. It moves to a better version when one is ready. - Pinned id (
newera-chat-large-v1). Frozen at one version. Use it in production when you need output to stay stable.
A model that is being retired returns a Sunset response header with its removal date, ahead of its removal. GET /v1/models shows status and sunset_at.
Core concepts
Errors
Every error is OpenAI-shaped: a JSON body with an error object.
{
"error": {
"message": "Your organisation is not entitled to `newera-chat-large`. Contact your Newera account manager to add it to your contract.",
"type": "invalid_request_error",
"param": "model",
"code": "model_not_entitled",
"request_id": "req_..."
}
}| Status | Meaning | Typical code |
|---|---|---|
400 | Malformed request | invalid_request_error |
401 | Missing, malformed, revoked or expired key | invalid_api_key |
403 | The model is not on your organisation's contract, or the account is inactive | model_not_entitled |
403 | This key's permissions do not include this kind of model. Use a key that can call it, or create one in the portal | key_permission_denied |
402 | This key has reached the monthly spending limit set on it in the portal. It works again on the 1st, or raise the limit under API keys | key_spend_limit_reached |
429 | Too many invalid API keys from your address in the last minute. Wait for the retry-after seconds, and check the key you are sending | too_many_invalid_keys |
404 | No such model | model_not_found |
429 | A limit on your key was reached (not on the model) | key_quota_exhausted |
502 | The model returned an error or could not be reached | upstream_error |
503 | Model temporarily unavailable; retry shortly | model_unavailable |
Quote the request_id (also on the x-request-id header) in any support request. Newera generates this id for every call; an x-request-id you send is not used as it. To tie a call to your own logs, send it anyway: it comes back unchanged in x-client-request-id. A 429 is always about your key. See Rate limits.
Core concepts
Rate limits
Limits live on the API key, and the error always tells you which limit and what to do. There are two kinds:
- Shape limits, rejected before any processing starts:
max_output_tokens,max_input_tokens,max_documents,max_images,max_pages. - Volume limits, a cap on how many requests a key may make, in total or per model type.
Core concepts
Streaming
Chat supports server-sent events. Set "stream": true and read data: lines until data: [DONE]. The final chunk carries the usage object. Core requests it on your behalf, so a streamed request is always billable and you always receive the token counts.
stream = client.chat.completions.create(
model="newera-chat-large",
messages=[{"role": "user", "content": "Count to five."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)If you disconnect mid-stream, the compute already spent is still metered.
Chat
Chat completions
POST /v1/chat/completions
OpenAI-compatible, three models. Pick by context length and depth of reasoning. All three share the parameters below; each model card adds only what is specific to it.
Parameters
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
model | string | yes | Which model to use, for example newera-chat-large. A pinned id (newera-chat-large-v1) never changes, so use that in production. | |
messages | array | yes | The conversation so far: a list of messages, each with a role (system, user or assistant) and content. | |
max_tokens | integer | no | see below | Output limit: the most tokens the model may write in one reply, thinking included. It is not a limit on your input. Input and reply share the model's context window, so the input plus max_tokens must fit in it. If you leave it out, the reply is limited to the model's default (see the model's own limits) and ends with finish_reason "length" if it hits it. |
temperature | number | no | 1 | 0 - 2. How varied the answers are. Lower is steadier; 0 gives the closest to the same answer every time. |
top_p | number | no | 1 | 0 - 1. Only choose from the most likely words that add up to this share of the probability. Lower is more focused. Change this or temperature, rarely both. |
top_k | integer | no | Only choose from the k most likely next words. Lower is more focused. Off unless you set it. | |
min_p | number | no | 0 - 1. Drop next words that are much less likely than the best one. 0.05 drops any word under 5% of the best word's chance. Higher is more focused. Not available on newera-chat-large, which refuses it. | |
presence_penalty | number | no | 0 | -2 - 2. Above 0 nudges the model toward new topics: a word that has already appeared is less likely to come up again. |
frequency_penalty | number | no | 0 | -2 - 2. Above 0 reduces repetition: the more often a word has appeared, the less likely it is to appear again. |
repetition_penalty | number | no | 1 | Above 1 discourages repeating words already used; 1 does nothing. Try 1.05 to 1.2. |
stop | string | array | no | Text that ends the reply when the model writes it. The stop text itself is not included. | |
seed | integer | no | With the same seed and a temperature of 0, repeating a request gives the same answer. | |
stream | boolean | no | False | Send the reply in pieces as it is written. The last piece carries the token counts. |
response_format | object | no | {"type": "json_object"} makes the reply valid JSON. {"type": "json_schema", "json_schema": {...}} makes it match your schema. | |
tools | array | no | Functions the model may ask you to call, each with a name, a description and a JSON schema for its arguments. | |
tool_choice | string | object | no | auto | "auto" lets the model decide, "none" never calls a tool, "required" forces a call, or name one function. "none" is the default when you send no tools. |
logprobs | boolean | no | False | Return how likely each chosen word was. |
top_logprobs | integer | no | 0 | 0 - 20. With logprobs on, also return the n likeliest alternatives for each word. |
max_tokens is the output limit. It caps the tokens in the reply, thinking included, and never limits your input. The input and the reply share one context window, so the input plus max_tokens must fit in it; a request that does not is refused with context_length_exceeded. Each model's own limits are listed in its section below.
If you do not set max_tokens, a reply is limited to the smaller of two things: the model's default reply length, and what your prepaid balance covers. The reply then ends with finish_reason: "length". Set max_tokens to choose your own limit instead.
newera-chat-large: a default reply of 32,768 tokens.newera-chat-medium: no fixed default; a reply can use all the room your input leaves in its 128,000-token window.newera-chat-small: no fixed default; a reply can use all the room your input leaves in its 64,000-token window.
With n above 1 the limit applies to each reply. A request that sets max_tokens is checked against your balance as before, and refused with insufficient_quota if it cannot be covered.
Response
The OpenAI chat completion shape: choices[].message.content, plus a usage object with prompt_tokens, completion_tokens, prompt_tokens_details.cached_tokens (the part of the prompt served from the model's prefix cache) and, where the model reasons, completion_tokens_details.reasoning_tokens.
Chat model
newera-chat-large
The flagship: the longest context in the catalogue, native vision, and the deepest reasoning. Reach for it when the task is hard or the input is large.
Best for
Long-document analysis, multi-step reasoning, agentic and tool-use flows, and anything mixing text with images.
Not for
High-volume, latency-sensitive or trivial calls. As the most capable tier, it is best suited to demanding tasks; for simple or high-throughput work, a smaller tier is the better fit.
- Model id
newera-chat-large- Pinned id
newera-chat-large-v1- Context window
- 262,000 tokens
- Inputs
- text + image
- Billed
- per input and output token
- Endpoint
/chat/completions
Images
Send message content as parts: a {"type":"text"} part and an {"type":"image_url"} part whose url is a data: URL or a public https: URL. Image pixels are tokenised into the prompt, so they count as input tokens, and a larger image uses more of them.
Thinking
This model thinks step by step before it answers, and thinking is on by default. That helps on hard, multi-step problems and costs time and tokens. For a faster direct answer, turn it off with chat_template_kwargs, as the example below shows. completion_tokens_details.reasoning_tokens shows how much a call thought.
Limits for this model
| Limit | Value |
|---|---|
| Context window | 262,000 tokens, shared by the input and the reply |
| Max input tokens | 261,000 (input plus reply must fit the window) |
| Max output tokens | 261,000: the max_tokens ceiling, thinking included (input plus reply must fit the window) |
| Default output tokens | 32,768 when max_tokens is left out |
| Image input | yes |
| Audio input | no: an audio part is refused |
Parameters for this model
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
chat_template_kwargs | object | no | {"enable_thinking": true} | Thinking on or off. This model thinks step by step before it answers, which helps on hard problems and costs time and tokens. Send {"enable_thinking": false} for a faster, cheaper direct answer, but we recommend keeping thinking on: it helps the model follow your instructions and ignore instructions hidden inside user messages or documents. The choice is yours. Nothing else is accepted inside it. |
curl https://api.newera.ai/v1/chat/completions \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-chat-large",
"messages": [{"role": "user", "content": "Explain the difference between TCP and UDP."}],
"max_tokens": 512,
"chat_template_kwargs": {"enable_thinking": false}
}'
resp = client.chat.completions.create(
model="newera-chat-large",
messages=[{"role": "user", "content": "Explain the difference between TCP and UDP."}],
max_tokens=512,
# Answer directly; omit to let the model reason first.
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)
const resp = await client.chat.completions.create({
model: "newera-chat-large",
messages: [{ role: "user", content: "Explain the difference between TCP and UDP." }],
max_tokens: 512,
// Answer directly; omit to let the model reason first.
chat_template_kwargs: { enable_thinking: false },
});
console.log(resp.choices[0].message.content);
const resp = await client.chat.completions.create({
model: "newera-chat-large",
messages: [{ role: "user", content: "Explain the difference between TCP and UDP." }],
max_tokens: 512,
// chat_template_kwargs is a Newera extension, not in the OpenAI types.
chat_template_kwargs: { enable_thinking: false },
} as OpenAI.Chat.ChatCompletionCreateParamsNonStreaming);
console.log(resp.choices[0].message.content);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-chat-large",
"messages": []map[string]string{
{"role": "user", "content": "Explain the difference between TCP and UDP."},
},
"max_tokens": 512,
"chat_template_kwargs": map[string]any{"enable_thinking": false},
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
fmt.Println(out.Choices[0].Message.Content)
}
Chat model
newera-chat-medium
A stronger model for long, complex work: it steps up from newera-chat-small with twice the context window, and is better at harder reasoning and coding. It reasons before it answers by default and can be told to answer directly. Reads images as well as text.
Best for
Long, multi-step tasks, harder reasoning and coding, tool-using assistants, and long documents of up to 128,000 tokens.
Not for
Requests whose input and reply together need more than its 128,000-token window, or the deepest multi-step reasoning: use newera-chat-large for those. Simple everyday requests, structured output and audio input are better served by newera-chat-small, which is faster.
- Model id
newera-chat-medium- Pinned id
newera-chat-medium-v1- Context window
- 128,000 tokens
- Inputs
- text + image
- Billed
- per input and output token
- Endpoint
/chat/completions
Thinking
This model thinks step by step before it answers, and thinking is on by default. That helps on hard, multi-step problems and costs time and tokens. For a faster direct answer, turn it off with chat_template_kwargs, as the example below shows. completion_tokens_details.reasoning_tokens shows how much a call thought.
Limits for this model
| Limit | Value |
|---|---|
| Context window | 128,000 tokens, shared by the input and the reply |
| Max input tokens | 127,000 (input plus reply must fit the window) |
| Max output tokens | 127,000: the max_tokens ceiling, thinking included (input plus reply must fit the window) |
| Default output tokens | when max_tokens is left out, up to all the room your input leaves in the window |
| Image input | yes |
| Audio input | no: an audio part is refused |
Parameters for this model
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
chat_template_kwargs | object | no | {"enable_thinking": true} | Thinking on or off. This model thinks step by step before it answers, which helps on hard problems and costs time and tokens. Send {"enable_thinking": false} for a faster, cheaper direct answer, but we recommend keeping thinking on: it helps the model follow your instructions and ignore instructions hidden inside user messages or documents. The choice is yours. Nothing else is accepted inside it. |
curl https://api.newera.ai/v1/chat/completions \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-chat-medium",
"messages": [{"role": "user", "content": "Summarise the water cycle."}],
"max_tokens": 256,
"temperature": 0.7,
"chat_template_kwargs": {"enable_thinking": false}
}'
resp = client.chat.completions.create(
model="newera-chat-medium",
messages=[{"role": "user", "content": "Summarise the water cycle."}],
max_tokens=256,
temperature=0.7,
# Answer directly; omit to let the model reason first.
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)
print(resp.usage)
const resp = await client.chat.completions.create({
model: "newera-chat-medium",
messages: [{ role: "user", content: "Summarise the water cycle." }],
max_tokens: 256,
temperature: 0.7,
// Answer directly; omit to let the model reason first.
chat_template_kwargs: { enable_thinking: false },
});
console.log(resp.choices[0].message.content);
console.log(resp.usage);
const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [
{ role: "user", content: "Summarise the water cycle." },
];
const resp = await client.chat.completions.create({
model: "newera-chat-medium",
messages,
max_tokens: 256,
temperature: 0.7,
// chat_template_kwargs is a Newera extension, not in the OpenAI types.
chat_template_kwargs: { enable_thinking: false },
} as OpenAI.Chat.ChatCompletionCreateParamsNonStreaming);
console.log(resp.choices[0].message.content);
console.log(resp.usage);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-chat-medium",
"messages": []map[string]string{
{"role": "user", "content": "Summarise the water cycle."},
},
"max_tokens": 256,
"temperature": 0.7,
"chat_template_kwargs": map[string]any{"enable_thinking": false},
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
Usage map[string]int `json:"usage"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
fmt.Println(out.Choices[0].Message.Content)
fmt.Println(out.Usage)
}
Chat model
newera-chat-small
Fast chat for everyday requests. Reads text, images and audio, and writes text. It is the smallest of the three tiers: for long, complex or harder work, step up to newera-chat-medium.
Best for
JSON and structured output, tool calling, everyday questions and answers, audio understanding, images, and high-volume tasks where speed matters most.
Not for
Long, complex multi-step tasks and harder reasoning or coding: use newera-chat-medium, or newera-chat-large for the longest inputs. Requests whose input and reply together need more than its 64,000-token window are refused, as are a clip longer than 30 seconds and more than 8 images, 4 clips or 1 video.
- Model id
newera-chat-small- Pinned id
newera-chat-small-v1- Context window
- 64,000 tokens
- Inputs
- text + image + audio
- Billed
- per input and output token
- Endpoint
/chat/completions
Images
Send message content as parts: a {"type":"text"} part and an {"type":"image_url"} part whose url is a data: URL or a public https: URL. Image pixels count as input tokens, and a larger image uses more of them. A request can carry up to 8 images.
Audio
Add an {"type":"input_audio"} part to the message content: input_audio.data is the clip as base64 and input_audio.format is wav or mp3. A request can carry up to 4 clips, each at most 30 seconds long; a longer clip is refused with a 400 (audio_too_long). Audio becomes input tokens (about 25 for every second of sound) and is billed as input tokens.
Thinking
This model answers directly unless you ask it to think. Send chat_template_kwargs with {"enable_thinking": true} to have it think step by step first: better on harder problems, slower and more output tokens. completion_tokens_details.reasoning_tokens shows how much a call thought.
Limits for this model
| Limit | Value |
|---|---|
| Context window | 64,000 tokens, shared by the input and the reply |
| Max input tokens | 63,000 (input plus reply must fit the window) |
| Max output tokens | 63,000: the max_tokens ceiling, thinking included (input plus reply must fit the window) |
| Default output tokens | when max_tokens is left out, up to all the room your input leaves in the window |
| Image input | yes, up to 8 images in a request |
| Audio input | yes, up to 4 clips in a request, each of up to 30 seconds |
Parameters for this model
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
chat_template_kwargs | object | no | {"enable_thinking": false} | Thinking on or off. This model answers directly unless you ask it to think. Send {"enable_thinking": true} to have it think step by step first, which helps on hard problems and costs time and tokens. For content you do not control, such as user messages or documents, we recommend turning thinking on: it helps the model follow your instructions and ignore instructions hidden in that content. It is faster without it. The choice is yours. Nothing else is accepted inside it. |
curl https://api.newera.ai/v1/chat/completions \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-chat-small",
"messages": [{"role": "user", "content": "Classify sentiment: the invoice is overdue"}],
"max_tokens": 16
}'
resp = client.chat.completions.create(
model="newera-chat-small",
messages=[{"role": "user", "content": "Classify sentiment: the invoice is overdue"}],
max_tokens=16,
)
print(resp.choices[0].message.content)
const resp = await client.chat.completions.create({
model: "newera-chat-small",
messages: [{ role: "user", content: "Classify sentiment: the invoice is overdue" }],
max_tokens: 16,
});
console.log(resp.choices[0].message.content);
const resp: OpenAI.Chat.ChatCompletion = await client.chat.completions.create({
model: "newera-chat-small",
messages: [{ role: "user", content: "Classify sentiment: the invoice is overdue" }],
max_tokens: 16,
});
const label: string | null = resp.choices[0].message.content;
console.log(label);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-chat-small",
"messages": []map[string]string{
{"role": "user", "content": "Classify sentiment: the invoice is overdue"},
},
"max_tokens": 16,
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
fmt.Println(out.Choices[0].Message.Content)
}
Embeddings
newera-embedding
POST /v1/embeddings
Turn text into 1,024-number vectors for semantic search, retrieval, clustering and deduplication. The vector size is fixed at 1,024.
Best for
Building a vector index for search or RAG, measuring semantic similarity, grouping or de-duplicating text.
Not for
Generating text. Pair it with newera-rerank to sharpen the top of your results.
- Model id
newera-embedding- Pinned id
newera-embedding-v1- Vector size
- 1,024
- Max input
- 8,192 tokens per text
- Billed
- per input token
- Endpoint
/embeddings
Parameters
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
model | string | yes | ||
input | string | array | yes | One text, or a list of texts. Each text may be up to 8,192 tokens; longer is refused. A list is one request, billed as the total of its tokens. | |
encoding_format | string | no | float | float, base64. float returns numbers. base64 returns the same vector packed as text, which is smaller to send. |
Response
OpenAI embeddings shape: data[].embedding, with a usage object carrying prompt_tokens.
curl https://api.newera.ai/v1/embeddings \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-embedding",
"input": ["first passage", "second passage"]
}'
resp = client.embeddings.create(
model="newera-embedding",
input=["first passage", "second passage"],
)
print(len(resp.data), len(resp.data[0].embedding))
const resp = await client.embeddings.create({
model: "newera-embedding",
input: ["first passage", "second passage"],
});
console.log(resp.data.length, resp.data[0].embedding.length);
const resp: OpenAI.CreateEmbeddingResponse = await client.embeddings.create({
model: "newera-embedding",
input: ["first passage", "second passage"],
});
const vectors: number[][] = resp.data.map((d) => d.embedding);
console.log(vectors.length, vectors[0].length);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-embedding",
"input": []string{"first passage", "second passage"},
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/embeddings", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Data []struct {
Embedding []float64 `json:"embedding"`
} `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
fmt.Println(len(out.Data), len(out.Data[0].Embedding))
}
Rerank
newera-rerank
POST /v1/rerank
Score a list of documents against a query and return them by relevance. The second step of a good search.
Best for
Reordering the top candidates from a vector or keyword search so the most relevant rise to the top; two-stage RAG.
Not for
First-pass retrieval over a large corpus. Use embeddings and a vector store for that, then rerank the shortlist.
- Model id
newera-rerank- Pinned id
newera-rerank-v1- Max documents
- 1,000 per request
- Billed
- per document scored
- Endpoint
/rerank- Shape
- Newera-native
Parameters
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
model | string | yes | ||
query | string | yes | What you are searching for. | |
documents | array | yes | up to 1,000. The texts to rank. Each one, with the query, may be up to 5,000 tokens. Every one is scored and billed, even if top_n then leaves it out. | |
top_n | integer | no | Return only the n best. It does not lower the billed count. | |
return_documents | boolean | no | False | Include each document's text next to its score. |
Usage
Usage counts every candidate the model scores, not only the top_n you keep. Scoring 500 documents and returning 5 counts as 500 documents.
curl https://api.newera.ai/v1/rerank \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-rerank",
"query": "how do I rotate a key?",
"documents": ["Keys are shown once.", "Rotate from the portal."],
"top_n": 2
}'
import os, httpx
BASE = "https://api.newera.ai/v1"
AUTH = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
r = httpx.post(BASE + "/rerank", headers=AUTH, json={
"model": "newera-rerank",
"query": "how do I rotate a key?",
"documents": ["Keys are shown once.", "Rotate from the portal."],
"top_n": 2,
}, timeout=60)
r.raise_for_status()
for row in r.json()["results"]:
print(row["index"], round(row["relevance_score"], 4))
const r = await fetch("https://api.newera.ai/v1/rerank", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.NEWERA_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "newera-rerank",
query: "how do I rotate a key?",
documents: ["Keys are shown once.", "Rotate from the portal."],
top_n: 2,
}),
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
const { results } = await r.json();
for (const row of results) {
console.log(row.index, row.relevance_score.toFixed(4));
}
interface RerankResult {
index: number;
relevance_score: number;
}
const r = await fetch("https://api.newera.ai/v1/rerank", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.NEWERA_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "newera-rerank",
query: "how do I rotate a key?",
documents: ["Keys are shown once.", "Rotate from the portal."],
top_n: 2,
}),
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
const { results } = (await r.json()) as { results: RerankResult[] };
for (const row of results) {
console.log(row.index, row.relevance_score.toFixed(4));
}
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-rerank",
"query": "how do I rotate a key?",
"documents": []string{"Keys are shown once.", "Rotate from the portal."},
"top_n": 2,
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/rerank", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Results []struct {
Index int `json:"index"`
RelevanceScore float64 `json:"relevance_score"`
} `json:"results"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
for _, row := range out.Results {
fmt.Printf("%d %.4f\n", row.Index, row.RelevanceScore)
}
}
Document extraction
newera-ocr
POST /v1/documents/extract
Pull text out of a PDF or an image. Handles Arabic and mixed scripts. multipart/form-data.
Best for
Reading scanned or born-digital PDFs and images into plain text; feeding documents into search or an LLM.
Not for
Structured field extraction with coordinates. It returns the document's text, not a form-field map.
- Model id
newera-ocr- Pinned id
newera-ocr-v1- Max upload
- 50 MB
- Billed
- per page processed
- Endpoint
/documents/extract- Shape
- Newera-native · multipart
Parameters (form fields)
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
file | file | yes | up to 50 MB. A PDF or an image. | |
model | string | no | newera-ocr |
Response
{ "text": "...", "pages": [...] | null, "usage": { "pages": N } }
curl https://api.newera.ai/v1/documents/extract \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-F "file=@contract.pdf" \
-F "model=newera-ocr"
import os, httpx
BASE = "https://api.newera.ai/v1"
AUTH = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
with open("contract.pdf", "rb") as fh:
r = httpx.post(BASE + "/documents/extract", headers=AUTH,
files={"file": fh},
data={"model": "newera-ocr"},
timeout=300)
r.raise_for_status()
doc = r.json()
print(doc["text"][:500])
print(doc["usage"]["pages"], "pages billed")
import { openAsBlob } from "node:fs";
const form = new FormData();
form.append("file", await openAsBlob("contract.pdf"), "contract.pdf");
form.append("model", "newera-ocr");
const r = await fetch("https://api.newera.ai/v1/documents/extract", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}` },
body: form,
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
const doc = await r.json();
console.log(doc.text.slice(0, 500));
console.log(doc.usage.pages, "pages billed");
import { openAsBlob } from "node:fs";
interface ExtractResponse {
text: string;
usage: { pages: number };
}
const form = new FormData();
form.append("file", await openAsBlob("contract.pdf"), "contract.pdf");
form.append("model", "newera-ocr");
const r = await fetch("https://api.newera.ai/v1/documents/extract", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}` },
body: form,
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
const doc = (await r.json()) as ExtractResponse;
console.log(doc.text.slice(0, 500));
console.log(doc.usage.pages, "pages billed");
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
)
func main() {
var buf bytes.Buffer
form := multipart.NewWriter(&buf)
f, err := os.Open("contract.pdf")
if err != nil {
panic(err)
}
part, _ := form.CreateFormFile("file", "contract.pdf")
io.Copy(part, f)
f.Close()
form.WriteField("model", "newera-ocr")
form.Close()
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/documents/extract", &buf)
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", form.FormDataContentType())
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var doc struct {
Text string `json:"text"`
Usage struct {
Pages int `json:"pages"`
} `json:"usage"`
}
if err := json.NewDecoder(resp.Body).Decode(&doc); err != nil {
panic(err)
}
fmt.Println(doc.Text[:min(500, len(doc.Text))])
fmt.Println(doc.Usage.Pages, "pages billed")
}
Segmentation
newera-segment
POST /v1/images/segment
Find and outline the concepts you name in an image, returning a mask, a bounding box and a confidence for each. multipart/form-data.
Best for
Locating named objects in imagery: buildings, roads, vegetation and the like in aerial or satellite pictures.
Not for
Open-ended "segment everything" requests. It finds the concepts you name, so name what you want found.
- Model id
newera-segment- Pinned id
newera-segment-v1- Max upload
- 25 MB
- Billed
- per image submitted
- Endpoint
/images/segment- Shape
- Newera-native · multipart
Parameters (form fields)
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
image | file | yes | up to 25 MB. A PNG or JPEG image. | |
model | string | no | newera-segment | |
prompt | string | yes | What to look for, for example "building, road, tree". Only what you name is found: there is no find-everything mode. |
Response
detections[], each with label, confidence, a box, and a full-resolution PNG mask as a data: URL; plus not_found, image dimensions, and usage.
Finding nothing is a success, not an error. Ask for three things and find one, and detections has one entry while not_found lists the other two. Check not_found rather than diffing the response against your request. Either way the request counts as one image. The first request after a quiet period can take about 40 seconds; later requests return in under a second.
curl https://api.newera.ai/v1/images/segment \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-F "image=@aerial.png" \
-F "model=newera-segment" \
-F "prompt=building, road"
import os, base64, httpx
BASE = "https://api.newera.ai/v1"
AUTH = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
with open("aerial.png", "rb") as fh:
r = httpx.post(BASE + "/images/segment", headers=AUTH,
files={"image": fh},
data={"model": "newera-segment", "prompt": "building, road"},
timeout=300)
r.raise_for_status()
for det in r.json()["detections"]:
print(det["label"], round(det["confidence"], 3))
head, _, b64 = det["mask"].partition(",") # data:image/png;base64,....
open(det["label"] + ".png", "wb").write(base64.b64decode(b64))
import { openAsBlob, writeFileSync } from "node:fs";
const form = new FormData();
form.append("image", await openAsBlob("aerial.png"), "aerial.png");
form.append("model", "newera-segment");
form.append("prompt", "building, road");
const r = await fetch("https://api.newera.ai/v1/images/segment", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}` },
body: form,
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
const { detections } = await r.json();
for (const det of detections) {
console.log(det.label, det.confidence.toFixed(3));
const b64 = det.mask.split(",")[1]; // data:image/png;base64,....
writeFileSync(`${det.label}.png`, Buffer.from(b64, "base64"));
}
import { openAsBlob, writeFileSync } from "node:fs";
interface Detection {
label: string;
confidence: number;
mask: string; // data:image/png;base64,....
}
const form = new FormData();
form.append("image", await openAsBlob("aerial.png"), "aerial.png");
form.append("model", "newera-segment");
form.append("prompt", "building, road");
const r = await fetch("https://api.newera.ai/v1/images/segment", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}` },
body: form,
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
const { detections } = (await r.json()) as { detections: Detection[] };
for (const det of detections) {
console.log(det.label, det.confidence.toFixed(3));
const b64 = det.mask.split(",")[1];
writeFileSync(`${det.label}.png`, Buffer.from(b64, "base64"));
}
package main
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
"strings"
)
func main() {
var buf bytes.Buffer
form := multipart.NewWriter(&buf)
f, err := os.Open("aerial.png")
if err != nil {
panic(err)
}
part, _ := form.CreateFormFile("image", "aerial.png")
io.Copy(part, f)
f.Close()
form.WriteField("model", "newera-segment")
form.WriteField("prompt", "building, road")
form.Close()
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/images/segment", &buf)
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", form.FormDataContentType())
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Detections []struct {
Label string `json:"label"`
Confidence float64 `json:"confidence"`
Mask string `json:"mask"` // data:image/png;base64,....
} `json:"detections"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
for _, det := range out.Detections {
fmt.Printf("%s %.3f\n", det.Label, det.Confidence)
_, b64, _ := strings.Cut(det.Mask, ",")
png, _ := base64.StdEncoding.DecodeString(b64)
os.WriteFile(det.Label+".png", png, 0o644)
}
}
Speech
newera-tts
POST /v1/audio/speech
Natural Saudi Arabic speech from text. OpenAI-shaped; returns 24 kHz audio bytes. Tuned for spoken Saudi Arabic, it answers fast enough for live calls. It is the voice of newera-s2s.
Best for
Voicing short-to-medium text: notifications, prompts, call-centre replies and read-aloud. Mainly Saudi Arabic, with English supported as well.
Not for
Languages beyond Arabic and English, or inputs over 4,096 characters per request.
- Model id
newera-tts- Pinned id
newera-tts-v1- Max input
- 4,096 characters
- Billed
- per second produced, rounded up
- Endpoint
/audio/speech- Returns
- 24 kHz mono audio
Parameters
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
model | string | yes | ||
input | string | yes | 1 - 4,096 characters. The text to speak. | |
voice | string | no | male | male, female. male (Khaled) or female (Lama). |
quality | string | no | studio | instant, balanced, studio. instant starts fastest, studio sounds most natural. |
normalize | boolean | no | True | Say numbers, times, percentages, phone numbers, currency and brand names the way a person would. false speaks the text exactly as written. |
response_format | string | no | wav | wav, pcm. wav is a complete audio file. pcm is raw audio: 24 kHz, 16-bit, mono. |
speed | number | no | 1.0 | 0.25 - 4. How fast it speaks. 1 is the natural pace. |
Numbers and names, spoken naturally
Write text the way you normally would. Digits, times, percentages, amounts, phone numbers and brand names are spoken in natural Saudi Arabic automatically: 10:30
is heard as عشر ونص
, and 15%
as خمسطعش بالمية
. Send "normalize": false to speak the text exactly as given.
Duration
With wav, the response carries x-newera-audio-seconds and x-newera-billed-seconds, so you can reconcile usage without decoding the audio.
curl https://api.newera.ai/v1/audio/speech \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "newera-tts",
"input": "أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
"voice": "male",
"quality": "balanced",
"response_format": "wav"
}' \
--output reply.wav
audio = client.audio.speech.create(
model="newera-tts",
input="أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
voice="male",
response_format="wav",
extra_body={"quality": "balanced"}, # instant | balanced | studio
)
with open("reply.wav", "wb") as fh:
fh.write(audio.read())
import { writeFile } from "node:fs/promises";
const audio = await client.audio.speech.create({
model: "newera-tts",
input: "أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
voice: "male",
response_format: "wav",
quality: "balanced", // instant | balanced | studio
});
await writeFile("reply.wav", Buffer.from(await audio.arrayBuffer()));
import { writeFile } from "node:fs/promises";
const params: OpenAI.Audio.SpeechCreateParams = {
model: "newera-tts",
input: "أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
voice: "male" as OpenAI.Audio.SpeechCreateParams["voice"],
response_format: "wav",
// quality is a Newera extension, not in the OpenAI types.
quality: "balanced",
} as OpenAI.Audio.SpeechCreateParams;
const audio = await client.audio.speech.create(params);
await writeFile("reply.wav", Buffer.from(await audio.arrayBuffer()));
package main
import (
"bytes"
"encoding/json"
"io"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "newera-tts",
"input": "أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
"voice": "male",
"quality": "balanced",
"response_format": "wav",
})
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/audio/speech", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
out, err := os.Create("reply.wav")
if err != nil {
panic(err)
}
defer out.Close()
io.Copy(out, resp.Body)
}
Speech
newera-stt
POST /v1/audio/transcriptions
Transcribe Arabic audio to text. OpenAI-shaped, multipart/form-data.
Best for
Arabic transcription: meetings, calls, voice notes and media, including everyday Arabic-English code-switching (Arabic speech with English words mixed in).
Not for
Languages other than Arabic and English. It is tuned for Arabic; send language en for English. verbose_json output is not supported (use json or text).
- Model id
newera-stt- Pinned id
newera-stt-v1- Max upload
- 25 MB
- Billed
- per second submitted, rounded up
- Endpoint
/audio/transcriptions- Language
- Arabic (send
languagear)
Parameters (form fields)
| Name | Type | Req | Default | What it does |
|---|---|---|---|---|
file | file | yes | up to 25 MB. The audio: wav, mp3, m4a, ogg, flac or webm. | |
model | string | yes | ||
language | string | no | en | ar, en. The language spoken. Send ar for Arabic: the language is not detected, and Arabic audio sent without it comes back written in English letters. |
response_format | string | no | json | json, text. json returns {"text": ...}. text returns the plain text. |
temperature | number | no | 0 | 0 - 1. 0 gives the most consistent transcript. |
Response
{ "text": "..." } for json, plain text for text; duration is on the x-newera-audio-seconds header either way.
curl https://api.newera.ai/v1/audio/transcriptions \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-F "file=@meeting.wav" \
-F "model=newera-stt" \
-F "response_format=json"
with open("meeting.wav", "rb") as fh:
result = client.audio.transcriptions.create(
model="newera-stt",
file=fh,
response_format="json",
)
print(result.text)
import { createReadStream } from "node:fs";
const result = await client.audio.transcriptions.create({
model: "newera-stt",
file: createReadStream("meeting.wav"),
response_format: "json",
});
console.log(result.text);
import { createReadStream } from "node:fs";
const result: OpenAI.Audio.Transcription = await client.audio.transcriptions.create({
model: "newera-stt",
file: createReadStream("meeting.wav"),
response_format: "json",
});
console.log(result.text);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
)
func main() {
var buf bytes.Buffer
form := multipart.NewWriter(&buf)
f, err := os.Open("meeting.wav")
if err != nil {
panic(err)
}
part, _ := form.CreateFormFile("file", "meeting.wav")
io.Copy(part, f)
f.Close()
form.WriteField("model", "newera-stt")
form.WriteField("response_format", "json")
form.Close()
req, _ := http.NewRequest("POST", "https://api.newera.ai/v1/audio/transcriptions", &buf)
req.Header.Set("Authorization", "Bearer "+os.Getenv("NEWERA_API_KEY"))
req.Header.Set("Content-Type", form.FormDataContentType())
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(resp.Body)
panic(resp.Status + ": " + string(msg))
}
var out struct {
Text string `json:"text"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
panic(err)
}
fmt.Println(out.Text)
}
Realtime voice
newera-s2s
WSS /v1/realtime
A live speech-to-speech conversation in Arabic or English over one WebSocket: audio in, audio out. Stream the caller's audio in; the assistant's speech, its written words and the caller's transcript stream back. The assistant notices when the caller has finished, answers almost instantly, and stops talking when the caller cuts in.
The events follow the standard Realtime event set (beta version): the same event names, fields and order. A standard Realtime client works by changing the URL and the API key, with no proxy. Compatibility notes lists what is not the same.
By default the language of the call is detected automatically, on every caller turn (Arabic or English), and the assistant answers in it. If you know the language up front, fix it with newera.detect_language and detection is off for the whole call. See Language.
Best for
Phone agents and customer service in Saudi Arabic: bookings, order and account questions, short guided conversations.
Not for
Reading long text aloud (use newera-tts) or languages other than Arabic and English.
- Model id
newera-s2s(default) ornewera-s2s-mini, chosen in the URL- Pinned id
newera-s2s-v1- URL
wss://api.newera.ai/v1/realtime- Audio in
- PCM 16-bit mono, 24 kHz (or 8 to 48 kHz, or G.711)
- Audio out
- PCM 16-bit mono, 24 kHz
Connect
From a server, send your API key on the WebSocket handshake as Authorization: Bearer …. Choose the model in the URL: wss://api.newera.ai/v1/realtime?model=newera-s2s-mini (the default is newera-s2s; see Choosing the model). The key must be entitled to the model you connect with. A refused connection still opens, sends one error event saying why, and closes with one of these codes:
| Close code | Meaning |
|---|---|
4400 | Unknown or non-voice model (invalid_model) |
4401 | Missing or invalid API key |
4402 | The balance cannot cover the call (insufficient_quota) |
4403 | Key not entitled to the model (model_not_entitled) |
4429 | Too many calls on this key right now (concurrent_call_limit). Wait and retry. |
4503 | Voice is temporarily unavailable; retry |
Connect-time refusals are error events, and the socket then closes. Two messages: invalid_model, close 4400: "foo is not a voice model. Choose newera-s2s or newera-s2s-mini, for example ?model=newera-s2s-mini." and insufficient_quota, close 4402: "Your prepaid balance cannot cover this call. Top up your credit and connect again."
If a key reaches its call capacity, a new call is refused with concurrent_call_limit (close 4429) and calls already running are not affected; contact your account manager to raise it.
From a browser, never put the API key in the page. Your server trades it for a client secret that lasts 60 seconds and opens one connection, and the page offers the secret as a WebSocket subprotocol, newera-client-secret.<secret>, beside realtime.
curl -X POST https://api.newera.ai/v1/realtime/sessions \
-H "Authorization: Bearer $NEWERA_API_KEY"
# {"object": "realtime.session", "model": "newera-s2s",
# "client_secret": {"value": "ek_...", "expires_at": 1760000000}}GET /v1/realtime/voices with your API key lists the voices before you connect: {"default", "data": [{"id", "name", "gender"}]}. There are two: male (Khaled) and female (Lama).
Audio
Both directions carry audio base64-encoded inside JSON events. Send the caller's audio in small chunks of 20 to 100 ms, continuously and in real time, silence included: the call decides that a turn is over by hearing the silence. Turn on echo cancellation on the microphone, or the assistant hears itself and stops. If you capture audio in a browser, turn off autoGainControl in getUserMedia (Chrome turns it on by default).
input_audio_format: "pcm16" (the default) is 16-bit little-endian mono PCM at 24 kHz; g711_ulaw and g711_alaw are 8 kHz telephony. For PCM at 16 kHz, 8 kHz or 48 kHz set newera.input_audio_sample_rate. Output is output_audio_format: "pcm16" at 24 kHz (the default) or G.711 at 8 kHz.
A call, event by event
→ (connect)
← session.created settings and voices
← conversation.created
→ session.update optional: instructions, voice, turn control
→ input_audio_buffer.append {audio} continuously, silence included
← input_audio_buffer.speech_started {item_id} the caller began talking
← input_audio_buffer.speech_stopped {item_id} the caller paused
← input_audio_buffer.committed {item_id, previous_item_id} the pause was long enough: a turn
← conversation.item.created {item} the caller's audio as an item (transcript null)
← conversation.item.input_audio_transcription.completed {item_id, transcript}
(with create_response: true, the default, the reply follows)
← response.created {response} status in_progress, output [], your metadata
← response.output_item.added {item} the assistant's message item
← conversation.item.created {item} the same item, in the conversation
← response.content_part.added {part}
← response.audio_transcript.delta {delta} text of the next piece
← response.audio.delta {delta} its audio, base64 PCM 24 kHz
...repeated per piece...
← response.audio.done
← response.audio_transcript.done {transcript}
← response.content_part.done {part}
← response.output_item.done {item}
← response.done {response} status, output[], usageEvery reply has this shape, whether the caller's voice, a typed message or a response.create started it. With tools, function calls appear in the middle of it (see Tools). In a text-only reply (modalities: ["text"]) the audio events are replaced by response.text.delta and response.text.done, and the content part is {type: "text", text}.
Events you send
| Event | Fields | What it does |
|---|---|---|
session.update | session | Change the call's settings (below). Answered by session.updated. A field that is unknown, unsupported or has a bad value gets an error event naming it (unknown_parameter, unsupported_parameter or invalid_value, with param); the rest of the update still applies. |
input_audio_buffer.append | audio | Stream the caller's audio. |
input_audio_buffer.commit | With turn_detection: null: add the audio sent so far to the conversation as the caller's turn (at least 100 ms). It does not start a reply: send response.create. | |
input_audio_buffer.clear | Discard audio not yet committed (manual turns). | |
conversation.item.create | item, previous_item_id? | Add an item to the conversation: a message from the user, the assistant or the system, or a function call and its output. See Items. |
conversation.item.delete | item_id | Remove an item. Answered by conversation.item.deleted. |
conversation.item.retrieve | item_id | Answered by conversation.item.retrieved. |
conversation.item.truncate | item_id, content_index: 0, audio_end_ms | Tell Newera how much of an assistant reply was actually played, so its memory of the conversation matches what the caller heard. Answered by conversation.item.truncated. |
response.create | response? | Ask for a reply. See Replies. |
response.cancel | response_id? | Stop the current reply. response.done has status: "cancelled" and reason: "client_cancelled". |
session.end | Newera extension. Finish the call: you receive session.summary with the call's connected time and latency, then the socket closes. |
Session settings
Send these in a session.update event. A name with a dot is a field inside an object: turn_detection.threshold goes in session.turn_detection, and newera.voice_quality in session.newera.
| Name | Type | Default | What it does |
|---|---|---|---|
modalities | array | ["text", "audio"] | text, audio. Replies are spoken, and the written words of each reply arrive with the audio. ["audio"] gives the same. Send ["text"] for written replies with no speech. |
instructions | string | Who the assistant is and how it should behave: its name, its job, what it may talk about. Newera adds its own rules first: short replies, natural Saudi Arabic, numbers said as words. | |
voice | string | male | male, female. male (Khaled) or female (Lama). Voice names from other realtime clients are matched to the closest of these two. |
input_audio_format | string | pcm16 | pcm16, g711_ulaw, g711_alaw. The audio you send. pcm16 is 16-bit mono at 24 kHz unless you set newera.input_audio_sample_rate. g711 is 8 kHz phone audio. |
output_audio_format | string | pcm16 | pcm16, g711_ulaw, g711_alaw. The audio you get back. pcm16 is 16-bit mono at 24 kHz. g711 is 8 kHz phone audio. |
input_audio_transcription | object | null | Set {"language": "en"} or {"language": "ar"} to fix the call's language. null turns off the written transcript of what the caller said. | |
turn_detection.silence_duration_ms | integer | 500 | 100 - 5000. How long a pause ends the caller's turn. Shorter answers sooner but may cut in on a slow speaker. |
turn_detection.threshold | number | 0.6 | 0 - 1. How clearly something must sound like speech to count as the caller talking. Higher ignores more background noise. |
turn_detection.prefix_padding_ms | integer | 300 | 0 - 1000. How much audio from just before the caller starts speaking is kept, so the first word is not clipped. |
turn_detection.create_response | boolean | True | Reply automatically when the caller's turn ends. With false, the assistant replies only when you send response.create. |
turn_detection.interrupt_response | boolean | True | Stop talking when the caller speaks over the assistant. |
turn_detection | object | null | Set to null to end turns yourself: send input_audio_buffer.commit, then response.create. | |
temperature | number | 0.4 | 0 - 1.5. How varied the replies are. Values outside the range are set to the nearest limit. |
max_response_output_tokens | integer | "inf" | no limit | The most tokens in one reply. "inf" means no limit. |
tools | array | Functions the assistant may call during the call. | |
tool_choice | string | object | auto | "auto", "none", "required", or one named function. |
newera.voice_quality | string | studio | instant, balanced, studio. instant starts each reply fastest, studio sounds most natural. Can change mid-call and applies from the next reply. |
newera.voice_clone | object | Speak in your own voice, or one whose owner has agreed: {"audio": base64, "text": what the clip says}. The clip is 3 to 8 seconds and at most 3 MB. | |
newera.input_audio_sample_rate | integer | 24000 | 8000, 16000, 24000, 48000. The sample rate of the pcm16 audio you send. |
newera.detect_language | string | auto | auto, ar, en. auto works out Arabic or English on every turn. ar or en fixes the call to that language. |
newera.timezone | string | Asia/Riyadh | The time zone the assistant uses for today, now and weekdays, for example Asia/Riyadh. |
newera.tool_auto_continue | boolean | True | After you return a tool's result, the assistant carries on and speaks the answer by itself. |
newera.max_tool_steps | integer | 6 | The most tool rounds in a single reply. |
An unknown field anywhere in session, session.newera, session.turn_detection or response is an error event naming it, never silently ignored. A bad value gets an error too (for example invalid_value with param: "session.newera.voice_quality"), and the call keeps its setting. The model is chosen in the URL and cannot change during a call: sending the same model back is accepted, a different one is refused with unsupported_parameter.
Choosing the model
Pick the model when the call connects, in the URL: wss://api.newera.ai/v1/realtime?model=newera-s2s-mini. Without model the call uses newera-s2s. The model is fixed for the whole call.
| Model | Use it for |
|---|---|
newera-s2s (default) | Conversations that need deeper reasoning: multi-step troubleshooting, consultative sales, bookings that balance several constraints, workflows that chain many tools, long calls with a lot of context, nuanced policy questions. |
newera-s2s-mini | Fast and efficient for high-volume, well-defined conversations: customer support and FAQs, appointment booking and rescheduling, order and delivery status, reservations, lead qualification and intake, surveys and feedback calls, reminders and confirmations, replacing IVR menus, and tool-driven flows such as looking up a record or opening a ticket. |
Both models support tool calling and the same protocol, events and voices.
Language
session.newera.detect_language sets how the call's language is decided: "auto" (the default), "ar" or "en".
"auto"detects the language on every caller turn. Newera listens to what the caller says and decides between Arabic and English. The assistant replies in the detected language, and a typed message follows the script it is written in. A single short or accented sentence does not flip the call: switching takes strong evidence.- A fixed language (
"ar"or"en") turns detection off. For the whole call, from the next turn: no language detection runs at all, the caller's speech is always taken as that language, the assistant always replies in it, and the voice and the way numbers and dates are spoken follow it. Typed text does not change the language either: English typed into an"ar"call is answered in Arabic. - Use a fixed language when the call's language is known up front, such as interviews in one language, or a line that serves a single language. It is also the right choice for callers who speak one language with a strong accent in it (an English interview with an Arabic accent), where detection could guess the other language. Keep
"auto"when callers may speak either. session.createdand everysession.updatedshow the setting undernewera.detect_language. It can be changed at any time and applies from the next turn: a reply already being generated or spoken keeps the language it started in.- This works too:
session.input_audio_transcription: {"language": "en"}(or"ar") fixes the language exactly likedetect_language. Any other language is anerrorwithinvalid_valueandparam: "session.input_audio_transcription.language", naming the supported ones. If onesession.updatecarries both and they disagree,newera.detect_languagewins. - Any other value of
detect_languageis anerrorwithinvalid_valueandparam: "session.newera.detect_language", and the call keeps its setting.
{"type": "session.update", "session": {"newera": {"detect_language": "en"}}}Turn control
turn_detection decides who ends the caller's turn and who starts the reply.
Automatic turns (the default). The call listens for the caller's speech and tells you what it hears: speech_started, speech_stopped, committed and a conversation.item.created for the caller's audio, then its transcript. silence_duration_ms is how long a pause ends the caller's turn: 500 ms by default, 100 to 5000, counted from the caller's last word to the commit. threshold (0 to 1, default 0.6) is how clearly a sound must be speech to count, so a higher value ignores more background noise. prefix_padding_ms (0 to 1000, default 300) is the audio kept from just before the caller starts, so the first word is not clipped. The assistant starts preparing its answer after 200 ms of silence and holds it until the commit; if the caller resumes first, the draft is dropped and the turn simply continues. Shorter pauses answer sooner and cut into the caller's pauses more often.
create_response(defaulttrue): the assistant answers each committed turn. Withfalseit does everything above, the transcript and the item included, but starts no reply: the client decides, withresponse.create. Aresponse.createsent before the transcript is ready waits for it.interrupt_response(defaulttrue): when the caller talks over a reply, Newera cancels it (response.donewithstatus: "cancelled",reason: "turn_detected") and the conversation keeps what was heard. Withfalsethe reply plays on and the next one starts after it.
turn_detection: null (manual). The call does not listen for turns. Send input_audio_buffer.append, then input_audio_buffer.commit: the caller's audio becomes an item (committed, conversation.item.created, its transcript) and nothing else happens until response.create. Several commits before one response.create are all part of its context.
Items
The conversation is a list of items with ids: the caller's audio and typed messages, the assistant's replies, and whatever the client adds. A reply is always generated from the conversation as it stands. conversation.item.create takes these (previous_item_id places the item after that one; absent, at the end; "root", first):
| Item | Content |
|---|---|
{type: "message", role: "user"} | input_text {text} (or text); input_audio {audio, transcript?}, audio in the session's input format: with a transcript it is used as is, without one, the transcript is filled in from the audio. |
{type: "message", role: "assistant"} | text {text} (or output_text); audio {transcript}. |
{type: "message", role: "system"} | input_text {text}: facts and context for the assistant, shown to it with the instructions, not as caller speech. |
{type: "function_call", call_id, name, arguments} and {type: "function_call_output", call_id, output} | A tool call and its result, to restore a conversation that used tools. |
Use it to restore a conversation after a reconnect: create the items, then response.create. The answer is conversation.item.created {previous_item_id, item}. An unknown previous_item_id, a duplicate id and an unsupported content type or role are each an error naming the field. The history is trimmed from its oldest turn when it outgrows the model's budget, as in a live call.
conversation.item.truncate applies to an assistant reply Newera spoke: the conversation then holds only the words heard up to audio_end_ms, followed by "…", and Newera answers conversation.item.truncated. An unknown item_id is an invalid_request_error (item_not_found) naming it; a user item, or an item without audio, cannot be truncated.
Replies
response.create asks for a reply to the conversation as it stands; there is no need to add input first. With nothing to answer and no instructions, the assistant opens the call from the session instructions.
{"type": "response.create", "response": {
"instructions": "Greet the caller by name, then ask how you can help.",
"metadata": {"client_event_id": "abc"}}}| response field | |
|---|---|
instructions | For this reply only: added after the session's instructions and winning where they disagree. They are instructions, not caller text: nothing is stored as a user turn, and the next reply does not see them. |
modalities, voice, temperature, max_output_tokens | This reply's own value instead of the session's. |
metadata | Up to 16 string pairs, echoed in response.created and response.done. |
conversation | Only "auto" (or "default"): every reply is added to the conversation. |
Not supported, with an unsupported_parameter error naming the field and no reply started: conversation: "none" (out-of-band responses), input, and per-response tools, tool_choice or output_audio_format. One reply runs at a time: a response.create while another is running is conversation_already_has_active_response, and response.cancel with nothing running is response_cancel_not_active.
Barge-in
When the caller talks over a reply (and interrupt_response is on), Newera cancels it and sends response.done with status: "cancelled", then input_audio_buffer.speech_started. On speech_started, stop playback and drop any queued audio. Audio arrives faster than real time, so a player usually holds several seconds in hand. If you know how much was played, send conversation.item.truncate; otherwise Newera estimates it.
response.done
{"type": "response.done", "response": {
"id": "resp_...", "object": "realtime.response", "status": "completed", "status_details": null,
"conversation_id": "conv_...", "metadata": {"client_event_id": "abc"},
"output": [{"id": "item_...", "object": "realtime.item", "type": "message", "status": "completed",
"role": "assistant", "content": [{"type": "audio", "transcript": "أبشر، متى يناسبك الموعد؟"}]}],
"usage": null,
"latency": {"voice_to_voice_ms": …},
"output_transcript": "أبشر، متى يناسبك الموعد؟"}}output is always filled: the assistant's message first, then any function_call items. usage is null: a reply carries no token or audio counts. latency.voice_to_voice_ms runs from the caller's last word to the first audio of the reply leaving Newera; add your network time to get what the caller hears. output_transcript is the reply's full text.
status is completed, cancelled (barge-in or response.cancel) or failed; status_details says why ({type: "cancelled", reason: "turn_detected" | "client_cancelled"}, or {type: "failed", error}). A reply that was cancelled still closes its item first (response.audio.done, response.audio_transcript.done, response.content_part.done, response.output_item.done) with the words spoken so far.
Compatibility notes
The call follows the beta Realtime events. What is not the same as a standard client expects:
- Beta event names. The beta event names (
response.audio.delta,response.audio_transcript.delta,conversation.item.created) and a flatsessionobject. Fields from the newer version of the protocol (session.audio,output_modalities,type) are refused by name. - Voices are
maleandfemale. A voice name from another realtime client is accepted so that client's default works, and matched to the closest of the two:session.voicethen readsmaleorfemale. Any other name isinvalid_voice. The model is selected in the URL (?model=).input_audio_transcription.languageof"en"or"ar"fixes the call's language, likenewera.detect_language(which wins if both are sent); other languages are refused withinvalid_value. Without either, the language is detected on every caller turn. - Turn detection has one kind, so
typecan be left out.silence_duration_mscounts from the last word to the commit, butspeech_stoppedis sent after the first 200 ms of it. - Not supported (an error naming the field): replies outside the conversation (
response.conversation: "none"),response.input, per-responsetools,tool_choiceandoutput_audio_format,input_audio_noise_reduction,tracing, aspeedother than 1.0, content types other than text and audio,item_reference. - Transcription events:
conversation.item.input_audio_transcription.completedonly; there is no.delta. - Function calls the assistant makes are listed in
response.done.output; they are not individually addressable withconversation.item.retrieveordelete(the reply item that holds them is). - Tool calls are one response by default (
newera.tool_auto_continue: true): the assistant continues by itself after yourfunction_call_outputs. Set it tofalseto send a secondresponse.createyourself. See Tools. - No
rate_limits.updated. The only limit is your balance. - Newera extras, safe to ignore:
session.newera,output_transcript,latencyonresponse.done,latencyonsession.summary, and thenewera.function_call.*events. - Usage is
nullonresponse.done, andconversation.item.input_audio_transcription.completedcarries nousage.
Tools
The assistant can call functions that you run: look up a booking, check an order, create an appointment. Declare them in session.update. When the assistant decides to use one, Newera sends you a function_call; you run it and send back the result; the assistant carries on and speaks the answer. The shapes are the standard Realtime ones, plus Newera additions marked as such.
{"type": "session.update", "session": {
"tools": [
{"type": "function", "name": "get_free_slots",
"description": "Free appointment times on a day",
"parameters": {"type": "object",
"properties": {"date": {"type": "string", "description": "YYYY-MM-DD"}},
"required": ["date"]}},
{"type": "function", "name": "book_appointment",
"description": "Book an appointment",
"parameters": {"type": "object",
"properties": {"patient_name": {"type": "string"}, "date": {"type": "string"},
"time": {"type": "string"}},
"required": ["patient_name", "date", "time"]},
"newera": {"requires_confirmation": true, "timeout_ms": 20000}}],
"tool_choice": "auto"}}| Field | Notes |
|---|---|
name | ^[a-zA-Z0-9_-]{1,64}$, unique. |
description | Tell the assistant when to use the function; it reads this. |
parameters | JSON Schema, type: "object". |
newera.requires_confirmation | Newera extension. Default false. Set true for anything that changes something: the assistant reads the details back and the caller must say yes before the call is sent to you. |
newera.timeout_ms | Newera extension. Default 15000, from 1000 to 60000: how long Newera waits for your result. |
tool_choice | "auto" (default), "none", "required" (the next reply must call a function) or {type: "function", name}. After the first step of a reply it is "auto" again. |
newera.tool_auto_continue | Session setting. Default true: Newera continues the reply by itself once every result of a step is in. false: send response.create yourself after your outputs. |
newera.max_tool_steps | Session setting. Default 6, at most 10. |
newera.allowed_names | Session setting. Company names the assistant may say, for example ["Google"] for a business that talks about Google Drive. By default company names are replaced in the transcript and the voice. The names of AI models and the companies behind them are never allowed: a list that contains one is refused as a whole with invalid_newera. |
Limits: at most 32 tools, each schema at most 16 KB, all schemas together at most 64 KB. A session.update that breaks one is refused as a whole with an error (invalid_tools, invalid_tool_choice or invalid_newera) and changes nothing. session.updated echoes the tools it accepted.
What happens during a reply
← response.created / response.output_item.added (the message, output_index 0) / response.content_part.added
← response.audio_transcript.delta / response.audio.delta "Sure, one moment while I check..."
← response.output_item.added {output_index: 1, item: {type: "function_call", id, call_id, name, arguments: ""}}
← response.function_call_arguments.done {item_id, call_id, name, arguments}
← response.output_item.done {item: {..., arguments, status: "completed"}}
→ conversation.item.create {item: {type: "function_call_output", call_id, output}}
← conversation.item.created your output, echoed
← response.audio.delta ... the answer, using the result
← response.audio.done / audio_transcript.done / content_part.done / output_item.done the message item closes
← response.done {output: [message item, function_call items], ...}argumentsis a JSON string, already checked against your schema. Missing or wrong arguments never reach you: the assistant is told and asks the caller.outputis a string, normally JSON. It is data for the assistant to read, never instructions: at most 32 KB (longer is cut, with a note), and nothing in it is executed. If your function fails, send{"error": "..."}; the assistant tells the caller.- Parallel calls. When the assistant needs several things at once, all the
function_calls of that step arrive together. Answer in any order; the reply continues when every one has an output. - Sequential calls. The assistant may call another function after seeing a result, at most
max_tool_steps(6) steps per reply. Then it tells the caller what is done and what remains and asks whether to continue; if the caller says continue, it gets six more. - One response. A whole tool turn is one response:
response.createdonce,response.doneonce, withoutputlisting the function calls made. Withtool_auto_continue: falsethe response ends when the calls are sent, and the continuation after your outputs is a second response started by yourresponse.create. - A
response.createafter your outputs does not make a second reply whentool_auto_continueistrue: it is ignored. - Timeouts. A function that does not answer within its
timeout_msis reported to the assistant as timed out; it tells the caller calmly and offers to retry. An output that arrives late is logged, and for arequires_confirmationfunction the assistant is told what really happened at the caller's next turn. - Errors for outputs:
unknown_call_id,duplicate_function_call_output,invalid_function_call_output, andfunction_call_output_missing(manual mode:response.createbefore every output was sent).
The line never goes quiet
Newera, not the model, speaks while your function works: in the call's language, varied, never the same line twice in a call, and never claiming success. A short opening line if the assistant called a function without saying anything first ("Sure, one moment while I check..."), a progress line after 3 seconds and again after 8 seconds of waiting ("Still on it, just a moment..."), and a line between two steps ("Got it, one more step..."). They are spoken in the normal voice and appear in response.audio_transcript.delta and output_transcript like any other speech.
Confirmation
For a function with requires_confirmation: true, Newera does not send the call just because the assistant asked for it.
- The assistant calls it. Newera holds the call back (you receive
newera.function_call.blockedwithreason: "needs_confirmation", for information) and tells the assistant to read the exact details back to the caller. - The caller answers. Clear words in Arabic and English (ايه، أيوه، تمام، أكيد، yes, sure, go ahead... / لا، استنى، لحظة، غيّر، no, wait, stop...) decide at once; anything else is checked before it counts.
- Only if the assistant then calls the same function with the same arguments in the very next turn and the answer was a yes is the
function_callsent to you, withitem.newera.confirmed: true. The approval is single use and expires after that turn. Different arguments are asked about again. A no is told to the assistant and nothing is sent.
The assistant may also read the details back in words without calling the function first. A yes then counts when the call that follows is the first confirmed call of the reply, repeats the details the assistant had just said, and the assistant's previous reply ended with that question (item.newera.confirmation is held_then_asked or asked_in_words). Arguments that differ only in spelling (spacing, case, Arabic letter variants, 10:00 and 10 AM, +9665... and 05..., Eastern digits) count as the same call.
Your function is therefore only called with a spoken yes behind it, and you never need to ask twice.
Interruptions
If the caller talks over the reply while a function is running, a function without confirmation is cancelled: you receive newera.function_call.cancelled with its call_id, stop the work if you can, and an output you send later is ignored. A function with confirmation is never cancelled, because it may already have happened. Newera waits for its real result and, at the caller's next turn, has the assistant say what actually happened ("the booking for Saturday already went through. Shall I cancel it and book Sunday?").
import asyncio, base64, json, os, websockets
TOOLS = [{"type": "function", "name": "get_free_slots",
"description": "Free appointment times on a day",
"parameters": {"type": "object", "properties": {"date": {"type": "string"}},
"required": ["date"]}}]
def get_free_slots(date): # your code
return {"date": date, "slots": ["09:00", "10:30"]}
async def main():
async with websockets.connect("wss://api.newera.ai/v1/realtime", additional_headers={
"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}) as ws:
send = lambda ev: ws.send(json.dumps(ev))
await send({"type": "session.update", "session": {"tools": TOOLS,
"instructions": "أنت موظف حجوزات في عيادة. استخدم الأدوات."}})
await send({"type": "conversation.item.create", "item": {"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "أبي موعد يوم السبت 2026-10-24"}]}})
await send({"type": "response.create"})
async for message in ws:
ev = json.loads(message)
if ev["type"] == "response.function_call_arguments.done":
args = json.loads(ev["arguments"])
try:
result = {"get_free_slots": get_free_slots}[ev["name"]](**args)
except Exception as exc:
result = {"error": str(exc)}
await send({"type": "conversation.item.create", "item": {
"type": "function_call_output", "call_id": ev["call_id"],
"output": json.dumps(result, ensure_ascii=False)}})
# no response.create needed: Newera continues by itself
elif ev["type"] == "response.audio_transcript.delta":
print(ev["delta"], end="")
elif ev["type"] == "response.done":
print("\n", ev["response"]["status"])
break
asyncio.run(main())from newera import VoiceAgent
agent = VoiceAgent(api_key=os.environ["NEWERA_API_KEY"], voice="male",
model="newera-s2s-mini", # or the default, "newera-s2s"
instructions="أنت موظف حجوزات في عيادة.")
@agent.tool
def get_free_slots(date: str) -> dict:
"""Free appointment times on a day.
Args:
date: The day, as YYYY-MM-DD.
"""
return {"date": date, "slots": ["09:00", "10:30"]}
@agent.tool(requires_confirmation=True, timeout_ms=20000)
async def book_appointment(patient_name: str, date: str, time: str) -> dict:
"""Book an appointment."""
return {"booking_id": "B-1"}
async with agent.connect() as call:
await call.send_text("أبي موعد يوم السبت")
async for event in call:
if event.type == "response.audio_transcript.delta":
print(event["delta"], end="")The newera Python package (pip install newera-sdk; source at github.com/Newera-ai-Repo/newera-sdk) builds the schema from the type hints and docstring, runs the calls for you, JSON-encodes the results and turns an exception into {"error": "..."}.
# On your server: trade the API key for a client secret
# (60 seconds, one connection) before a browser opens a call.
curl -X POST https://api.newera.ai/v1/realtime/sessions \
-H "Authorization: Bearer $NEWERA_API_KEY"
# {"object": "realtime.session", "model": "newera-s2s",
# "client_secret": {"value": "ek_...", "expires_at": 1760000000}}
# The browser then opens:
# new WebSocket("wss://api.newera.ai/v1/realtime",
# ["realtime", "newera-client-secret.ek_..."])
import asyncio, base64, json, os, wave
import websockets # pip install websockets
URL = "wss://api.newera.ai/v1/realtime?model=newera-s2s"
HEADERS = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
async def main():
with open("question.pcm", "rb") as fh: # 16 kHz, 16-bit, mono
question = fh.read()
async with websockets.connect(URL, additional_headers=HEADERS) as ws:
async def send(event):
await ws.send(json.dumps(event))
await send({"type": "session.update", "session": {
"voice": "male",
"instructions": "You are a friendly receptionist. Keep answers short.",
"turn_detection": None, # we end the turn ourselves
"newera": {"input_audio_sample_rate": 16000},
}})
for i in range(0, len(question), 3200): # 100 ms per chunk
chunk = base64.b64encode(question[i:i + 3200]).decode()
await send({"type": "input_audio_buffer.append", "audio": chunk})
await send({"type": "input_audio_buffer.commit"})
await send({"type": "response.create"})
reply = bytearray() # 24 kHz, 16-bit, mono
async for message in ws:
event = json.loads(message)
if event["type"] == "response.audio.delta":
reply += base64.b64decode(event["delta"])
elif event["type"] == "error":
raise RuntimeError(event["error"]["message"])
elif event["type"] == "response.done":
break
with wave.open("reply.wav", "wb") as out:
out.setnchannels(1)
out.setsampwidth(2)
out.setframerate(24000)
out.writeframes(bytes(reply))
asyncio.run(main())
import { readFileSync, writeFileSync } from "node:fs";
import WebSocket from "ws"; // npm install ws
const question = readFileSync("question.pcm"); // 16 kHz, 16-bit, mono
const ws = new WebSocket("wss://api.newera.ai/v1/realtime?model=newera-s2s", {
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}` },
});
const send = (event) => ws.send(JSON.stringify(event));
const reply = []; // 24 kHz, 16-bit, mono
ws.on("open", () => {
send({
type: "session.update",
session: {
voice: "male",
instructions: "You are a friendly receptionist. Keep answers short.",
turn_detection: null, // we end the turn ourselves
newera: { input_audio_sample_rate: 16000 },
},
});
for (let i = 0; i < question.length; i += 3200) { // 100 ms per chunk
const audio = question.subarray(i, i + 3200).toString("base64");
send({ type: "input_audio_buffer.append", audio });
}
send({ type: "input_audio_buffer.commit" });
send({ type: "response.create" });
});
ws.on("message", (data) => {
const event = JSON.parse(data.toString());
if (event.type === "response.audio.delta") {
reply.push(Buffer.from(event.delta, "base64"));
} else if (event.type === "error") {
throw new Error(event.error.message);
} else if (event.type === "response.done") {
writeFileSync("reply.pcm", Buffer.concat(reply)); // play: ffplay -f s16le -ar 24000 -ac 1 reply.pcm
ws.close();
}
});
import { readFileSync, writeFileSync } from "node:fs";
import WebSocket from "ws"; // npm install ws @types/ws
interface RealtimeEvent {
type: string;
delta?: string;
error?: { message: string };
}
const question: Buffer = readFileSync("question.pcm"); // 16 kHz, 16-bit, mono
const ws = new WebSocket("wss://api.newera.ai/v1/realtime?model=newera-s2s", {
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}` },
});
const send = (event: object): void => ws.send(JSON.stringify(event));
const reply: Buffer[] = []; // 24 kHz, 16-bit, mono
ws.on("open", () => {
send({
type: "session.update",
session: {
voice: "male",
instructions: "You are a friendly receptionist. Keep answers short.",
turn_detection: null, // we end the turn ourselves
newera: { input_audio_sample_rate: 16000 },
},
});
for (let i = 0; i < question.length; i += 3200) { // 100 ms per chunk
const audio = question.subarray(i, i + 3200).toString("base64");
send({ type: "input_audio_buffer.append", audio });
}
send({ type: "input_audio_buffer.commit" });
send({ type: "response.create" });
});
ws.on("message", (data: WebSocket.RawData) => {
const event: RealtimeEvent = JSON.parse(data.toString());
if (event.type === "response.audio.delta" && event.delta) {
reply.push(Buffer.from(event.delta, "base64"));
} else if (event.type === "error") {
throw new Error(event.error?.message);
} else if (event.type === "response.done") {
writeFileSync("reply.pcm", Buffer.concat(reply)); // play: ffplay -f s16le -ar 24000 -ac 1 reply.pcm
ws.close();
}
});
package main
import (
"encoding/base64"
"log"
"net/http"
"os"
"github.com/gorilla/websocket" // go get github.com/gorilla/websocket
)
func main() {
question, err := os.ReadFile("question.pcm") // 16 kHz, 16-bit, mono
if err != nil {
log.Fatal(err)
}
header := http.Header{"Authorization": {"Bearer " + os.Getenv("NEWERA_API_KEY")}}
ws, _, err := websocket.DefaultDialer.Dial("wss://api.newera.ai/v1/realtime?model=newera-s2s", header)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
send := func(event map[string]any) {
if err := ws.WriteJSON(event); err != nil {
log.Fatal(err)
}
}
send(map[string]any{"type": "session.update", "session": map[string]any{
"voice": "male",
"instructions": "You are a friendly receptionist. Keep answers short.",
"turn_detection": nil, // we end the turn ourselves
"newera": map[string]any{"input_audio_sample_rate": 16000},
}})
for i := 0; i < len(question); i += 3200 { // 100 ms per chunk
end := min(i+3200, len(question))
send(map[string]any{"type": "input_audio_buffer.append",
"audio": base64.StdEncoding.EncodeToString(question[i:end])})
}
send(map[string]any{"type": "input_audio_buffer.commit"})
send(map[string]any{"type": "response.create"})
var reply []byte // 24 kHz, 16-bit, mono
for {
var event struct {
Type string `json:"type"`
Delta string `json:"delta"`
Error struct {
Message string `json:"message"`
} `json:"error"`
}
if err := ws.ReadJSON(&event); err != nil {
log.Fatal(err)
}
switch event.Type {
case "response.audio.delta":
chunk, err := base64.StdEncoding.DecodeString(event.Delta)
if err != nil {
log.Fatal(err)
}
reply = append(reply, chunk...)
case "error":
log.Fatal(event.Error.Message)
case "response.done":
if err := os.WriteFile("reply.pcm", reply, 0o644); err != nil {
log.Fatal(err)
}
return
}
}
}
In a browser, open the socket with a client secret and play the audio as it arrives. This sample stops playback the moment the caller cuts in.
// On your server: trade the API key for a one-use secret that lasts 60 seconds.
// POST https://api.newera.ai/v1/realtime/sessions (Authorization: Bearer <API key>)
// -> {"client_secret": {"value": "ek_...", "expires_at": ...}}
// In the browser: never the API key, only the secret.
const { client_secret } = await (await fetch("/api/voice-secret")).json();
const ws = new WebSocket("wss://api.newera.ai/v1/realtime",
["realtime", "newera-client-secret." + client_secret.value]);
const speaker = new AudioContext({ sampleRate: 24000 });
let playAt = 0, queued = [];
ws.onopen = () => ws.send(JSON.stringify({
type: "session.update",
session: { voice: "male", instructions: "أنت موظف حجوزات في عيادة أسنان في الرياض.",
newera: { input_audio_sample_rate: 16000 } }, // the default is 24000
}));
// Call this with every 20-100 ms of microphone audio: 16 kHz mono Int16Array,
// captured with echoCancellation on and autoGainControl off.
function sendMicrophone(frame) {
const bytes = new Uint8Array(frame.buffer, frame.byteOffset, frame.byteLength);
let bin = "";
for (const b of bytes) bin += String.fromCharCode(b);
ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: btoa(bin) }));
}
ws.onmessage = ({ data }) => {
const ev = JSON.parse(data);
if (ev.type === "response.audio.delta") {
const bin = atob(ev.delta);
const pcm = new Int16Array(bin.length / 2);
for (let i = 0; i < pcm.length; i++)
pcm[i] = bin.charCodeAt(2 * i) | (bin.charCodeAt(2 * i + 1) << 8);
const buf = speaker.createBuffer(1, pcm.length, 24000);
buf.getChannelData(0).set(Float32Array.from(pcm, (v) => v / 32768));
const src = speaker.createBufferSource();
src.buffer = buf;
src.connect(speaker.destination);
playAt = Math.max(playAt, speaker.currentTime);
src.start(playAt);
playAt += buf.duration;
queued.push(src);
} else if (ev.type === "input_audio_buffer.speech_started") {
queued.forEach((src) => src.stop()); // the caller cut in: stop at once
queued = [];
playAt = 0;
} else if (ev.type === "conversation.item.input_audio_transcription.completed") {
console.log("caller:", ev.transcript);
} else if (ev.type === "response.done") {
console.log("assistant:", ev.response.output_transcript);
}
};Avatar
Avatar
WSS /v1/avatar
A talking face that answers by voice, in real time. It is realtime voice with a face: you stream the caller's microphone audio in, and the assistant's speech comes back as video with its sound, 512 × 512 pixels at 25 frames per second, lip-synced, together with the written words of the reply and of the caller. It speaks Arabic and English; the language is detected on every turn, as in realtime voice. Use the preset face, lama, or your own face from a photo.
The protocol is the realtime voice protocol: the same events, the same session settings, the same turn handling. A client that already speaks realtime voice changes the URL, adds the avatar fields to its first session.update, and plays response.video.chunk events instead of response.audio.delta. This section is complete on its own: you can integrate from it without reading the realtime voice section, which it links to for the session settings.
Best for
A visible assistant on a web page, kiosk or screen: reception, guided onboarding, support and learning, in Arabic or English.
Not for
Phone-only calls (use newera-s2s), camera or video input (the input is audio only), or animating a face without the person's agreement.
- URL
wss://api.newera.ai/v1/avatar- Voice model
newera-s2s(default) ornewera-s2s-mini, chosen in the URL- Input
- The caller's microphone audio only: PCM 16-bit mono, 24 kHz (or 8 to 48 kHz)
- Output
- JPEG video frames, 512 × 512 at 25 fps, with their PCM 16-bit mono 24 kHz audio, plus transcripts
- Languages
- Arabic and English
- Length of a call
- Indefinite
Quickest start: the Python SDK
The Newera Python SDK does the protocol for you: it opens the call, streams the audio, decodes the video and the sound that belong together, and maps every refusal to an exception. Install it with the avatar extra, set your key, and talk to the avatar from the terminal:
pip install "newera-sdk[avatar]"
export NEWERA_API_KEY=nwc_live_...
newera avatarnewera avatar uses your microphone, plays the avatar's voice on your speaker, shows the face in a window and prints what you and the avatar say. Use headphones, or the avatar hears its own voice and stops to listen. Press Ctrl+C (or Esc) to end the call. Add --voice male, --language ar, --image me.jpg (your own face: only of a person who agreed) or --record call.mp4; newera avatar --help lists them all.
The same in Python: ask a question from a recording and save the answer as a video.
import asyncio
from newera import Newera
async def main():
client = Newera() # reads NEWERA_API_KEY
async with client.avatar.connect(avatar="lama", voice="female", record=True) as call:
await call.send_file("question.wav") # sent in real time, as a microphone would
reply = await call.wait_reply()
print(reply.user_text, "->", reply.text)
call.save_mp4("reply.mp4") # the frames and the sound as a video (needs ffmpeg)
asyncio.run(main())The events are typed (UserTranscript, ReplyTextDelta, VideoChunk with its frames and audio, Interrupted, Unavailable, ErrorEvent): iterate them with async for event in call.events(). A refused call raises NotEnabled, AuthenticationError or another NeweraError with the same codes as the table in Errors and close codes.
For a web page, your server trades its key for a one-use client secret with create_session(); the page opens the WebSocket with that secret (the browser example below shows the page). This is the whole endpoint:
# pip install newera-sdk fastapi uvicorn
# NEWERA_API_KEY=... uvicorn server:app
from fastapi import FastAPI, HTTPException
from newera import AvatarError, Newera
app = FastAPI()
client = Newera()
@app.post("/avatar-token")
async def avatar_token():
# Check here that the person is signed in and allowed to start a call.
try:
secret = await client.avatar.create_session(model="newera-s2s")
except AvatarError as exc:
raise HTTPException(exc.http_status or 502, {"code": exc.code, "message": exc.message})
return secret.as_dict() # {"client_secret": {"value": "avs_...", "expires_at": ...}, "model": ...}Access
Access is on request. Your organisation must be enabled for the avatar, and for the voice model you call with. Ask your Newera account manager. A key that is not enabled is refused with avatar_not_enabled (close code 4403).
Quickstart with a raw WebSocket
You need an API key (API keys in the portal), an organisation enabled for the avatar, and a short recording of a question: 16-bit mono WAV at 8, 16, 24 or 48 kHz. To make one from any audio file: ffmpeg -i question.m4a -ac 1 -ar 16000 -sample_fmt s16 question.wav.
Python sends the recording as if it came from a microphone, prints the reply as it is written, and saves the avatar's answer as reply.mp4 (it needs ffmpeg for the last step). JavaScript is a complete browser page: it listens through the microphone and shows the avatar on a canvas, and a small server of yours hands the page a short-lived client secret, so the API key never reaches the browser.
import asyncio, base64, json, os, shutil, subprocess, time, wave
import websockets # pip install websockets
URL = "wss://api.newera.ai/v1/avatar?model=newera-s2s"
HEADERS = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
FPS = 25
BYTES_PER_FRAME = 960 * 2 # 24 kHz / 25 fps = 960 samples of 16 bits
async def main():
with wave.open("question.wav") as w: # 16-bit mono PCM: 8, 16, 24 or 48 kHz
rate, pcm = w.getframerate(), w.readframes(w.getnframes())
frames, audio = {}, {} # frame number -> JPEG bytes / PCM bytes of that frame
# max_size=None: one video event is a few hundred KB, above the 1 MiB default
async with websockets.connect(URL, additional_headers=HEADERS, max_size=None) as ws:
async def send(event):
await ws.send(json.dumps(event))
# The first message chooses the face: send it before anything else.
await send({"type": "session.update", "session": {
"avatar": "lama",
"voice": "female",
"instructions": "You are a friendly assistant. Answer in one short sentence.",
"newera": {"input_audio_sample_rate": rate},
}})
async def microphone():
step = rate // 50 * 2 # 20 ms of audio, in bytes
for i in range(0, len(pcm), step):
await send({"type": "input_audio_buffer.append",
"audio": base64.b64encode(pcm[i:i + step]).decode()})
await asyncio.sleep(0.02) # in real time
silence = base64.b64encode(bytes(step)).decode()
while True: # keep sending silence: the call hears the question end in it
await send({"type": "input_audio_buffer.append", "audio": silence})
await asyncio.sleep(0.02)
mic = asyncio.create_task(microphone())
replied, last_video = False, time.monotonic()
while not (replied and time.monotonic() - last_video > 3):
try:
event = json.loads(await asyncio.wait_for(ws.recv(), 1))
except asyncio.TimeoutError:
continue
kind = event["type"]
if kind == "response.video.chunk" and event["speech"]:
last_video = time.monotonic()
sound = base64.b64decode(event["audio"])
for i, jpeg in enumerate(event["frames"]):
n = event["frame_index"] + i
frames[n] = base64.b64decode(jpeg)
audio[n] = sound[i * BYTES_PER_FRAME:(i + 1) * BYTES_PER_FRAME]
elif kind == "response.audio_transcript.delta":
print(event["delta"], end="", flush=True)
elif kind == "response.done":
reason = (event["response"].get("status_details") or {}).get("reason")
if reason != "turn_detected": # else: a draft dropped because the caller kept talking
replied, last_video = True, time.monotonic()
elif kind == "error":
raise RuntimeError(event["error"]["code"] + ": " + event["error"]["message"])
mic.cancel()
if not frames:
raise SystemExit("no video received")
# Lay the frames out by frame_index; a missing frame repeats the previous one, in silence.
os.makedirs("frames", exist_ok=True)
sound, last = bytearray(), None
for n, number in enumerate(range(min(frames), max(frames) + 1)):
last = frames.get(number, last)
with open("frames/%06d.jpg" % n, "wb") as fh:
fh.write(last)
sound += audio.get(number, bytes(BYTES_PER_FRAME))
with wave.open("reply.wav", "wb") as out:
out.setnchannels(1)
out.setsampwidth(2)
out.setframerate(24000)
out.writeframes(bytes(sound))
if shutil.which("ffmpeg"):
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-framerate", str(FPS), "-i", "frames/%06d.jpg",
"-i", "reply.wav", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
"-shortest", "reply.mp4"], check=True)
print("\nsaved reply.mp4")
else:
print("\nsaved frames/ and reply.wav; join them with ffmpeg (see Saving the reply as a video)")
asyncio.run(main())
<!doctype html>
<meta charset="utf-8">
<title>Avatar</title>
<button id="start">Start the call</button> <button id="stop" disabled>End</button>
<canvas id="face" width="512" height="512"></canvas>
<p id="text"></p>
<script>
// Plays response.video.chunk events: frames and audio on ONE clock, the audio clock.
class AvatarPlayer {
constructor(canvas, { prebuffer = 2.0 } = {}) {
this.canvas = canvas;
this.g = canvas.getContext("2d");
this.audio = new AudioContext(); // create it inside a click handler, or it starts suspended
this.prebuffer = prebuffer; // seconds of video to hold back before a reply starts
this.queue = []; // frames waiting for their time: { at, bitmap }
this.sources = []; // scheduled audio, so it can be cancelled
this.nextFrame = -1; // the frame_index that continues the current run
this.playhead = 0; // audio-clock time at which the next frame belongs
requestAnimationFrame(() => this.draw());
}
// Call this for every response.video.chunk, in the order they arrive.
add(ev) {
const now = this.audio.currentTime;
const contiguous = ev.frame_index === this.nextFrame;
if (!contiguous || this.playhead < now) {
// The start of a reply (or of idle video), or we ran dry: restart the clock with a cushion.
const lead = contiguous ? 0.4 : ev.speech ? this.prebuffer : 0.2;
this.playhead = Math.max(this.playhead, now + lead);
}
const at = this.playhead; // when this event's first frame and audio start
this.playhead += ev.frames.length / ev.fps;
this.nextFrame = ev.frame_index + ev.frames.length;
if (ev.audio) { // PCM16 mono 24 kHz, exactly these frames' sound
const pcm = new Int16Array(bytes(ev.audio).buffer);
const buf = this.audio.createBuffer(1, pcm.length, ev.audio_sample_rate);
const ch = buf.getChannelData(0);
for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
const src = this.audio.createBufferSource();
src.buffer = buf;
src.connect(this.audio.destination);
src.start(at);
src.onended = () => { this.sources = this.sources.filter((s) => s !== src); };
this.sources.push(src);
}
ev.frames.forEach((jpeg, i) => {
const item = { at: at + i / ev.fps, bitmap: null };
this.queue.push(item);
createImageBitmap(new Blob([bytes(jpeg)], { type: "image/jpeg" })).then((b) => { item.bitmap = b; });
});
}
// The caller started talking (avatar.interrupted or input_audio_buffer.speech_started):
// stop the sound, drop every frame not shown yet. The last frame stays on the canvas.
interrupt() {
this.sources.forEach((s) => { try { s.stop(); } catch (e) {} });
this.sources = [];
this.queue = [];
this.nextFrame = -1;
this.playhead = 0;
}
// Show the newest frame whose time has come, by the audio clock.
draw() {
const now = this.audio.currentTime;
let last = -1;
while (last + 1 < this.queue.length && this.queue[last + 1].at <= now) last++;
if (last >= 0) {
const shown = this.queue.splice(0, last + 1).pop();
if (shown.bitmap) this.g.drawImage(shown.bitmap, 0, 0, this.canvas.width, this.canvas.height);
}
requestAnimationFrame(() => this.draw());
}
}
const bytes = (b64) => Uint8Array.from(atob(b64), (c) => c.charCodeAt(0));
// Microphone capture. The worklet runs on the audio thread: it brings whatever rate the device runs at
// (usually 44.1 or 48 kHz) down to 24 kHz and posts 20 ms blocks of 16-bit samples.
const WORKLET = `
class Capture extends AudioWorkletProcessor {
constructor() { super(); this.ratio = sampleRate / 24000; this.pos = 0; this.prev = 0; this.out = []; }
process(inputs) {
const ch = inputs[0] && inputs[0][0];
if (!ch) return true;
let p = this.pos;
while (p < ch.length) { // linear interpolation down to 24 kHz
const i = Math.floor(p), f = p - i;
const a = i === 0 ? this.prev : ch[i - 1], b = ch[i];
const s = Math.max(-1, Math.min(1, a + (b - a) * f));
this.out.push(Math.round(s * 32767));
p += this.ratio;
}
this.pos = p - ch.length;
this.prev = ch[ch.length - 1];
while (this.out.length >= 480) { // 480 samples = 20 ms
this.port.postMessage(Int16Array.from(this.out.splice(0, 480)));
}
return true;
}
}
registerProcessor("capture", Capture);`;
async function startMicrophone(onBlock) { // onBlock(base64 string): one 20 ms chunk
const stream = await navigator.mediaDevices.getUserMedia({ audio: {
channelCount: 1,
echoCancellation: true, // or the avatar hears itself and stops talking
noiseSuppression: true,
autoGainControl: false, // Chrome turns this on by default; leave it off
} });
const ctx = new AudioContext();
const url = URL.createObjectURL(new Blob([WORKLET], { type: "text/javascript" }));
await ctx.audioWorklet.addModule(url);
const node = new AudioWorkletNode(ctx, "capture");
node.port.onmessage = (e) => {
let s = "";
new Uint8Array(e.data.buffer).forEach((b) => { s += String.fromCharCode(b); });
onBlock(btoa(s));
};
ctx.createMediaStreamSource(stream).connect(node); // not connected to the speakers
return () => { stream.getTracks().forEach((t) => t.stop()); ctx.close(); }; // call to stop
}
const $ = (id) => document.getElementById(id);
let ws, player, stopMic;
$("start").onclick = async () => {
player = new AvatarPlayer($("face")); // inside the click: audio may start
await player.audio.resume();
// Your server trades its API key for a one-use client secret (see Browsers).
const { client_secret } = await (await fetch("/api/avatar-secret", { method: "POST" })).json();
ws = new WebSocket("wss://api.newera.ai/v1/avatar", ["realtime", "newera-client-secret." + client_secret.value]);
ws.onopen = () => ws.send(JSON.stringify({ type: "session.update", session: { // first message
avatar: "lama",
voice: "female",
instructions: "You are a friendly assistant. Answer in one short sentence.",
newera: { input_audio_sample_rate: 24000 },
} }));
ws.onmessage = async (m) => {
const ev = JSON.parse(m.data);
switch (ev.type) {
case "session.created": // the face is up: start sending the microphone
stopMic = await startMicrophone((audio) =>
ws.readyState === 1 && ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio })));
break;
case "response.video.chunk": player.add(ev); break;
case "avatar.interrupted":
case "input_audio_buffer.speech_started": player.interrupt(); break;
case "avatar.unavailable": console.warn(ev.message); break; // sound continues as response.audio.delta
case "response.audio_transcript.delta": $("text").textContent += ev.delta; break;
case "input_audio_buffer.speech_stopped": $("text").textContent = ""; break;
case "error": console.error(ev.error.code, ev.error.message); break;
}
};
ws.onclose = (e) => { console.log("closed", e.code, e.reason); stopMic && stopMic(); };
$("start").disabled = true; $("stop").disabled = false;
};
$("stop").onclick = () => ws.send(JSON.stringify({ type: "session.end" }));
</script>
The server for the browser page. Put your own sign-in check where the comment is.
import { createServer } from "node:http";
import { readFile } from "node:fs/promises";
createServer(async (req, res) => {
if (req.method === "POST" && req.url === "/api/avatar-secret") {
// Check here that the person is signed in and allowed to start a call.
const r = await fetch("https://api.newera.ai/v1/avatar/sessions", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}`, "Content-Type": "application/json" },
body: JSON.stringify({ model: "newera-s2s" }),
});
res.writeHead(r.status, { "Content-Type": "application/json" });
res.end(await r.text());
} else {
res.writeHead(200, { "Content-Type": "text/html" });
res.end(await readFile("index.html"));
}
}).listen(3000);Run it with NEWERA_API_KEY=… node server.mjs, save the page as index.html next to it, and open http://localhost:3000. Use headphones while you try it: without echo cancellation the avatar hears itself. The pieces of that page, and why they are written as they are, are explained in Playing the video and Capturing the microphone.
Connect
From a server, send your API key on the WebSocket handshake and choose the voice model in the URL:
GET wss://api.newera.ai/v1/avatar?model=newera-s2s
Authorization: Bearer nwc_live_...model is the voice model of the call: newera-s2s (default) or newera-s2s-mini (see Choosing the model). The key must be entitled to the avatar and to that model. A refused connection still opens, sends one error event saying why, and closes with a code from Errors and close codes.
Send session.update first, as soon as the socket opens: the face is chosen in it. The server waits up to two seconds for that message; a client that sends nothing (or sends something else first) gets the default face, and the face cannot be changed afterwards. Do not wait for session.created before sending it: session.created is sent only after the face has started.
Browsers: client secrets
Never put an API key in a web page. Your server trades its key for a client secret that works for one connection and expires after 60 seconds, and the page offers the secret as a WebSocket subprotocol, newera-client-secret.<secret>, beside realtime. Request a new secret for every call. Call POST /v1/avatar/sessions from your server: browsers are not allowed to call it from your own site.
curl -X POST https://api.newera.ai/v1/avatar/sessions \
-H "Authorization: Bearer $NEWERA_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "newera-s2s"}' # the body is optional
# {"object": "avatar.session", "model": "newera-s2s",
# "client_secret": {"value": "avs_...", "expires_at": 1763300000}}const ws = new WebSocket("wss://api.newera.ai/v1/avatar",
["realtime", "newera-client-secret." + secret]);
ws.onopen = () => ws.send(JSON.stringify({type: "session.update", session: {avatar: "lama"}}));The model was fixed when the secret was made, so a connection with a secret takes no ?model=. The request for a secret is checked as a call is: a key that is not enabled gets 403 avatar_not_enabled there, not a closed socket later. Its errors are JSON, {"error": {"type", "code", "message", "param"}}, with the HTTP status listed under Errors and close codes.
Session settings
The avatar adds four optional fields, at the top level of session in a session.update (not inside newera). Every realtime voice session setting also applies: instructions, voice, turn_detection, newera.input_audio_sample_rate, newera.detect_language, tools and the rest. They are in the realtime voice session settings table, and work the same here. Leave input_audio_format and output_audio_format at their default, pcm16.
| Name | Type | Default | What it does |
|---|---|---|---|
avatar | string | lama | A preset face. Currently available: lama. An unknown name is refused with invalid_avatar. |
avatar_image | string | Your own face instead of a preset: a base64 JPEG, PNG or WebP photo (a data: URL is accepted too). At most 8 MB once decoded; 1024 to 2048 pixels on the long side is plenty. It replaces avatar. See the photo notes below. | |
avatar_face_crop | boolean | true | Find the face in avatar_image and crop around it. Set false for an image that is already a centred head shot. |
avatar_idle_video | boolean | false | Also stream the face while it is not speaking (it blinks and moves a little), so it looks alive. With false, video arrives only while the avatar speaks, about a tenth of the data. It can be changed during the call. |
The voice is independent of the face. voice is the realtime voice setting, male or female, whichever face you choose. The default voice is male, so send voice when your face is a woman's.
The face is fixed for the call. The avatar fields are read from the first session.update. Sending avatar, avatar_image or avatar_face_crop later gets an error event with code avatar_locked (the rest of that update is still applied). Only avatar_idle_video can change during the call.
Your own photo
Send a clear photo of one person, facing the camera, with the face fully visible (no hand, mask or sunglasses across it). A photo with no usable face is refused with invalid_avatar_image (close 4400), and so is one that is larger than 8 MB or not a JPEG, PNG or WebP. Send only a photo of a person who has agreed to be animated, and tell them what it is for: you are responsible for that agreement. The photo is used for the call only. Animating anyone without their consent is prohibited and may lead to loss of access.
{"type": "session.update", "session": {
"avatar": "lama",
"avatar_idle_video": false,
"voice": "female",
"instructions": "You are a friendly assistant. Answer in one short sentence.",
"newera": {"input_audio_sample_rate": 24000}
}}A call, event by event
→ (connect)
→ session.update first: the face, the voice, your instructions
← avatar.ready the face has started: fps, size, frames per event
← session.created the call is open: start sending audio
← conversation.created
← session.updated
→ input_audio_buffer.append {audio} continuously, silence included
← input_audio_buffer.speech_started the caller began talking
← avatar.interrupted discard what you had queued
← input_audio_buffer.speech_stopped the caller paused
← input_audio_buffer.committed, conversation.item.created
← conversation.item.input_audio_transcription.completed {transcript}
← response.created
← response.audio_transcript.delta {delta}
← response.audio.done, response.audio_transcript.done
← response.done the VOICE is complete; the video is still coming
← response.video.chunk {frame_index, frames, audio}
...one event per 7 frames (0.28 s), for as long as the reply is...
→ session.end when you are finished
← session.summary the end of the call; then the socket closesresponse.done can arrive before the end of the video. A reply is finished when its video has played out, not when response.done arrives.
Events you send
Every client event of realtime voice is accepted and passed on unchanged. Send JSON text only: a binary message gets an error event, binary_not_supported.
| Event | Fields | What it does |
|---|---|---|
session.update | session | The call's settings: the avatar fields above and every realtime voice setting. Send it first, within two seconds. |
input_audio_buffer.append | audio | The caller's audio: base64 PCM 16-bit mono, at 24 kHz unless you set newera.input_audio_sample_rate to 8000, 16000 or 48000. Send it in real time in chunks of about 20 ms, and keep sending, silence included, so the end of a turn is heard. |
input_audio_buffer.commit / .clear | Manual turns (turn_detection: null), as in realtime voice. | |
conversation.item.create | item | Add a message to the conversation, for example typed text. See Items. |
response.create | response? | Ask for a reply, for example after typed text. See Replies. |
response.cancel | Stop the current reply. | |
conversation.item.truncate | item_id, content_index, audio_end_ms | Tell Newera how much of a reply the caller actually saw and heard. |
session.end | Finish the call: you receive session.summary, then the socket closes. |
Events you receive
Four events are the avatar's own. Every other event is the realtime voice event, unchanged, with one difference: response.audio.delta is not sent while the face is up. The reply's sound is inside the video events, paired with its frames. Ignore event types you do not know.
| Event | When | Fields |
|---|---|---|
avatar.ready | Once, when the face has started, before session.created. | fps (25), width and height (512), video_format ("jpeg"), audio_sample_rate (24000), frames_per_event (7) |
response.video.chunk | While the avatar speaks, and with avatar_idle_video: true while it listens. | See below. |
avatar.interrupted | The caller started speaking. | None. Stop the sound and discard everything not yet played. |
avatar.unavailable | The face stopped in the middle of a call. | message. The call goes on audio only: the reply's sound resumes as response.audio.delta events, so the voice is never lost. Play those as you would in realtime voice, and show a still image. |
Every event has type and event_id.
{"type": "avatar.ready", "event_id": "evt_fe8f1133c07c4dd4adf6",
"fps": 25, "width": 512, "height": 512, "video_format": "jpeg",
"audio_sample_rate": 24000, "frames_per_event": 7}
{"type": "avatar.interrupted", "event_id": "evt_c301efb4fdea48168d68"}
{"type": "avatar.unavailable", "event_id": "evt_...",
"message": "The avatar's face stopped. The reply continues as audio (response.audio.delta) without video."}response.video.chunk
A piece of video and of its sound, in one self-contained JSON event.
{
"type": "response.video.chunk", "event_id": "evt_d35d7b4b081b4e62be7e",
"index": 1,
"frame_index": 28,
"start_ms": 1120,
"frames": ["<base64 JPEG>", "<base64 JPEG>", "... up to 7"],
"fps": 25, "width": 512, "height": 512,
"speech": true,
"audio": "<base64 PCM16 mono 24 kHz: 960 samples per frame>",
"audio_sample_rate": 24000,
"generation_ms": …,
"queued_ms": 6530
}| Field | What it is |
|---|---|
frame_index | The number of the first frame of this event, counted from the start of the call's video. It is the timeline: a frame is shown at frame_index / fps seconds. Consecutive events of one reply are contiguous. A reply does not start at 0, and with avatar_idle_video: false the numbers jump between replies. |
start_ms | The same time in milliseconds (frame_index × 1000 / fps). |
frames | 1 to 7 (frames_per_event) JPEG frames, base64, in display order. |
speech | true when the event carries reply speech; false for the listening face (only with avatar_idle_video: true). |
audio | The reply's sound for exactly these frames: PCM 16-bit mono, little-endian, base64, frames.length × 960 samples (24 kHz ÷ 25 fps). null when speech is false. Play it together with its frames, on one clock, and the lips and the voice stay together. |
audio_sample_rate, fps, width, height | 24000, 25, 512 and 512, as in avatar.ready. |
generation_ms | How long the face took to draw this part. Informational. |
queued_ms | Reply sound already accepted and still to come after this event. On the first event of a reply it tells you how long the reply is. |
index | The face's own chunk number (28 frames each, so four events share one). Informational. |
The frames are inside the event, as base64, rather than sent as separate binary messages. Every realtime client already reads JSON text events, and one event needs no ordering rules between a header and its frames. An event of 7 frames is about 230 KB, so it stays under the 1 MiB message limit that common WebSocket clients have.
Playing the video
Frames and sound of one event belong together, and you must keep them together. The rule is one clock, the audio clock:
- Schedule by the timeline, not by arrival. Events arrive in bursts, not at the pace of playback. Give the sound of each event a start time on the audio clock, and show each frame when the audio clock reaches that frame's time: the event's start plus the frame's position (
i / fps). Do not draw frames fromsetIntervalor as they arrive. - Hold some video back before you start. Video arrives in pieces of 1.12 seconds. Buffer before you start: about 2 seconds for a reply up to 15 seconds, 3.5 seconds for one up to 30 seconds.
queued_mson the first event tells you how long the reply will be. - If the buffer runs empty, hold the last frame and pause the sound, then start again with a small cushion, instead of letting the picture and the voice drift apart.
- On
avatar.interrupted(and oninput_audio_buffer.speech_started, which comes with it) stop the sound that is playing and drop every frame not shown yet. Keep showing the last frame. The next reply starts a new run on the clock. - With
avatar_idle_video: falsenothing arrives between replies, and before the first reply nothing has arrived at all: show your own still image (for your own photo, that photo) and keep the last frame on screen after a reply.
This is a complete browser player: a canvas for the frames, Web Audio for the sound.
// Plays response.video.chunk events: frames and audio on ONE clock, the audio clock.
class AvatarPlayer {
constructor(canvas, { prebuffer = 2.0 } = {}) {
this.canvas = canvas;
this.g = canvas.getContext("2d");
this.audio = new AudioContext(); // create it inside a click handler, or it starts suspended
this.prebuffer = prebuffer; // seconds of video to hold back before a reply starts
this.queue = []; // frames waiting for their time: { at, bitmap }
this.sources = []; // scheduled audio, so it can be cancelled
this.nextFrame = -1; // the frame_index that continues the current run
this.playhead = 0; // audio-clock time at which the next frame belongs
requestAnimationFrame(() => this.draw());
}
// Call this for every response.video.chunk, in the order they arrive.
add(ev) {
const now = this.audio.currentTime;
const contiguous = ev.frame_index === this.nextFrame;
if (!contiguous || this.playhead < now) {
// The start of a reply (or of idle video), or we ran dry: restart the clock with a cushion.
const lead = contiguous ? 0.4 : ev.speech ? this.prebuffer : 0.2;
this.playhead = Math.max(this.playhead, now + lead);
}
const at = this.playhead; // when this event's first frame and audio start
this.playhead += ev.frames.length / ev.fps;
this.nextFrame = ev.frame_index + ev.frames.length;
if (ev.audio) { // PCM16 mono 24 kHz, exactly these frames' sound
const pcm = new Int16Array(bytes(ev.audio).buffer);
const buf = this.audio.createBuffer(1, pcm.length, ev.audio_sample_rate);
const ch = buf.getChannelData(0);
for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
const src = this.audio.createBufferSource();
src.buffer = buf;
src.connect(this.audio.destination);
src.start(at);
src.onended = () => { this.sources = this.sources.filter((s) => s !== src); };
this.sources.push(src);
}
ev.frames.forEach((jpeg, i) => {
const item = { at: at + i / ev.fps, bitmap: null };
this.queue.push(item);
createImageBitmap(new Blob([bytes(jpeg)], { type: "image/jpeg" })).then((b) => { item.bitmap = b; });
});
}
// The caller started talking (avatar.interrupted or input_audio_buffer.speech_started):
// stop the sound, drop every frame not shown yet. The last frame stays on the canvas.
interrupt() {
this.sources.forEach((s) => { try { s.stop(); } catch (e) {} });
this.sources = [];
this.queue = [];
this.nextFrame = -1;
this.playhead = 0;
}
// Show the newest frame whose time has come, by the audio clock.
draw() {
const now = this.audio.currentTime;
let last = -1;
while (last + 1 < this.queue.length && this.queue[last + 1].at <= now) last++;
if (last >= 0) {
const shown = this.queue.splice(0, last + 1).pop();
if (shown.bitmap) this.g.drawImage(shown.bitmap, 0, 0, this.canvas.width, this.canvas.height);
}
requestAnimationFrame(() => this.draw());
}
}
const bytes = (b64) => Uint8Array.from(atob(b64), (c) => c.charCodeAt(0));Use it like this: const player = new AvatarPlayer(canvas) inside a click handler (a browser does not start sound before the user acts), then player.add(ev) for every response.video.chunk, and player.interrupt() on avatar.interrupted and input_audio_buffer.speech_started. Pass {prebuffer: 3.5} if your replies are long.
Outside a browser, the same rule holds: place each frame and each frame's 1,920 bytes of sound by frame_index and mix them against one audio clock.
Capturing the microphone
The input is the caller's voice only: there is no camera input. Send 16-bit little-endian mono PCM, base64, in chunks of about 20 ms, continuously and in real time, silence included: the call decides that a turn is over by hearing the silence. 24 kHz is the default; for 8, 16 or 48 kHz set newera.input_audio_sample_rate. Turn on echo cancellation, or the avatar hears itself and stops talking, and turn off autoGainControl (Chrome turns it on by default). A page that uses the microphone must be served over HTTPS (or be localhost).
Browsers run the microphone at 44.1 or 48 kHz. This AudioWorklet brings it down to 24 kHz and hands you a base64 string for every 20 ms. The worklet is written as a string and loaded from a blob: URL, so it needs no extra file; if your page's content security policy blocks that, put the class in a file and load it with audioWorklet.addModule("capture.js").
// Microphone capture. The worklet runs on the audio thread: it brings whatever rate the device runs at
// (usually 44.1 or 48 kHz) down to 24 kHz and posts 20 ms blocks of 16-bit samples.
const WORKLET = `
class Capture extends AudioWorkletProcessor {
constructor() { super(); this.ratio = sampleRate / 24000; this.pos = 0; this.prev = 0; this.out = []; }
process(inputs) {
const ch = inputs[0] && inputs[0][0];
if (!ch) return true;
let p = this.pos;
while (p < ch.length) { // linear interpolation down to 24 kHz
const i = Math.floor(p), f = p - i;
const a = i === 0 ? this.prev : ch[i - 1], b = ch[i];
const s = Math.max(-1, Math.min(1, a + (b - a) * f));
this.out.push(Math.round(s * 32767));
p += this.ratio;
}
this.pos = p - ch.length;
this.prev = ch[ch.length - 1];
while (this.out.length >= 480) { // 480 samples = 20 ms
this.port.postMessage(Int16Array.from(this.out.splice(0, 480)));
}
return true;
}
}
registerProcessor("capture", Capture);`;
async function startMicrophone(onBlock) { // onBlock(base64 string): one 20 ms chunk
const stream = await navigator.mediaDevices.getUserMedia({ audio: {
channelCount: 1,
echoCancellation: true, // or the avatar hears itself and stops talking
noiseSuppression: true,
autoGainControl: false, // Chrome turns this on by default; leave it off
} });
const ctx = new AudioContext();
const url = URL.createObjectURL(new Blob([WORKLET], { type: "text/javascript" }));
await ctx.audioWorklet.addModule(url);
const node = new AudioWorkletNode(ctx, "capture");
node.port.onmessage = (e) => {
let s = "";
new Uint8Array(e.data.buffer).forEach((b) => { s += String.fromCharCode(b); });
onBlock(btoa(s));
};
ctx.createMediaStreamSource(stream).connect(node); // not connected to the speakers
return () => { stream.getTracks().forEach((t) => t.stop()); ctx.close(); }; // call to stop
}Send each block as {"type": "input_audio_buffer.append", "audio": block}. The Python quickstart does the same from a file, one 20 ms chunk every 20 ms.
Saving the reply as a video file
The quickstart's Python script lays the reply out by frame_index: it writes each JPEG as a numbered file, the sound as one 24 kHz WAV, and joins them with ffmpeg. A frame that is missing repeats the previous one, in silence, so the picture and the sound keep the same length. You can run the join yourself:
ffmpeg -framerate 25 -i frames/%06d.jpg -i reply.wav \
-c:v libx264 -pix_fmt yuv420p -c:a aac -shortest reply.mp4Errors and close codes
Errors are realtime voice's: {"type": "error", "error": {"type", "code", "message", "param", "event_id"}}. A connection that is refused still opens, sends one error event, and closes with one of these codes; the close reason is the error.code. An error from the voice session itself, for example insufficient_quota, is passed on with its close code.
| Close code | error.code | Meaning |
|---|---|---|
4400 | invalid_avatar, invalid_avatar_image, invalid_model, invalid_value | A bad session field or model: an unknown preset, a photo that is too large, not a JPEG, PNG or WebP, or with no usable face, a model that is not newera-s2s or newera-s2s-mini, or a field that is not true or false. param names it. |
4401 | invalid_api_key, invalid_client_secret | No key, a wrong key, or a client secret that is wrong, expired (60 seconds) or already used. Request a new secret. |
4402 | insufficient_quota | Your balance cannot cover the call. Top up and connect again. |
4403 | avatar_not_enabled, model_not_entitled | Your organisation is not enabled for the avatar, or for the voice model. Contact your account manager. |
4429 | rate_limit_exceeded, concurrent_call_limit | Too many requests, or too many calls on this key right now. Wait and retry. |
4503 | avatar_unavailable | The face, or Newera Core, is temporarily unavailable. Retry shortly. |
Some errors do not close the call: avatar_locked (an avatar field sent after the first session.update), binary_not_supported (a binary message), and the realtime voice errors for a bad setting later in the call (unknown_parameter, invalid_value). If the call itself fails, you get call_dropped and the socket closes with 1011: connect again. A normal end is close code 1000.
The request POST /v1/avatar/sessions answers with HTTP status and a JSON error: 400 (invalid_model), 401 (invalid_api_key), 403 (avatar_not_enabled, model_not_entitled), 429 (rate_limit_exceeded), 503 (avatar_unavailable). Every response carries an x-request-id.
Limits
| Limit | Value |
|---|---|
| Your own photo | 8 MB once decoded; JPEG, PNG or WebP; 1024 to 2048 pixels is plenty. |
| Video | 512 × 512 JPEG frames at 25 frames per second. |
| Message size | At most 7 frames per response.video.chunk, about 230 KB as sent. A Python websockets client should pass max_size=None. |
| Bandwidth to you | About 0.9 MB per second (about 7 Mbit/s) while the avatar speaks, mostly the frames; the same all the time with avatar_idle_video: true, and about a tenth of the data with false. To Newera: the caller's audio, about 64 KB per second. |