Newera Core Docs

Newera Core · API

API Documentation

A single sovereign API for every model your organisation runs, deployed inside the Kingdom with no data egress.

Newera Core is OpenAI-compatible wherever the response shape allows, so you can integrate with the OpenAI SDK using a Newera key. Prompts and completions are never stored. Every request is metered, and every response carries an x-request-id for billing and support.

Base URL
https://api.newera.ai/v1
Auth
Bearer <API key>
Format
JSON · multipart

Get started

Quickstart

Create a key in the portal (API keys in the Newera portal). A key is shown once, at creation. Store it, and issue another if you lose it.

Set your key
Shell
export NEWERA_API_KEY="nwc_live_..."

Your first request:

Your first request
curl https://api.newera.ai/v1/chat/completions \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-chat-large",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.newera.ai/v1",
    api_key=os.environ["NEWERA_API_KEY"],
)

resp = client.chat.completions.create(
    model="newera-chat-large",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(resp.choices[0].message.content)
print(resp.usage)   # what you are billed on

Point base_url at Newera and nothing else about your OpenAI client changes.

Get started

Authentication

Send your key as a bearer token. x-api-key is accepted too, for clients that cannot set an Authorization header.

Authorization header
HTTP
Authorization: Bearer nwc_live_...

Keys look like nwc_live_<prefix>_<secret>. Only a hash is stored, so the full secret exists only in your copy.

  • One tenant per key. A key sees only the models on your organisation's contract, and may be limited further to some kinds of model (chat, embeddings, voice calls and so on), chosen when you create it.
  • Shown once. Treat a key like a password; rotate it from the portal.
  • Revocation is immediate. A revoked or expired key stops working everywhere within about a second.

A request with no key, or a malformed one, returns 401. A well-formed key that is not entitled to the model you asked for returns 403, not 404: the model exists, and access is a contract question.

Get started

Keeping keys safe

Anyone holding a key can spend your credit, so treat it like a password. Most leaked keys leak from the client side: committed to a repository, built into an app, or pasted into a chat.

  • Keep keys on your server. Read them from an environment variable or a secret manager. Never write a key into source code or a config file you commit.
  • Never put a key in a browser page or a mobile app. Anything shipped to a device can be read from it. Call Newera Core from your server; for voice calls from a browser, trade the key for a 60-second client secret on your server (see newera-s2s).
  • One key per application and environment. Label each one in the portal, so a leak can be revoked without stopping everything else, and usage per key shows which application spent what.
  • Give a key only what it needs. Limit it to the kinds of model it calls, set an expiry date, and cap what it may spend in a month.
  • Rotate keys. Replace keys every 90 days or so, and whenever someone who had one leaves. Issue the new key, deploy it, then revoke the old one; we email a reminder when a key is 90 days old.
  • Revoke at once if a key may have leaked. Revoke it from API keys in the portal; it stops working everywhere within about a second.

If a key shows use you do not recognise, revoke it and contact us from the Support button in the portal.

Get started

Making requests

Base URLhttps://api.newera.ai/v1
Chat · embeddings · rerank · speech synthesisContent-Type: application/json
OCR · segmentation · transcriptionmultipart/form-data (file upload)
Every responsecarries x-request-id, always generated by Newera

Pass a model by its alias (newera-chat-large) or its pinned id (newera-chat-large-v1). The alias always points at the current best version of a capability; the pinned id is frozen. See Model lifecycle.

The OpenAI-shaped endpoints (chat, embeddings, speech) work with the OpenAI SDK unchanged. The Newera-native endpoints (rerank, OCR, segmentation) are plain HTTP; call them with httpx or curl. Both styles appear in each model's examples.

Core concepts

Models

The models, grouped by what they do. Each has a stable public id beginning newera-. Call GET /v1/models to see the ones your key is entitled to.

ModelDoesEndpointCounted per
newera-chat-largeChat + vision, longest context, deepest reasoning/chat/completionstoken
newera-chat-mediumLong, complex tasks and harder reasoning, 128,000-token context/chat/completionstoken
newera-chat-smallFast everyday chat with image and audio input, 64,000-token context/chat/completionstoken
newera-embeddingText embeddings for search and RAG/embeddingsinput token
newera-rerankRe-order retrieval candidates by relevance/rerankdocument scored
newera-ocrExtract text from PDFs and images/documents/extractpage
newera-segmentSegment named concepts in an image/images/segmentimage
newera-ttsNatural Saudi Arabic text to speech/audio/speechsecond of audio
newera-sttArabic speech to text/audio/transcriptionssecond of audio
newera-s2sRealtime voice conversations in Saudi Arabicwss://…/realtimeper connected second
newera-s2s-miniThe same voice conversations on a lighter model, for high-volume, well-defined callswss://…/realtime?model=newera-s2s-miniper connected second

For images, use any of the three chat models: newera-chat-small, newera-chat-medium or newera-chat-large; all accept image input. For audio clips of up to 30 seconds, use newera-chat-small.

For rates, open a model on the Nawah page in the portal.

GET /v1/models

Lists the models on your contract, and nothing more. It is not a catalogue browser.

List entitled models
Shell
curl https://api.newera.ai/v1/models \
  -H "Authorization: Bearer $NEWERA_API_KEY"
Response
JSON
{
  "object": "list",
  "data": [
    {
      "id": "newera-chat-large",
      "object": "model",
      "owned_by": "newera",
      "newera": {
        "class": "chat",
        "modality": "text+image",
        "unit_type": "tokens",
        "context_length": 262144,
        "status": "hot",
        "pinned_id": "newera-chat-large-v1",
        "sunset_at": null
      }
    }
  ]
}

Model lifecycle

Two identifiers, so a model can improve without your code changing:

  • Alias (newera-chat-large). The permanent name for a capability. It moves to a better version when one is ready.
  • Pinned id (newera-chat-large-v1). Frozen at one version. Use it in production when you need output to stay stable.

A model that is being retired returns a Sunset response header with its removal date, ahead of its removal. GET /v1/models shows status and sunset_at.

Core concepts

Errors

Every error is OpenAI-shaped: a JSON body with an error object.

Error body
JSON
{
  "error": {
    "message": "Your organisation is not entitled to `newera-chat-large`. Contact your Newera account manager to add it to your contract.",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_entitled",
    "request_id": "req_..."
  }
}
StatusMeaningTypical code
400Malformed requestinvalid_request_error
401Missing, malformed, revoked or expired keyinvalid_api_key
403The model is not on your organisation's contract, or the account is inactivemodel_not_entitled
403This key's permissions do not include this kind of model. Use a key that can call it, or create one in the portalkey_permission_denied
402This key has reached the monthly spending limit set on it in the portal. It works again on the 1st, or raise the limit under API keyskey_spend_limit_reached
429Too many invalid API keys from your address in the last minute. Wait for the retry-after seconds, and check the key you are sendingtoo_many_invalid_keys
404No such modelmodel_not_found
429A limit on your key was reached (not on the model)key_quota_exhausted
502The model returned an error or could not be reachedupstream_error
503Model temporarily unavailable; retry shortlymodel_unavailable

Quote the request_id (also on the x-request-id header) in any support request. Newera generates this id for every call; an x-request-id you send is not used as it. To tie a call to your own logs, send it anyway: it comes back unchanged in x-client-request-id. A 429 is always about your key. See Rate limits.

Core concepts

Rate limits

Limits live on the API key, and the error always tells you which limit and what to do. There are two kinds:

  • Shape limits, rejected before any processing starts: max_output_tokens, max_input_tokens, max_documents, max_images, max_pages.
  • Volume limits, a cap on how many requests a key may make, in total or per model type.

Core concepts

Streaming

Chat supports server-sent events. Set "stream": true and read data: lines until data: [DONE]. The final chunk carries the usage object. Core requests it on your behalf, so a streamed request is always billable and you always receive the token counts.

Stream a completion
Python
stream = client.chat.completions.create(
    model="newera-chat-large",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

If you disconnect mid-stream, the compute already spent is still metered.

Chat

Chat completions

POST /v1/chat/completions

OpenAI-compatible, three models. Pick by context length and depth of reasoning. All three share the parameters below; each model card adds only what is specific to it.

Parameters

NameTypeReqDefaultWhat it does
modelstringyesWhich model to use, for example newera-chat-large. A pinned id (newera-chat-large-v1) never changes, so use that in production.
messagesarrayyesThe conversation so far: a list of messages, each with a role (system, user or assistant) and content.
max_tokensintegernosee belowOutput limit: the most tokens the model may write in one reply, thinking included. It is not a limit on your input. Input and reply share the model's context window, so the input plus max_tokens must fit in it. If you leave it out, the reply is limited to the model's default (see the model's own limits) and ends with finish_reason "length" if it hits it.
temperaturenumberno10 - 2. How varied the answers are. Lower is steadier; 0 gives the closest to the same answer every time.
top_pnumberno10 - 1. Only choose from the most likely words that add up to this share of the probability. Lower is more focused. Change this or temperature, rarely both.
top_kintegernoOnly choose from the k most likely next words. Lower is more focused. Off unless you set it.
min_pnumberno0 - 1. Drop next words that are much less likely than the best one. 0.05 drops any word under 5% of the best word's chance. Higher is more focused. Not available on newera-chat-large, which refuses it.
presence_penaltynumberno0-2 - 2. Above 0 nudges the model toward new topics: a word that has already appeared is less likely to come up again.
frequency_penaltynumberno0-2 - 2. Above 0 reduces repetition: the more often a word has appeared, the less likely it is to appear again.
repetition_penaltynumberno1Above 1 discourages repeating words already used; 1 does nothing. Try 1.05 to 1.2.
stopstring | arraynoText that ends the reply when the model writes it. The stop text itself is not included.
seedintegernoWith the same seed and a temperature of 0, repeating a request gives the same answer.
streambooleannoFalseSend the reply in pieces as it is written. The last piece carries the token counts.
response_formatobjectno{"type": "json_object"} makes the reply valid JSON. {"type": "json_schema", "json_schema": {...}} makes it match your schema.
toolsarraynoFunctions the model may ask you to call, each with a name, a description and a JSON schema for its arguments.
tool_choicestring | objectnoauto"auto" lets the model decide, "none" never calls a tool, "required" forces a call, or name one function. "none" is the default when you send no tools.
logprobsbooleannoFalseReturn how likely each chosen word was.
top_logprobsintegerno00 - 20. With logprobs on, also return the n likeliest alternatives for each word.

max_tokens is the output limit. It caps the tokens in the reply, thinking included, and never limits your input. The input and the reply share one context window, so the input plus max_tokens must fit in it; a request that does not is refused with context_length_exceeded. Each model's own limits are listed in its section below.

If you do not set max_tokens, a reply is limited to the smaller of two things: the model's default reply length, and what your prepaid balance covers. The reply then ends with finish_reason: "length". Set max_tokens to choose your own limit instead.

  • newera-chat-large: a default reply of 32,768 tokens.
  • newera-chat-medium: no fixed default; a reply can use all the room your input leaves in its 128,000-token window.
  • newera-chat-small: no fixed default; a reply can use all the room your input leaves in its 64,000-token window.

With n above 1 the limit applies to each reply. A request that sets max_tokens is checked against your balance as before, and refused with insufficient_quota if it cannot be covered.

Response

The OpenAI chat completion shape: choices[].message.content, plus a usage object with prompt_tokens, completion_tokens, prompt_tokens_details.cached_tokens (the part of the prompt served from the model's prefix cache) and, where the model reasons, completion_tokens_details.reasoning_tokens.

Chat model

newera-chat-large

The flagship: the longest context in the catalogue, native vision, and the deepest reasoning. Reach for it when the task is hard or the input is large.

Best for

Long-document analysis, multi-step reasoning, agentic and tool-use flows, and anything mixing text with images.

Not for

High-volume, latency-sensitive or trivial calls. As the most capable tier, it is best suited to demanding tasks; for simple or high-throughput work, a smaller tier is the better fit.

Model id
newera-chat-large
Pinned id
newera-chat-large-v1
Context window
262,000 tokens
Inputs
text + image
Billed
per input and output token
Endpoint
/chat/completions

Images

Send message content as parts: a {"type":"text"} part and an {"type":"image_url"} part whose url is a data: URL or a public https: URL. Image pixels are tokenised into the prompt, so they count as input tokens, and a larger image uses more of them.

Thinking

This model thinks step by step before it answers, and thinking is on by default. That helps on hard, multi-step problems and costs time and tokens. For a faster direct answer, turn it off with chat_template_kwargs, as the example below shows. completion_tokens_details.reasoning_tokens shows how much a call thought.

Limits for this model

LimitValue
Context window262,000 tokens, shared by the input and the reply
Max input tokens261,000 (input plus reply must fit the window)
Max output tokens261,000: the max_tokens ceiling, thinking included (input plus reply must fit the window)
Default output tokens32,768 when max_tokens is left out
Image inputyes
Audio inputno: an audio part is refused

Parameters for this model

NameTypeReqDefaultWhat it does
chat_template_kwargsobjectno{"enable_thinking": true}Thinking on or off. This model thinks step by step before it answers, which helps on hard problems and costs time and tokens. Send {"enable_thinking": false} for a faster, cheaper direct answer, but we recommend keeping thinking on: it helps the model follow your instructions and ignore instructions hidden inside user messages or documents. The choice is yours. Nothing else is accepted inside it.
Chat completion
curl https://api.newera.ai/v1/chat/completions \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-chat-large",
    "messages": [{"role": "user", "content": "Explain the difference between TCP and UDP."}],
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
resp = client.chat.completions.create(
    model="newera-chat-large",
    messages=[{"role": "user", "content": "Explain the difference between TCP and UDP."}],
    max_tokens=512,
    # Answer directly; omit to let the model reason first.
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

Chat model

newera-chat-medium

A stronger model for long, complex work: it steps up from newera-chat-small with twice the context window, and is better at harder reasoning and coding. It reasons before it answers by default and can be told to answer directly. Reads images as well as text.

Best for

Long, multi-step tasks, harder reasoning and coding, tool-using assistants, and long documents of up to 128,000 tokens.

Not for

Requests whose input and reply together need more than its 128,000-token window, or the deepest multi-step reasoning: use newera-chat-large for those. Simple everyday requests, structured output and audio input are better served by newera-chat-small, which is faster.

Model id
newera-chat-medium
Pinned id
newera-chat-medium-v1
Context window
128,000 tokens
Inputs
text + image
Billed
per input and output token
Endpoint
/chat/completions

Thinking

This model thinks step by step before it answers, and thinking is on by default. That helps on hard, multi-step problems and costs time and tokens. For a faster direct answer, turn it off with chat_template_kwargs, as the example below shows. completion_tokens_details.reasoning_tokens shows how much a call thought.

Limits for this model

LimitValue
Context window128,000 tokens, shared by the input and the reply
Max input tokens127,000 (input plus reply must fit the window)
Max output tokens127,000: the max_tokens ceiling, thinking included (input plus reply must fit the window)
Default output tokenswhen max_tokens is left out, up to all the room your input leaves in the window
Image inputyes
Audio inputno: an audio part is refused

Parameters for this model

NameTypeReqDefaultWhat it does
chat_template_kwargsobjectno{"enable_thinking": true}Thinking on or off. This model thinks step by step before it answers, which helps on hard problems and costs time and tokens. Send {"enable_thinking": false} for a faster, cheaper direct answer, but we recommend keeping thinking on: it helps the model follow your instructions and ignore instructions hidden inside user messages or documents. The choice is yours. Nothing else is accepted inside it.
Chat completion
curl https://api.newera.ai/v1/chat/completions \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-chat-medium",
    "messages": [{"role": "user", "content": "Summarise the water cycle."}],
    "max_tokens": 256,
    "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
resp = client.chat.completions.create(
    model="newera-chat-medium",
    messages=[{"role": "user", "content": "Summarise the water cycle."}],
    max_tokens=256,
    temperature=0.7,
    # Answer directly; omit to let the model reason first.
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)
print(resp.usage)

Chat model

newera-chat-small

Fast chat for everyday requests. Reads text, images and audio, and writes text. It is the smallest of the three tiers: for long, complex or harder work, step up to newera-chat-medium.

Best for

JSON and structured output, tool calling, everyday questions and answers, audio understanding, images, and high-volume tasks where speed matters most.

Not for

Long, complex multi-step tasks and harder reasoning or coding: use newera-chat-medium, or newera-chat-large for the longest inputs. Requests whose input and reply together need more than its 64,000-token window are refused, as are a clip longer than 30 seconds and more than 8 images, 4 clips or 1 video.

Model id
newera-chat-small
Pinned id
newera-chat-small-v1
Context window
64,000 tokens
Inputs
text + image + audio
Billed
per input and output token
Endpoint
/chat/completions

Images

Send message content as parts: a {"type":"text"} part and an {"type":"image_url"} part whose url is a data: URL or a public https: URL. Image pixels count as input tokens, and a larger image uses more of them. A request can carry up to 8 images.

Audio

Add an {"type":"input_audio"} part to the message content: input_audio.data is the clip as base64 and input_audio.format is wav or mp3. A request can carry up to 4 clips, each at most 30 seconds long; a longer clip is refused with a 400 (audio_too_long). Audio becomes input tokens (about 25 for every second of sound) and is billed as input tokens.

Thinking

This model answers directly unless you ask it to think. Send chat_template_kwargs with {"enable_thinking": true} to have it think step by step first: better on harder problems, slower and more output tokens. completion_tokens_details.reasoning_tokens shows how much a call thought.

Limits for this model

LimitValue
Context window64,000 tokens, shared by the input and the reply
Max input tokens63,000 (input plus reply must fit the window)
Max output tokens63,000: the max_tokens ceiling, thinking included (input plus reply must fit the window)
Default output tokenswhen max_tokens is left out, up to all the room your input leaves in the window
Image inputyes, up to 8 images in a request
Audio inputyes, up to 4 clips in a request, each of up to 30 seconds

Parameters for this model

NameTypeReqDefaultWhat it does
chat_template_kwargsobjectno{"enable_thinking": false}Thinking on or off. This model answers directly unless you ask it to think. Send {"enable_thinking": true} to have it think step by step first, which helps on hard problems and costs time and tokens. For content you do not control, such as user messages or documents, we recommend turning thinking on: it helps the model follow your instructions and ignore instructions hidden in that content. It is faster without it. The choice is yours. Nothing else is accepted inside it.
Classify a string
curl https://api.newera.ai/v1/chat/completions \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-chat-small",
    "messages": [{"role": "user", "content": "Classify sentiment: the invoice is overdue"}],
    "max_tokens": 16
  }'
resp = client.chat.completions.create(
    model="newera-chat-small",
    messages=[{"role": "user", "content": "Classify sentiment: the invoice is overdue"}],
    max_tokens=16,
)
print(resp.choices[0].message.content)

Embeddings

newera-embedding

POST /v1/embeddings

Turn text into 1,024-number vectors for semantic search, retrieval, clustering and deduplication. The vector size is fixed at 1,024.

Best for

Building a vector index for search or RAG, measuring semantic similarity, grouping or de-duplicating text.

Not for

Generating text. Pair it with newera-rerank to sharpen the top of your results.

Model id
newera-embedding
Pinned id
newera-embedding-v1
Vector size
1,024
Max input
8,192 tokens per text
Billed
per input token
Endpoint
/embeddings

Parameters

NameTypeReqDefaultWhat it does
modelstringyes
inputstring | arrayyesOne text, or a list of texts. Each text may be up to 8,192 tokens; longer is refused. A list is one request, billed as the total of its tokens.
encoding_formatstringnofloatfloat, base64. float returns numbers. base64 returns the same vector packed as text, which is smaller to send.

Response

OpenAI embeddings shape: data[].embedding, with a usage object carrying prompt_tokens.

Embed a batch
curl https://api.newera.ai/v1/embeddings \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-embedding",
    "input": ["first passage", "second passage"]
  }'
resp = client.embeddings.create(
    model="newera-embedding",
    input=["first passage", "second passage"],
)
print(len(resp.data), len(resp.data[0].embedding))

Rerank

newera-rerank

POST /v1/rerank

Score a list of documents against a query and return them by relevance. The second step of a good search.

Best for

Reordering the top candidates from a vector or keyword search so the most relevant rise to the top; two-stage RAG.

Not for

First-pass retrieval over a large corpus. Use embeddings and a vector store for that, then rerank the shortlist.

Model id
newera-rerank
Pinned id
newera-rerank-v1
Max documents
1,000 per request
Billed
per document scored
Endpoint
/rerank
Shape
Newera-native

Parameters

NameTypeReqDefaultWhat it does
modelstringyes
querystringyesWhat you are searching for.
documentsarrayyesup to 1,000. The texts to rank. Each one, with the query, may be up to 5,000 tokens. Every one is scored and billed, even if top_n then leaves it out.
top_nintegernoReturn only the n best. It does not lower the billed count.
return_documentsbooleannoFalseInclude each document's text next to its score.

Usage

Usage counts every candidate the model scores, not only the top_n you keep. Scoring 500 documents and returning 5 counts as 500 documents.

Rerank candidates
curl https://api.newera.ai/v1/rerank \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-rerank",
    "query": "how do I rotate a key?",
    "documents": ["Keys are shown once.", "Rotate from the portal."],
    "top_n": 2
  }'
import os, httpx
BASE = "https://api.newera.ai/v1"
AUTH = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}

r = httpx.post(BASE + "/rerank", headers=AUTH, json={
    "model": "newera-rerank",
    "query": "how do I rotate a key?",
    "documents": ["Keys are shown once.", "Rotate from the portal."],
    "top_n": 2,
}, timeout=60)
r.raise_for_status()
for row in r.json()["results"]:
    print(row["index"], round(row["relevance_score"], 4))

Document extraction

newera-ocr

POST /v1/documents/extract

Pull text out of a PDF or an image. Handles Arabic and mixed scripts. multipart/form-data.

Best for

Reading scanned or born-digital PDFs and images into plain text; feeding documents into search or an LLM.

Not for

Structured field extraction with coordinates. It returns the document's text, not a form-field map.

Model id
newera-ocr
Pinned id
newera-ocr-v1
Max upload
50 MB
Billed
per page processed
Endpoint
/documents/extract
Shape
Newera-native · multipart

Parameters (form fields)

NameTypeReqDefaultWhat it does
filefileyesup to 50 MB. A PDF or an image.
modelstringnonewera-ocr

Response

{ "text": "...", "pages": [...] | null, "usage": { "pages": N } }

Extract a PDF
curl https://api.newera.ai/v1/documents/extract \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -F "file=@contract.pdf" \
  -F "model=newera-ocr"
import os, httpx
BASE = "https://api.newera.ai/v1"
AUTH = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}

with open("contract.pdf", "rb") as fh:
    r = httpx.post(BASE + "/documents/extract", headers=AUTH,
                   files={"file": fh},
                   data={"model": "newera-ocr"},
                   timeout=300)
r.raise_for_status()
doc = r.json()
print(doc["text"][:500])
print(doc["usage"]["pages"], "pages billed")

Segmentation

newera-segment

POST /v1/images/segment

Find and outline the concepts you name in an image, returning a mask, a bounding box and a confidence for each. multipart/form-data.

Best for

Locating named objects in imagery: buildings, roads, vegetation and the like in aerial or satellite pictures.

Not for

Open-ended "segment everything" requests. It finds the concepts you name, so name what you want found.

Model id
newera-segment
Pinned id
newera-segment-v1
Max upload
25 MB
Billed
per image submitted
Endpoint
/images/segment
Shape
Newera-native · multipart

Parameters (form fields)

NameTypeReqDefaultWhat it does
imagefileyesup to 25 MB. A PNG or JPEG image.
modelstringnonewera-segment
promptstringyesWhat to look for, for example "building, road, tree". Only what you name is found: there is no find-everything mode.

Response

detections[], each with label, confidence, a box, and a full-resolution PNG mask as a data: URL; plus not_found, image dimensions, and usage.

Finding nothing is a success, not an error. Ask for three things and find one, and detections has one entry while not_found lists the other two. Check not_found rather than diffing the response against your request. Either way the request counts as one image. The first request after a quiet period can take about 40 seconds; later requests return in under a second.

Segment concepts
curl https://api.newera.ai/v1/images/segment \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -F "image=@aerial.png" \
  -F "model=newera-segment" \
  -F "prompt=building, road"
import os, base64, httpx
BASE = "https://api.newera.ai/v1"
AUTH = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}

with open("aerial.png", "rb") as fh:
    r = httpx.post(BASE + "/images/segment", headers=AUTH,
                   files={"image": fh},
                   data={"model": "newera-segment", "prompt": "building, road"},
                   timeout=300)
r.raise_for_status()
for det in r.json()["detections"]:
    print(det["label"], round(det["confidence"], 3))
    head, _, b64 = det["mask"].partition(",")   # data:image/png;base64,....
    open(det["label"] + ".png", "wb").write(base64.b64decode(b64))

Speech

newera-tts

POST /v1/audio/speech

Natural Saudi Arabic speech from text. OpenAI-shaped; returns 24 kHz audio bytes. Tuned for spoken Saudi Arabic, it answers fast enough for live calls. It is the voice of newera-s2s.

Best for

Voicing short-to-medium text: notifications, prompts, call-centre replies and read-aloud. Mainly Saudi Arabic, with English supported as well.

Not for

Languages beyond Arabic and English, or inputs over 4,096 characters per request.

Model id
newera-tts
Pinned id
newera-tts-v1
Max input
4,096 characters
Billed
per second produced, rounded up
Endpoint
/audio/speech
Returns
24 kHz mono audio

Parameters

NameTypeReqDefaultWhat it does
modelstringyes
inputstringyes1 - 4,096 characters. The text to speak.
voicestringnomalemale, female. male (Khaled) or female (Lama).
qualitystringnostudioinstant, balanced, studio. instant starts fastest, studio sounds most natural.
normalizebooleannoTrueSay numbers, times, percentages, phone numbers, currency and brand names the way a person would. false speaks the text exactly as written.
response_formatstringnowavwav, pcm. wav is a complete audio file. pcm is raw audio: 24 kHz, 16-bit, mono.
speednumberno1.00.25 - 4. How fast it speaks. 1 is the natural pace.

Numbers and names, spoken naturally

Write text the way you normally would. Digits, times, percentages, amounts, phone numbers and brand names are spoken in natural Saudi Arabic automatically: 10:30 is heard as عشر ونص, and 15% as خمسطعش بالمية. Send "normalize": false to speak the text exactly as given.

Duration

With wav, the response carries x-newera-audio-seconds and x-newera-billed-seconds, so you can reconcile usage without decoding the audio.

Speak a Saudi sentence
curl https://api.newera.ai/v1/audio/speech \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "newera-tts",
    "input": "أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
    "voice": "male",
    "quality": "balanced",
    "response_format": "wav"
  }' \
  --output reply.wav
audio = client.audio.speech.create(
    model="newera-tts",
    input="أهلًا وسهلًا، موعدك بكرة الساعة 10:30.",
    voice="male",
    response_format="wav",
    extra_body={"quality": "balanced"},   # instant | balanced | studio
)
with open("reply.wav", "wb") as fh:
    fh.write(audio.read())

Speech

newera-stt

POST /v1/audio/transcriptions

Transcribe Arabic audio to text. OpenAI-shaped, multipart/form-data.

Best for

Arabic transcription: meetings, calls, voice notes and media, including everyday Arabic-English code-switching (Arabic speech with English words mixed in).

Not for

Languages other than Arabic and English. It is tuned for Arabic; send language en for English. verbose_json output is not supported (use json or text).

Model id
newera-stt
Pinned id
newera-stt-v1
Max upload
25 MB
Billed
per second submitted, rounded up
Endpoint
/audio/transcriptions
Language
Arabic (send language ar)

Parameters (form fields)

NameTypeReqDefaultWhat it does
filefileyesup to 25 MB. The audio: wav, mp3, m4a, ogg, flac or webm.
modelstringyes
languagestringnoenar, en. The language spoken. Send ar for Arabic: the language is not detected, and Arabic audio sent without it comes back written in English letters.
response_formatstringnojsonjson, text. json returns {"text": ...}. text returns the plain text.
temperaturenumberno00 - 1. 0 gives the most consistent transcript.

Response

{ "text": "..." } for json, plain text for text; duration is on the x-newera-audio-seconds header either way.

Transcribe a clip
curl https://api.newera.ai/v1/audio/transcriptions \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -F "file=@meeting.wav" \
  -F "model=newera-stt" \
  -F "response_format=json"
with open("meeting.wav", "rb") as fh:
    result = client.audio.transcriptions.create(
        model="newera-stt",
        file=fh,
        response_format="json",
    )
print(result.text)

Realtime voice

newera-s2s

WSS /v1/realtime

A live speech-to-speech conversation in Arabic or English over one WebSocket: audio in, audio out. Stream the caller's audio in; the assistant's speech, its written words and the caller's transcript stream back. The assistant notices when the caller has finished, answers almost instantly, and stops talking when the caller cuts in.

The events follow the standard Realtime event set (beta version): the same event names, fields and order. A standard Realtime client works by changing the URL and the API key, with no proxy. Compatibility notes lists what is not the same.

By default the language of the call is detected automatically, on every caller turn (Arabic or English), and the assistant answers in it. If you know the language up front, fix it with newera.detect_language and detection is off for the whole call. See Language.

Best for

Phone agents and customer service in Saudi Arabic: bookings, order and account questions, short guided conversations.

Not for

Reading long text aloud (use newera-tts) or languages other than Arabic and English.

Model id
newera-s2s (default) or newera-s2s-mini, chosen in the URL
Pinned id
newera-s2s-v1
URL
wss://api.newera.ai/v1/realtime
Audio in
PCM 16-bit mono, 24 kHz (or 8 to 48 kHz, or G.711)
Audio out
PCM 16-bit mono, 24 kHz

Connect

From a server, send your API key on the WebSocket handshake as Authorization: Bearer …. Choose the model in the URL: wss://api.newera.ai/v1/realtime?model=newera-s2s-mini (the default is newera-s2s; see Choosing the model). The key must be entitled to the model you connect with. A refused connection still opens, sends one error event saying why, and closes with one of these codes:

Close codeMeaning
4400Unknown or non-voice model (invalid_model)
4401Missing or invalid API key
4402The balance cannot cover the call (insufficient_quota)
4403Key not entitled to the model (model_not_entitled)
4429Too many calls on this key right now (concurrent_call_limit). Wait and retry.
4503Voice is temporarily unavailable; retry

Connect-time refusals are error events, and the socket then closes. Two messages: invalid_model, close 4400: "foo is not a voice model. Choose newera-s2s or newera-s2s-mini, for example ?model=newera-s2s-mini." and insufficient_quota, close 4402: "Your prepaid balance cannot cover this call. Top up your credit and connect again."

If a key reaches its call capacity, a new call is refused with concurrent_call_limit (close 4429) and calls already running are not affected; contact your account manager to raise it.

From a browser, never put the API key in the page. Your server trades it for a client secret that lasts 60 seconds and opens one connection, and the page offers the secret as a WebSocket subprotocol, newera-client-secret.<secret>, beside realtime.

Get a client secret (on your server)
Shell
curl -X POST https://api.newera.ai/v1/realtime/sessions \
  -H "Authorization: Bearer $NEWERA_API_KEY"
# {"object": "realtime.session", "model": "newera-s2s",
#  "client_secret": {"value": "ek_...", "expires_at": 1760000000}}

GET /v1/realtime/voices with your API key lists the voices before you connect: {"default", "data": [{"id", "name", "gender"}]}. There are two: male (Khaled) and female (Lama).

Audio

Both directions carry audio base64-encoded inside JSON events. Send the caller's audio in small chunks of 20 to 100 ms, continuously and in real time, silence included: the call decides that a turn is over by hearing the silence. Turn on echo cancellation on the microphone, or the assistant hears itself and stops. If you capture audio in a browser, turn off autoGainControl in getUserMedia (Chrome turns it on by default).

input_audio_format: "pcm16" (the default) is 16-bit little-endian mono PCM at 24 kHz; g711_ulaw and g711_alaw are 8 kHz telephony. For PCM at 16 kHz, 8 kHz or 48 kHz set newera.input_audio_sample_rate. Output is output_audio_format: "pcm16" at 24 kHz (the default) or G.711 at 8 kHz.

A call, event by event

Event flow
Events
→ (connect)
← session.created                        settings and voices
← conversation.created
→ session.update                         optional: instructions, voice, turn control
→ input_audio_buffer.append {audio}      continuously, silence included
← input_audio_buffer.speech_started {item_id}     the caller began talking
← input_audio_buffer.speech_stopped {item_id}     the caller paused
← input_audio_buffer.committed {item_id, previous_item_id}   the pause was long enough: a turn
← conversation.item.created {item}       the caller's audio as an item (transcript null)
← conversation.item.input_audio_transcription.completed {item_id, transcript}
   (with create_response: true, the default, the reply follows)
← response.created {response}            status in_progress, output [], your metadata
← response.output_item.added {item}      the assistant's message item
← conversation.item.created {item}       the same item, in the conversation
← response.content_part.added {part}
← response.audio_transcript.delta {delta}     text of the next piece
← response.audio.delta {delta}                its audio, base64 PCM 24 kHz
   ...repeated per piece...
← response.audio.done
← response.audio_transcript.done {transcript}
← response.content_part.done {part}
← response.output_item.done {item}
← response.done {response}               status, output[], usage

Every reply has this shape, whether the caller's voice, a typed message or a response.create started it. With tools, function calls appear in the middle of it (see Tools). In a text-only reply (modalities: ["text"]) the audio events are replaced by response.text.delta and response.text.done, and the content part is {type: "text", text}.

Events you send

EventFieldsWhat it does
session.updatesessionChange the call's settings (below). Answered by session.updated. A field that is unknown, unsupported or has a bad value gets an error event naming it (unknown_parameter, unsupported_parameter or invalid_value, with param); the rest of the update still applies.
input_audio_buffer.appendaudioStream the caller's audio.
input_audio_buffer.commitWith turn_detection: null: add the audio sent so far to the conversation as the caller's turn (at least 100 ms). It does not start a reply: send response.create.
input_audio_buffer.clearDiscard audio not yet committed (manual turns).
conversation.item.createitem, previous_item_id?Add an item to the conversation: a message from the user, the assistant or the system, or a function call and its output. See Items.
conversation.item.deleteitem_idRemove an item. Answered by conversation.item.deleted.
conversation.item.retrieveitem_idAnswered by conversation.item.retrieved.
conversation.item.truncateitem_id, content_index: 0, audio_end_msTell Newera how much of an assistant reply was actually played, so its memory of the conversation matches what the caller heard. Answered by conversation.item.truncated.
response.createresponse?Ask for a reply. See Replies.
response.cancelresponse_id?Stop the current reply. response.done has status: "cancelled" and reason: "client_cancelled".
session.endNewera extension. Finish the call: you receive session.summary with the call's connected time and latency, then the socket closes.

Session settings

Send these in a session.update event. A name with a dot is a field inside an object: turn_detection.threshold goes in session.turn_detection, and newera.voice_quality in session.newera.

NameTypeDefaultWhat it does
modalitiesarray["text", "audio"]text, audio. Replies are spoken, and the written words of each reply arrive with the audio. ["audio"] gives the same. Send ["text"] for written replies with no speech.
instructionsstringWho the assistant is and how it should behave: its name, its job, what it may talk about. Newera adds its own rules first: short replies, natural Saudi Arabic, numbers said as words.
voicestringmalemale, female. male (Khaled) or female (Lama). Voice names from other realtime clients are matched to the closest of these two.
input_audio_formatstringpcm16pcm16, g711_ulaw, g711_alaw. The audio you send. pcm16 is 16-bit mono at 24 kHz unless you set newera.input_audio_sample_rate. g711 is 8 kHz phone audio.
output_audio_formatstringpcm16pcm16, g711_ulaw, g711_alaw. The audio you get back. pcm16 is 16-bit mono at 24 kHz. g711 is 8 kHz phone audio.
input_audio_transcriptionobject | nullSet {"language": "en"} or {"language": "ar"} to fix the call's language. null turns off the written transcript of what the caller said.
turn_detection.silence_duration_msinteger500100 - 5000. How long a pause ends the caller's turn. Shorter answers sooner but may cut in on a slow speaker.
turn_detection.thresholdnumber0.60 - 1. How clearly something must sound like speech to count as the caller talking. Higher ignores more background noise.
turn_detection.prefix_padding_msinteger3000 - 1000. How much audio from just before the caller starts speaking is kept, so the first word is not clipped.
turn_detection.create_responsebooleanTrueReply automatically when the caller's turn ends. With false, the assistant replies only when you send response.create.
turn_detection.interrupt_responsebooleanTrueStop talking when the caller speaks over the assistant.
turn_detectionobject | nullSet to null to end turns yourself: send input_audio_buffer.commit, then response.create.
temperaturenumber0.40 - 1.5. How varied the replies are. Values outside the range are set to the nearest limit.
max_response_output_tokensinteger | "inf"no limitThe most tokens in one reply. "inf" means no limit.
toolsarrayFunctions the assistant may call during the call.
tool_choicestring | objectauto"auto", "none", "required", or one named function.
newera.voice_qualitystringstudioinstant, balanced, studio. instant starts each reply fastest, studio sounds most natural. Can change mid-call and applies from the next reply.
newera.voice_cloneobjectSpeak in your own voice, or one whose owner has agreed: {"audio": base64, "text": what the clip says}. The clip is 3 to 8 seconds and at most 3 MB.
newera.input_audio_sample_rateinteger240008000, 16000, 24000, 48000. The sample rate of the pcm16 audio you send.
newera.detect_languagestringautoauto, ar, en. auto works out Arabic or English on every turn. ar or en fixes the call to that language.
newera.timezonestringAsia/RiyadhThe time zone the assistant uses for today, now and weekdays, for example Asia/Riyadh.
newera.tool_auto_continuebooleanTrueAfter you return a tool's result, the assistant carries on and speaks the answer by itself.
newera.max_tool_stepsinteger6The most tool rounds in a single reply.

An unknown field anywhere in session, session.newera, session.turn_detection or response is an error event naming it, never silently ignored. A bad value gets an error too (for example invalid_value with param: "session.newera.voice_quality"), and the call keeps its setting. The model is chosen in the URL and cannot change during a call: sending the same model back is accepted, a different one is refused with unsupported_parameter.

Choosing the model

Pick the model when the call connects, in the URL: wss://api.newera.ai/v1/realtime?model=newera-s2s-mini. Without model the call uses newera-s2s. The model is fixed for the whole call.

ModelUse it for
newera-s2s (default)Conversations that need deeper reasoning: multi-step troubleshooting, consultative sales, bookings that balance several constraints, workflows that chain many tools, long calls with a lot of context, nuanced policy questions.
newera-s2s-miniFast and efficient for high-volume, well-defined conversations: customer support and FAQs, appointment booking and rescheduling, order and delivery status, reservations, lead qualification and intake, surveys and feedback calls, reminders and confirmations, replacing IVR menus, and tool-driven flows such as looking up a record or opening a ticket.

Both models support tool calling and the same protocol, events and voices.

Language

session.newera.detect_language sets how the call's language is decided: "auto" (the default), "ar" or "en".

  • "auto" detects the language on every caller turn. Newera listens to what the caller says and decides between Arabic and English. The assistant replies in the detected language, and a typed message follows the script it is written in. A single short or accented sentence does not flip the call: switching takes strong evidence.
  • A fixed language ("ar" or "en") turns detection off. For the whole call, from the next turn: no language detection runs at all, the caller's speech is always taken as that language, the assistant always replies in it, and the voice and the way numbers and dates are spoken follow it. Typed text does not change the language either: English typed into an "ar" call is answered in Arabic.
  • Use a fixed language when the call's language is known up front, such as interviews in one language, or a line that serves a single language. It is also the right choice for callers who speak one language with a strong accent in it (an English interview with an Arabic accent), where detection could guess the other language. Keep "auto" when callers may speak either.
  • session.created and every session.updated show the setting under newera.detect_language. It can be changed at any time and applies from the next turn: a reply already being generated or spoken keeps the language it started in.
  • This works too: session.input_audio_transcription: {"language": "en"} (or "ar") fixes the language exactly like detect_language. Any other language is an error with invalid_value and param: "session.input_audio_transcription.language", naming the supported ones. If one session.update carries both and they disagree, newera.detect_language wins.
  • Any other value of detect_language is an error with invalid_value and param: "session.newera.detect_language", and the call keeps its setting.
{"type": "session.update", "session": {"newera": {"detect_language": "en"}}}

Turn control

turn_detection decides who ends the caller's turn and who starts the reply.

Automatic turns (the default). The call listens for the caller's speech and tells you what it hears: speech_started, speech_stopped, committed and a conversation.item.created for the caller's audio, then its transcript. silence_duration_ms is how long a pause ends the caller's turn: 500 ms by default, 100 to 5000, counted from the caller's last word to the commit. threshold (0 to 1, default 0.6) is how clearly a sound must be speech to count, so a higher value ignores more background noise. prefix_padding_ms (0 to 1000, default 300) is the audio kept from just before the caller starts, so the first word is not clipped. The assistant starts preparing its answer after 200 ms of silence and holds it until the commit; if the caller resumes first, the draft is dropped and the turn simply continues. Shorter pauses answer sooner and cut into the caller's pauses more often.

  • create_response (default true): the assistant answers each committed turn. With false it does everything above, the transcript and the item included, but starts no reply: the client decides, with response.create. A response.create sent before the transcript is ready waits for it.
  • interrupt_response (default true): when the caller talks over a reply, Newera cancels it (response.done with status: "cancelled", reason: "turn_detected") and the conversation keeps what was heard. With false the reply plays on and the next one starts after it.

turn_detection: null (manual). The call does not listen for turns. Send input_audio_buffer.append, then input_audio_buffer.commit: the caller's audio becomes an item (committed, conversation.item.created, its transcript) and nothing else happens until response.create. Several commits before one response.create are all part of its context.

Items

The conversation is a list of items with ids: the caller's audio and typed messages, the assistant's replies, and whatever the client adds. A reply is always generated from the conversation as it stands. conversation.item.create takes these (previous_item_id places the item after that one; absent, at the end; "root", first):

ItemContent
{type: "message", role: "user"}input_text {text} (or text); input_audio {audio, transcript?}, audio in the session's input format: with a transcript it is used as is, without one, the transcript is filled in from the audio.
{type: "message", role: "assistant"}text {text} (or output_text); audio {transcript}.
{type: "message", role: "system"}input_text {text}: facts and context for the assistant, shown to it with the instructions, not as caller speech.
{type: "function_call", call_id, name, arguments} and {type: "function_call_output", call_id, output}A tool call and its result, to restore a conversation that used tools.

Use it to restore a conversation after a reconnect: create the items, then response.create. The answer is conversation.item.created {previous_item_id, item}. An unknown previous_item_id, a duplicate id and an unsupported content type or role are each an error naming the field. The history is trimmed from its oldest turn when it outgrows the model's budget, as in a live call.

conversation.item.truncate applies to an assistant reply Newera spoke: the conversation then holds only the words heard up to audio_end_ms, followed by "…", and Newera answers conversation.item.truncated. An unknown item_id is an invalid_request_error (item_not_found) naming it; a user item, or an item without audio, cannot be truncated.

Replies

response.create asks for a reply to the conversation as it stands; there is no need to add input first. With nothing to answer and no instructions, the assistant opens the call from the session instructions.

Ask for a greeting
JSON
{"type": "response.create", "response": {
  "instructions": "Greet the caller by name, then ask how you can help.",
  "metadata": {"client_event_id": "abc"}}}
response field
instructionsFor this reply only: added after the session's instructions and winning where they disagree. They are instructions, not caller text: nothing is stored as a user turn, and the next reply does not see them.
modalities, voice, temperature, max_output_tokensThis reply's own value instead of the session's.
metadataUp to 16 string pairs, echoed in response.created and response.done.
conversationOnly "auto" (or "default"): every reply is added to the conversation.

Not supported, with an unsupported_parameter error naming the field and no reply started: conversation: "none" (out-of-band responses), input, and per-response tools, tool_choice or output_audio_format. One reply runs at a time: a response.create while another is running is conversation_already_has_active_response, and response.cancel with nothing running is response_cancel_not_active.

Barge-in

When the caller talks over a reply (and interrupt_response is on), Newera cancels it and sends response.done with status: "cancelled", then input_audio_buffer.speech_started. On speech_started, stop playback and drop any queued audio. Audio arrives faster than real time, so a player usually holds several seconds in hand. If you know how much was played, send conversation.item.truncate; otherwise Newera estimates it.

response.done

A completed reply
JSON
{"type": "response.done", "response": {
  "id": "resp_...", "object": "realtime.response", "status": "completed", "status_details": null,
  "conversation_id": "conv_...", "metadata": {"client_event_id": "abc"},
  "output": [{"id": "item_...", "object": "realtime.item", "type": "message", "status": "completed",
              "role": "assistant", "content": [{"type": "audio", "transcript": "أبشر، متى يناسبك الموعد؟"}]}],
  "usage": null,
  "latency": {"voice_to_voice_ms": …},
  "output_transcript": "أبشر، متى يناسبك الموعد؟"}}

output is always filled: the assistant's message first, then any function_call items. usage is null: a reply carries no token or audio counts. latency.voice_to_voice_ms runs from the caller's last word to the first audio of the reply leaving Newera; add your network time to get what the caller hears. output_transcript is the reply's full text.

status is completed, cancelled (barge-in or response.cancel) or failed; status_details says why ({type: "cancelled", reason: "turn_detected" | "client_cancelled"}, or {type: "failed", error}). A reply that was cancelled still closes its item first (response.audio.done, response.audio_transcript.done, response.content_part.done, response.output_item.done) with the words spoken so far.

Compatibility notes

The call follows the beta Realtime events. What is not the same as a standard client expects:

  • Beta event names. The beta event names (response.audio.delta, response.audio_transcript.delta, conversation.item.created) and a flat session object. Fields from the newer version of the protocol (session.audio, output_modalities, type) are refused by name.
  • Voices are male and female. A voice name from another realtime client is accepted so that client's default works, and matched to the closest of the two: session.voice then reads male or female. Any other name is invalid_voice. The model is selected in the URL (?model=). input_audio_transcription.language of "en" or "ar" fixes the call's language, like newera.detect_language (which wins if both are sent); other languages are refused with invalid_value. Without either, the language is detected on every caller turn.
  • Turn detection has one kind, so type can be left out. silence_duration_ms counts from the last word to the commit, but speech_stopped is sent after the first 200 ms of it.
  • Not supported (an error naming the field): replies outside the conversation (response.conversation: "none"), response.input, per-response tools, tool_choice and output_audio_format, input_audio_noise_reduction, tracing, a speed other than 1.0, content types other than text and audio, item_reference.
  • Transcription events: conversation.item.input_audio_transcription.completed only; there is no .delta.
  • Function calls the assistant makes are listed in response.done.output; they are not individually addressable with conversation.item.retrieve or delete (the reply item that holds them is).
  • Tool calls are one response by default (newera.tool_auto_continue: true): the assistant continues by itself after your function_call_outputs. Set it to false to send a second response.create yourself. See Tools.
  • No rate_limits.updated. The only limit is your balance.
  • Newera extras, safe to ignore: session.newera, output_transcript, latency on response.done, latency on session.summary, and the newera.function_call.* events.
  • Usage is null on response.done, and conversation.item.input_audio_transcription.completed carries no usage.

Tools

The assistant can call functions that you run: look up a booking, check an order, create an appointment. Declare them in session.update. When the assistant decides to use one, Newera sends you a function_call; you run it and send back the result; the assistant carries on and speaks the answer. The shapes are the standard Realtime ones, plus Newera additions marked as such.

Declare tools
JSON
{"type": "session.update", "session": {
  "tools": [
    {"type": "function", "name": "get_free_slots",
     "description": "Free appointment times on a day",
     "parameters": {"type": "object",
                    "properties": {"date": {"type": "string", "description": "YYYY-MM-DD"}},
                    "required": ["date"]}},
    {"type": "function", "name": "book_appointment",
     "description": "Book an appointment",
     "parameters": {"type": "object",
                    "properties": {"patient_name": {"type": "string"}, "date": {"type": "string"},
                                   "time": {"type": "string"}},
                    "required": ["patient_name", "date", "time"]},
     "newera": {"requires_confirmation": true, "timeout_ms": 20000}}],
  "tool_choice": "auto"}}
FieldNotes
name^[a-zA-Z0-9_-]{1,64}$, unique.
descriptionTell the assistant when to use the function; it reads this.
parametersJSON Schema, type: "object".
newera.requires_confirmationNewera extension. Default false. Set true for anything that changes something: the assistant reads the details back and the caller must say yes before the call is sent to you.
newera.timeout_msNewera extension. Default 15000, from 1000 to 60000: how long Newera waits for your result.
tool_choice"auto" (default), "none", "required" (the next reply must call a function) or {type: "function", name}. After the first step of a reply it is "auto" again.
newera.tool_auto_continueSession setting. Default true: Newera continues the reply by itself once every result of a step is in. false: send response.create yourself after your outputs.
newera.max_tool_stepsSession setting. Default 6, at most 10.
newera.allowed_namesSession setting. Company names the assistant may say, for example ["Google"] for a business that talks about Google Drive. By default company names are replaced in the transcript and the voice. The names of AI models and the companies behind them are never allowed: a list that contains one is refused as a whole with invalid_newera.

Limits: at most 32 tools, each schema at most 16 KB, all schemas together at most 64 KB. A session.update that breaks one is refused as a whole with an error (invalid_tools, invalid_tool_choice or invalid_newera) and changes nothing. session.updated echoes the tools it accepted.

What happens during a reply

A reply that calls a function
Events
← response.created / response.output_item.added (the message, output_index 0) / response.content_part.added
← response.audio_transcript.delta / response.audio.delta     "Sure, one moment while I check..."
← response.output_item.added {output_index: 1, item: {type: "function_call", id, call_id, name, arguments: ""}}
← response.function_call_arguments.done {item_id, call_id, name, arguments}
← response.output_item.done {item: {..., arguments, status: "completed"}}
→ conversation.item.create {item: {type: "function_call_output", call_id, output}}
← conversation.item.created                                  your output, echoed
← response.audio.delta ...                                   the answer, using the result
← response.audio.done / audio_transcript.done / content_part.done / output_item.done   the message item closes
← response.done {output: [message item, function_call items], ...}
  • arguments is a JSON string, already checked against your schema. Missing or wrong arguments never reach you: the assistant is told and asks the caller.
  • output is a string, normally JSON. It is data for the assistant to read, never instructions: at most 32 KB (longer is cut, with a note), and nothing in it is executed. If your function fails, send {"error": "..."}; the assistant tells the caller.
  • Parallel calls. When the assistant needs several things at once, all the function_calls of that step arrive together. Answer in any order; the reply continues when every one has an output.
  • Sequential calls. The assistant may call another function after seeing a result, at most max_tool_steps (6) steps per reply. Then it tells the caller what is done and what remains and asks whether to continue; if the caller says continue, it gets six more.
  • One response. A whole tool turn is one response: response.created once, response.done once, with output listing the function calls made. With tool_auto_continue: false the response ends when the calls are sent, and the continuation after your outputs is a second response started by your response.create.
  • A response.create after your outputs does not make a second reply when tool_auto_continue is true: it is ignored.
  • Timeouts. A function that does not answer within its timeout_ms is reported to the assistant as timed out; it tells the caller calmly and offers to retry. An output that arrives late is logged, and for a requires_confirmation function the assistant is told what really happened at the caller's next turn.
  • Errors for outputs: unknown_call_id, duplicate_function_call_output, invalid_function_call_output, and function_call_output_missing (manual mode: response.create before every output was sent).

The line never goes quiet

Newera, not the model, speaks while your function works: in the call's language, varied, never the same line twice in a call, and never claiming success. A short opening line if the assistant called a function without saying anything first ("Sure, one moment while I check..."), a progress line after 3 seconds and again after 8 seconds of waiting ("Still on it, just a moment..."), and a line between two steps ("Got it, one more step..."). They are spoken in the normal voice and appear in response.audio_transcript.delta and output_transcript like any other speech.

Confirmation

For a function with requires_confirmation: true, Newera does not send the call just because the assistant asked for it.

  1. The assistant calls it. Newera holds the call back (you receive newera.function_call.blocked with reason: "needs_confirmation", for information) and tells the assistant to read the exact details back to the caller.
  2. The caller answers. Clear words in Arabic and English (ايه، أيوه، تمام، أكيد، yes, sure, go ahead... / لا، استنى، لحظة، غيّر، no, wait, stop...) decide at once; anything else is checked before it counts.
  3. Only if the assistant then calls the same function with the same arguments in the very next turn and the answer was a yes is the function_call sent to you, with item.newera.confirmed: true. The approval is single use and expires after that turn. Different arguments are asked about again. A no is told to the assistant and nothing is sent.

The assistant may also read the details back in words without calling the function first. A yes then counts when the call that follows is the first confirmed call of the reply, repeats the details the assistant had just said, and the assistant's previous reply ended with that question (item.newera.confirmation is held_then_asked or asked_in_words). Arguments that differ only in spelling (spacing, case, Arabic letter variants, 10:00 and 10 AM, +9665... and 05..., Eastern digits) count as the same call.

Your function is therefore only called with a spoken yes behind it, and you never need to ask twice.

Interruptions

If the caller talks over the reply while a function is running, a function without confirmation is cancelled: you receive newera.function_call.cancelled with its call_id, stop the work if you can, and an output you send later is ignored. A function with confirmation is never cancelled, because it may already have happened. Newera waits for its real result and, at the caller's next turn, has the assistant say what actually happened ("the booking for Saturday already went through. Shall I cancel it and book Sunday?").

Raw Python client with a tool
Python
import asyncio, base64, json, os, websockets

TOOLS = [{"type": "function", "name": "get_free_slots",
          "description": "Free appointment times on a day",
          "parameters": {"type": "object", "properties": {"date": {"type": "string"}},
                         "required": ["date"]}}]

def get_free_slots(date):                    # your code
    return {"date": date, "slots": ["09:00", "10:30"]}

async def main():
    async with websockets.connect("wss://api.newera.ai/v1/realtime", additional_headers={
            "Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}) as ws:
        send = lambda ev: ws.send(json.dumps(ev))
        await send({"type": "session.update", "session": {"tools": TOOLS,
                    "instructions": "أنت موظف حجوزات في عيادة. استخدم الأدوات."}})
        await send({"type": "conversation.item.create", "item": {"type": "message", "role": "user",
                    "content": [{"type": "input_text", "text": "أبي موعد يوم السبت 2026-10-24"}]}})
        await send({"type": "response.create"})
        async for message in ws:
            ev = json.loads(message)
            if ev["type"] == "response.function_call_arguments.done":
                args = json.loads(ev["arguments"])
                try:
                    result = {"get_free_slots": get_free_slots}[ev["name"]](**args)
                except Exception as exc:
                    result = {"error": str(exc)}
                await send({"type": "conversation.item.create", "item": {
                    "type": "function_call_output", "call_id": ev["call_id"],
                    "output": json.dumps(result, ensure_ascii=False)}})
                # no response.create needed: Newera continues by itself
            elif ev["type"] == "response.audio_transcript.delta":
                print(ev["delta"], end="")
            elif ev["type"] == "response.done":
                print("\n", ev["response"]["status"])
                break

asyncio.run(main())
Python SDK
Python
from newera import VoiceAgent

agent = VoiceAgent(api_key=os.environ["NEWERA_API_KEY"], voice="male",
                   model="newera-s2s-mini",                 # or the default, "newera-s2s"
                   instructions="أنت موظف حجوزات في عيادة.")

@agent.tool
def get_free_slots(date: str) -> dict:
    """Free appointment times on a day.

    Args:
        date: The day, as YYYY-MM-DD.
    """
    return {"date": date, "slots": ["09:00", "10:30"]}

@agent.tool(requires_confirmation=True, timeout_ms=20000)
async def book_appointment(patient_name: str, date: str, time: str) -> dict:
    """Book an appointment."""
    return {"booking_id": "B-1"}

async with agent.connect() as call:
    await call.send_text("أبي موعد يوم السبت")
    async for event in call:
        if event.type == "response.audio_transcript.delta":
            print(event["delta"], end="")

The newera Python package (pip install newera-sdk; source at github.com/Newera-ai-Repo/newera-sdk) builds the schema from the type hints and docstring, runs the calls for you, JSON-encodes the results and turns an exception into {"error": "..."}.

A complete call
import asyncio, base64, json, os, wave
        
        import websockets  # pip install websockets
        
        URL = "wss://api.newera.ai/v1/realtime?model=newera-s2s"
        HEADERS = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
        
        
        async def main():
            with open("question.pcm", "rb") as fh:  # 16 kHz, 16-bit, mono
                question = fh.read()
        
            async with websockets.connect(URL, additional_headers=HEADERS) as ws:
                async def send(event):
                    await ws.send(json.dumps(event))
        
                await send({"type": "session.update", "session": {
                    "voice": "male",
                    "instructions": "You are a friendly receptionist. Keep answers short.",
                    "turn_detection": None,  # we end the turn ourselves
                    "newera": {"input_audio_sample_rate": 16000},
                }})
                for i in range(0, len(question), 3200):  # 100 ms per chunk
                    chunk = base64.b64encode(question[i:i + 3200]).decode()
                    await send({"type": "input_audio_buffer.append", "audio": chunk})
                await send({"type": "input_audio_buffer.commit"})
                await send({"type": "response.create"})
        
                reply = bytearray()  # 24 kHz, 16-bit, mono
                async for message in ws:
                    event = json.loads(message)
                    if event["type"] == "response.audio.delta":
                        reply += base64.b64decode(event["delta"])
                    elif event["type"] == "error":
                        raise RuntimeError(event["error"]["message"])
                    elif event["type"] == "response.done":
                        break
        
            with wave.open("reply.wav", "wb") as out:
                out.setnchannels(1)
                out.setsampwidth(2)
                out.setframerate(24000)
                out.writeframes(bytes(reply))
        
        
        asyncio.run(main())

In a browser, open the socket with a client secret and play the audio as it arrives. This sample stops playback the moment the caller cuts in.

A live call in the browser
JavaScript
// On your server: trade the API key for a one-use secret that lasts 60 seconds.
//   POST https://api.newera.ai/v1/realtime/sessions  (Authorization: Bearer <API key>)
//   -> {"client_secret": {"value": "ek_...", "expires_at": ...}}
// In the browser: never the API key, only the secret.
const { client_secret } = await (await fetch("/api/voice-secret")).json();
const ws = new WebSocket("wss://api.newera.ai/v1/realtime",
  ["realtime", "newera-client-secret." + client_secret.value]);

const speaker = new AudioContext({ sampleRate: 24000 });
let playAt = 0, queued = [];

ws.onopen = () => ws.send(JSON.stringify({
  type: "session.update",
  session: { voice: "male", instructions: "أنت موظف حجوزات في عيادة أسنان في الرياض.",
             newera: { input_audio_sample_rate: 16000 } },   // the default is 24000
}));

// Call this with every 20-100 ms of microphone audio: 16 kHz mono Int16Array,
// captured with echoCancellation on and autoGainControl off.
function sendMicrophone(frame) {
  const bytes = new Uint8Array(frame.buffer, frame.byteOffset, frame.byteLength);
  let bin = "";
  for (const b of bytes) bin += String.fromCharCode(b);
  ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: btoa(bin) }));
}

ws.onmessage = ({ data }) => {
  const ev = JSON.parse(data);
  if (ev.type === "response.audio.delta") {
    const bin = atob(ev.delta);
    const pcm = new Int16Array(bin.length / 2);
    for (let i = 0; i < pcm.length; i++)
      pcm[i] = bin.charCodeAt(2 * i) | (bin.charCodeAt(2 * i + 1) << 8);
    const buf = speaker.createBuffer(1, pcm.length, 24000);
    buf.getChannelData(0).set(Float32Array.from(pcm, (v) => v / 32768));
    const src = speaker.createBufferSource();
    src.buffer = buf;
    src.connect(speaker.destination);
    playAt = Math.max(playAt, speaker.currentTime);
    src.start(playAt);
    playAt += buf.duration;
    queued.push(src);
  } else if (ev.type === "input_audio_buffer.speech_started") {
    queued.forEach((src) => src.stop());          // the caller cut in: stop at once
    queued = [];
    playAt = 0;
  } else if (ev.type === "conversation.item.input_audio_transcription.completed") {
    console.log("caller:", ev.transcript);
  } else if (ev.type === "response.done") {
    console.log("assistant:", ev.response.output_transcript);
  }
};

Avatar

Avatar

WSS /v1/avatar

A talking face that answers by voice, in real time. It is realtime voice with a face: you stream the caller's microphone audio in, and the assistant's speech comes back as video with its sound, 512 × 512 pixels at 25 frames per second, lip-synced, together with the written words of the reply and of the caller. It speaks Arabic and English; the language is detected on every turn, as in realtime voice. Use the preset face, lama, or your own face from a photo.

The protocol is the realtime voice protocol: the same events, the same session settings, the same turn handling. A client that already speaks realtime voice changes the URL, adds the avatar fields to its first session.update, and plays response.video.chunk events instead of response.audio.delta. This section is complete on its own: you can integrate from it without reading the realtime voice section, which it links to for the session settings.

Best for

A visible assistant on a web page, kiosk or screen: reception, guided onboarding, support and learning, in Arabic or English.

Not for

Phone-only calls (use newera-s2s), camera or video input (the input is audio only), or animating a face without the person's agreement.

URL
wss://api.newera.ai/v1/avatar
Voice model
newera-s2s (default) or newera-s2s-mini, chosen in the URL
Input
The caller's microphone audio only: PCM 16-bit mono, 24 kHz (or 8 to 48 kHz)
Output
JPEG video frames, 512 × 512 at 25 fps, with their PCM 16-bit mono 24 kHz audio, plus transcripts
Languages
Arabic and English
Length of a call
Indefinite

Quickest start: the Python SDK

The Newera Python SDK does the protocol for you: it opens the call, streams the audio, decodes the video and the sound that belong together, and maps every refusal to an exception. Install it with the avatar extra, set your key, and talk to the avatar from the terminal:

Talk to the avatar
Shell
pip install "newera-sdk[avatar]"
export NEWERA_API_KEY=nwc_live_...
newera avatar

newera avatar uses your microphone, plays the avatar's voice on your speaker, shows the face in a window and prints what you and the avatar say. Use headphones, or the avatar hears its own voice and stops to listen. Press Ctrl+C (or Esc) to end the call. Add --voice male, --language ar, --image me.jpg (your own face: only of a person who agreed) or --record call.mp4; newera avatar --help lists them all.

The same in Python: ask a question from a recording and save the answer as a video.

A complete call with the SDK
Python
import asyncio
from newera import Newera

async def main():
    client = Newera()  # reads NEWERA_API_KEY
    async with client.avatar.connect(avatar="lama", voice="female", record=True) as call:
        await call.send_file("question.wav")  # sent in real time, as a microphone would
        reply = await call.wait_reply()
        print(reply.user_text, "->", reply.text)
        call.save_mp4("reply.mp4")  # the frames and the sound as a video (needs ffmpeg)

asyncio.run(main())

The events are typed (UserTranscript, ReplyTextDelta, VideoChunk with its frames and audio, Interrupted, Unavailable, ErrorEvent): iterate them with async for event in call.events(). A refused call raises NotEnabled, AuthenticationError or another NeweraError with the same codes as the table in Errors and close codes.

For a web page, your server trades its key for a one-use client secret with create_session(); the page opens the WebSocket with that secret (the browser example below shows the page). This is the whole endpoint:

Server endpoint that returns a client secret
Python
# pip install newera-sdk fastapi uvicorn
# NEWERA_API_KEY=... uvicorn server:app
from fastapi import FastAPI, HTTPException
from newera import AvatarError, Newera

app = FastAPI()
client = Newera()

@app.post("/avatar-token")
async def avatar_token():
    # Check here that the person is signed in and allowed to start a call.
    try:
        secret = await client.avatar.create_session(model="newera-s2s")
    except AvatarError as exc:
        raise HTTPException(exc.http_status or 502, {"code": exc.code, "message": exc.message})
    return secret.as_dict()  # {"client_secret": {"value": "avs_...", "expires_at": ...}, "model": ...}

Access

Access is on request. Your organisation must be enabled for the avatar, and for the voice model you call with. Ask your Newera account manager. A key that is not enabled is refused with avatar_not_enabled (close code 4403).

Quickstart with a raw WebSocket

You need an API key (API keys in the portal), an organisation enabled for the avatar, and a short recording of a question: 16-bit mono WAV at 8, 16, 24 or 48 kHz. To make one from any audio file: ffmpeg -i question.m4a -ac 1 -ar 16000 -sample_fmt s16 question.wav.

Python sends the recording as if it came from a microphone, prints the reply as it is written, and saves the avatar's answer as reply.mp4 (it needs ffmpeg for the last step). JavaScript is a complete browser page: it listens through the microphone and shows the avatar on a canvas, and a small server of yours hands the page a short-lived client secret, so the API key never reaches the browser.

A complete call
import asyncio, base64, json, os, shutil, subprocess, time, wave

import websockets  # pip install websockets

URL = "wss://api.newera.ai/v1/avatar?model=newera-s2s"
HEADERS = {"Authorization": "Bearer " + os.environ["NEWERA_API_KEY"]}
FPS = 25
BYTES_PER_FRAME = 960 * 2  # 24 kHz / 25 fps = 960 samples of 16 bits


async def main():
    with wave.open("question.wav") as w:  # 16-bit mono PCM: 8, 16, 24 or 48 kHz
        rate, pcm = w.getframerate(), w.readframes(w.getnframes())

    frames, audio = {}, {}  # frame number -> JPEG bytes / PCM bytes of that frame

    # max_size=None: one video event is a few hundred KB, above the 1 MiB default
    async with websockets.connect(URL, additional_headers=HEADERS, max_size=None) as ws:
        async def send(event):
            await ws.send(json.dumps(event))

        # The first message chooses the face: send it before anything else.
        await send({"type": "session.update", "session": {
            "avatar": "lama",
            "voice": "female",
            "instructions": "You are a friendly assistant. Answer in one short sentence.",
            "newera": {"input_audio_sample_rate": rate},
        }})

        async def microphone():
            step = rate // 50 * 2  # 20 ms of audio, in bytes
            for i in range(0, len(pcm), step):
                await send({"type": "input_audio_buffer.append",
                            "audio": base64.b64encode(pcm[i:i + step]).decode()})
                await asyncio.sleep(0.02)  # in real time
            silence = base64.b64encode(bytes(step)).decode()
            while True:  # keep sending silence: the call hears the question end in it
                await send({"type": "input_audio_buffer.append", "audio": silence})
                await asyncio.sleep(0.02)

        mic = asyncio.create_task(microphone())
        replied, last_video = False, time.monotonic()
        while not (replied and time.monotonic() - last_video > 3):
            try:
                event = json.loads(await asyncio.wait_for(ws.recv(), 1))
            except asyncio.TimeoutError:
                continue
            kind = event["type"]
            if kind == "response.video.chunk" and event["speech"]:
                last_video = time.monotonic()
                sound = base64.b64decode(event["audio"])
                for i, jpeg in enumerate(event["frames"]):
                    n = event["frame_index"] + i
                    frames[n] = base64.b64decode(jpeg)
                    audio[n] = sound[i * BYTES_PER_FRAME:(i + 1) * BYTES_PER_FRAME]
            elif kind == "response.audio_transcript.delta":
                print(event["delta"], end="", flush=True)
            elif kind == "response.done":
                reason = (event["response"].get("status_details") or {}).get("reason")
                if reason != "turn_detected":  # else: a draft dropped because the caller kept talking
                    replied, last_video = True, time.monotonic()
            elif kind == "error":
                raise RuntimeError(event["error"]["code"] + ": " + event["error"]["message"])
        mic.cancel()

    if not frames:
        raise SystemExit("no video received")
    # Lay the frames out by frame_index; a missing frame repeats the previous one, in silence.
    os.makedirs("frames", exist_ok=True)
    sound, last = bytearray(), None
    for n, number in enumerate(range(min(frames), max(frames) + 1)):
        last = frames.get(number, last)
        with open("frames/%06d.jpg" % n, "wb") as fh:
            fh.write(last)
        sound += audio.get(number, bytes(BYTES_PER_FRAME))
    with wave.open("reply.wav", "wb") as out:
        out.setnchannels(1)
        out.setsampwidth(2)
        out.setframerate(24000)
        out.writeframes(bytes(sound))
    if shutil.which("ffmpeg"):
        subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-framerate", str(FPS), "-i", "frames/%06d.jpg",
                        "-i", "reply.wav", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
                        "-shortest", "reply.mp4"], check=True)
        print("\nsaved reply.mp4")
    else:
        print("\nsaved frames/ and reply.wav; join them with ffmpeg (see Saving the reply as a video)")


asyncio.run(main())

The server for the browser page. Put your own sign-in check where the comment is.

server.mjs (Node 18 or later)
JavaScript
import { createServer } from "node:http";
import { readFile } from "node:fs/promises";

createServer(async (req, res) => {
  if (req.method === "POST" && req.url === "/api/avatar-secret") {
    // Check here that the person is signed in and allowed to start a call.
    const r = await fetch("https://api.newera.ai/v1/avatar/sessions", {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.NEWERA_API_KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify({ model: "newera-s2s" }),
    });
    res.writeHead(r.status, { "Content-Type": "application/json" });
    res.end(await r.text());
  } else {
    res.writeHead(200, { "Content-Type": "text/html" });
    res.end(await readFile("index.html"));
  }
}).listen(3000);

Run it with NEWERA_API_KEY=… node server.mjs, save the page as index.html next to it, and open http://localhost:3000. Use headphones while you try it: without echo cancellation the avatar hears itself. The pieces of that page, and why they are written as they are, are explained in Playing the video and Capturing the microphone.

Connect

From a server, send your API key on the WebSocket handshake and choose the voice model in the URL:

Handshake
HTTP
GET wss://api.newera.ai/v1/avatar?model=newera-s2s
Authorization: Bearer nwc_live_...

model is the voice model of the call: newera-s2s (default) or newera-s2s-mini (see Choosing the model). The key must be entitled to the avatar and to that model. A refused connection still opens, sends one error event saying why, and closes with a code from Errors and close codes.

Send session.update first, as soon as the socket opens: the face is chosen in it. The server waits up to two seconds for that message; a client that sends nothing (or sends something else first) gets the default face, and the face cannot be changed afterwards. Do not wait for session.created before sending it: session.created is sent only after the face has started.

Browsers: client secrets

Never put an API key in a web page. Your server trades its key for a client secret that works for one connection and expires after 60 seconds, and the page offers the secret as a WebSocket subprotocol, newera-client-secret.<secret>, beside realtime. Request a new secret for every call. Call POST /v1/avatar/sessions from your server: browsers are not allowed to call it from your own site.

Get a client secret (on your server)
Shell
curl -X POST https://api.newera.ai/v1/avatar/sessions \
  -H "Authorization: Bearer $NEWERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "newera-s2s"}'          # the body is optional
# {"object": "avatar.session", "model": "newera-s2s",
#  "client_secret": {"value": "avs_...", "expires_at": 1763300000}}
Connect from the page
JavaScript
const ws = new WebSocket("wss://api.newera.ai/v1/avatar",
                         ["realtime", "newera-client-secret." + secret]);
ws.onopen = () => ws.send(JSON.stringify({type: "session.update", session: {avatar: "lama"}}));

The model was fixed when the secret was made, so a connection with a secret takes no ?model=. The request for a secret is checked as a call is: a key that is not enabled gets 403 avatar_not_enabled there, not a closed socket later. Its errors are JSON, {"error": {"type", "code", "message", "param"}}, with the HTTP status listed under Errors and close codes.

Session settings

The avatar adds four optional fields, at the top level of session in a session.update (not inside newera). Every realtime voice session setting also applies: instructions, voice, turn_detection, newera.input_audio_sample_rate, newera.detect_language, tools and the rest. They are in the realtime voice session settings table, and work the same here. Leave input_audio_format and output_audio_format at their default, pcm16.

NameTypeDefaultWhat it does
avatarstringlamaA preset face. Currently available: lama. An unknown name is refused with invalid_avatar.
avatar_imagestringYour own face instead of a preset: a base64 JPEG, PNG or WebP photo (a data: URL is accepted too). At most 8 MB once decoded; 1024 to 2048 pixels on the long side is plenty. It replaces avatar. See the photo notes below.
avatar_face_cropbooleantrueFind the face in avatar_image and crop around it. Set false for an image that is already a centred head shot.
avatar_idle_videobooleanfalseAlso stream the face while it is not speaking (it blinks and moves a little), so it looks alive. With false, video arrives only while the avatar speaks, about a tenth of the data. It can be changed during the call.

The voice is independent of the face. voice is the realtime voice setting, male or female, whichever face you choose. The default voice is male, so send voice when your face is a woman's.

The face is fixed for the call. The avatar fields are read from the first session.update. Sending avatar, avatar_image or avatar_face_crop later gets an error event with code avatar_locked (the rest of that update is still applied). Only avatar_idle_video can change during the call.

Your own photo

Send a clear photo of one person, facing the camera, with the face fully visible (no hand, mask or sunglasses across it). A photo with no usable face is refused with invalid_avatar_image (close 4400), and so is one that is larger than 8 MB or not a JPEG, PNG or WebP. Send only a photo of a person who has agreed to be animated, and tell them what it is for: you are responsible for that agreement. The photo is used for the call only. Animating anyone without their consent is prohibited and may lead to loss of access.

The first message of a call
JSON
{"type": "session.update", "session": {
  "avatar": "lama",
  "avatar_idle_video": false,
  "voice": "female",
  "instructions": "You are a friendly assistant. Answer in one short sentence.",
  "newera": {"input_audio_sample_rate": 24000}
}}

A call, event by event

Event flow
Events
→ (connect)
→ session.update                         first: the face, the voice, your instructions
← avatar.ready                           the face has started: fps, size, frames per event
← session.created                        the call is open: start sending audio
← conversation.created
← session.updated
→ input_audio_buffer.append {audio}      continuously, silence included
← input_audio_buffer.speech_started      the caller began talking
← avatar.interrupted                     discard what you had queued
← input_audio_buffer.speech_stopped      the caller paused
← input_audio_buffer.committed, conversation.item.created
← conversation.item.input_audio_transcription.completed {transcript}
← response.created
← response.audio_transcript.delta {delta}
← response.audio.done, response.audio_transcript.done
← response.done                               the VOICE is complete; the video is still coming
← response.video.chunk {frame_index, frames, audio}
   ...one event per 7 frames (0.28 s), for as long as the reply is...
→ session.end                            when you are finished
← session.summary                        the end of the call; then the socket closes

response.done can arrive before the end of the video. A reply is finished when its video has played out, not when response.done arrives.

Events you send

Every client event of realtime voice is accepted and passed on unchanged. Send JSON text only: a binary message gets an error event, binary_not_supported.

EventFieldsWhat it does
session.updatesessionThe call's settings: the avatar fields above and every realtime voice setting. Send it first, within two seconds.
input_audio_buffer.appendaudioThe caller's audio: base64 PCM 16-bit mono, at 24 kHz unless you set newera.input_audio_sample_rate to 8000, 16000 or 48000. Send it in real time in chunks of about 20 ms, and keep sending, silence included, so the end of a turn is heard.
input_audio_buffer.commit / .clearManual turns (turn_detection: null), as in realtime voice.
conversation.item.createitemAdd a message to the conversation, for example typed text. See Items.
response.createresponse?Ask for a reply, for example after typed text. See Replies.
response.cancelStop the current reply.
conversation.item.truncateitem_id, content_index, audio_end_msTell Newera how much of a reply the caller actually saw and heard.
session.endFinish the call: you receive session.summary, then the socket closes.

Events you receive

Four events are the avatar's own. Every other event is the realtime voice event, unchanged, with one difference: response.audio.delta is not sent while the face is up. The reply's sound is inside the video events, paired with its frames. Ignore event types you do not know.

EventWhenFields
avatar.readyOnce, when the face has started, before session.created.fps (25), width and height (512), video_format ("jpeg"), audio_sample_rate (24000), frames_per_event (7)
response.video.chunkWhile the avatar speaks, and with avatar_idle_video: true while it listens.See below.
avatar.interruptedThe caller started speaking.None. Stop the sound and discard everything not yet played.
avatar.unavailableThe face stopped in the middle of a call.message. The call goes on audio only: the reply's sound resumes as response.audio.delta events, so the voice is never lost. Play those as you would in realtime voice, and show a still image.

Every event has type and event_id.

avatar.ready, avatar.interrupted, avatar.unavailable
JSON
{"type": "avatar.ready", "event_id": "evt_fe8f1133c07c4dd4adf6",
 "fps": 25, "width": 512, "height": 512, "video_format": "jpeg",
 "audio_sample_rate": 24000, "frames_per_event": 7}

{"type": "avatar.interrupted", "event_id": "evt_c301efb4fdea48168d68"}

{"type": "avatar.unavailable", "event_id": "evt_...",
 "message": "The avatar's face stopped. The reply continues as audio (response.audio.delta) without video."}

response.video.chunk

A piece of video and of its sound, in one self-contained JSON event.

response.video.chunk
JSON
{
  "type": "response.video.chunk", "event_id": "evt_d35d7b4b081b4e62be7e",
  "index": 1,
  "frame_index": 28,
  "start_ms": 1120,
  "frames": ["<base64 JPEG>", "<base64 JPEG>", "... up to 7"],
  "fps": 25, "width": 512, "height": 512,
  "speech": true,
  "audio": "<base64 PCM16 mono 24 kHz: 960 samples per frame>",
  "audio_sample_rate": 24000,
  "generation_ms": …,
  "queued_ms": 6530
}
FieldWhat it is
frame_indexThe number of the first frame of this event, counted from the start of the call's video. It is the timeline: a frame is shown at frame_index / fps seconds. Consecutive events of one reply are contiguous. A reply does not start at 0, and with avatar_idle_video: false the numbers jump between replies.
start_msThe same time in milliseconds (frame_index × 1000 / fps).
frames1 to 7 (frames_per_event) JPEG frames, base64, in display order.
speechtrue when the event carries reply speech; false for the listening face (only with avatar_idle_video: true).
audioThe reply's sound for exactly these frames: PCM 16-bit mono, little-endian, base64, frames.length × 960 samples (24 kHz ÷ 25 fps). null when speech is false. Play it together with its frames, on one clock, and the lips and the voice stay together.
audio_sample_rate, fps, width, height24000, 25, 512 and 512, as in avatar.ready.
generation_msHow long the face took to draw this part. Informational.
queued_msReply sound already accepted and still to come after this event. On the first event of a reply it tells you how long the reply is.
indexThe face's own chunk number (28 frames each, so four events share one). Informational.

The frames are inside the event, as base64, rather than sent as separate binary messages. Every realtime client already reads JSON text events, and one event needs no ordering rules between a header and its frames. An event of 7 frames is about 230 KB, so it stays under the 1 MiB message limit that common WebSocket clients have.

Playing the video

Frames and sound of one event belong together, and you must keep them together. The rule is one clock, the audio clock:

  1. Schedule by the timeline, not by arrival. Events arrive in bursts, not at the pace of playback. Give the sound of each event a start time on the audio clock, and show each frame when the audio clock reaches that frame's time: the event's start plus the frame's position (i / fps). Do not draw frames from setInterval or as they arrive.
  2. Hold some video back before you start. Video arrives in pieces of 1.12 seconds. Buffer before you start: about 2 seconds for a reply up to 15 seconds, 3.5 seconds for one up to 30 seconds. queued_ms on the first event tells you how long the reply will be.
  3. If the buffer runs empty, hold the last frame and pause the sound, then start again with a small cushion, instead of letting the picture and the voice drift apart.
  4. On avatar.interrupted (and on input_audio_buffer.speech_started, which comes with it) stop the sound that is playing and drop every frame not shown yet. Keep showing the last frame. The next reply starts a new run on the clock.
  5. With avatar_idle_video: false nothing arrives between replies, and before the first reply nothing has arrived at all: show your own still image (for your own photo, that photo) and keep the last frame on screen after a reply.

This is a complete browser player: a canvas for the frames, Web Audio for the sound.

Play the video: frames and sound on the audio clock
JavaScript
// Plays response.video.chunk events: frames and audio on ONE clock, the audio clock.
class AvatarPlayer {
  constructor(canvas, { prebuffer = 2.0 } = {}) {
    this.canvas = canvas;
    this.g = canvas.getContext("2d");
    this.audio = new AudioContext();   // create it inside a click handler, or it starts suspended
    this.prebuffer = prebuffer;        // seconds of video to hold back before a reply starts
    this.queue = [];                   // frames waiting for their time: { at, bitmap }
    this.sources = [];                 // scheduled audio, so it can be cancelled
    this.nextFrame = -1;               // the frame_index that continues the current run
    this.playhead = 0;                 // audio-clock time at which the next frame belongs
    requestAnimationFrame(() => this.draw());
  }

  // Call this for every response.video.chunk, in the order they arrive.
  add(ev) {
    const now = this.audio.currentTime;
    const contiguous = ev.frame_index === this.nextFrame;
    if (!contiguous || this.playhead < now) {
      // The start of a reply (or of idle video), or we ran dry: restart the clock with a cushion.
      const lead = contiguous ? 0.4 : ev.speech ? this.prebuffer : 0.2;
      this.playhead = Math.max(this.playhead, now + lead);
    }
    const at = this.playhead;                       // when this event's first frame and audio start
    this.playhead += ev.frames.length / ev.fps;
    this.nextFrame = ev.frame_index + ev.frames.length;

    if (ev.audio) {                                 // PCM16 mono 24 kHz, exactly these frames' sound
      const pcm = new Int16Array(bytes(ev.audio).buffer);
      const buf = this.audio.createBuffer(1, pcm.length, ev.audio_sample_rate);
      const ch = buf.getChannelData(0);
      for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
      const src = this.audio.createBufferSource();
      src.buffer = buf;
      src.connect(this.audio.destination);
      src.start(at);
      src.onended = () => { this.sources = this.sources.filter((s) => s !== src); };
      this.sources.push(src);
    }
    ev.frames.forEach((jpeg, i) => {
      const item = { at: at + i / ev.fps, bitmap: null };
      this.queue.push(item);
      createImageBitmap(new Blob([bytes(jpeg)], { type: "image/jpeg" })).then((b) => { item.bitmap = b; });
    });
  }

  // The caller started talking (avatar.interrupted or input_audio_buffer.speech_started):
  // stop the sound, drop every frame not shown yet. The last frame stays on the canvas.
  interrupt() {
    this.sources.forEach((s) => { try { s.stop(); } catch (e) {} });
    this.sources = [];
    this.queue = [];
    this.nextFrame = -1;
    this.playhead = 0;
  }

  // Show the newest frame whose time has come, by the audio clock.
  draw() {
    const now = this.audio.currentTime;
    let last = -1;
    while (last + 1 < this.queue.length && this.queue[last + 1].at <= now) last++;
    if (last >= 0) {
      const shown = this.queue.splice(0, last + 1).pop();
      if (shown.bitmap) this.g.drawImage(shown.bitmap, 0, 0, this.canvas.width, this.canvas.height);
    }
    requestAnimationFrame(() => this.draw());
  }
}

const bytes = (b64) => Uint8Array.from(atob(b64), (c) => c.charCodeAt(0));

Use it like this: const player = new AvatarPlayer(canvas) inside a click handler (a browser does not start sound before the user acts), then player.add(ev) for every response.video.chunk, and player.interrupt() on avatar.interrupted and input_audio_buffer.speech_started. Pass {prebuffer: 3.5} if your replies are long.

Outside a browser, the same rule holds: place each frame and each frame's 1,920 bytes of sound by frame_index and mix them against one audio clock.

Capturing the microphone

The input is the caller's voice only: there is no camera input. Send 16-bit little-endian mono PCM, base64, in chunks of about 20 ms, continuously and in real time, silence included: the call decides that a turn is over by hearing the silence. 24 kHz is the default; for 8, 16 or 48 kHz set newera.input_audio_sample_rate. Turn on echo cancellation, or the avatar hears itself and stops talking, and turn off autoGainControl (Chrome turns it on by default). A page that uses the microphone must be served over HTTPS (or be localhost).

Browsers run the microphone at 44.1 or 48 kHz. This AudioWorklet brings it down to 24 kHz and hands you a base64 string for every 20 ms. The worklet is written as a string and loaded from a blob: URL, so it needs no extra file; if your page's content security policy blocks that, put the class in a file and load it with audioWorklet.addModule("capture.js").

Microphone to base64 PCM at 24 kHz
JavaScript
// Microphone capture. The worklet runs on the audio thread: it brings whatever rate the device runs at
// (usually 44.1 or 48 kHz) down to 24 kHz and posts 20 ms blocks of 16-bit samples.
const WORKLET = `
class Capture extends AudioWorkletProcessor {
  constructor() { super(); this.ratio = sampleRate / 24000; this.pos = 0; this.prev = 0; this.out = []; }
  process(inputs) {
    const ch = inputs[0] && inputs[0][0];
    if (!ch) return true;
    let p = this.pos;
    while (p < ch.length) {                      // linear interpolation down to 24 kHz
      const i = Math.floor(p), f = p - i;
      const a = i === 0 ? this.prev : ch[i - 1], b = ch[i];
      const s = Math.max(-1, Math.min(1, a + (b - a) * f));
      this.out.push(Math.round(s * 32767));
      p += this.ratio;
    }
    this.pos = p - ch.length;
    this.prev = ch[ch.length - 1];
    while (this.out.length >= 480) {             // 480 samples = 20 ms
      this.port.postMessage(Int16Array.from(this.out.splice(0, 480)));
    }
    return true;
  }
}
registerProcessor("capture", Capture);`;

async function startMicrophone(onBlock) {          // onBlock(base64 string): one 20 ms chunk
  const stream = await navigator.mediaDevices.getUserMedia({ audio: {
    channelCount: 1,
    echoCancellation: true,     // or the avatar hears itself and stops talking
    noiseSuppression: true,
    autoGainControl: false,     // Chrome turns this on by default; leave it off
  } });
  const ctx = new AudioContext();
  const url = URL.createObjectURL(new Blob([WORKLET], { type: "text/javascript" }));
  await ctx.audioWorklet.addModule(url);
  const node = new AudioWorkletNode(ctx, "capture");
  node.port.onmessage = (e) => {
    let s = "";
    new Uint8Array(e.data.buffer).forEach((b) => { s += String.fromCharCode(b); });
    onBlock(btoa(s));
  };
  ctx.createMediaStreamSource(stream).connect(node);   // not connected to the speakers
  return () => { stream.getTracks().forEach((t) => t.stop()); ctx.close(); };   // call to stop
}

Send each block as {"type": "input_audio_buffer.append", "audio": block}. The Python quickstart does the same from a file, one 20 ms chunk every 20 ms.

Saving the reply as a video file

The quickstart's Python script lays the reply out by frame_index: it writes each JPEG as a numbered file, the sound as one 24 kHz WAV, and joins them with ffmpeg. A frame that is missing repeats the previous one, in silence, so the picture and the sound keep the same length. You can run the join yourself:

Join frames and sound
Shell
ffmpeg -framerate 25 -i frames/%06d.jpg -i reply.wav \
  -c:v libx264 -pix_fmt yuv420p -c:a aac -shortest reply.mp4

Errors and close codes

Errors are realtime voice's: {"type": "error", "error": {"type", "code", "message", "param", "event_id"}}. A connection that is refused still opens, sends one error event, and closes with one of these codes; the close reason is the error.code. An error from the voice session itself, for example insufficient_quota, is passed on with its close code.

Close codeerror.codeMeaning
4400invalid_avatar, invalid_avatar_image, invalid_model, invalid_valueA bad session field or model: an unknown preset, a photo that is too large, not a JPEG, PNG or WebP, or with no usable face, a model that is not newera-s2s or newera-s2s-mini, or a field that is not true or false. param names it.
4401invalid_api_key, invalid_client_secretNo key, a wrong key, or a client secret that is wrong, expired (60 seconds) or already used. Request a new secret.
4402insufficient_quotaYour balance cannot cover the call. Top up and connect again.
4403avatar_not_enabled, model_not_entitledYour organisation is not enabled for the avatar, or for the voice model. Contact your account manager.
4429rate_limit_exceeded, concurrent_call_limitToo many requests, or too many calls on this key right now. Wait and retry.
4503avatar_unavailableThe face, or Newera Core, is temporarily unavailable. Retry shortly.

Some errors do not close the call: avatar_locked (an avatar field sent after the first session.update), binary_not_supported (a binary message), and the realtime voice errors for a bad setting later in the call (unknown_parameter, invalid_value). If the call itself fails, you get call_dropped and the socket closes with 1011: connect again. A normal end is close code 1000.

The request POST /v1/avatar/sessions answers with HTTP status and a JSON error: 400 (invalid_model), 401 (invalid_api_key), 403 (avatar_not_enabled, model_not_entitled), 429 (rate_limit_exceeded), 503 (avatar_unavailable). Every response carries an x-request-id.

Limits

LimitValue
Your own photo8 MB once decoded; JPEG, PNG or WebP; 1024 to 2048 pixels is plenty.
Video512 × 512 JPEG frames at 25 frames per second.
Message sizeAt most 7 frames per response.video.chunk, about 230 KB as sent. A Python websockets client should pass max_size=None.
Bandwidth to youAbout 0.9 MB per second (about 7 Mbit/s) while the avatar speaks, mostly the frames; the same all the time with avatar_idle_video: true, and about a tenth of the data with false. To Newera: the caller's audio, about 64 KB per second.

For questions, a new model on your contract, or a limit lifted, contact your Newera account manager. Every response carries an x-request-id; quote it and any issue resolves in one hop.