The architecture of a real-time voice AI agent: what happens between question and answer

01The flow of a call in ten steps

A call follows a precise sequence of ten steps. First, a visitor calls the museum's number. The institution keeps its own phone number: either the existing number redirects to the telephony platform's number, or it is ported directly. The caller perceives no change. Second, the telephony platform receives the call and contacts AI ARTEDUSA. Third, the system identifies the institution in under 200 milliseconds, based on the called phone number. This identification enables instant loading of the institution's specific configuration: its voice, default language, activated tools and knowledge base. Fourth, a bidirectional audio connection opens in real time via a real-time connection protocol that remains active throughout the call's duration. Fifth, the visitor's voice is transcribed into text by the speech recognition engine, with latency under 500 milliseconds. Sixth, the transcribed text is sent to the language model with the institution's full context. Seventh, the language model reasons, analyses the visitor's intent and uses its tools if necessary: booking a visit, searching for information in the knowledge base, checking a price. Eighth, the textual response is converted into natural speech by the text-to-speech engine, with latency under 500 milliseconds. Ninth, the visitor hears the response and the conversation continues fluidly and naturally. Tenth, at the end of the call, automatic post-processing generates a summary, a satisfaction score and follow-up actions.

02The technology stack, component by component

Each step in this flow relies on a specific technology component. The user interface, the dashboard where the museum director configures the agent and reviews statistics, is built with Next.js 16, React 19 and TypeScript. The application server that orchestrates calls and coordinates all components is an API built with our API in Python, deployed on the our server server. The business database, which stores institutions, calls, client records and configurations, is our secure database 17 with data isolation through per-institution data isolation: each institution sees only its own data. The cache, which accelerates configuration loading and manages rate limiting, is our caching system 7 with a two-minute time-to-live for configurations and our verification algorithm-hashed keys to ensure no personal data transits in plain text.

03Conversational intelligence

The agent's brain is the our artificial intelligence language model, orchestrated via our AI orchestrator. This choice is deliberate. The model must be fast enough to maintain acceptable conversational latency, between one and three seconds of reasoning, while being capable enough to understand complex requests and use its tools autonomously. The conversation window retains the last twenty messages, allowing the agent to remember everything said during the call. The maximum depth of nested tools is five levels, and the agent can make up to fifteen tool calls per speaking turn. The maximum input character count is 2,000.

04The voice pipeline: from speech to text and text to speech

Voice-to-text transcription relies on our voice transcription engine, a speech recognition engine designed for real-time operation. The visitor's voice is transmitted continuously via the real-time connection, and our transcription engine returns transcribed text with latency under 500 milliseconds. Text-to-speech synthesis, which converts the AI's textual response into audible voice, uses our speech synthesis engine, a multilingual engine that produces natural voices in fifteen languages. Each language has its own voice, ensuring authentic pronunciation and intonation. The latency of this conversion is also under 500 milliseconds. The telephony platform managing the connection between the traditional phone network and the digital audio stream is our telephony partner, via its Media Streams service providing a bidirectional real-time audio channel.

05The knowledge base: when the museum speaks through AI

Before the visitor even asks their first question, the agent already possesses deep knowledge about the institution. This knowledge comes from the RAG knowledge base, fed by documents the institution has uploaded through its dashboard. Accepted formats are PDF, Word and plain text, with a maximum size of 10 megabytes per file. Each document is split into segments of intelligent segments with an overlap of overlap to preserve context between segments. Each segment is transformed into a mathematical vector of high-dimensional by the our vectorisation engine 2 model. These vectors are stored in our vector database, a vector database that enables semantic search: the system understands the meaning of words, not just their form. Automatic deduplication by our verification algorithm hash eliminates duplicates. During a call, the agent queries this database and retrieves the five most relevant results, with a minimum score of 0.8 out of 1.0. The three most relevant documents are automatically injected into the AI's context at the beginning of each call.

06Resilience: when things go wrong

A real-time voice system cannot afford to crash in the middle of a call. AI ARTEDUSA integrates several resilience mechanisms at every level of the architecture. If the primary language model, our AI model, does not respond, the system automatically switches to the fallback model, a backup model, without the caller perceiving any interruption. If three consecutive language model errors occur during the same call, the agent ends the call gracefully with an apology message, rather than leaving the visitor in silence. If the server crashes during a call's post-processing, the orphan call recovery mechanism automatically resumes processing on restart. If our payment partner billing fails, an automatic retry is triggered every hour. If text-to-speech synthesis encounters a problem, automatic reconnection is launched, with an alert triggered after two or more consecutive failures.

07Performance figures

End-to-end latency, meaning the total time between the moment the visitor finishes asking their question and the moment they hear the beginning of the response, sits between two and four seconds. This delay is comparable to a human thinking before answering. The database connection pool manages thirty simultaneous connections, with twenty in reserve and ten in overflow. The maximum call duration is five minutes, a parameter configurable by the institution. The number of simultaneous calls per institution defaults to five, is configurable and has no technical limit on the server side. The number of monthly calls per institution defaults to five hundred, also configurable. Standard API endpoints are protected by rate limiting of one hundred requests per minute. Telephony webhooks accept three hundred requests per minute. Document uploads and logins are limited to ten requests per minute each. All monitoring relies on our monitoring system for error tracking and our tracing system for distributed tracing.

08Architecture in service of art

This technical architecture is not an engineering exercise for its own sake. Every component, every resilience mechanism, every latency optimisation was designed for a single objective: enabling a visitor to call a museum, gallery, auction house or theater and receive a competent, natural and immediate answer, in their language, at any hour. Consumption-based pricing at 0.15 dollars per minute, with an actual cost of approximately 0.03 dollars per minute and a of 80 percent, makes this technology accessible to any cultural institution regardless of size. GDPR compliance and preparation for the EU AI Act, which comes into force in August 2026, ensure that this adoption takes place within the European regulatory framework. File storage is provided by our secure storage, S3-compatible, and transactional emails are sent via SMTP with Jinja2 templates.

Discover ai.artedusa: https://ai.artedusa.com/