BYOI (Bring Your Own Infrastructure)

Connect your own infrastructure components for complete control over your chatbot's backend.

Overview

BYOI allows you to replace WizChat's default infrastructure with your own:

• Custom LLM - Use your own OpenAI-compatible endpoint (or several, one per role)
• Custom Embeddings - Use your own embedding service
• Custom Vector Store - Use your own PostgreSQL with pgvector
• Custom FalkorDB - Use your own graph database for GraphRAG
• Custom Conversations DB - Use your own PostgreSQL for chat history

Who Is This For?

BYOI is ideal for:

• Enterprise customers with data residency requirements
• Organizations with existing AI infrastructure investments
• Teams requiring specific model configurations
• Companies with compliance needs requiring self-hosted components

Paid Feature

BYOI is available on Pro, Pro+, Business, and Enterprise plans.

Supported Components

Custom LLM

Use any OpenAI-compatible API endpoint as your language model. You can register a single endpoint, or several and assign a different model to each role — see Using more than one model.

Compatible with:

• Azure OpenAI
• AWS Bedrock (via proxy)
• Google Vertex AI (via proxy)
• Ollama
• vLLM
• LocalAI
• Any OpenAI-compatible endpoint

Configuration:

Field	Description
API Endpoint	Base URL (e.g., https://your-api.com/v1)
API Key	Authentication key
Model Name	Model identifier (e.g., gpt-4o, llama-3.1-70b)

Custom Embeddings

Use your own embedding service for document vectorization.

Compatible with:

• Azure OpenAI Embeddings
• Cohere Embeddings
• Voyage AI
• Any OpenAI-compatible embedding endpoint

Configuration:

Field	Description
API Endpoint	Base URL for embeddings
API Key	Authentication key
Model Name	Embedding model (e.g., text-embedding-3-large)
Dimensions	Vector dimensions (default: 3072)

Custom Vector Store

Use your own PostgreSQL database with pgvector extension for document storage.

Requirements:

• PostgreSQL 14+
• pgvector extension installed
• SSL connection supported

Configuration:

Field	Description
Host	Database host
Port	Database port (default: 5432)
Database	Database name
Username	Database user
Password	Database password
SSL Mode	SSL connection mode

note

WizChat will create required tables automatically on first use.

Custom FalkorDB

Use your own FalkorDB instance for GraphRAG capabilities.

Requirements:

• FalkorDB 4.0+
• Redis-compatible connection

Configuration:

Field	Description
Host	FalkorDB host
Port	FalkorDB port (default: 6379)
Password	Connection password (optional)
SSL	Enable SSL connection

Custom Conversations Database

Use your own PostgreSQL for storing chat conversations.

Requirements:

• PostgreSQL 14+
• SSL connection supported

Configuration:

Field	Description
Host	Database host
Port	Database port (default: 5432)
Database	Database name
Username	Database user
Password	Database password
SSL Mode	SSL connection mode

Preparing Your Self-Hosted LLM Endpoint

When you add a custom LLM, WizChat runs a one-time capability probe against your endpoint — it confirms the endpoint works and detects what it supports (tools/function calling, JSON mode, streaming, vision). The probe runs from WizChat's servers, not your browser, so your endpoint needs a few things set up correctly.

Endpoint requirements

• Publicly reachable over HTTPS. Because the probe connects from WizChat's cloud, localhost, private/internal IP ranges (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12), link-local, and cloud-metadata addresses are rejected for security. Expose the endpoint behind a public HTTPS URL (reverse proxy, cloud load balancer, or a tunnel).
• Serve the OpenAI-compatible API under /v1. Enter the base URL including /v1 — e.g. https://llm.example.com/v1. If you omit it, WizChat may auto-append /v1.
• Expose GET /v1/models. WizChat uses it to normalize the URL and to show your available model names if the one you entered isn't found.
• Use the exact model id your server reports at /v1/models (e.g. Qwen/Qwen2.5-7B-Instruct), not a friendly alias.
• The probe can take up to ~2 minutes on a cold or slow endpoint. That's expected — it isn't a hang.

Using a tunnel (ngrok / Cloudflare Tunnel)?

Keep the tunnel running while you set up and test. An offline tunnel returns an HTML error page, which the probe reports as "endpoint appears offline."

What the probe detects (and how to enable each)

Only a basic chat completion is required — if that works, your endpoint is accepted. Every other capability is detected, and if it's missing that feature simply falls back to WizChat's platform model, so your chatbot still works.

Capability	Used for	How to enable on your server
Chat completion (required)	All text generation	Serve POST /v1/chat/completions
Tools / function calling	MCP tools, agent actions	Launch with a tool-call parser (see examples)
JSON mode	Structured extraction, classifiers	Support response_format: {"type":"json_object"} / structured outputs
Streaming	Live token-by-token replies	Support stream: true (SSE)
Vision	Answering questions about images	Serve a vision model (VLM) that accepts image data: URLs
Audio input	Listening to uploaded audio directly (otherwise audio is transcribed)	Serve a chat model that accepts an input_audio content part
Video	Answering questions about uploaded video (required — no platform fallback)	Serve a chat model that accepts a video_url content part

Server setup examples

vLLM

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice --tool-call-parser hermes
# OpenAI-compatible API at http://<host>:8000/v1

SGLang

python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct \
  --tool-call-parser qwen25 --host 0.0.0.0
# OpenAI-compatible API at http://<host>:30000/v1

Ollama

ollama serve            # OpenAI-compatible API at http://<host>:11434/v1
ollama pull llama3.1    # model name: "llama3.1"

For tools, use a tool-capable model; for vision, pull a vision model such as llava.

LocalAI — start the server and use its /v1 base URL; the model name is the configured model id.

Tools & vision are model- and flag-dependent

• Tools: most servers only emit tool calls when launched with the matching parser flag (vLLM --enable-auto-tool-choice --tool-call-parser, SGLang --tool-call-parser) and the model's tool chat template. Without it, WizChat detects no tool support and routes tool-using steps to the platform model.
• Vision: requires a vision-language model (e.g. Qwen-VL, Llava). A text-only model will show vision as unsupported.

Using more than one model (Endpoint Pool)

You can register several endpoints and choose which one handles each part of a conversation — for example a fast model for the main answer, a stronger model for code, and an image-capable model for pictures. Open your chatbot's settings, find the BYOI panel, and use the Endpoint Pool:

1. Add endpoint — give it a name, its base URL (ending in /v1), the model name, and an API key if it needs one. WizChat runs the capability check (the Test & Probe button) and shows what the endpoint supports.
2. Role routing — point each role at one of your endpoints, or leave it on Platform default to keep using WizChat's managed model.

Role	Handles
Chat answers	The main answer your users see
Utility	Rephrasing questions, summarizing, and sorting requests
Code generation	Running code and database queries to build an answer
Quality evaluation	Automatic answer-quality checks
Excel & CSV analysis	Understanding the structure of uploaded spreadsheets
Vision	Answering questions about images (needs an image-capable model)
Speech-to-text	Transcribing microphone input (see below)

Anything left on Platform default keeps working with WizChat's model, so you can move one role at a time and test as you go.

tip

A role is only offered an endpoint that can actually serve it — for example Vision only appears for image-capable endpoints. If an endpoint can't do what a role needs, that role stays on the platform default.

Changes take effect right away for new chats. A chat that's already open picks them up the next time you return to its tab.

Voice input (speech-to-text)

By default the microphone uses your browser's built-in speech recognition or WizChat's transcription. Assign the Speech-to-text role to one of your endpoints and the in-chat microphone transcribes through your model instead. Two kinds of endpoint work:

• A dedicated transcription model — a Whisper-compatible endpoint that serves POST /v1/audio/transcriptions.
• A multimodal model that accepts audio — a chat model that can listen to an audio clip and write down what it hears.

You can set an optional language hint if your users speak a specific language.

The recorded audio is relayed to your endpoint server-side — it still passes through WizChat's servers in transit, but it is only ever sent to your own model, never to the platform's speech-to-text provider.

• Fail-closed. When a speech-to-text endpoint is configured, WizChat transcribes through it or returns an error; it never silently falls back to the platform service.
• Loading state. The microphone button is disabled and shows a "Preparing microphone…" tooltip until the chatbot's configuration loads, then enables. If the configuration fails to load, the mic stays disabled (a fail-safe, so voice input can't bypass a configured endpoint) — reload the page to retry.

Setting Up BYOI

Step 1: Create a New Chatbot

You can turn BYOI on while creating a chatbot, and manage it any time afterwards from the chatbot's settings:

1. Go to Dashboard > Chatbots
2. Create a new chatbot, or open an existing one and go to its settings
3. Find the "BYOI (Bring Your Own Infrastructure)" panel

note

Your first endpoint can be set up at creation. The full Endpoint Pool and Role routing (assigning models to each role, and adding more endpoints) are managed from an existing chatbot's settings — see Using more than one model above.

Step 2: Enable and Configure Components

1. Toggle "Enable BYOI" to expand the configuration panel
2. For each component you want to customize:
   • Toggle the component switch (LLM, Embeddings, Vector Store, etc.)
   • Fill in the required connection details
   • Click "Test Connection" to validate
3. Complete the rest of the chatbot creation form
4. Click "Create Chatbot"

Step 3: Verify Configuration

After creating the chatbot:

1. Check that all BYOI components show green checkmarks
2. Test the chatbot with a sample query
3. Verify data flows to your infrastructure correctly

Environment Variables

When deploying your chatbot, BYOI settings are injected as environment variables:

LLM:

CUSTOM_LLM_ENDPOINT=https://your-api.com/v1
CUSTOM_LLM_API_KEY=***
CUSTOM_LLM_MODEL=gpt-4o

Embeddings:

CUSTOM_EMBEDDINGS_ENDPOINT=https://your-api.com/v1
CUSTOM_EMBEDDINGS_API_KEY=***
CUSTOM_EMBEDDINGS_MODEL=text-embedding-3-large
CUSTOM_EMBEDDINGS_DIMENSIONS=3072

Vector Store:

CUSTOM_PGVECTOR_HOST=your-db.com
CUSTOM_PGVECTOR_PORT=5432
CUSTOM_PGVECTOR_DATABASE=vectors
CUSTOM_PGVECTOR_USER=***
CUSTOM_PGVECTOR_PASSWORD=***
CUSTOM_PGVECTOR_SSL_MODE=require

FalkorDB:

CUSTOM_FALKORDB_HOST=your-falkor.com
CUSTOM_FALKORDB_PORT=6379
CUSTOM_FALKORDB_PASSWORD=***
CUSTOM_FALKORDB_SSL=true

Conversations:

CUSTOM_CONVERSATIONS_HOST=your-db.com
CUSTOM_CONVERSATIONS_PORT=5432
CUSTOM_CONVERSATIONS_DATABASE=conversations
CUSTOM_CONVERSATIONS_USER=***
CUSTOM_CONVERSATIONS_PASSWORD=***
CUSTOM_CONVERSATIONS_SSL_MODE=require

Security

Credential Storage

• All credentials are encrypted at rest
• Credentials are stored in secure, isolated storage
• Access is limited to your account only
• Credentials can be rotated at any time

Network Security

• Connections use SSL/TLS when available
• API keys are never logged
• Connection strings are not exposed in UI after saving

Best Practices

• Use dedicated service accounts with minimal permissions
• Rotate credentials regularly
• Enable SSL for all connections
• Use network firewalls to restrict access
• Monitor connection logs for anomalies

warning

Never use production database admin credentials. Create dedicated service accounts with only the required permissions.

Validation & Testing

Testing Connections

Use the "Test Connection" button to verify:

• Network connectivity
• Authentication
• Required permissions
• SSL configuration

Validation Checklist

Component	Validation
LLM	Test chat completion request
Embeddings	Test vector generation
Vector Store	Test insert and query operations
FalkorDB	Test graph write and read
Conversations	Test table creation and queries

Troubleshooting

Connection Timeout:

• Check firewall rules
• Verify host is reachable
• Confirm port is open

Authentication Failed:

• Verify API key or password
• Check user permissions
• Confirm account is active

SSL Error:

• Match SSL mode with server configuration
• Verify certificate validity
• Try different SSL modes

Capability probe failed (custom LLM):

• "502 Bad Gateway" / "503 Service Unavailable" / "504 Gateway Timeout" (or an error page mentioning your proxy or web server) — your endpoint's address is reachable, but the model server behind it isn't responding. This usually means the model server hasn't started yet, is still loading a large model, has crashed, or the proxy in front of it is pointing at the wrong place. Make sure your model server is running and that your proxy forwards requests to it, then run Test & Probe again.
• "not an OpenAI-compatible API" / 404 — the base URL is missing /v1, points at the wrong service, or (for a tunnel) the tunnel is offline. Confirm GET /v1/models returns JSON.
• "Model not found — Available: …" — use one of the exact model ids listed (from your server's /v1/models).
• "private / loopback / link-local / cloud-metadata address refused" — the endpoint isn't publicly reachable; expose it over a public HTTPS URL.
• Tools or vision show as unsupported — add the tool-call parser launch flag, or use a vision model. See Preparing Your Self-Hosted LLM Endpoint.

A chat against a slow self-hosted model fails after ~2 minutes:

• A query can take a while on a slow or cold-starting self-hosted model, especially when the answer needs several reasoning steps. WizChat now allows a single chat to run up to five minutes end-to-end and gives each individual call to your endpoint up to 90 seconds, so a slow or cold-starting model has time to respond instead of being cut off.
• On very slow endpoints, WizChat also bounds how many reasoning steps a single answer takes, so the reply still comes back within the limit rather than looping until it times out.
• If a query still doesn't finish in time, the model is likely far slower than expected — check that the endpoint isn't cold-starting on every request (keep the model loaded/warm), that it has enough GPU/CPU for the model size, and that a reverse proxy or tunnel in front of it isn't adding its own timeout.

Migration

Moving to BYOI

1. Set up your infrastructure components
2. Configure BYOI settings in WizChat
3. Test with a new chatbot first
4. Migrate existing chatbots gradually

Moving from BYOI

1. Disable custom components
2. WizChat will use default infrastructure
3. Note: Data in custom infrastructure is not migrated

Related

• Data Residency
• Encryption
• PostgreSQL Integration