The OpenAI-compatible endpoint

Point an OpenAI SDK at /v1, the chat completions and embeddings shapes, which features work, and which silently do not.

Existing SDKs against a local model

from openai import OpenAI

# the only change: a base_url and an API key that Ollama ignores
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="llama3.2:3b-instruct-q4_K_M",
    messages=[
        {"role": "system", "content": "You answer in at most two sentences."},
        {"role": "user", "content": "What is the point of an index in a database?"},
    ],
    temperature=0.2,
    max_tokens=200,
)
print(response.choices[0].message.content)
print(response.usage)                 # prompt/completion token counts

# streaming uses the same interface
stream = client.chat.completions.create(
    model="llama3.2:3b-instruct-q4_K_M",
    messages=[{"role": "user", "content": "Write a haiku about latency."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

# embeddings are exposed on the same base URL
vectors = client.embeddings.create(model="nomic-embed-text:v1.5",
                                   input=["first document", "second document"])
print(len(vectors.data), len(vectors.data[0].embedding))

• One client configuration change is the whole migration: LangChain, LlamaIndex, the OpenAI SDK and most agent frameworks work unchanged.
• max_tokens maps to num_predict, temperature and top_p map directly, and stop maps to stop sequences.
• The API key is required by the SDK but never validated by Ollama. Never put a real key there, and never expose the endpoint to a network you do not control.
• Tool calling through tools works for models that support it. Check per model — a small model may accept the parameter and never emit a tool call.

What differs from OpenAI

# provider-agnostic code: switch by changing one factory
import os
from openai import OpenAI

def make_client():
    if os.environ.get("LLM_PROVIDER") == "local":
        return OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    return OpenAI()                       # reads OPENAI_API_KEY from the environment

def model_name():
    return ("llama3.2:3b-instruct-q4_K_M"
            if os.environ.get("LLM_PROVIDER") == "local"
            else "gpt-4o-mini")

client = make_client()

def complete(prompt, **kwargs):
    result = client.chat.completions.create(
        model=model_name(),
        messages=[{"role": "user", "content": prompt}],
        **kwargs,
    )
    return result.choices[0].message.content

print(complete("Name one advantage of local inference.", max_tokens=60))

• Write against the shared subset and a provider switch becomes an environment variable. Code that leans on logprobs or fine-tuning endpoints will not port.
• Local models are more sensitive to prompt format. A prompt tuned for a hosted model may lose several points of accuracy when moved; rerun your evaluation after the switch.
• Behavioural defaults differ: context length, default temperature and stop sequences come from the model's Modelfile rather than from the OpenAI API's defaults.
• Error handling differs too. A local server returns connection errors when the model is loading, whereas a hosted provider returns structured rate-limit errors with headers.

When to use the native API instead

import requests

# the native API exposes things the compatibility layer does not
response = requests.post("http://localhost:11434/api/chat", json={
    "model": "qwen2.5:7b-instruct-q4_K_M",
    "messages": [{"role": "user", "content": "Return the capital of France as JSON."}],
    "format": {                                  # a real JSON schema, not just "json"
        "type": "object",
        "properties": {"country": {"type": "string"}, "capital": {"type": "string"}},
        "required": ["country", "capital"],
    },
    "keep_alive": "15m",                         # per-request residency control
    "options": {
        "num_ctx": 8192,
        "num_predict": 128,
        "temperature": 0.0,
        "seed": 42,                              # reproducible sampling
        "stop": ["\n\n"],
    },
    "stream": False,
}, timeout=180).json()

print(response["message"]["content"])
print(response["total_duration"] / 1e9, "seconds")
print(response["eval_count"], response["prompt_eval_count"])

• Use the native API when you need a JSON schema, per-request keep_alive, a sampling seed, or the timing fields that make performance work possible.
• Use the compatibility endpoint when you want an existing framework, SDK or client library to work with no changes.
• Mixing both is fine and common: the OpenAI path for the application code and the native path for the ingestion and evaluation scripts.
• Timing fields are returned only by the native API. If you are diagnosing latency, that alone decides the choice.

FAQ

Related

Calling Ollama from code Serving Ollama in Docker and over a network

Last refreshed 2026-09-18.