SIE: One Inference Server for All Your Agent's Models

By Prahlad Menon Published 2026-08-07 3 min read

If you’re building AI agents, you’ve probably hit the model sprawl problem: one server for embeddings, another for reranking, a third for OCR, maybe a fourth for entity extraction. Each needs its own deployment, scaling config, and monitoring. It’s a mess.

SIE (Superlinked Inference Engine) solves this by serving 100+ small AI models from a single cluster, loading them on demand and evicting via LRU. One API endpoint, one deployment, one set of dashboards.

What It Actually Does

SIE isn’t trying to replace your LLM provider — it’s handling all the other models your agents need:

Task	What It Does	Models
Search	Embed, match, rerank	bge-m3, splade-v3, colbertv2, qwen3-reranker
Document → Markdown	PDFs, Office, scans	lightonocr, glm-ocr, mineru, paddleocr-vl
Structured Output	Schema-valid JSON extraction	gliner2, nuner-zero, qwen3.6-27b
Content Safety	Safety verdicts with confidence	granite-guardian-2b
Agent Loop	Planning + tool calling	qwen3.6-27b

The key insight: these models are small enough to run on your own GPU but too numerous to justify individual deployments.

OpenAI-Compatible API

Drop-in replacement for existing code:

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": "Hello world"}'

Same endpoints you know: /v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses.

Quick Start

# macOS (Apple Silicon) or Linux
pip install "sie-server[local]" && sie-server serve

# Linux with NVIDIA GPU
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-default

Then use the SDK:

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

# Embeddings
result = client.encode("sentence-transformers/all-MiniLM-L6-v2", Item(text="Hello"))

# Reranking
scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text="What is machine learning?"),
    [Item(text="ML learns from data."), Item(text="The weather is sunny.")]
)

# Entity extraction
result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text="Tim Cook is the CEO of Apple."),
    labels=["person", "organization"]
)

Production-Ready

This is the part that impressed me. SIE ships the full production stack:

Load-balancing gateway for distributing requests
KEDA autoscaling (scale to zero when idle)
Grafana dashboards for monitoring
Terraform modules for GKE, EKS, and AKS

helm upgrade --install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster \
  --namespace sie --create-namespace \
  --set hfToken.create=true \
  --set hfToken.value=YOUR_HF_TOKEN \
  -f values-gke.yaml

All Apache 2.0 licensed. No enterprise upsell for production features.

Integrations

Works with the frameworks you’re already using:

Orchestration: LangChain, LlamaIndex, Haystack, DSPy, CrewAI
Vector stores: Chroma, Qdrant, Weaviate, LanceDB

There’s even an MCP edge package for offloading document processing from Claude — saves tokens by handling PDF/image work locally.

Why This Matters

The trend in AI infrastructure is clear: large models stay with cloud providers (OpenAI, Anthropic, etc.), but smaller models are moving in-house. Embeddings, rerankers, and extractors don’t need 70B parameters — they need low latency and predictable costs.

SIE gives you a single abstraction for this entire class of models. One cluster, one API, one bill.

Links

GitHub: superlinked/sie
Docs: superlinked.com/docs
Model Catalog: superlinked.com/models

If you’re running multiple model servers for your agent stack, this is worth evaluating. The production story alone — Helm charts, autoscaling, Terraform — puts it ahead of most OSS inference projects.