LLMs & services

In one line. Two Studio surfaces that supply an agent’s engine: the LLMs page (which provider models your agents may run on) and the Services library (registering external long-running workers a mesh can hand jobs to). You’ll be able to. Import a prebuilt provider model, register and test a custom one, understand how an agent selects a model, and recognise when a job belongs to a flag-gated async Service. Where this lives. Studio ▸ Agent Builder ▸ LLMs (/studio/.../agent_builder/llms) and Studio ▸ Agent Builder ▸ Services (/studio/.../agent_builder/services)

Why it matters

Every agent needs an engine: a large-language model to think with. The LLMs page is where you tell the platform which provider models exist and how to reach them, so the agent’s Model tab has something to pick. Most of the time the prebuilt models are enough; you import one and move on. You only come here to add a custom model when you have your own provider account, region, or deployment.

The Services library is a different, more advanced surface. It registers external long-running workers (a GPU model, an OCR farm, a slow partner API) that take longer than the platform’s in-run time limit and so can’t run as an ordinary step. It is flag-gated and only relevant once you are building meshes. Treat Part A as everyday and Part B as on-demand.

Watch out: two different “models”. This page’s LLM / Model is a provider model (Azure OpenAI, Gemini, and so on), the thing an agent thinks with. That is not the AI Model of AI Builder, which is a trainable extractor you teach with corrected examples. Same English word, unrelated objects.

Part A — LLMs

What an LLM config is

An LLM config is one row that says: “this provider model exists and here is how to reach it.” It bundles:

a provider (Azure OpenAI, Azure AI Foundry, Google Gemini, AWS Bedrock, a Managed Gateway, …),
the model name / deployment (e.g. gpt-4o, gemini-1.5-pro),
an endpoint (the URL the platform calls), and
credentials (an API key or cloud identity), stored encrypted, never shown back.

Once a config exists and is verified, it appears in the LLM Model picker on every agent’s Model tab.

Under the hood. A config is a bm.ai_models row; the orchestrator decrypts its credentials at run time and builds the actual provider client. The agent only stores a reference to the config, not the secret.

The LLMs page

┌ Studio ▸ Agent Builder ▸ LLMs ───────────────────────────────────────────────┐
│ ┌ LLMs ─────────────────┐  │  Detail — selected model                        │
│ │ [⇅ sort]   [search]   │  │  ┌──────────────────────────────────────────┐  │
│ │ ───────────────────── │  │  │  Model Provider:  Azure OpenAI           │  │
│ │ • gpt-4o   (Azure)    │◄─┤  │  (name, provider — read-only projection) │  │
│ │ • gemini-1.5-pro      │  │  └──────────────────────────────────────────┘  │
│ │ • my-bedrock-claude   │  │                                                │
│ │ … (server-paged)      │  │                                                │
│ └───────────────────────┘  │                                                │
│                [ + Add Model ]  ← footer button opens the import/add dialog │
└──────────────────────────────────────────────────────────────────────────────┘

A two-pane shell: a server-paged list of registered models on the left with a sort toggle (date to A-Z and back) and a search field, and a detail pane showing the selected model’s Model Provider. If the project has no models yet, the empty state offers an Add Model call to action. The one action that matters is + Add Model in the footer.

The Add Model dialog — two tabs

Tab 1 — Import LLM. A card grid of prebuilt platform models. Each card shows the model and a chip: Already Imported or Click to import. Subscription-level defaults carry a Default badge. Load More pages through the catalogue. This is the one-click path.

Tab 2 — Add New LLM. The full form, for a custom model:

Section	Field	What it does
Provider & Model	Provider (select)	Which provider family this model belongs to
	Model dropdown (Managed Gateway / LiteLLM only)	Pick from gateway-served models; empty-state Retry re-fetches
	LLM Name (text) (all other providers)	The model / deployment name you call
Connection (non-gateway, provider-driven)	API Type	Provider API flavour
	Azure Endpoint / Endpoint	The base URL the platform calls
	AWS Access Key ID / Secret / Region	Bedrock credentials
	Client ID / API Key / Tenant ID / Authority / Scope	Identity/key fields (masked, create-only)
	API Version	Provider API version string
	Model ID	Provider-side model/deployment id
Options	Is Embedding (toggle)	Marks this as an embedding model (for vectorising/RAG) rather than a chat model
Footer	Test	Validates the connection — required
	Submit	Saves the config — disabled until Test passes

Test gates Submit. You cannot save a custom LLM until Test reports “Connection verified”. This is deliberate: it stops a typo in an endpoint or a wrong key from silently breaking every agent that later selects the model.

Which fields appear depends on the provider. Choose Azure and you get Azure Endpoint + API Version; choose AWS Bedrock and you get the three AWS fields; choose the Managed Gateway (LiteLLM) and the free-text name turns into a Model dropdown of what the gateway already serves (no endpoint or keys to enter at all).

How agents pick a model

You don’t bind a model here; you bind it on the agent’s Model tab. There, LLM Model is a multi-select, and selecting more than one is a feature, not a mistake:

The agent runs on a pool of the models you select.
At run time the platform picks one and, if a call fails or rate-limits, fails over to another in the pool.

This page stocks the shelf; the agent’s Model tab picks from it. Add a second model to a pool when you want resilience against a single provider hiccup. Full Model-tab behaviour (Agent Mode, thinking budget, sampling) is in Your first agent.

Prebuilt vs custom

Use the prebuilt models (Import tab) when…	Add a custom LLM (Add New tab) when…
You just need a capable general model	You must use your own Azure/Gemini/Bedrock account or region (data-residency, billing, quota)
You’re prototyping or following a use-case	You have a private deployment or fine-tuned model at a specific endpoint
The Default-badged subscription model is fine	You need an embedding model the catalogue doesn’t offer (set Is Embedding)

Publishing note. Credentials are DECLARE-class: when you package an agent or solution for the Hub, the model’s endpoint and keys are never shipped. The package carries only a requirement (“needs an Azure-OpenAI class model”); whoever installs it supplies their own LLM config. Structure travels, secrets don’t.

Part B — Services (async, advanced)

Flag-gated. The Services library and async services run only when BM_MESH_SERVICES_ENABLED is on. If you don’t see Services in the Agent Builder rail, the flag is off; skip Part B. Nothing in Part A or your everyday agents depends on it.

What a Service is

A Service is a registered external, long-running worker (a GPU inference model, a heavy OCR pipeline, a slow partner API) that exceeds the in-platform run-time limit: a mesh member must finish within roughly 240 seconds. Work that can’t fit that budget can’t run as an ordinary step, so you register it as a Service instead.

The shape is pull, not push:

A mesh member of kind Service doesn’t do the work; it parks (status awaiting), holding no worker slot, and waits.
Your external worker pulls the job over a tiny HTTP gateway — three verbs, claim → complete / fail — authenticated with a per-service secret (svcsec_...).
When the worker reports back, the parked member wakes and the mesh continues.

Because the worker dials out to claim work, the platform never needs the worker’s address or to reach into your network. That is what lets a Service safely be a GPU box in your own data-centre.

Under the hood. The parked member consumes no Prefect/worker slot while awaiting, so a single slow Service can’t starve the rest of a mesh run. A run is done only when nothing is running, pending, or awaiting.

The Services library page

A menu-panel list (columns name, service id, transport) with a footer + Service button. The empty state reads “Register a long-running external service…”. The detail pane has three tabs:

Tab	What it shows
Details	The registration: Service ID (code, copy), Transport chip, Binding chip (if set), Callback event (when binding = callback), Input mode, Result keys, Max wait (s), Max retries. The ⋯ menu offers Edit · Rotate secret · Delete.
Runs	Live invocations: `job_id`, a status pill (`submitted` / `claimed` / `done` / `failed` / `dead_letter`), attempt count, and the document, with Refresh. Empty: “No invocations yet”.
Callbacks	Inbound callback routing rows: outcome (`resumed` / `ambiguous` / `no_match` / `error`), event, document, duplicate count, job, and detail, with Refresh. Shows why a callback did or didn’t resume a parked job.

The register dialog

Field	What it does
Service ID (required)	The stable id your worker uses to claim jobs
Name	Human-friendly label
Transport	How jobs reach the worker: mesh-bus / servicebus / redis / webhook
Input mode	How the payload is passed: baton (inline) or blob_ref (a pointer to blob storage, for large inputs)
Result keys	Comma-separated names of the values the worker returns
Callback type	The inbound event type that resumes the job (when applicable)
Max wait (s)	How long the platform parks before giving up
Max retries	How many times to re-offer a failed job

On success the dialog shows “Service registered” and reveals the service secret (svcsec_...) once.

The secret is shown once. Copy svcsec_... immediately. If you lose it, Rotate secret on the Details tab issues a new one (and invalidates the old); there is no “show again”.

When you need a Service

A Service is only useful inside a mesh: in the mesh designer you add a member that references it ({kind: "service", id}), and when the baton reaches it, the member parks and your external worker takes over. Building the mesh itself is Teams & mesh.

You don’t need a Service when…	You do need a Service when…
The work finishes in seconds (ordinary tool / agent step)	The work takes minutes: GPU inference, large-batch OCR, a slow partner API
You can call an external API as a normal Tool or MCP server within the time limit	The worker must run on your hardware and pull jobs (no inbound access)
You’re not building a mesh	You’re building a mesh and one stage exceeds the ~240 s member ceiling

Try it yourself

A — Import and verify an LLM (everyone can do this):

Go to Studio ▸ Agent Builder ▸ LLMs, click + Add Model, and on the Import LLM tab import a card marked Click to import. Confirm it flips to Already Imported.
Click + Add Model again, switch to Add New LLM, pick a Provider, and watch the Connection fields change to match it.
Fill in the endpoint and credentials for a model you control, click Test, and note that Submit stays disabled until Test reports “Connection verified”. Then Submit.
Open any agent’s Model tab and confirm your model is selectable. Add a second model to the pool to see failover in action.

B — Register a Service (only if you see the Services tab):

Click + Service. Set a Service ID, pick a Transport (try redis), leave Input mode = baton, add a Result key or two, and a sensible Max wait / Max retries.
Save and copy the svcsec_... secret from the success state.
Check the Details tab, then peek at Runs and Callbacks (empty until a mesh invokes the service).

Where to go next

Your first agent — the Model tab, pooling, failover.
XFlows & pipelines — the visual pipelines that string steps together.
Teams & mesh — where Services actually get wired in.
AI Builder — the other kind of “model”: trainable extractors.