LLMs & services

In one line. Two Studio surfaces that supply an agent’s engine: the Language Models page (which provider models your agents may run on) and the Services library (registering external long-running workers a mesh can hand jobs to). You’ll be able to. Import a prebuilt provider model, register and test a custom one, understand how an agent selects a model, and recognise when a job belongs to an asynchronous Service. Where this lives. Studio > Agents > Agent Toolkit > Language Models, and Studio > Agents > Agent Toolkit > Services

Watch out: two names, one page. The menu entry reads Language Models. Older screenshots and some copy inside the page still say LLMs. Same page.

Why it matters

Every agent needs an engine: a large-language model to think with. The Language Models page is where you tell the platform which provider models exist and how to reach them, so the agent’s Model tab has something to pick. Most of the time the prebuilt models are enough; you import one and move on. You only come here to add a custom model when you have your own provider account, region, or deployment.

The Services library is a different, more advanced surface. It registers external long-running workers (a GPU model, an OCR farm, a slow partner API) that take longer than the platform’s in-run time limit and so can’t run as an ordinary step. Treat Part A as everyday and Part B as on-demand.

Watch out: two different “models”. This page’s Model is a provider model (Azure OpenAI, Gemini, and so on), the thing an agent thinks with. That is not the trained AI Model you teach with corrected examples under Studio > Agents > Classic ML > AI Models (see Trained models). Same English word, unrelated objects.

Part A — Language models

What a model entry is

A model entry is one row that says: “this provider model exists and here is how to reach it.” It bundles:

a provider (Azure OpenAI, Azure AI Foundry, Google Gemini, AWS Bedrock, the Managed Gateway, …),
the model name / deployment (e.g. gpt-4o, gemini-1.5-pro),
an endpoint (the URL the platform calls), and
credentials (an API key or cloud identity), stored encrypted, never shown back.

Once an entry exists and is verified, it appears in the LLM Model picker on every agent’s Model tab.

The agent stores a pointer to this entry, not the credentials — so changing a key here fixes every agent that uses it, and no agent ever holds the secret.

The Language Models page

┌ Studio > Agents > Agent Toolkit > Language Models ────────────────────────────┐
│ ┌ Models ───────────────┐  │  Detail — selected model                        │
│ │ [⇅ sort]   [search]   │  │  ┌──────────────────────────────────────────┐  │
│ │ ───────────────────── │  │  │  Model Provider:  Azure OpenAI           │  │
│ │ • gpt-4o   (Azure)    │◄─┤  │  (name and provider, read-only)          │  │
│ │ • gemini-1.5-pro      │  │  └──────────────────────────────────────────┘  │
│ │ • my-bedrock-claude   │  │                                                │
│ │ … (more load as you   │  │                                                │
│ │    scroll)            │  │                                                │
│ └───────────────────────┘  │                                                │
│                [ + Add Model ]  ← footer button opens the import/add dialog │
└──────────────────────────────────────────────────────────────────────────────┘

A two-pane shell: a list of registered models on the left with a sort toggle (date to A-Z and back) and a search field, and a detail pane showing the selected model’s Model Provider. If the project has no models yet, the empty state offers an Add Model call to action. The one action that matters is + Add Model in the footer.

The Add Model dialog — two tabs

Tab 1 — Import LLM. A card grid of prebuilt platform models. Each card shows the model and a chip: Already Imported or Click to import. Workspace-level defaults carry a Default badge. Load More pages through the catalogue. This is the one-click path.

Tab 2 — Add New LLM. The full form, for a custom model:

Section	Field	What it does
Provider & Model	Provider (select)	Which provider family this model belongs to
	Model dropdown (Managed Gateway only)	Pick from the models the gateway already serves; empty-state Retry re-fetches
	LLM Name (text) (all other providers)	The model / deployment name you call
Connection (non-gateway, provider-driven)	API Type	Provider API flavour
	Azure Endpoint / Endpoint	The base URL the platform calls
	AWS Access Key ID / Secret / Region	Bedrock credentials
	Client ID / API Key / Tenant ID / Authority / Scope	Identity/key fields (masked, entered once)
	API Version	Provider API version string
	Model ID	Provider-side model/deployment id
Options	Is Embedding (toggle)	Marks this as an embedding model (for vectorising and retrieval) rather than a chat model
Footer	Test	Validates the connection — required
	Submit	Saves the entry — disabled until Test passes

Test gates Submit. You cannot save a custom model until Test reports “Connection verified”. This is deliberate: it stops a typo in an endpoint or a wrong key from silently breaking every agent that later selects the model.

Which fields appear depends on the provider. Choose Azure and you get Azure Endpoint + API Version; choose AWS Bedrock and you get the three AWS fields; choose the Managed Gateway and the free-text name turns into a Model dropdown of what the gateway already serves — no endpoint or keys to enter at all.

How agents pick a model

You don’t bind a model here; you bind it on the agent’s Model tab. There, LLM Model is a multi-select, and selecting more than one is a feature, not a mistake:

The agent runs on a pool of the models you select.
At run time the platform picks one and, if a call fails or rate-limits, fails over to another in the pool.

This page stocks the shelf; the agent’s Model tab picks from it. Add a second model to a pool when you want resilience against a single provider hiccup. Full Model-tab behaviour (Agent Mode, thinking budget, sampling) is in Your first agent.

Prebuilt vs custom

Use the prebuilt models (Import tab) when…	Add a custom model (Add New tab) when…
You just need a capable general model	You must use your own Azure/Gemini/Bedrock account or region (data-residency, billing, quota)
You’re prototyping or following a use-case	You have a private deployment or fine-tuned model at a specific endpoint
The Default-badged workspace model is fine	You need an embedding model the catalogue doesn’t offer (set Is Embedding)

Publishing note. When you package an agent or solution for the Hub, the model’s endpoint and keys are never shipped. The package carries only a requirement (“needs an Azure-OpenAI class model”); whoever installs it supplies their own model entry. Structure travels, secrets don’t.

Part B — Services (asynchronous, advanced)

Advanced. Services is an advanced surface. It sits in the Agent Toolkit group alongside Tools, Guard Rails, MCP Servers and Skills, and only matters once you are building meshes. If you don’t see Services under Studio > Agents > Agent Toolkit, ask your administrator; nothing in Part A or your everyday agents depends on it.

What a Service is

A Service is a registered external, long-running worker (a GPU inference model, a heavy OCR pipeline, a slow partner API) that exceeds the in-platform run-time limit: a mesh member must finish within roughly 240 seconds. Work that can’t fit that budget can’t run as an ordinary step, so you register it as a Service instead.

The shape is pull, not push:

A mesh member of kind Service doesn’t do the work; it parks, waits, and holds no processing capacity.
Your external worker asks the platform for waiting jobs, does the work, and reports the result back — using the secret shown once when you register the Service.
When the worker reports back, the parked member wakes and the mesh continues.

Because the worker reaches out, the platform never needs an address inside your network. That is what lets a Service safely be a GPU box in your own data-centre. Hand the Service ID and the secret to whoever builds the worker; everything else on this page is yours to configure.

Since a waiting Service member holds no processing capacity, one slow Service can’t hold up the rest of the run. A run counts as finished only when nothing is still running, queued, or waiting on a Service.

The Services library page

A list (columns name, service id, transport) with a footer + Service button. The empty state reads “Register a long-running external service…”. The detail pane has three tabs:

Tab	What it shows
Details	The registration: Service ID (with a copy control), Transport chip, Binding chip (if set), Callback event (when binding = callback), Input mode, Result keys, Max wait (s), Max retries. The ⋯ menu offers Edit · Rotate secret · Delete.
Runs	Live invocations: the job id, a status pill (submitted / claimed / done / failed / dead letter), attempt count, and the document, with Refresh. Empty: “No invocations yet”.
Callbacks	Inbound callback routing rows: outcome (resumed / ambiguous / no match / error), event, document, duplicate count, job, and detail, with Refresh. Shows why a callback did or didn’t resume a parked job.

The register dialog

Field	What it does
Service ID (required)	The stable id your worker uses to claim jobs
Name	Human-friendly label
Transport	How jobs reach the worker: mesh-bus / servicebus / redis / webhook
Input mode	How the payload is passed: baton (inline) or blob_ref (a pointer to blob storage, for large inputs)
Result keys	Comma-separated names of the values the worker returns
Callback type	The inbound event type that resumes the job (when applicable)
Max wait (s)	How long the platform parks before giving up
Max retries	How many times to re-offer a failed job

On success the dialog shows “Service registered” and reveals the service secret once.

The secret is shown once. Copy it immediately. If you lose it, Rotate secret on the Details tab issues a new one (and invalidates the old); there is no “show again”.

When you need a Service

A Service is only useful inside a mesh: in the mesh designer you add a member that points at the registered Service, and when the work reaches it, the member parks and your external worker takes over. Building the mesh itself is Teams & mesh.

You don’t need a Service when…	You do need a Service when…
The work finishes in seconds (ordinary tool / agent step)	The work takes minutes: GPU inference, large-batch OCR, a slow partner API
You can call an external API as a normal Tool or MCP server within the time limit	The worker must run on your hardware and pull jobs (no inbound access)
You’re not building a mesh	You’re building a mesh and one stage exceeds the ~240 s member ceiling

Try it yourself

A — Import and verify a model (everyone can do this):

Go to Studio > Agents > Agent Toolkit > Language Models, click + Add Model, and on the Import LLM tab import a card marked Click to import. Confirm it flips to Already Imported.
Click + Add Model again, switch to Add New LLM, pick a Provider, and watch the Connection fields change to match it.
Fill in the endpoint and credentials for a model you control, click Test, and note that Submit stays disabled until Test reports “Connection verified”. Then Submit.
Open any agent’s Model tab and confirm your model is selectable. Add a second model to the pool to see failover in action.

B — Register a Service (advanced; skip it unless you are heading for meshes):

Go to Studio > Agents > Agent Toolkit > Services and click + Service. Set a Service ID, pick a Transport (try redis), leave Input mode = baton, add a Result key or two, and a sensible Max wait / Max retries.
Save and copy the service secret from the success state — it is shown only once.
Check the Details tab, then peek at Runs and Callbacks (empty until a mesh invokes the service).

Where to go next

Your first agent — the Model tab, pooling, failover.
XFlows & pipelines — the visual pipelines that string steps together.
Teams & mesh — where Services actually get wired in.
Trained models — the other kind of “model”: trainable extractors.