Ingestion & connectors

In one line. How documents get into a collection — upload, pull, feed, or form — and how processed results get out through Export. You’ll be able to. Wire a reusable connection to an external system, schedule a job that pulls documents into a collection, run a one-shot upload, watch a run, and set up an export. Where this lives. Studio ▸ Data ▸ Ingestion, Studio ▸ Data ▸ Input Form, and Studio ▸ Data ▸ Export — three entries in the Data pillar.

Why it matters

A solution is only as good as the documents flowing through it. The Data pillar of Studio is ordered the way data actually flows — define it, bring it in, look at it, send it out — and it is one flat list of seven entries:

Collections · Ingestion · Input Form · Drive · Datasheets · Views · Export

This page covers three of them. Ingestion is the loading dock: it is where you connect to SharePoint, a Google/internal Drive, a mailbox, or a public data feed; schedule pulls; and drop in one-off uploads. Input Form is for the times you have no document at all and want a person to type structured input instead. Export is the shipping dock: it pushes finished documents and their extracted data back out to your own systems.

The other four entries have their own pages: Collections & schema for Collections, and Drive & datasheets for Drive and Datasheets. Views is covered alongside the data it slices.

Everything here feeds the collections you built in Collections & schema. Get the model right once and you rarely think about it again — documents just arrive, show a “Processing…” chip, and fill in.

The model: Connector vs Job vs Run

This is the single idea that makes Ingestion make sense. From Core concepts:

Term	What it is	In one word	Example
Connector	A tested connection — credentials + a target system	identity	“Our Finance SharePoint”
Job	A scoped, scheduled fetch — a connector + a scope + a schedule + a target collection	scope	“The /Invoices library, every 6 hours, into Vendor Invoices”
Run	One execution of a job (or an upload) — the record of what happened	history	“Tuesday 06:00 run: 14 new, 2 duplicates”

You build a connector once and reuse it across many jobs. A job is where you say which documents, how often, and where they land. A run is what you inspect when you want to know what happened.

What you’ll see. A document appears in the list straight away with a Processing… chip and fills in as it is worked. Knowledge collections index it for search instead — watch that on the collection’s Index Health tab (Search, indexes & vectorization). If a document sits at 0% for a long time, tell your administrator.

The Ingestion screen

Studio ▸ Data ▸ Ingestion. One screen, three tabs. It is also embedded inside a collection’s own Ingestion tab, where one extra Add-Data settings toolbar button appears.

┌─ Ingestion ─────────────────────────────────────────────────────────────────┐
│                        [ Upload ] [ + Connector ] [ + Job ]  (Add-Data       │
│                                                              settings*)      │
│  ┌──────────┬────────────┬────────┐                                          │
│  │  Jobs ▸  │ Connectors │  Runs  │   ← three tabs (Jobs is default)         │
│  └──────────┴────────────┴────────┘                                          │
│                                                                              │
│  Invoices nightly   on Finance SP · cron 0 6 * * * · next 06:00 · active     │
│        [ Run now ] [ Pause ] [ edit ] [ delete ]                             │
│  Contracts drip     on Botminds Drive · every 6h · next 12:00 · active       │
│        [ Run now ] [ Pause ] [ edit ] [ delete ]                             │
│  ──────────────────────────────────────────────────────────────────────     │
│  * Add-Data settings appears only when embedded in a collection's tab.      │
└──────────────────────────────────────────────────────────────────────────────┘

• Toolbar (top-right): Upload (one-shot drop), + Connector (the connection wizard), + Job (the primary action — schedule a pull). When embedded in a collection, an Add-Data settings button also appears.
• Tabs: Jobs (scheduled pulls — the default tab), Connectors (your saved connections), Runs (execution history). The body shows Loading… while fetching.
• Each row carries a kind icon: SharePoint, Email, Drive, Public Feed, or a generic package icon for anything else.

Watch out. An older multi-source setup wizard still opens from old bookmarks but is no longer in the menu. Start new work on Ingestion.

What every control does

Tab	Row shows	Row actions
Jobs	kind icon + job name + “on <connector> · <schedule> · next <date> · active/paused”	Run now, Pause/Resume, edit, delete (confirms; run history is preserved)
Connectors	icon + name + “<kind> · active/paused · test OK/failed/never · used by N jobs”	Reconnect (only when the last test failed — SharePoint only today), Test (live test + latency), Pause/Resume, delete (disabled with a tooltip if jobs still use it — delete the jobs first)
Runs	When · Trigger · Status · funnel counters	row click opens the Run detail dialog

Empty states. Jobs: “No jobs yet · Click + Job to schedule one. (You’ll need a Connector first.)” — Connectors: “No connectors yet”. The order matters: a connector must exist before a job can use it.

The Runs tab and Run detail

The Runs table tracks each execution with a funnel of counters — the eight stages a batch of items passes through:

Enum (enumerated) → Fetch (downloaded) → New → Upd (updated) → Dup (duplicate, skipped) → Disp (dispatched) → Merge → Fail.

Status colours: completed is green, partial amber, failed red. Click a row to open the Run detail dialog: pipeline-summary chips (Enumerate → Fetch → New → Updated → Duplicate → Dispatch → Merge), the status + failed count + last error, then a per-item event log (When · Stage · Item · Status · Message) so you can see exactly which file failed and why.

Tip. Dup climbing on a recurring job is normal and good — it means dedup is working and you are not re-processing files you already have.

The Add-Connector wizard

+ Connector opens a wizard. There are six kinds, and the number of steps varies by kind: SharePoint has 3 steps (kind → auth pattern → form); every other kind has 2 steps (kind → form — the auth-pattern step is skipped). SharePoint’s auth-pattern step offers two tiles: Quick (connect with Microsoft — smaller teams and pilots) and Enterprise (use your own Entra app — enterprise IT).

Step 1 is a tile grid of the six kinds:

Kind	Tile	Use it for
SharePoint	“Pull docs from a SharePoint site”	Microsoft 365 document libraries
Email (IMAP)	“Pull docs from a mailbox”	Gmail / Outlook / generic IMAP inboxes
Botminds Drive	“Pull docs from this project’s internal drive. No setup needed”	Files already inside the project’s Drive
Public Feed	“Pull from public gov data sources (SEC, CFPB, Federal Register)”	Curated public data feeds
Datasheet	“Re-ingest rows from a datasheet in this project as documents”	Turning datasheet rows into documents
Azure Blob Storage	“Ingest from an Azure Blob container”	Files in your own Azure storage

The per-kind credential forms and their test rules are enumerated in the Connector catalog — this page teaches the shape; the catalog enumerates every field.

Behaviours to know

• Test-before-save. Email, Public Feed, Drive, Azure Blob, and SharePoint-Enterprise connectors cannot be saved until a live connection test passes. Editing any field after a green test clears it — you must re-test. (Datasheet is the exception: it’s internal — just pick the datasheet and name it.)
• SharePoint Quick has no Save button — it completes through the Connect with Microsoft sign-in popup. A closed popup reads as “Sign-in cancelled”; a blocked popup shows a warning.
• Edit / Reconnect jumps straight to the form step, pre-filled — but secrets are never pre-filled; re-enter them. The stepper is hidden in edit mode.

The Add-Job wizard

+ Job is the primary action — this is where documents actually start flowing. Three steps (editing an existing job locks the connector, dropping it to two):

1. Pick a connector — choose from the project’s active connectors (radio rows with kind icon + test status + “used by N jobs”). No connectors? The wizard tells you to “Cancel and create a Connector first.” Hidden when editing.
2. Scope & metadata mapping — the Job name (required), then a kind-specific scope (the folder/library/mailbox/feed slice to pull), the Target collection (a real dropdown of your project’s collections — never free text; “Default” is always present), and optional Secondary XFlows (extra flows that run after each item is registered). Scope fields per kind:
   • SharePoint — Document library (with Test Drive to resolve a name to a stable id), Folder path, File extensions, Recursive, Max docs/run, plus a “Pull metadata from SharePoint?” toggle that reveals a metadata-mapping grid (map a taxonomy field to a SharePoint column; Discover columns lists the available columns).
   • Email — Folder (default INBOX), Include attachments, Max docs/run.
   • Drive — Folder (read-only; Browse… opens a picker), Extensions filter, Recursive, “Explode structured rows” (Excel/CSV/JSON → one document per row, with an optional Unique ID column), Max docs/run.
   • Public Feed — fields rendered dynamically from the feed’s own shape.
   • Azure Blob — Container, Folder path (blob prefix), Extensions filter, Recursive.
   • Datasheet — Document granularity (one document per row / whole sheet), Max rows/run. New and edited rows are picked up automatically each run.
3. Schedule — choose On-demand only (you press Run now), Recurring (every N Seconds/Minutes/Hours/Days), or a Cron expression (+ timezone). Public feeds also offer preset chips (Hourly / Every 6h / Daily 9am UTC / Weekly Mon 9am / On-demand). A live “Next 5 runs” preview confirms your schedule before you save.

The final button reads Create job (or Save changes when editing).

Scheduling, drip, and dedup

• Scheduling is per-job. A scheduled job fires on its cadence; Run now triggers it immediately regardless of schedule.
• Drip — public feeds (and other recurring jobs) pull only new items each run rather than re-downloading the whole source; the feed’s drip behaviour is described on its tile.
• Dedup — each run reports Dup for items it has seen before, so a recurring job is safe to run often without reprocessing. For exploded structured rows, set a Unique ID column as the business key; leave it blank and the platform dedups on the content itself.

The Upload modal (one-shot)

The Upload toolbar button opens a modal for ad-hoc files — no connector or job needed:

1. Drag-drop or browse multiple files. The list shows each file with its size, a remove control, and Clear all.
2. If multi-file grouping is on, a note explains the files are “grouped into one document set (first file primary).”
3. Choose the Target collection (unless it is pre-filled by context).
4. Submit opens a result panel — complete / partial / failed, “N of M registered, K failed”, the run id, and a per-file breakdown. Done closes.

Tip. Use Upload to smoke-test a new collection’s schema before you wire a recurring job — drop in five representative files, watch them process, fix the schema, then automate.

Export — getting results out

Studio ▸ Data ▸ Export. Ingestion is the loading dock; Export is the shipping dock — it pushes processed documents (with their extracted data, optionally split or bookmarked) out to Azure Blob storage or the Botminds Drive.

The screen is a standard list + details split: the left rail lists your exports, the right panel shows the selected export’s details — Type, Taxonomy (the fields included, as chips), Storage Type (Azure Blob or Botminds Drive), and Storage Details (account + container, or the drive URL), with edit/delete row actions. Empty state: “Setup an export to see more details” plus + Export.

An export can fire on ingestion, on a workflow transition, or on demand.

The Add Export dialog

+ Export opens a dialog whose fields change based on the Export Type you pick — choose the type first, the rest follows. Always present: a Name, the Export Type, and the destination (External Sources: Azure Blob → Storage Name/Key/Container, or Botminds Drive → drive URL).

The notable toggles for a split/bookmark export type:

• Split Input Document, Split Document on Sequential Values, Split By Start Page — break one input PDF into several output files.
• Grouped Bookmark, First Value As Bookmark, Group By Label, Order Date — build bookmarks/structure in the output.
• Suffix for document name — append a value to output filenames.

And common toggles across types: Export Documents from current job alone, Add date-wise folders, Add Document-wise folders, Export input Document (ship the original alongside results), Delete input document from project (clean up after export). Save is disabled while the form is invalid. The full per-type field matrix is in the Connector catalog.

Input Form — capturing structured input directly

Studio ▸ Data ▸ Input Form. Sometimes you don’t have documents at all — you want a person to type structured input that becomes a record. The Input Form builder defines those form templates, in the familiar list + details split. Empty state: “No Form. Add new form to process” plus + Add Form. If you don’t see this entry, ask your administrator.

Every template has a Name, Description, and a Type:

• Create Source — the form creates documents/records. You build its Columns (a repeatable list): each column has a Name, a Type (Text / Multiline / Checkbox / Dropdown / Radio), a Label (mapped to a taxonomy field, grouped by taxonomy), and flags Required, Is Title, Is Template. Each column can carry rules (Field + Value conditions that show/validate it). A Use Excel shortcut lets you define columns from a spreadsheet instead. If a column is Is Template, a Templates section appears (Value + an Export Template + a Rendering Template).
• Publish Template — a seed/launch form rather than a record-creator. Its Sub Type is either Connector (pick an ingestion source) or Global Variables (pick variables to expose).

Submit with Create (or Update when editing). The full field/rule reference is in the Connector catalog.

Try it yourself

Goal: a recurring Drive-folder job that feeds a collection.

1. Open Studio ▸ Data ▸ Ingestion.
2. Click + Connector, pick Botminds Drive. Give it a display name, click Test connection (it reports a sub-folder count), then Save connector. No credentials needed.
3. Click + Job. Step 1: select your new Drive connector.
4. Step 2: name it (e.g. “Contracts intake”), click Browse… and pick a folder, set Extensions to pdf, tick Recursive, and choose your Target collection.
5. Step 3: pick Recurring → every 6 Hours. Check the Next 5 runs preview, then Create job.
6. Back on the Jobs tab, click Run now on your job. Switch to the Runs tab and watch the funnel counters fill; click the run to open Run detail and confirm each file landed.

Faster variant: skip connectors entirely — click Upload, drag in a batch of PDFs, pick the target collection, Submit, and watch the result panel.

Recap

• The Data pillar is one flat list, in flow order: Collections, Ingestion, Input Form, Drive, Datasheets, Views, Export. This page covered Ingestion, Input Form and Export.
• Connector = identity (a tested connection), Job = scope (a scheduled, targeted pull), Run = history (one execution, with a funnel + event log).
• Ingestion (Studio ▸ Data ▸ Ingestion) has Jobs / Connectors / Runs tabs. An older setup wizard opens from old bookmarks but is no longer in the menu.
• Build a connector first (six kinds; most require a passing test before save), then a job (connector → scope + mapping → schedule), or just Upload one-shot files.
• Scheduling is per-job; drip pulls only new items; dedup reports Dup and is keyed by a Unique ID column for exploded rows.
• Export (Studio ▸ Data ▸ Export) ships processed results out; Input Form (Studio ▸ Data ▸ Input Form) captures structured input directly.

Where to go next

• Drive & datasheets — the internal Drive a Drive connector reads from, and the datasheet layer beside it.
• Build: Invoice settlement — a full ingest-to-export loop, end to end.
• Connector catalog — every connector kind’s fields, the full Export field matrix, and the Input Form column/rule reference.
• Collections & schema — the collections your jobs target.