Scheduling, MCP and deployment
Run pipelines unattended, connect any LLM over MCP, and ship a self-contained server bundle.
Schedules and triggers
Pipelines can run on a cron expression, a fixed interval, or a file-change watch. Schedules live on the pipeline right-click menu and in the Runs/History tab, so you can review past runs and set up future ones from the same place.
Watermark incremental loads (xf.incremental) and DuckLake CDC persist state to the workspace and advance only on a fully successful run. A partial "Run from here" never advances the watermark, so you can safely re-run a slice of a pipeline without skipping data on the next scheduled load.
Cron
Schedule with a standard cron expression for precise recurring runs.
Interval
Run on a fixed interval for simple, regular polling.
File watch
Trigger a run when a watched file changes.
Connections and contexts
Connections are reusable, type-filtered credential sets stored under connections/ and AES-256-GCM encrypted at rest with a per-workspace key. Each database, warehouse, or storage node has a "Saved connection" picker that auto-fills host, port, database, and credentials, so the same credentials are shared across every node that needs them.
Context variables let you parameterize any field with ${VAR} or ${ContextName.VAR} (for example, a database password), resolved at run time on the canvas and on scheduled runs alike. Never hardcode secrets; use ${ENV:KEY} placeholders for env-supplied values.
Built-in placeholders need no context: ${workspace} (alias ${projectroot}) resolves to the project root, and ${date}, ${time}, ${datetime}, ${timestamp}, and ${now} stamp the run time (UTC) into a path. They resolve fresh on every run, and a sink's parent folder is created automatically, so ${workspace}/exports/${date}/orders.parquet lands in a new dated folder each day.
MCP server (connect Claude or any LLM)
Duckle embeds an MCP (Model Context Protocol) stdio server. Connect it in one click from the in-app MCP popup, or run it headless. Over MCP an LLM can drive the studio end to end:
| Tool | What it does |
|---|---|
| List components | Enumerate the available sources, transforms, sinks, quality, control, and code nodes. |
| Get a component schema | Fetch the option schema for a single component. |
| Create a pipeline | Generate a pipeline, validated before writing. |
| Validate a pipeline | Check a pipeline for errors without running it. |
| Run a pipeline | Execute a pipeline headlessly. |
| Read run logs | Read the logs from a run. |
| Build a standalone executable | Export the pipeline as a self-contained binary. |
| Manage connections | List and create reusable connections. |
${ENV:KEY} placeholders. The run pipeline and build pipeline tools need a DuckDB binary (DUCKLE_DUCKDB_BIN); build pipeline also needs the duckle-runner binary.Headless runner and server deployment (Build Pipeline)
The duckle-runner CLI executes a pipeline file headlessly. "Build Pipeline" exports a self-contained server bundle (engine plus runner) with env/passphrase secret handling, so a pipeline can run on a server or in CI without the desktop app.
Target OS. The Build dialog has a Target OS selector: build natively for Windows, cross-build a Linux server file from any host (the Linux engine and DuckDB are bundled into the file for you), or build for macOS on a Mac. The output stays one self-contained file per target.
There is also a File Transfer sink (snk.ftp, FTP/FTPS/SFTP) for delivering outputs to a remote host as part of the run.
Engines
Duckle runs each pipeline on a selectable engine. DuckDB is the default; the others are opt-in or planned.
| Engine | Role |
|---|---|
| DuckDB | Default execution engine, tracking v1.5.3. |
| Duckie AI | Local llama.cpp plus Qwen, opt-in. |
| SlothDB | Alternate embedded analytical engine, selectable per pipeline. |
| Native | In-process Rust streaming engine, planned. |