Run Duckle on your own cloud

You draw a pipeline on a laptop and it runs, on a schedule, on a server you own. Here is the whole path.

Three crossings, and the boundary is drawn only where something crosses it. Deploying needs the admin role and turning the schedule on needs only operator, because shipping code to a host and deciding when trusted code runs are different sizes of decision.

Two server modes, and only one of them schedules

This is the single most common surprise, so it goes first.

Three things that will stop you

Reference

Image and ports

ghcr.io/slothflowlabs/duckle-web:latest    # runner + DuckDB CLI + editor bundle
port 8080                                 # plain HTTP, TLS lives in your proxy
volume /workspace                         # pipelines, state, logs, keys

Environment

Credentials

A shared token gets you running. Accounts and API keys are covered in Who can do what.

Who can do what

Start from where you actually are.

How a request is decided

A refusal is recorded as carefully as a success, which is what makes duckle-runner audit --outcome denied worth running: it answers who is reaching for what they do not have. And a route with no entry in the permission table requires admin, so one added later is locked down rather than accidentally left open.

The three roles

The split follows what an action can destroy, not which screen it lives on. Each role includes the ones above it.

Giving people and machines access

# people, each with their own so the audit log can name them
duckle-runner console add-user alice --role admin    --workspace /workspace
duckle-runner console add-user bob   --role operator --workspace /workspace

# machines: a key of their own, with its own role and an optional expiry
duckle-runner console key-add ci-deployer --role admin --expires-days 90
duckle-runner console key-list      # role, state, and when each was last used
duckle-runner console key-revoke ci-deployer

A token or key is printed once and kept only as a hash, so a lost one is replaced rather than recovered. Revoking a key takes effect on a console that is already running, not at the next restart, and revoked keys are marked rather than deleted so one that turns up in an old log can still be named. Before revoking anything, key-list answers the question that actually matters: whether it is still in use.

Where credentials live

All three in <workspace>/.duckle/console.db. A copy of that file, or a backup of the workspace, admits nobody.

The browser holds a random session id, never the credential. Its cookie is:

• HttpOnly so page scripts cannot read it
• SameSite=Strict so another site cannot ride it
• Secure whenever your proxy reports the browser is on https
• good for 12 hours, and it survives a restart, so a rolling deploy does not sign your team out

Upgrading from an older release carries console-users.json into the database on first start and renames it .migrated, so nobody is locked out and the only copy of a credential store is not destroyed.

EC2, the straightforward one

A compute-optimised instance with local NVMe is the sweet spot: the engine is parallel, so cores turn directly into throughput, and spill wants fast disk. c7i.4xlarge or m7i.2xlarge is a sensible start; go wider before you go clever.

#!/bin/bash
set -euo pipefail

dnf install -y docker && systemctl enable --now docker
mkdir -p /srv/duckle

# Token from Secrets Manager, never baked into the instance or the image.
TOKEN=$(aws secretsmanager get-secret-value \
  --secret-id duckle/console-token --query SecretString --output text)

cat >/etc/systemd/system/duckle.service <<UNIT
[Unit]
Description=Duckle console
After=docker.service
Requires=docker.service

[Service]
Restart=always
RestartSec=5
Environment=DUCKLE_CONSOLE_TOKEN=${TOKEN}
Environment=DUCKLE_MAX_CONCURRENT_RUNS=4
ExecStartPre=-/usr/bin/docker rm -f duckle
ExecStart=/usr/bin/docker run --rm --name duckle \\
  -p 127.0.0.1:8080:8080 \\
  -v /srv/duckle:/workspace \\
  -e DUCKLE_CONSOLE_TOKEN \\
  -e DUCKLE_MAX_CONCURRENT_RUNS \\
  --entrypoint duckle-runner \\
  ghcr.io/slothflowlabs/duckle-web:latest \\
  serve --host 0.0.0.0 --port 8080 --workspace /workspace

[Install]
WantedBy=multi-user.target
UNIT

systemctl daemon-reload && systemctl enable --now duckle

ECS on Fargate

Managed containers with no instance to patch. Fargate has no local disk worth using, so the workspace goes on EFS.

{
  "family": "duckle",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "4096", "memory": "16384",
  "volumes": [{
    "name": "workspace",
    "efsVolumeConfiguration": { "fileSystemId": "fs-0abc123", "transitEncryption": "ENABLED" }
  }],
  "containerDefinitions": [{
    "name": "duckle",
    "image": "ghcr.io/slothflowlabs/duckle-web:latest",
    "entryPoint": ["duckle-runner"],
    "command": ["serve", "--host", "0.0.0.0", "--port", "8080", "--workspace", "/workspace"],
    "portMappings": [{ "containerPort": 8080 }],
    "mountPoints": [{ "sourceVolume": "workspace", "containerPath": "/workspace" }],
    "secrets": [{
      "name": "DUCKLE_CONSOLE_TOKEN",
      "valueFrom": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:duckle/console-token"
    }]
  }]
}

Keep the service at desired count 1. See Scaling for why, and for how to add capacity instead.

EKS

A StatefulSet rather than a Deployment, because the workspace is state and the identity is stable.

apiVersion: v1
kind: Secret
metadata: { name: duckle-console, namespace: data }
stringData:
  token: "REPLACE_ME"        # or use External Secrets / IRSA + Secrets Manager
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: duckle, namespace: data }
spec:
  serviceName: duckle
  replicas: 1                # one scheduler. See Scaling.
  selector: { matchLabels: { app: duckle } }
  template:
    metadata: { labels: { app: duckle } }
    spec:
      containers:
        - name: duckle
          image: ghcr.io/slothflowlabs/duckle-web:latest
          command: ["duckle-runner"]
          args: ["serve", "--host", "0.0.0.0", "--port", "8080", "--workspace", "/workspace"]
          ports: [{ containerPort: 8080 }]
          env:
            - name: DUCKLE_CONSOLE_TOKEN
              valueFrom: { secretKeyRef: { name: duckle-console, key: token } }
            - name: DUCKLE_MAX_CONCURRENT_RUNS
              value: "4"
            - name: DUCKLE_TEMP_DIR
              value: /spill
          resources:
            requests: { cpu: "4", memory: 16Gi }
            limits:   { cpu: "8", memory: 32Gi }
          volumeMounts:
            - { name: workspace, mountPath: /workspace }
            - { name: spill,     mountPath: /spill }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 20
      volumes:
        - name: spill
          emptyDir: { sizeLimit: 100Gi }
  volumeClaimTemplates:
    - metadata: { name: workspace }
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources: { requests: { storage: 100Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: duckle, namespace: data }
spec:
  selector: { app: duckle }
  ports: [{ port: 80, targetPort: 8080 }]

Front it with an ingress that terminates TLS. DUCKLE_TEMP_DIR pointed at an emptyDir keeps spill off the persistent volume, where it would otherwise compete with your data for space.

Virtual Machine

Same shape as EC2. Standard_D8s_v5 or an F-series if the work is compute-heavy. Attach a Premium SSD for the workspace and use the ephemeral temp disk for spill.

# Token from Key Vault, using the VM's managed identity.
TOKEN=$(az keyvault secret show --vault-name my-vault \
  --name duckle-console-token --query value -o tsv)

docker run -d --restart=always --name duckle \
  -p 127.0.0.1:8080:8080 \
  -v /srv/duckle:/workspace \
  -v /mnt/spill:/spill \
  -e DUCKLE_CONSOLE_TOKEN="$TOKEN" \
  -e DUCKLE_TEMP_DIR=/spill \
  --entrypoint duckle-runner \
  ghcr.io/slothflowlabs/duckle-web:latest \
  serve --host 0.0.0.0 --port 8080 --workspace /workspace

Publish it through Application Gateway or an Azure Front Door origin for TLS.

Container Apps

az containerapp create \
  --name duckle --resource-group data --environment data-env \
  --image ghcr.io/slothflowlabs/duckle-web:latest \
  --min-replicas 1 --max-replicas 1 \
  --cpu 4 --memory 16Gi \
  --target-port 8080 --ingress external \
  --secrets console-token=keyvaultref:https://my-vault.vault.azure.net/secrets/duckle-console-token,identityref:system \
  --env-vars DUCKLE_CONSOLE_TOKEN=secretref:console-token DUCKLE_MAX_CONCURRENT_RUNS=4 \
  --command duckle-runner \
  --args serve --host 0.0.0.0 --port 8080 --workspace /workspace

# Mount an Azure Files share at /workspace so state survives a revision.
az containerapp update --name duckle --resource-group data \
  --yaml volume-mount.yaml

AKS

The EKS manifest above works unchanged. Swap the storage class:

storageClassName: managed-csi-premium     # instead of gp3

Use Workload Identity with the Secrets Store CSI driver to bring DUCKLE_CONSOLE_TOKEN in from Key Vault rather than a Kubernetes Secret.

Compute Engine

c3-standard-8 upward, with a Local SSD for spill. A container-optimised image keeps the host minimal.

gcloud compute instances create-with-container duckle \
  --machine-type=c3-standard-8 \
  --container-image=ghcr.io/slothflowlabs/duckle-web:latest \
  --container-command=duckle-runner \
  --container-arg=serve --container-arg=--host --container-arg=0.0.0.0 \
  --container-arg=--port --container-arg=8080 \
  --container-arg=--workspace --container-arg=/workspace \
  --container-mount-disk=mount-path=/workspace,name=duckle-ws \
  --container-env=DUCKLE_MAX_CONCURRENT_RUNS=4 \
  --disk=name=duckle-ws,auto-delete=no \
  --no-address

Read the token from Secret Manager on boot rather than passing it on the command line, where it would land in instance metadata.

Cloud Run

GKE

The EKS manifest works as-is. Swap the storage class:

storageClassName: premium-rwo             # instead of gp3

Bind the pod's service account to a Google service account with Workload Identity so the pipelines themselves can reach BigQuery or GCS without a key file.

Scaling

Duckle scales up before it scales out, and that goes a long way: the engine is parallel and uses every core you give it, so a larger instance is a faster pipeline with no change to the pipeline.

Adding capacity properly

Workers as a Kubernetes Deployment, which can scale freely because they pull work rather than schedule it:

apiVersion: apps/v1
kind: Deployment
metadata: { name: duckle-workers, namespace: data }
spec:
  replicas: 6                # scale this freely
  selector: { matchLabels: { app: duckle-worker } }
  template:
    metadata: { labels: { app: duckle-worker } }
    spec:
      containers:
        - name: worker
          image: ghcr.io/slothflowlabs/duckle-web:latest
          command: ["duckle-runner"]
          args: ["work", "--workspace", "/workspace"]
          volumeMounts: [{ name: workspace, mountPath: /workspace }]
      volumes:
        - name: workspace
          persistentVolumeClaim: { claimName: duckle-shared }

The one thing Duckle does not do is split a single query across a cluster. When a job needs that, push it down into the system that has it.

Before you call it production

Something here not match what you hit? Open an issue or ask in Discord. Deployment notes age faster than code, and we would rather hear about it.