clay.com

Command Palette

Search for a command to run...

Watching a Stalled Run and Raising an Alert over REST with TypeScript

Last updated: 10/20/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}.

Watching a Stalled Run and Raising an Alert over REST with TypeScript

A run that is still in_progress after an unexpected time is not an error to the API, so a caller that waits has to decide for itself when to stop waiting. This script polls one run, treats a status and finished count that have not changed for a set number of seconds as a stall, treats a total run time above a limit as a timeout, and in either case prints one JSON alert on stdout and exits with code 2.

What you will build

A script that starts a small Company Domain run, or follows an existing run id, and applies two time limits. Poll lines go to stderr, the final result or alert goes to stdout, and the exit code says which one happened.

POST /routines/{id}/run                  (skipped when RUN_ID is set)
    ↓
every POLL_SECONDS: GET /routines/run/{id}/results?limit=1
    status complete                      -> stdout {"ok":true,...}      exit 0
    status other than in_progress        -> stdout alert                exit 2
    (status, finished/total) unchanged for STALL_SECONDS -> stdout alert exit 2
    elapsed >= MAX_RUN_SECONDS           -> stdout alert                exit 2
    read error (for example 404)         -> stdout alert                exit 2
invalid settings or script error         -> stderr message              exit 1

AI Prompt

Implement a TypeScript script that watches a Clay routine run and exits non-zero with a structured alert when the run stops making progress.

Requirements:
- Base URL: https://api.clay.com/public/v0, auth header "clay-api-key".
- With no RUN_ID in the environment, start a run with POST /routines/function:t_0tjqm9rUjfGY4MunGDM/run (Company Domain) and body { items: [{ id, inputs: { "Company Name": ... } }] }, using DEMO_ROWS companies (default 8). With RUN_ID set, watch that run and start nothing.
- Poll GET /routines/run/{id}/results?limit=1 every POLL_SECONDS (default 3). The response has status, total and finished. While the run is going, status is "in_progress" (HTTP 202) and finished rises from 0 to total, in steps. A run of 8 rows stayed at finished 0 for about 4 seconds, then showed 7, then 8 at about 10 seconds. A run of 3 rows stayed at finished 0 for 9 seconds before completing.
- Define progress as a change in the pair of status and finished/total. If it has not changed for STALL_SECONDS (default 120), raise an alert. Also raise an alert when the elapsed time reaches MAX_RUN_SECONDS (default 900).
- An alert is one line of JSON on stdout with the keys alert ("clay_run_watchdog"), severity, at (ISO time), reason, run_id, status, finished, total, elapsed_s, and a note that the alert does not cancel the run. Then exit with code 2. Poll lines go to stderr.
- A read failure raises an alert as well. An unknown run id returns 404 with the message "Routine run '<id>' not found".
- A run whose status is neither in_progress nor complete raises an alert.
- On complete, print {"ok":true,...} on stdout and exit 0.
- Reject settings that are not positive numbers with a message on stderr and exit 1.
- The REST API has no cancel call that was found, so the script reports and does not stop the run.
- Run the verification step below before finishing.

Prerequisites

  • Node.js 18+ and TypeScript
  • A Clay Public API key (clay_scoped_...)
  • About 9 credits: three runs of 3 rows each for the healthy run, the stall and the timeout. The alerted runs continue after the alert, so their rows are still billed

1. Create the project

mkdir clay-watchdog && cd clay-watchdog
npm init -y
npm install -D typescript @types/node
mkdir src

Use the tsconfig.json below. The types entry lists node because TypeScript 6 and later does not discover @types packages by default:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "types": ["node"],
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

2. Discover the real schema

A run's progress is visible in finished and total. A run of 3 rows, polled every 2 seconds, kept finished at 0 for the first 9 seconds. A stall limit shorter than that would raise an alert on a healthy run, which is how the alert is triggered below without altering any request.

Cancelling was also looked for. These paths were tried with a finished run's id, and each returned 404 with the HTML page Cannot <METHOD> /public/v0/...:

POST   /routines/run/{id}/cancel
POST   /routines/run/{id}/stop
DELETE /routines/run/{id}
PATCH  /routines/run/{id}
POST   /routines/run-batch/{id}/cancel
POST   /routines/runs/{id}/cancel
GET    /routines/run/{id}
GET    /routines/run/{id}/status

Those are the paths tried, not a list of every possible one. The CLI has a separate cancel command for runs, which this example did not use.

3. Configure credentials

Create .env:

CLAY_API_KEY=

Keep .env out of version control.

4. Implement the script

src/clay.ts is the fetch wrapper and the response types:

const BASE = "https://api.clay.com/public/v0";

export async function clay<T>(path: string, init: { method?: string; body?: unknown } = {}): Promise<T> {
  const key = process.env.CLAY_API_KEY;
  if (!key) throw new Error("CLAY_API_KEY is not set");
  const res = await fetch(`${BASE}${path}`, {
    method: init.method ?? "GET",
    headers: { "clay-api-key": key, "content-type": "application/json" },
    body: init.body === undefined ? undefined : JSON.stringify(init.body),
  });
  const text = await res.text();
  if (!res.ok) throw new Error(`${res.status} ${path}: ${text}`);
  return JSON.parse(text) as T;
}

export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

export async function balance(): Promise<number> {
  return (await clay<{ balance: number }>("/credits/balance")).balance;
}

export interface RunRow {
  id: string;
  status: string;
  result?: Record<string, unknown>;
  error?: { message: string };
}

export interface RunPage {
  status: string;
  total: number;
  finished: number;
  data?: RunRow[];
  cursor?: string;
}

// Starts an inline run and reads every page of the finished result.
export async function runRoutine(routineId: string, items: { id: string; inputs: Record<string, unknown> }[]): Promise<RunRow[]> {
  const started = await clay<{ routine_run_id: string }>(`/routines/${routineId}/run`, { method: "POST", body: { items } });
  const path = `/routines/run/${started.routine_run_id}/results`;
  for (let i = 0; i < 100; i++) {
    const head = await clay<RunPage>(`${path}?limit=1`);
    if (head.status !== "in_progress") break;
    await sleep(3000);
  }
  const rows: RunRow[] = [];
  let page = await clay<RunPage>(`${path}?limit=100`);
  rows.push(...(page.data ?? []));
  while (page.cursor) {
    page = await clay<RunPage>(`${path}?limit=100&cursor=${encodeURIComponent(page.cursor)}`);
    rows.push(...(page.data ?? []));
  }
  return rows;
}

src/watchdog.ts holds the polling loop and the alert:

import { clay, sleep, RunPage } from "./clay";

const COMPANY_DOMAIN = "function:t_0tjqm9rUjfGY4MunGDM";
const allDemoCompanies = ["Zendesk", "Intercom", "Segment", "Mixpanel", "Amplitude", "PagerDuty", "Sentry", "Fastly"];
const demoCompanies = allDemoCompanies.slice(0, Number(process.env.DEMO_ROWS ?? allDemoCompanies.length));

const STALL_SECONDS = Number(process.env.STALL_SECONDS ?? 120); // no change in the run's progress for this long
const MAX_RUN_SECONDS = Number(process.env.MAX_RUN_SECONDS ?? 900); // total time allowed, even if progress keeps moving
const POLL_SECONDS = Number(process.env.POLL_SECONDS ?? 3);

// One JSON object per alert on stdout, shaped so it can be piped to a webhook or a log shipper.
function alert(fields: Record<string, unknown>): never {
  console.log(JSON.stringify({ alert: "clay_run_watchdog", severity: "error", at: new Date().toISOString(), ...fields }));
  process.exit(2);
}

async function main() {
  if (![STALL_SECONDS, MAX_RUN_SECONDS, POLL_SECONDS].every((n) => Number.isFinite(n) && n > 0)) {
    throw new Error("STALL_SECONDS, MAX_RUN_SECONDS and POLL_SECONDS must be positive numbers");
  }

  let runId = process.env.RUN_ID;
  if (!runId) {
    const items = demoCompanies.map((name, i) => ({ id: `w${i}`, inputs: { "Company Name": name } }));
    runId = (await clay<{ routine_run_id: string }>(`/routines/${COMPANY_DOMAIN}/run`, { method: "POST", body: { items } })).routine_run_id;
  }

  const startedAt = Date.now();
  let lastProgressAt = startedAt;
  let lastSignature = "";
  let polls = 0;

  for (;;) {
    let page: RunPage;
    try {
      page = await clay<RunPage>(`/routines/run/${runId}/results?limit=1`);
    } catch (err) {
      alert({ reason: "could not read the run", run_id: runId, detail: err instanceof Error ? err.message : String(err) });
    }
    polls++;
    const elapsed = Math.round((Date.now() - startedAt) / 1000);
    const signature = `${page.status}:${page.finished}/${page.total}`;
    if (signature !== lastSignature) {
      lastSignature = signature;
      lastProgressAt = Date.now();
    }
    console.error(JSON.stringify({ poll: polls, elapsed_s: elapsed, status: page.status, finished: page.finished, total: page.total }));

    if (page.status === "complete") {
      console.log(JSON.stringify({ ok: true, run_id: runId, status: page.status, finished: page.finished, total: page.total, elapsed_s: elapsed }));
      return;
    }
    if (page.status !== "in_progress") {
      alert({ reason: "run ended without completing", run_id: runId, status: page.status, finished: page.finished, total: page.total, elapsed_s: elapsed });
    }
    const idle = Math.round((Date.now() - lastProgressAt) / 1000);
    if (idle >= STALL_SECONDS) {
      alert({ reason: `no progress for ${idle}s`, run_id: runId, status: page.status, finished: page.finished, total: page.total, elapsed_s: elapsed, stall_limit_s: STALL_SECONDS, note: "the alert does not cancel the run" });
    }
    if (elapsed >= MAX_RUN_SECONDS) {
      alert({ reason: `run exceeded ${MAX_RUN_SECONDS}s`, run_id: runId, status: page.status, finished: page.finished, total: page.total, elapsed_s: elapsed, note: "the alert does not cancel the run" });
    }
    await sleep(POLL_SECONDS * 1000);
  }
}

main().catch((err) => {
  console.error(err instanceof Error ? err.message : err);
  process.exit(1);
});

5. Run

npx tsc

# healthy run with the default limits
DEMO_ROWS=3 POLL_SECONDS=2 node --env-file=.env dist/watchdog.js; echo "exit=$?"

# a stall limit shorter than the time to first progress
DEMO_ROWS=3 STALL_SECONDS=2 POLL_SECONDS=1 node --env-file=.env dist/watchdog.js 2>watch.log | tee alert.json; echo "exit=${PIPESTATUS[0]}"

# a total-time limit shorter than the run
DEMO_ROWS=3 MAX_RUN_SECONDS=3 POLL_SECONDS=1 node --env-file=.env dist/watchdog.js; echo "exit=$?"

# an existing finished run, and an id that does not exist
RUN_ID=<a finished run id> node --env-file=.env dist/watchdog.js; echo "exit=$?"
RUN_ID=run_doesnotexist node --env-file=.env dist/watchdog.js; echo "exit=$?"

STALL_SECONDS=0 node --env-file=.env dist/watchdog.js; echo "exit=$?"

Use bash for PIPESTATUS.

6. Verify the result

Output on 2026-10-20, workspace 1322578. The healthy run, with the default limits (stderr and stdout shown together):

{"poll":1,"elapsed_s":0,"status":"in_progress","finished":0,"total":3}
{"poll":2,"elapsed_s":2,"status":"in_progress","finished":0,"total":3}
{"poll":3,"elapsed_s":4,"status":"in_progress","finished":0,"total":3}
{"poll":4,"elapsed_s":7,"status":"in_progress","finished":0,"total":3}
{"poll":5,"elapsed_s":9,"status":"in_progress","finished":0,"total":3}
{"poll":6,"elapsed_s":11,"status":"complete","finished":3,"total":3}
{"ok":true,"run_id":"run_0tn6kko2MfFmRFd7qBF","status":"complete","finished":3,"total":3,"elapsed_s":11}
exit=0

The stall limit of 2 seconds:

{"poll":1,"elapsed_s":0,"status":"in_progress","finished":0,"total":3}
{"poll":2,"elapsed_s":1,"status":"in_progress","finished":0,"total":3}
{"poll":3,"elapsed_s":3,"status":"in_progress","finished":0,"total":3}
{"alert":"clay_run_watchdog","severity":"error","at":"2026-10-20T00:41:26.417Z","reason":"no progress for 2s","run_id":"run_0tn6kkzypD7nnwUDo7J","status":"in_progress","finished":0,"total":3,"elapsed_s":3,"stall_limit_s":2,"note":"the alert does not cancel the run"}
exit=2

The total-time limit of 3 seconds (stdout only):

{"alert":"clay_run_watchdog","severity":"error","at":"2026-10-20T00:35:27.138Z","reason":"run exceeded 3s","run_id":"run_0tn6kazSxZjVzQ3xNWt","status":"in_progress","finished":0,"total":3,"elapsed_s":4,"note":"the alert does not cancel the run"}
exit=2

A finished run, an unknown run id, and an invalid setting:

{"ok":true,"run_id":"run_0tn6ka4uJRAH3ucNcE7","status":"complete","finished":8,"total":8,"elapsed_s":0}
exit=0
{"alert":"clay_run_watchdog","severity":"error","at":"2026-10-20T00:35:27.709Z","reason":"could not read the run","run_id":"run_doesnotexist","detail":"404 /routines/run/run_doesnotexist/results?limit=1: {\"message\":\"Routine run 'run_doesnotexist' not found\"}"}
exit=2
STALL_SECONDS, MAX_RUN_SECONDS and POLL_SECONDS must be positive numbers
exit=1

The stall and the timeout in these runs were produced by deliberately short limits on runs that were healthy. No run during this work stopped progressing on its own, so the alert has not been exercised on a run that was stuck. The checks are properties of the exit codes and the alert: a healthy run under the default limits exits 0 with ok: true, each limit that is crossed exits 2 with a reason that names it, the run id is in the alert, and bad settings exit 1. After each alert, the alerted run was read again and had status complete with 3 of 3 rows finished, so the alert did not stop the run, and the credits for it were spent.

How it works

Every poll reduces the response to a signature made of the status and the finished over total count. When the signature differs from the previous one, the script records the time as the last progress. The stall rule compares the current time with that record. The timeout rule compares the elapsed time since the start with MAX_RUN_SECONDS, whether or not progress is still being made, which covers a run that moves forward too slowly.

The alert is a single JSON line on stdout and the poll lines are on stderr, so watchdog.js | jq . or a pipe to a log shipper sees only the final record. The exit code is the same signal for a scheduler: 0 for done, 2 for an alert, 1 for a problem with the script itself.

Common issues

A short stall limit alerts on a healthy run

The healthy run above kept finished at 0 for 9 seconds before completing, and a run of 8 rows stayed at 0 for the first 4 seconds and then reported 7 of 8 at the next poll. A STALL_SECONDS under that time-to-first-progress alerts for runs that are fine, which is how the stall alert was produced here. Choose a limit above the longest wait to first progress that the routine and run size produce. Work Email runs took 1 to 4 minutes in an earlier example, which can exceed the 120-second default for a stall. This example did not run Work Email.

The alert does not cancel the run

Both alerted runs finished afterwards. The script only reports. The cancel paths tried on 2026-10-20 all returned 404, and a run left going still spends credits. A cancel was not found over REST in this example, so stopping a run needs a different tool than this script.

One failed read is an alert

A single read error, such as the 404 for an unknown id, raises an alert at once. The script has no retry. The first attempt to run it while building this example failed before the first poll with the message fetch failed on stderr and exit code 1, which is the path for an error outside the loop. That error did not raise an alert, and whether the run had started was not determined. A deployment that sees network errors from time to time would add a few retries around the read.

Stall and timeout are different checks

By the logic of the two rules, a run that is slow but moving, for example 1 row finishing every 20 seconds, never triggers the stall rule and does trigger MAX_RUN_SECONDS. A run that stops completely would trigger the stall rule first, if STALL_SECONDS is less than MAX_RUN_SECONDS. Setting both lets the alert say which of the two happened in its reason.

Batch runs use a different endpoint

This script watches an inline run through GET /routines/run/{id}/results. A run started with the batch flow is read from GET /routines/run-batch/{id}/results, and the inline endpoint answers 404 for it. The watchdog was not run against a batch run.

Next steps


verification:
  status: verified
  tested_at: "2026-10-20"
  product_version: "public/v0"
  command: "npx tsc && DEMO_ROWS=3 STALL_SECONDS=2 POLL_SECONDS=1 node --env-file=.env dist/watchdog.js; echo exit=$?"
  expected_result: "The script prints one JSON alert on stdout with reason `no progress for 2s` and the run id, and exits 2. With the default limits a healthy run prints {\"ok\":true,...} and exits 0."

Related Articles