Giving Up on a Stalled Run After a No-Progress Window with Python

A run that never finishes keeps a script waiting. This example polls a routine run, treats a stretch with no new finished rows as a stall, stops waiting, and records the run id so the rows can be collected later. The REST paths tried here had no way to cancel the run, so giving up stops the script and not the run.

What you will build

A script, stall_watch.py, with a watch command that starts or attaches to a run and a collect command that reads a run by id. The stall rule lives in a small StallTracker class that takes the clock as an argument, so the decision logic is tested without waiting.

POST /routines/{id}/run  (or --run-id)
   ↓
poll GET /routines/run/{id}/results?limit=100   every --poll seconds
   ├─ status complete          -> fetch every page, print row statuses, exit 0
   ├─ finished unchanged for --stall-window seconds
   │                           -> write stalled.json, print gave_up, exit 3
   └─ otherwise                -> keep polling

collect <run_id>  -> same results endpoint, later: which ids finished (exit 4 if still running)

POST /routines/{id}/run  (or --run-id)
   ↓
poll GET /routines/run/{id}/results?limit=100   every --poll seconds
   ├─ status complete          -> fetch every page, print row statuses, exit 0
   ├─ finished unchanged for --stall-window seconds
   │                           -> write stalled.json, print gave_up, exit 3
   └─ otherwise                -> keep polling

collect <run_id>  -> same results endpoint, later: which ids finished (exit 4 if still running)

AI Prompt

Implement a Python script that stops waiting on a Clay routine run when it makes no progress, and reports what finished.

Requirements:
- Base URL: https://api.clay.com/public/v0, auth header "clay-api-key".
- Use the "Company Domain" routine, function:t_0tjqm9rUjfGY4MunGDM. Input "Company Name", result { "Domain": "<domain>" }, about 1 credit per row.
- Start a run with POST /routines/{id}/run and { items: [{ id, inputs }] }. It returns { routine_run_id, status: "in_progress" }.
- Poll GET /routines/run/{routine_run_id}/results?limit=100. While the run is in progress the body has status, total and finished and no data rows, even when finished is above 0. Only a completed run returns data, 20 rows per page unless limit is set, with a cursor while more pages remain.
- Progress is the finished count increasing. A finished count that stays at its starting value is not progress. A run can show finished 0 for several seconds at the start, so the window must be longer than that.
- Give up when finished has not increased for --stall-window seconds. Write stalled.json with the run id, finished count, total and window, print the outcome, and exit with code 3. State in the output that the run may still be running server side, because no REST path to cancel it was found.
- A collect command reads a run by id and prints the status counts and the ids that completed. It exits 4 when the run is still in progress.
- Take the clock as a parameter of the stall rule and test it with simulated times.
- Run the verification step below before finishing.

Implement a Python script that stops waiting on a Clay routine run when it makes no progress, and reports what finished.

Prerequisites

Python 3.10+
pip install requests
A Clay Public API key (clay_scoped_...)
12 credits for one 12-row run

1. Set up the project

mkdir clay-stall-watch && cd clay-stall-watch
python3 -m venv .venv && source .venv/bin/activate
pip

mkdir clay-stall-watch && cd clay-stall-watch
python3 -m venv .venv && source .venv/bin/activate
pip install requests

2. Discover the real schema

Two behaviors of the results endpoint decide the design, and both came from polling real runs at 0.5 to 1 second intervals.

First, a run in progress reports counts and nothing else. A separate 25-row run of the same routine, printed whenever the response changed:

0.1 s   status in_progress, total 25, finished 0, no data
7.7 s   status in_progress, total 25, finished 5, no data
10.3 s  status complete,    total 25, finished 25, data: 25 rows

At 7.7 seconds five rows were finished and the response listed none of them. Rows can only be read by id once the run is complete.

Second, progress arrives in steps after a delay. In that 25-row run, nothing changed for about 7.7 seconds. A 25-row run through run-batch showed the same shape, with total at 0 for the first poll, then finished going 0, 4, 23, 25 between 1.3 and 12.9 seconds. A 12-row run went from finished 0 to 12 in a single step at about 6 seconds.

Next, look for a way to cancel. These five requests were sent for a completed run id, and each returned 404 with Cannot POST or Cannot DELETE:

POST   /routines/run/{id}/cancel
POST   /routines/run-batch/{id}/cancel
POST   /routines/runs/{id}/cancel
POST   /routines/run/{id}/stop
DELETE /routines/run/{id}

Those are guesses at five paths and do not prove that no cancel path exists. The CLI documentation set has a clay routines runs cancel command, which needs an OAuth session, and a key alone was reported to get auth_forbidden there.

3. Configure credentials

export CLAY_API_KEY=clay_scoped_...

export CLAY_API_KEY=clay_scoped_...

4. Implement the script

Save the helper as clay.py:

import os

import requests

BASE = "https://api.clay.com/public/v0"

session = requests.Session()
session.headers.update({"clay-api-key": os.environ["CLAY_API_KEY"],

import os

import requests

BASE = "https://api.clay.com/public/v0"

session = requests.Session()
session.headers.update({"clay-api-key": os.environ["CLAY_API_KEY"], "content-type": "application/json"})


def clay(path, method="GET", body=None):
    res = session.request(method, f"{BASE}{path}", json=body, timeout=60)
    if not res.ok:
        raise RuntimeError(f"{res.status_code} {path}: {res.text}")
    return res.json()


def all_results(run_id):
    """Every row of a finished run. The endpoint pages at 20 rows unless limit is set."""
    page = clay(f"/routines/run/{run_id}/results?limit=100")
    rows = list(page.get("data", []))
    while page.get("cursor"):
        page = clay(f"/routines/run/{run_id}/results?limit=100&cursor={page['cursor']}")
        rows.extend(page.get("data", []))
    page["data"] = rows
    return page

Save the script as stall_watch.py:

import argparse
import csv
import json
import sys
import time
from pathlib import Path

from clay import all_results, clay

COMPANY_DOMAIN = "function:t_0tjqm9rUjfGY4MunGDM"
STATE = Path("stalled.json")


class StallTracker:
    """Gives up when `finished` has not increased for `window` seconds. Takes the clock as an argument so it can be tested."""

    def __init__(self, window, now):
        self.window, self.best, self.last_change = window, 0, now

    def update(self, finished, now):
        if finished > self.best:
            self.best, self.last_change = finished, now
        return (now - self.last_change) >= self.window  # True means stalled


def start_run(path, column):
    with open(path, newline="", encoding="utf-8") as fh:
        names = [r[column].strip() for r in csv.DictReader(fh) if r[column].strip()]
    items = [{"id": f"r{i}", "inputs": {"Company Name": n}} for i, n in enumerate(names)]
    return clay(f"/routines/{COMPANY_DOMAIN}/run", "POST", {"items": items})["routine_run_id"]


def summarize(results):
    rows = results["data"]
    return {"status": results["status"], "total": results["total"], "finished": results["finished"],
            "row_statuses": {s: sum(1 for r in rows if r["status"] == s) for s in sorted
            "with_domain": sum(1 for r in rows if (r.get("result") or {}).get("Domain"))}


def watch(run_id, window, poll):
    started = time.monotonic()
    tracker = StallTracker(window, started)
    while True:
        body = clay(f"/routines/run/{run_id}/results?limit=100")
        now = time.monotonic()
        if body["status"] != "in_progress":
            report = summarize(all_results(run_id))
            print(json.dumps({"outcome": "finished", "run_id": run_id, "seconds": round(now - started, 1), **report}))
            return 0
        if tracker.update(body["finished"], now):
            state = {"run_id": run_id, "finished_at_give_up": body["finished"], "total": body["total"],
                     "rows_listed_by_endpoint": len(body.get("data", [])), "window_seconds": window,
                     "seconds_waited": round(now - started, 1)}
            STATE.write_text(json.dumps(state, indent=2))
            print(json.dumps({"outcome": "gave_up", **state, "run_still_running_server_side": True}))
            return 3
        time.sleep(poll)


def main():
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd", required=True)
    w = sub.add_parser("watch")
    src = w.add_mutually_exclusive_group(required=True)
    src.add_argument("--csv"); src.add_argument("--run-id")
    w.add_argument("--column", default="company")
    w.add_argument("--stall-window", type=float, required=True, help="seconds without a new finished row")
    w.add_argument("--poll", type=float, default=1.0)
    c = sub.add_parser("collect"); c.add_argument("run_id")
    a = ap.parse_args()
    if a.cmd == "collect":
        results = all_results(a.run_id)
        print(json.dumps({**summarize(results), "ids_finished": sorted(r["id"] for r in results["data"] if r["status"] 
        return 0 if results["status"] != "in_progress" else 4
    run_id = a.run_id or start_run(a.csv, a.column)
    print(json.dumps({"run_id": run_id, "stall_window": a.stall_window}))
    return watch(run_id, a.stall_window, a.poll)


if __name__ == "__main__":
    sys.exit(main())

import argparse
import csv
import json
import sys
import time
from pathlib import Path

from clay import all_results, clay

COMPANY_DOMAIN = "function:t_0tjqm9rUjfGY4MunGDM"
STATE = Path("stalled.json")


class StallTracker:
    """Gives up when `finished` has not increased for `window` seconds. Takes the clock as an argument so it can be tested."""

    def __init__(self, window, now):
        self.window, self.best, self.last_change = window, 0, now

    def update(self, finished, now):
        if finished > self.best:
            self.best, self.last_change = finished, now
        return (now - self.last_change) >= self.window  # True means stalled


def start_run(path, column):
    with open(path, newline="", encoding="utf-8") as fh:
        names = [r[column].strip() for r in csv.DictReader(fh) if r[column].strip()]
    items = [{"id": f"r{i}", "inputs": {"Company Name": n}} for i, n in enumerate(names)]
    return clay(f"/routines/{COMPANY_DOMAIN}/run", "POST", {"items": items})["routine_run_id"]


def summarize(results):
    rows = results["data"]
    return {"status": results["status"], "total": results["total"], "finished": results["finished"],
            "row_statuses": {s: sum(1 for r in rows if r["status"] == s) for s in sorted({r["status"] for r in rows})},
            "with_domain": sum(1 for r in rows if (r.get("result") or {}).get("Domain"))}


def watch(run_id, window, poll):
    started = time.monotonic()
    tracker = StallTracker(window, started)
    while True:
        body = clay(f"/routines/run/{run_id}/results?limit=100")
        now = time.monotonic()
        if body["status"] != "in_progress":
            report = summarize(all_results(run_id))
            print(json.dumps({"outcome": "finished", "run_id": run_id, "seconds": round(now - started, 1), **report}))
            return 0
        if tracker.update(body["finished"], now):
            state = {"run_id": run_id, "finished_at_give_up": body["finished"], "total": body["total"],
                     "rows_listed_by_endpoint": len(body.get("data", [])), "window_seconds": window,
                     "seconds_waited": round(now - started, 1)}
            STATE.write_text(json.dumps(state, indent=2))
            print(json.dumps({"outcome": "gave_up", **state, "run_still_running_server_side": True}))
            return 3
        time.sleep(poll)


def main():
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd", required=True)
    w = sub.add_parser("watch")
    src = w.add_mutually_exclusive_group(required=True)
    src.add_argument("--csv"); src.add_argument("--run-id")
    w.add_argument("--column", default="company")
    w.add_argument("--stall-window", type=float, required=True, help="seconds without a new finished row")
    w.add_argument("--poll", type=float, default=1.0)
    c = sub.add_parser("collect"); c.add_argument("run_id")
    a = ap.parse_args()
    if a.cmd == "collect":
        results = all_results(a.run_id)
        print(json.dumps({**summarize(results), "ids_finished": sorted(r["id"] for r in results["data"] if r["status"] == "complete")}))
        return 0 if results["status"] != "in_progress" else 4
    run_id = a.run_id or start_run(a.csv, a.column)
    print(json.dumps({"run_id": run_id, "stall_window": a.stall_window}))
    return watch(run_id, a.stall_window, a.poll)


if __name__ == "__main__":
    sys.exit(main())

Save the test as test_tracker.py:

from stall_watch import StallTracker

# Simulated clock and progress. This exercises the decision logic only; it is not a live stall.
t = StallTracker(window=10, now=0)
assert

from stall_watch import StallTracker

# Simulated clock and progress. This exercises the decision logic only; it is not a live stall.
t = StallTracker(window=10, now=0)
assert not t.update(0, 5)      # 5 s with no progress: inside the window
assert t.update(0, 10)         # 10 s with no progress: stalled
t = StallTracker(window=10, now=0)
assert not t.update(0, 9)
assert not t.update(3, 12)     # progress at 12 s resets the timer
assert not t.update(3, 21)     # 9 s since the last change
assert t.update(3, 22)         # 10 s since the last change
t = StallTracker(window=10, now=0)
assert not t.update(5, 4)
assert not t.update(4, 13)     # a lower count is not progress, 9 s since the last increase
assert t.update(4, 14)
print("tracker: 3 sequences passed")

Test data: companies12.csv has a company column with twelve names.

company
Stripe
Shopify
Airbnb
Slack
Zoom
Asana
Okta
Twilio
Atlassian
HubSpot
Box
Canva

5. Run

python3 test_tracker.py
python3 stall_watch.py watch --csv companies12.csv --stall-window 3
python3 stall_watch.py

python3 test_tracker.py
python3 stall_watch.py watch --csv companies12.csv --stall-window 3
python3 stall_watch.py collect <run_id from stalled.json>
python3 stall_watch.py watch --run-id <run_id from stalled.json> --stall-window 30
python3 stall_watch.py collect <run_id from stalled.json>

6. Verify the result

Tested live on 2026-10-20 against the real Clay API (workspace 1322578). No run was observed to be stuck, so the live give-up below was caused by a window of 3 seconds, shorter than the run's normal start-up delay. It shows the give-up path end to end and does not show a real stall.

The decision logic, with simulated times:

tracker: 3 sequences passed

The tracker starts its best count at 0, so a first poll of finished 0 does not count as progress. A variant that started at -1 reset the timer on that first poll and failed the first sequence of this test.

The give-up, then a read of the same run immediately afterwards:

$ watch --csv companies12.csv --stall-window 3
{"run_id": "run_0tn6k4nRFkgPuvNW489", "stall_window": 3.0}
{"outcome": "gave_up", "run_id": "run_0tn6k4nRFkgPuvNW489", "finished_at_give_up": 0, "total": 12, "rows_listed_by_endpoint": 0, "window_seconds": 3.0, "seconds_waited": 4.0, "run_still_running_server_side": true}
exit=3

$ watch --csv companies12.csv --stall-window 3
{"run_id": "run_0tn6k4nRFkgPuvNW489", "stall_window": 3.0}
{"outcome": "gave_up", "run_id": "run_0tn6k4nRFkgPuvNW489", "finished_at_give_up": 0, "total": 12, "rows_listed_by_endpoint": 0, "window_seconds": 3.0, "seconds_waited": 4.0, "run_still_running_server_side": true}
exit=3

$ cat stalled.json
{
  "run_id": "run_0tn6k4nRFkgPuvNW489",
  "finished_at_give_up": 0,
  "total": 12,
  "rows_listed_by_endpoint": 0,
  "window_seconds": 3.0,
  "seconds_waited": 4.0
}
$ collect (right after give-up)
{"status": "in_progress", "total": 12, "finished": 0, "row_statuses": {}, "with_domain": 0, "ids_finished": []}
exit=4

$ cat stalled.json
{
  "run_id": "run_0tn6k4nRFkgPuvNW489",
  "finished_at_give_up": 0,
  "total": 12,
  "rows_listed_by_endpoint": 0,
  "window_seconds": 3.0,
  "seconds_waited": 4.0
}
$ collect (right after give-up)
{"status": "in_progress", "total": 12, "finished": 0, "row_statuses": {}, "with_domain": 0, "ids_finished": []}
exit=4

The script exited 3 after 4.0 seconds for a 3-second window, because the check runs once per 1-second poll. At that point the endpoint listed no rows, so the report can state how many rows had finished (0) and not which ones.

Re-attaching to the same run with a 30-second window, which is the negative control for the stall rule on a healthy run, and a read of the run afterwards:

$ watch --run-id <same run> --stall-window 30
{"run_id": "run_0tn6k4nRFkgPuvNW489", "stall_window": 30.0}
{"outcome": "finished", "run_id": "run_0tn6k4nRFkgPuvNW489", "seconds": 3.6, "status": "complete", "total": 12, "finished": 12, "row_statuses": {"complete": 12}, "with_domain": 12}
exit=0

$ watch --run-id <same run> --stall-window 30
{"run_id": "run_0tn6k4nRFkgPuvNW489", "stall_window": 30.0}
{"outcome": "finished", "run_id": "run_0tn6k4nRFkgPuvNW489", "seconds": 3.6, "status": "complete", "total": 12, "finished": 12, "row_statuses": {"complete": 12}, "with_domain": 12}
exit=0

$ collect (later)
{"status": "complete", "total": 12, "finished": 12, "row_statuses": {"complete": 12}, "with_domain": 12, "ids_finished": ["r0", "r1", "r10", "r11", "r2", "r3", "r4", "r5", "r6", "r7", "r8", "r9"]}
exit=0

$ collect (later)
{"status": "complete", "total": 12, "finished": 12, "row_statuses": {"complete": 12}, "with_domain": 12, "ids_finished": ["r0", "r1", "r10", "r11", "r2", "r3", "r4", "r5", "r6", "r7", "r8", "r9"]}
exit=0

The run that the script had given up on completed 12 of 12 rows with a Domain each. The properties that define a pass are that watch with a short window exits 3 and writes stalled.json, that collect on an unfinished run exits 4, that watch with a long window on the same run exits 0 with finished equal to total, and that every row is complete. The domains, seconds and ids are one run on 2026-10-20 and are not part of the check.

How it works

The results endpoint exposes finished and total for a run in progress and the rows themselves only after completion. A stall rule therefore watches a single integer. StallTracker remembers the highest value seen and when it last rose, and reports a stall when the current time is at least window seconds past that moment. A count that goes down or stays level does not reset the timer.

Giving up is a client-side decision. The run id goes to stalled.json, the script exits with a distinct code, and collect can read the same run later through the same endpoint, because results are keyed by run id. The 12-row run in the verification step shows that the work continued after the script stopped waiting.

Common issues

A window shorter than the start-up delay reports a stall on a healthy run

The 25-row run showed no change for about 7.7 seconds and the 12-row run about 6. A window of 3 seconds gave up on a run that completed seconds later. Choose a window well above the longest start-up delay seen for the routine and batch size in use. Routines that run provider waterfalls can take longer; Work Email runs have taken 1 to 4 minutes in earlier testing, and a window for those needs to be sized in minutes.

Giving up does not stop spending

With no cancel path found over REST, a run that was abandoned can still finish and charge for its rows. The script prints run_still_running_server_side for this reason. Anything that needs to stop the run has to use another interface, and the CLI cancel command was documented as needing an OAuth session.

The rows are not listed while the run is in progress

finished is a count. rows_listed_by_endpoint was 0 in the give-up report. To learn which ids finished, wait for completion and call collect, or call it later with the saved run id.

`finished` can jump

Counts moved from 0 to 5 and from 0 to 12 in one step, and in the batch probe from 4 to 23. A tracker that expects a steady rise is wrong for this endpoint, and one that measures the gap between increases needs a window longer than the largest gap.

Not covered

A run that is genuinely stuck was not observed, so the time to a real stall and the behavior of a stuck run's status field are unknown. The run-batch endpoint was probed for progress shape only and the script does not watch it.

Next steps

verification:
  status: verified
  tested_at: "2026-10-20"
  product_version: "public/v0"
  command: "python3 test_tracker.py; python3 stall_watch.py watch --csv companies12.csv --stall-window 3; python3 stall_watch.py watch --run-id <run_id> --stall-window 30"

verification:
  status: verified
  tested_at: "2026-10-20"
  product_version: "public/v0"
  command: "python3 test_tracker.py; python3 stall_watch.py watch --csv companies12.csv --stall-window 3; python3 stall_watch.py watch --run-id <run_id> --stall-window 30"
  expected_result: "The tracker test passes. The 3-second watch exits 3 and writes stalled.json. Re-attaching with a 30-second window exits 0 with finished equal to total and every row complete."