clay.com

Command Palette

Search for a command to run...

Stopping Searches Before a Self-Imposed Quota with Python

Last updated: 10/20/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}.

Stopping Searches Before a Self-Imposed Quota with Python

A script that runs searches in a loop can use up a workspace's search quota without any error until the quota is gone. This example adds a client-side guard that counts runs and returned records in a session file and refuses the next search before any request is sent when it would pass a limit you set or the quota the API last reported.

What you will build

A script, search_guard.py, with init, search and status commands. init sets a maximum number of runs and records for the session. search checks four conditions before sending anything: the per-request limit, the run count, the record count, and the period_quota.remaining that the previous response reported. Each completed search prints the API's period_quota, the session counters and the credit balance before and after.

init --max-runs 3 --max-records 6          -> session.json
search "<query>" --limit 2
   ├─ guard fails                          -> {"refused": ...}, exit 2, http_posts_sent 0
   └─ guard passes
        POST /search/query-mode            -> search_id
        POST /search/query-mode/{id}/run   -> data, has_more, period_quota
        GET  /credits/balance              -> before and after
        update session.json

AI Prompt

Implement a Python script that stops Clay searches before a self-imposed per-session limit.

Requirements:
- Base URL: https://api.clay.com/public/v0, auth header "clay-api-key".
- POST /search/query-mode with { query } (for example: select from companies where industry = "Software Development") returns { search_id, source_type }. POST /search/query-mode/{search_id}/run with { limit } returns { data, has_more, source_type, period_quota: { limit, used, remaining, resets_at } }. Calling run again on the same search_id returns the next page.
- limit must be between 1 and 500. 0 and 501 return 400. Omitting limit returns 20 rows. An unknown or expired search_id returns 404 { message: "Search not found or expired" }. A query that does not parse returns 400 with a message.
- Creating a search does not use quota. Each record returned by run adds to period_quota.used. The API has no endpoint that reads the quota without returning a record, and the query-mode reference text contains no quota or plan information, so the first run of a session is the first chance to read it.
- Session limits: --max-runs counts run requests and --max-records counts returned records. Refuse before sending any request when the next run or its limit would pass either, when limit is outside 1..500, or when the last reported period_quota.remaining is below limit.
- Count POST requests in the HTTP helper and print the count, so a refusal can be shown to have sent none.
- Read GET /credits/balance before and after each search and print both.
- Keep volumes tiny: no run above 3 records during testing.
- Run the verification step below before finishing.

Prerequisites

  • Python 3.10+
  • pip install requests
  • A Clay Public API key (clay_scoped_...)
  • About 1 credit, for the balance control in controls.py. The searches themselves did not change the balance.

1. Set up the project

mkdir clay-search-guard && cd clay-search-guard
python3 -m venv .venv && source .venv/bin/activate
pip install requests

2. Discover the real schema

Probe the edges of the run endpoint and look in the reference text for quota information:

python3 api_edges.py

Save as api_edges.py:

import json
import re

import clay

QUERY = 'select from companies where industry = "Software Development"'
sid = clay.clay("/search/query-mode", "POST", {"query": QUERY})["search_id"]


def show(label, res):
    body = res.json()
    if "data" in body:
        body = {"rows": len(body["data"]), "has_more": body.get("has_more"), "used": body["period_quota"]["used"]}
    print(label, res.status_code, json.dumps(body))


run = f"/search/query-mode/{sid}/run"
show("limit 0     ", clay.clay(run, "POST", {"limit": 0}, raw=True))
show("limit 501   ", clay.clay(run, "POST", {"limit": 501}, raw=True))
show("limit omitted", clay.clay(run, "POST", {}, raw=True))
show("unknown id  ", clay.clay("/search/query-mode/search_nope/run", "POST", {"limit": 1}, raw=True))
show("bad query   ", clay.clay("/search/query-mode", "POST", {"query": "select from companys where"}, raw=True))

ref = clay.clay("/search/query-mode/reference")["reference"]
print("reference chars", len(ref), "| mentions of quota:", len(re.findall("quota", ref, re.I)), "| period_quota:", "period_quota" in ref)

Real output:

limit 0      400 {"message": "Invalid request parameter(s): Field \"limit\" - Too small: expected number to be >=1"}
limit 501    400 {"message": "Invalid request parameter(s): Field \"limit\" - Too big: expected number to be <=500"}
limit omitted 200 {"rows": 20, "has_more": true, "used": 52592}
unknown id   404 {"message": "Search not found or expired"}
bad query    400 {"message": "Expected PEOPLE, COMPANIES, or JOBS but got IDENT (at line 1, col 13)"}
reference chars 172831 | mentions of quota: 1 | period_quota: False

The single mention of "quota" in the 172,831-character reference is about sales quotas in a note on qualitative preferences, and period_quota does not appear in it. The only source of quota numbers over REST found here is the period_quota object in a run response.

Two more properties matter for a guard. Creating a search uses no quota, and a read-only quota check does not exist, so reading the quota costs at least one record, because the minimum limit is 1. controls.py checks the first property, and also checks that the balance reads can detect a change at all, by comparing a 5-record search with a 1-row enrichment:

python3 controls.py

Save as controls.py:

import json
import time

import clay

QUERY = 'select from companies where industry = "Software Development"'


def quota_after_one_record():
    sid = clay.clay("/search/query-mode", "POST", {"query": QUERY})["search_id"]
    page = clay.clay(f"/search/query-mode/{sid}/run", "POST", {"limit": 1})
    return page["period_quota"]["used"]


# 1. Does creating a search consume quota? Create three searches without running them, between two 1-record reads.
used_a = quota_after_one_record()
for _ in range(3):
    clay.clay("/search/query-mode", "POST", {"query": QUERY})
used_b = quota_after_one_record()
print(json.dumps({"used_before": used_a, "created_without_running": 3, "used_after": used_b, "difference": used_b - used_a}))

# 2. Control for the balance reads: a 1-row enrichment must move the balance, a search must not.
b0 = clay.balance()
sid = clay.clay("/search/query-mode", "POST", {"query": QUERY})["search_id"]
clay.clay(f"/search/query-mode/{sid}/run", "POST", {"limit": 5})
b1 = clay.balance()
start = clay.clay("/routines/function:t_0tjqm9rUjfGY4MunGDM/run", "POST",
                  {"items": [{"id": "c1", "inputs": {"Company Name": "Datadog"}}]})
while clay.clay(f"/routines/run/{start['routine_run_id']}/results?limit=100")["status"] == "in_progress":
    time.sleep(3)
time.sleep(6)
b2 = clay.balance()
print(json.dumps({"after_5_record_search": b1 - b0, "after_1_row_enrichment": b1 - b2}))

Real output:

{"used_before": 52566, "created_without_running": 3, "used_after": 52567, "difference": 1}
{"after_5_record_search": 0, "after_1_row_enrichment": 1}

Between the two 1-record reads, three searches were created and not run, and used rose by 1, which is the second read's own record. The 5-record search left the balance unchanged and the 1-row Company Domain run lowered it by 1, so the balance reads were able to show a change.

3. Configure credentials

export CLAY_API_KEY=clay_scoped_...

4. Implement the script

Save the helper as clay.py:

import os

import requests

BASE = "https://api.clay.com/public/v0"

session = requests.Session()
session.headers.update({"clay-api-key": os.environ["CLAY_API_KEY"], "content-type": "application/json"})
posts_sent = 0  # POST requests made by this process


def clay(path, method="GET", body=None, raw=False):
    global posts_sent
    if method == "POST":
        posts_sent += 1
    res = session.request(method, f"{BASE}{path}", json=body, timeout=60)
    if raw:
        return res
    if not res.ok:
        raise RuntimeError(f"{res.status_code} {path}: {res.text}")
    return res.json()


def balance():
    return clay("/credits/balance")["balance"]

Save the guard as search_guard.py:

import argparse
import json
import sys
from pathlib import Path

import clay

STATE = Path("session.json")
MAX_PER_REQUEST = 500  # the API answers limit=1000 with 400 "Too big: expected number to be <=500"


def load():
    if not STATE.exists():
        sys.exit("no session; run init first")
    return json.loads(STATE.read_text())


def save(state):
    STATE.write_text(json.dumps(state, indent=2))


def refuse(reason):
    print(json.dumps({"refused": reason, "http_posts_sent": clay.posts_sent}))
    sys.exit(2)


def search(query, limit, search_id):
    state = load()
    if not 1 <= limit <= MAX_PER_REQUEST:
        refuse(f"limit {limit} is outside 1..{MAX_PER_REQUEST}")
    if state["runs"] + 1 > state["max_runs"]:
        refuse(f"run {state['runs'] + 1} would exceed --max-runs {state['max_runs']}")
    if state["records"] + limit > state["max_records"]:
        refuse(f"{limit} more records would exceed --max-records {state['max_records']} (used {state['records']})")

    quota = state.get("last_period_quota")
    if quota and quota["remaining"] < limit:
        refuse(f"workspace period quota has {quota['remaining']} records left (resets {quota['resets_at']}), fewer than {limit}")

    before = clay.balance()
    if not search_id:
        created = clay.clay("/search/query-mode", "POST", {"query": query})
        search_id = created["search_id"]
    page = clay.clay(f"/search/query-mode/{search_id}/run", "POST", {"limit": limit})
    after = clay.balance()

    state["runs"] += 1
    state["records"] += len(page["data"])
    state["last_period_quota"] = page.get("period_quota")
    save(state)
    print(json.dumps({
        "search_id": search_id, "returned": len(page["data"]), "has_more": page.get("has_more"),
        "names": [r.get("name") for r in page["data"]],
        "period_quota": page.get("period_quota"),
        "session": {"runs": f"{state['runs']}/{state['max_runs']}", "records": f"{state['records']}/{state['max_records']}"},
        "balance_before": before, "balance_after": after, "http_posts_sent": clay.posts_sent,
    }))


def main():
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd", required=True)
    i = sub.add_parser("init"); i.add_argument("--max-runs", type=int, required=True); i.add_argument("--max-records", type=int, required=True)
    s = sub.add_parser("search"); s.add_argument("query", nargs="?"); s.add_argument("--limit", type=int, required=True); s.add_argument("--search-id")
    sub.add_parser("status")
    a = ap.parse_args()
    if a.cmd == "init":
        save({"max_runs": a.max_runs, "max_records": a.max_records, "runs": 0, "records": 0, "last_period_quota": None})
    elif a.cmd == "status":
        print(json.dumps(load()))
    else:
        if not a.query and not a.search_id:
            sys.exit("give a query or --search-id")
        search(a.query, a.limit, a.search_id)


if __name__ == "__main__":
    main()

5. Run

python3 search_guard.py init --max-runs 3 --max-records 6
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 2
python3 search_guard.py search --search-id <search_id from the previous output> --limit 2
python3 search_guard.py search 'select from companies where industry = "Advertising Services"' --limit 1
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 1
python3 search_guard.py status

Then a second session with a record limit, and a limit above the per-request maximum:

python3 search_guard.py init --max-runs 5 --max-records 4
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 3
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 2
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 501
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 1
python3 search_guard.py search 'select from companies where industry = "Software Development"' --limit 1

6. Verify the result

Tested live on 2026-10-20 against the real Clay API (workspace 1322578). The full output of both sessions, in the order run:

$ python3 search_guard.py init --max-runs 3 --max-records 6
exit=0
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 2
{"search_id": "search_0tn6kesEXyrGS9ekyQ8", "returned": 2, "has_more": true, "names": ["Google", "Amazon"], "period_quota": {"limit": 10000000, "used": 52597, "remaining": 9947403, "resets_at": "2026-10-21T00:00:00.000Z"}, "session": {"runs": "1/3", "records": "2/6"}, "balance_before": 391646, "balance_after": 391646, "http_posts_sent": 2}
exit=0
$ python3 search_guard.py search --search-id search_0tn6kesEXyrGS9ekyQ8 --limit 2
{"search_id": "search_0tn6kesEXyrGS9ekyQ8", "returned": 2, "has_more": true, "names": ["LinkedIn", "Microsoft"], "period_quota": {"limit": 10000000, "used": 52599, "remaining": 9947401, "resets_at": "2026-10-21T00:00:00.000Z"}, "session": {"runs": "2/3", "records": "4/6"}, "balance_before": 391646, "balance_after": 391646, "http_posts_sent": 1}
exit=0
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Advertising\ Services\" --limit 1
{"search_id": "search_0tn6kevBB6Zp97iEFnS", "returned": 1, "has_more": true, "names": ["Confidential"], "period_quota": {"limit": 10000000, "used": 52600, "remaining": 9947400, "resets_at": "2026-10-21T00:00:00.000Z"}, "session": {"runs": "3/3", "records": "5/6"}, "balance_before": 391646, "balance_after": 391646, "http_posts_sent": 2}
exit=0
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 1
{"refused": "run 4 would exceed --max-runs 3", "http_posts_sent": 0}
exit=2
$ python3 search_guard.py status
{"max_runs": 3, "max_records": 6, "runs": 3, "records": 5, "last_period_quota": {"limit": 10000000, "used": 52600, "remaining": 9947400, "resets_at": "2026-10-21T00:00:00.000Z"}}
exit=0

# second session: record limit
$ python3 search_guard.py init --max-runs 5 --max-records 4
exit=0
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 3
{"search_id": "search_0tn6kexmzem5NUHF6GB", "returned": 3, "has_more": true, "names": ["Google", "Amazon", "LinkedIn"], "period_quota": {"limit": 10000000, "used": 52603, "remaining": 9947397, "resets_at": "2026-10-21T00:00:00.000Z"}, "session": {"runs": "1/5", "records": "3/4"}, "balance_before": 391646, "balance_after": 391646, "http_posts_sent": 2}
exit=0
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 2
{"refused": "2 more records would exceed --max-records 4 (used 3)", "http_posts_sent": 0}
exit=2
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 501
{"refused": "limit 501 is outside 1..500", "http_posts_sent": 0}
exit=2
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 1
{"search_id": "search_0tn6kezkWYF6Rv5eWmi", "returned": 1, "has_more": true, "names": ["Google"], "period_quota": {"limit": 10000000, "used": 52604, "remaining": 9947396, "resets_at": "2026-10-21T00:00:00.000Z"}, "session": {"runs": "2/5", "records": "4/4"}, "balance_before": 391646, "balance_after": 391646, "http_posts_sent": 2}
exit=0
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 1
{"refused": "1 more records would exceed --max-records 4 (used 4)", "http_posts_sent": 0}
exit=2

In the first session, the third run reached the run limit of 3, and the fourth was refused with http_posts_sent: 0. The second run continued the first search and returned LinkedIn and Microsoft, which were not in the first page, so a repeated run on a search_id pages forward. In the second session the 2-record request was refused because 3 of 4 records were used, the 501 request was refused locally, the 1-record request used the last record, and the next request was refused.

period_quota.used rose by exactly the number of records returned in every consecutive pair of responses in that output (52597 to 52599 for 2 records, to 52600 for 1, to 52603 for 3, to 52604 for 1), and balance_before equalled balance_after for every search. Those two facts hold for this workspace in a window in which no other search or run was observed. They are checks of this run and not guarantees, because used and the balance belong to the whole workspace.

The branch that reads the API's reported quota was exercised with a hand-edited session.json that simulates a Free-plan workspace with 3 records left. No real quota near exhaustion was available, so this tests the guard logic and not the API's response at exhaustion:

# session.json edited by hand to simulate a Free-plan workspace with 3 records left
{"limit": 100, "used": 97, "remaining": 3, "resets_at": "2026-10-21T00:00:00.000Z"}
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 5
{"refused": "workspace period quota has 3 records left (resets 2026-10-21T00:00:00.000Z), fewer than 5", "http_posts_sent": 0}
exit=2
$ python3 search_guard.py search select\ from\ companies\ where\ industry\ =\ \"Software\ Development\" --limit 3
{"returned": 3, "period_quota": {"limit": 10000000, "used": 52607, "remaining": 9947393, "resets_at": "2026-10-21T00:00:00.000Z"}, "session": {"runs": "1/10", "records": "3/100"}, "http_posts_sent": 2}
exit=0

The properties that define a pass are that every refusal exits 2 with http_posts_sent equal to 0, that the runs and records counters never exceed their limits, and that returned equals the change in period_quota.used. The names, search ids and counts above are one run on 2026-10-20 and are not expected values.

How it works

The API reports the workspace's period quota only inside a run response, so the guard keeps two kinds of limit. The session limits are the script's own and are known before the first request. The workspace limit is learned from the previous response and is applied to the next one. The first run of a session therefore cannot be checked against the workspace quota, and a quota of 100 records would not stop a first request for 500.

Each guard check runs before the first POST. The counter in the HTTP helper makes "nothing was sent" a property of the process, which does not depend on the balance or used staying still while other jobs run.

Common issues

The first request of a session cannot be checked against the workspace quota

last_period_quota is empty until a run has returned. Keep the first --limit small, then rely on the reported remaining. The ordinary run response was the only quota source found over REST.

The API's own errors differ by cause

A limit of 0 or 501 returns 400 with a message naming the bound. A search id that is unknown or expired returns 404 Search not found or expired. A query that does not parse returns 400 and a position, such as Expected PEOPLE, COMPANIES, or JOBS but got IDENT (at line 1, col 13) for the misspelled companys. The guard's own range check avoids spending a request on the first of these.

Omitting limit returns 20 rows

A run without limit returned 20 rows in the probe above, and in an earlier probe of the same endpoint used rose by 20 for that call. The script requires --limit for that reason.

Paging a search uses quota again

A second run on the same search_id returns the next page and adds its rows to used, so a session counts each page as a run. Reusing the id does not avoid the cost of the records.

Quota exhaustion was not reached

The workspace used here reported a limit of 10,000,000, and nothing close to it was approached. The error the API returns when a quota is exceeded was not observed over REST, and the CLI documentation set describes it as quota_exceeded. The simulated state above tests only the guard's own refusal.

Other searches change used

used is shared by everything in the workspace. If another script searches between two of this script's runs, used rises by more than returned. The guard's own counters are the numbers it enforces for that reason, and last_period_quota is only the most recent reading.

Next steps

verification:
  status: verified
  tested_at: "2026-10-20"
  product_version: "public/v0"
  command: "python3 search_guard.py init --max-runs 3 --max-records 6; python3 search_guard.py search '<query>' --limit 2; python3 api_edges.py; python3 controls.py"
  expected_result: "Searches inside the session limits return rows and a period_quota, and used rises by the rows returned. The first search past a limit exits 2 with http_posts_sent 0. Balance before and after a search are equal."

Related Articles