Removing Duplicate Rows Before Enrichment with TypeScript

Normalizing names and removing exact duplicates before a run reduces the number of rows that are charged. In the example below, 6 input rows became 4 unique rows and the run spent 4 credits instead of 6.

What you will build

A script that normalizes company names, enriches each unique name once, and maps the results back onto every original row.

inputs[6]  ->  normalize()  ->  unique[4]
    ↓
POST /routines/{routineId}/run   (4 items)
    ↓
GET  /routines/run/{runId}/results
    ↓
map result back to all 6 inputs

AI Prompt

Implement a TypeScript script that removes duplicate company names before enriching them with Clay's Public API.

Requirements:
- Base URL: https://api.clay.com/public/v0, auth header "clay-api-key".
- Normalize each name by lowercasing, removing periods and commas, removing a trailing `inc`, `llc`, `ltd`, `corp`, or `co`, collapsing whitespace, and trimming.
- Send one item per unique normalized name to the Company Domain routine (`function:t_0tjqm9rUjfGY4MunGDM`, estimate 1 credit per row), using the index as the item `id`. Results can come back in a different order from the input, so map by `id`.
- Print every original input with the domain found for its normalized name, and print `creditsSpent` next to the row count without dedupe.
- Normalization only merges names that are identical after cleanup. A name with an extra word, such as `Notion Labs`, is a different name and is sent as its own row.
- Run the verification step below before finishing.

Implement a TypeScript script that removes duplicate company names before enriching them with Clay's Public API.

Requirements:
- Base URL: https://api.clay.com/public/v0, auth header "clay-api-key".
- Normalize each name by lowercasing, removing periods and commas, removing a trailing `inc`, `llc`, `ltd`, `corp`, or `co`, collapsing whitespace, and trimming.
- Send one item per unique normalized name to the Company Domain routine (`function:t_0tjqm9rUjfGY4MunGDM`, estimate 1 credit per row), using the index as the item `id`. Results can come back in a different order from the input, so map by `id`.
- Print every original input with the domain found for its normalized name, and print `creditsSpent` next to the row count without dedupe.
- Normalization only merges names that are identical after cleanup. A name with an extra word, such as `Notion Labs`, is a different name and is sent as its own row.
- Run the verification step below before finishing.

Prerequisites

Node.js 18+ and TypeScript
A Clay Public API key (clay_scoped_...)

1. Create the project

mkdir clay-example && cd clay-example
npm init -y
npm install -D typescript @types/node
mkdir src

mkdir clay-example && cd clay-example
npm init -y
npm install -D typescript @types/node
mkdir src

Create tsconfig.json. The types entry is required with TypeScript 6 and later, which no longer loads @types packages automatically, and the template from npx tsc --init turns on stricter module and optional-property options than this example's code is written for:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "types": ["node"],
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "types": ["node"],
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

Node.js 18 or later provides fetch, so no HTTP library is installed. The runs below used Node.js 24.

2. Discover the real schema

Names that differ only by case, spacing, or a legal suffix collapse into one row. The script's own output below shows which inputs merged:

{"inputRows":6,"uniqueRows":4,"unique":["notion","notion labs","figma","linear"]}

{"inputRows":6,"uniqueRows":4,"unique":["notion","notion labs","figma","linear"]}

3. Configure credentials

Create .env with a key from Creating an API Key and Making a First REST Call:

CLAY_API_KEY=

CLAY_API_KEY=

Keep .env out of version control.

4. Implement the script

Save this shared helper as src/clay.ts. It wraps fetch, reads the balance, and runs a routine to completion:

const BASE = "https://api.clay.com/public/v0";

function apiKey(): string {
  const key = process.env.CLAY_API_KEY;
  if (!key) throw new Error("CLAY_API_KEY is not set");
  return key;
}

export async function clay<T>(path: string, init: { method?: string; 
  const res = await fetch(`${BASE}${path}`, {
    method: init.method ?? "GET",
    headers: { "clay-api-key": apiKey(), "content-type": "application/json" },
    body: init.body === undefined ? undefined : JSON.stringify(init.body),
  });
  const text = await res.text();
  if (!res.ok) throw new Error(`${res.status} ${path}: ${text}`);
  return JSON.parse(text) as T;
}

export interface RunItem {
  id: string;
  status: string;
  result?: Record<string, unknown>;
}

export interface RunResults {
  routine_run_id: string;
  status: string;
  total: number;
  finished: number;
  data?: RunItem[];
}

export async function balance(): Promise<number> {
  return (await clay<{ balance: number }>("/credits/balance")).balance;
}

export async function runRoutine(routineId: string, rows: { id: string; inputs
  const started = await clay<{ routine_run_id: string }>(`/routines/${routineId}/run`, {
    method: "POST",
    body: { items: rows },
  });
  for (let i = 0; i < 30; i++) {
    await new Promise((r) => setTimeout(r, 4000));
    const results = await clay<RunResults>(`/routines/run/${started.routine_run_id}/results`);
    if (results.status !== "in_progress") return results;
  }
  throw new Error(`run ${started.routine_run_id} still in progress after 120 seconds`);
}

export const sleep = (ms: number) => new Promise((r) =>

const BASE = "https://api.clay.com/public/v0";

function apiKey(): string {
  const key = process.env.CLAY_API_KEY;
  if (!key) throw new Error("CLAY_API_KEY is not set");
  return key;
}

export async function clay<T>(path: string, init: { method?: string; body?: unknown } = {}): Promise<T> {
  const res = await fetch(`${BASE}${path}`, {
    method: init.method ?? "GET",
    headers: { "clay-api-key": apiKey(), "content-type": "application/json" },
    body: init.body === undefined ? undefined : JSON.stringify(init.body),
  });
  const text = await res.text();
  if (!res.ok) throw new Error(`${res.status} ${path}: ${text}`);
  return JSON.parse(text) as T;
}

export interface RunItem {
  id: string;
  status: string;
  result?: Record<string, unknown>;
}

export interface RunResults {
  routine_run_id: string;
  status: string;
  total: number;
  finished: number;
  data?: RunItem[];
}

export async function balance(): Promise<number> {
  return (await clay<{ balance: number }>("/credits/balance")).balance;
}

export async function runRoutine(routineId: string, rows: { id: string; inputs: Record<string, unknown> }[]): Promise<RunResults> {
  const started = await clay<{ routine_run_id: string }>(`/routines/${routineId}/run`, {
    method: "POST",
    body: { items: rows },
  });
  for (let i = 0; i < 30; i++) {
    await new Promise((r) => setTimeout(r, 4000));
    const results = await clay<RunResults>(`/routines/run/${started.routine_run_id}/results`);
    if (results.status !== "in_progress") return results;
  }
  throw new Error(`run ${started.routine_run_id} still in progress after 120 seconds`);
}

export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

Save the script as src/dedupe-before-enrich.ts:

import { balance, runRoutine, sleep } from "./clay";

const COMPANY_DOMAIN = "function:t_0tjqm9rUjfGY4MunGDM";
const inputs = ["Notion", "notion ", "Notion Labs, Inc.", "Figma", "FIGMA", "Linear"];

function normalize(name: string): string {
  return name
    .toLowerCase()
    .replace(/[.,]/g, "")
    .replace(/\s+(inc|llc|ltd|corp|co)$/, "")
    .replace(/\s+/g, " ")
    .trim();
}

async function main() {
  const unique = [...new Set(inputs.map(normalize))];
  console.log(JSON.stringify({ inputRows: inputs.length, uniqueRows: unique.length, unique }));

  const before = await balance();
  const results = await runRoutine(
    COMPANY_DOMAIN,
    unique.map((name, i) => ({ id: String(i), inputs: { "Company Name": name } })),
  );
  await sleep(5000);
  const after = await balance();

  const domainByName = new Map<string, unknown>();
  for (const r of results.data ?? []) domainByName.set(unique[Number(r.id)], r.result?.["Domain"]);
  for (const original of inputs) {
    console.log(JSON.stringify({ input: original, domain: domainByName.get(normalize(original)) }));
  }
  console.log(JSON.stringify({ status: results.status, creditsSpent: before - after, withoutDedupe: inputs.length }));
}

main().catch((err) => {
  console.error(err instanceof Error ? err.message : err);
  process.exit(1);
});

import { balance, runRoutine, sleep } from "./clay";

const COMPANY_DOMAIN = "function:t_0tjqm9rUjfGY4MunGDM";
const inputs = ["Notion", "notion ", "Notion Labs, Inc.", "Figma", "FIGMA", "Linear"];

function normalize(name: string): string {
  return name
    .toLowerCase()
    .replace(/[.,]/g, "")
    .replace(/\s+(inc|llc|ltd|corp|co)$/, "")
    .replace(/\s+/g, " ")
    .trim();
}

async function main() {
  const unique = [...new Set(inputs.map(normalize))];
  console.log(JSON.stringify({ inputRows: inputs.length, uniqueRows: unique.length, unique }));

  const before = await balance();
  const results = await runRoutine(
    COMPANY_DOMAIN,
    unique.map((name, i) => ({ id: String(i), inputs: { "Company Name": name } })),
  );
  await sleep(5000);
  const after = await balance();

  const domainByName = new Map<string, unknown>();
  for (const r of results.data ?? []) domainByName.set(unique[Number(r.id)], r.result?.["Domain"]);
  for (const original of inputs) {
    console.log(JSON.stringify({ input: original, domain: domainByName.get(normalize(original)) }));
  }
  console.log(JSON.stringify({ status: results.status, creditsSpent: before - after, withoutDedupe: inputs.length }));
}

main().catch((err) => {
  console.error(err instanceof Error ? err.message : err);
  process.exit(1);
});

5. Run

npx tsc
node --env-file=.env dist/dedupe-before-enrich.js

npx tsc
node --env-file=.env dist/dedupe-before-enrich.js

Real output on 2026-10-09:

{"inputRows":6,"uniqueRows":4,"unique":["notion","notion labs","figma","linear"]}
{"input":"Notion","domain":"notion.com"}
{"input":"notion ","domain":"notion.com"}
{"input":"Notion Labs, Inc.","domain":"firstround.com"}
{"input":"Figma","domain":"figma.com"}
{"input":"FIGMA","domain":"figma.com"}
{"input":"Linear","domain":"linear.app"}
{"status":"complete","creditsSpent":4,"withoutDedupe":6}

{"inputRows":6,"uniqueRows":4,"unique":["notion","notion labs","figma","linear"]}
{"input":"Notion","domain":"notion.com"}
{"input":"notion ","domain":"notion.com"}
{"input":"Notion Labs, Inc.","domain":"firstround.com"}
{"input":"Figma","domain":"figma.com"}
{"input":"FIGMA","domain":"figma.com"}
{"input":"Linear","domain":"linear.app"}
{"status":"complete","creditsSpent":4,"withoutDedupe":6}

Notion, notion and FIGMA merged with their duplicates. The run was charged 4 credits instead of 6. Notion Labs, Inc. normalized to notion labs, stayed a separate row, and returned firstround.com, which is not the domain of either Notion or a company called Notion Labs.

6. Verify the result

node --env-file=.env dist/dedupe-before-enrich.js | tail -1

node --env-file=.env dist/dedupe-before-enrich.js | tail -1

Expected: creditsSpent equal to the number of unique rows (4) and lower than withoutDedupe (6).

How it works

normalize() produces a comparison key, a Set removes repeated keys, and one routine run covers the unique keys. A Map from key to result puts the answer back on every original row, so the output has one line per input while the charge covers only the unique rows.

Common issues

Near duplicates are not merged

Notion and Notion Labs, Inc. are different strings after normalization, so both were enriched. Merging them would need a domain-level key or a manual alias list.

A returned domain is not proof of a match

Notion Labs, Inc. returned firstround.com. Check domains for rows whose names did not match a known company before using them in later steps. See Comparing What a Run Costs When It Finds Nothing.

The saving depends on the data

This input had 2 duplicates in 6 rows. A list with no repeats saves nothing, and the script then sends every row once.

Result order differs from input order

The script keys results by the item id, not by position.

Next steps

verification:
  status: verified
  tested_at: "2026-10-09"
  product_version: "public/v0"
  command: "npx tsc && node --env-file=.env dist/dedupe-before-enrich.js | tail -1"
  expected_result: "`creditsSpent` equal to the number of unique rows (4) and lower than `withoutDedupe` (6)."

verification:
  status: verified
  tested_at: "2026-10-09"
  product_version: "public/v0"
  command: "npx tsc && node --env-file=.env dist/dedupe-before-enrich.js | tail -1"
  expected_result: "`creditsSpent` equal to the number of unique rows (4) and lower than `withoutDedupe` (6)."