API documentation

Submit your first scrape job in three lines of curl. Replace sk_live_… with the key issued from /dashboard/api-keys.

curl -X POST https://api.scrappy.hu/v1/jobs \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

The response is a 202 with a job_id. Poll GET /v1/jobs/{id} or pass a webhook_url to receive the result via webhook (auto-signed with an HMAC secret returned in the same response).

Every endpoint requires an API key sent as a Bearer token:

Authorization: Bearer sk_live_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Keys are issued from the dashboard. Lost keys cannot be recovered — the server stores only the SHA-256 hash. Rotate via the dashboard.

Rate limits and monthly job quotas are scoped to your account, controlled by your subscription plan. Every authenticated response — including GETs — carries both header families:

X-RateLimit-Limit:     60          # per-minute cap
X-RateLimit-Remaining: 47          # requests left in the 60s sliding window
X-RateLimit-Reset:     38          # seconds until the oldest request expires

X-Quota-Limit:         10000       # monthly cap (omitted on unlimited plans)
X-Quota-Remaining:     8766        # max(0, limit - used)
X-Quota-Reset:         1748736000  # unix-seconds, 1st of next month UTC

GET endpoints tick the rate-limit bucket too — a tight polling loop is throttled the same as a stream of POSTs. The monthly quota is consumed only by job-creating endpoints (POST /v1/jobs, /batch, /:id/retry); GETs don't decrement it.

When the rate limit is hit:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
{
  "error": "rate_limit_exceeded",
  "message": "Rate limit of 60 requests per minute exceeded",
  "retry_after_seconds": 60
}

When the monthly quota is hit:

HTTP/1.1 403 Forbidden
{
  "error": "quota_exceeded",
  "message": "Monthly quota of 10000 requests exceeded",
  "monthly_used": 10000,
  "monthly_quota": 10000,
  "plan": "Free",
  "upgrade_url": "https://scrappy.hu/dashboard/billing"
}

When the requested feature isn't on your plan (e.g. screenshot: true on Free):

HTTP/1.1 402 Payment Required
{
  "error": "feature_not_in_plan",
  "feature": "screenshot",
  "plan": "Free",
  "message": "Screenshot capture is not available on the Free plan."
}

All errors are JSON with this shape:

{ "error": "<code>", "message": "<human readable>", ... }

Common codes:

• unauthorized (401)
• forbidden (403, includes quota_exceeded)
• feature_not_in_plan (402)
• validation_error (400)
• invalid_request (400, e.g. SSRF-blocked URL)
• not_found (404)
• conflict (409, e.g. cancelling a non-queued job)
• rate_limit_exceeded (429)
• internal_error (500)

Accepts a job and returns immediately. Workers pick it up via long-poll within seconds.

POST /v1/jobs HTTP/1.1
Authorization: Bearer sk_live_...
Content-Type: application/json

{
  "url": "https://example.com/product/42",
  "options": {
    "wait_until": "networkidle",
    "timeout_seconds": 30,
    "extract": ["title", "description", "tables", "text", "html"],
    "block_resources": ["image", "media", "font"],
    "screenshot": false,
    "headers": { "Accept-Language": "hu-HU" },
    "viewport": { "width": 1920, "height": 1080 }
  },
  "priority": "normal",
  "webhook_url": "https://your-app.example.com/api/webhook/scrappy",
  "client_reference_id": "wholesale_product_abc-123",
  "metadata": { "feed_id": "198", "tenant": "xupe", "attempt": 1 }
}

Fields

• url — required. http(s); not an internal/private IP; ≤ 2048 chars.
• options.wait_until — load | domcontentloaded | networkidle | commit. Default networkidle.
• options.timeout_seconds — int 5–300. Default 60.
• options.extract — array of title | description | text | tables | html. Default all.
• options.block_resources — array of image | media | font | stylesheet. Default [image, media, font] for ~50% faster fetches with no impact on extraction.
• options.screenshot — boolean. Captures a viewport PNG; available on paid plans only.
• options.headers — object. Extra request headers.
• options.viewport — { width, height }. Default 1920×1080.
• priority — low | normal | high. Default normal.
• webhook_url — HTTPS endpoint we POST job.completed / job.failed events to. The signing secret comes back in the response (webhook_secret), exactly once. SSRF-checked at submit (no private IPs / loopback).
• client_reference_id — your own correlation token (e.g. product id, feed entry). String ≤ 128 chars, allowed characters [a-zA-Z0-9-_:.]. Not unique — multiple jobs may share one. Indexed for GET /v1/jobs?client_reference_id=… lookups.
• metadata — flat scalar bag echoed back on reads and webhook payloads. ≤ 50 keys, key ≤ 40 chars matching [a-zA-Z0-9_], values must be string, number, or boolean (no nested objects, no arrays). Serialized JSON ≤ 8 KB.

Response — 202 Accepted

{
  "job_id": "01HZ4...",
  "status": "queued",
  "created_at": "2026-04-30T10:30:00Z",
  "estimated_wait_seconds": 5
}

Same per-job shape as POST /v1/jobs, wrapped in a jobs[] array. Atomic on quota: if the batch wouldn't fit under your plan's monthly cap, none of the jobs are queued and the response is 403 with quota_exceeded. Counts as a single rate-limit hit but N quota hits.

POST /v1/jobs/batch HTTP/1.1
Authorization: Bearer sk_live_...
Content-Type: application/json

{
  "jobs": [
    { "url": "https://example.com/product/1" },
    { "url": "https://example.com/product/2", "options": { "screenshot": true } },
    { "url": "https://example.com/product/3", "metadata": { "sku": "ABC" } }
  ]
}

Response — 202 Accepted

{
  "jobs": [
    { "job_id": "01HZ41...", "status": "queued", "created_at": "..." },
    { "job_id": "01HZ42...", "status": "queued", "created_at": "..." },
    { "job_id": "01HZ43...", "status": "queued", "created_at": "..." }
  ],
  "count": 3
}

Validation is up-front: a single bad URL or a screenshot flag on the Free plan rejects the whole batch with the offending index in the error body. No partial success — by design.

Returns the full job, plus its result if completed. Scoped to the authenticated key — you can only read jobs you submitted.

{
  "job_id": "01HZ4...",
  "status": "completed",
  "url": "https://example.com/product/42",
  "priority": "normal",
  "attempts": 1,
  "max_attempts": 3,
  "created_at": "2026-04-30T10:30:00Z",
  "started_at": "2026-04-30T10:30:02Z",
  "completed_at": "2026-04-30T10:30:05Z",
  "result": {
    "success": true,
    "http_status": 200,
    "final_url": "https://example.com/product/42",
    "title": "Product 42 — Example Shop",
    "description": "...",
    "text_content": "...",
    "tables": [],
    "html": "<!doctype html>...",
    "fetch_time_ms": 2340,
    "screenshot_url": "2026/05/03/<jobId>.png"
  }
}

While the job is queued, assigned, or running the result field is omitted. screenshot_url is a storage key — view the image in the dashboard.

Newest first. Filter via query string.

• status — exact status filter.
• before — ISO-8601 cursor; jobs created before this instant.
• limit — 1–100, default 20.

{
  "items": [
    { "job_id": "...", "status": "completed", "url": "...", "created_at": "...", "completed_at": "..." }
  ],
  "count": 20
}

Only allowed while status = queued. Returns 409 if the job already started.

Send Idempotency-Key: <opaque-token> (≤ 255 printable ASCII chars) on any POST /v1/jobs, POST /v1/jobs/batch, or POST /v1/jobs/:id/retry request to make the call safe to retry. The cache lives 24 hours.

• Same key + same body → returns the cached response from the first attempt; no duplicate job is created.
• Same key + different body → 422 idempotency_key_mismatch. Use a fresh key for a new logical request.
• Same key while the original is still in flight → 409 idempotent_request_in_progress. Retry shortly.
• 5xx responses are not cached, so transient server errors stay retryable.

Keys are scoped per API key — two different keys can use the same token without collision.

POST /v1/jobs HTTP/1.1
Authorization: Bearer sk_live_...
Idempotency-Key: order-7e3a-attempt-1
Content-Type: application/json

{ "url": "https://example.com/product/42" }

Clones a terminal-failed job's full configuration — url, options, priority, webhook_url, client_reference_id, metadata — into a fresh job and queues it. Returns the new job_id; the original stays in its terminal state for audit. The new job's retried_from points back to the original. A retry with a webhook gets a brand-new webhook_secret in the response — the original failed job's secret is never reused.

Counts against your monthly quota and rate limit like a normal POST /v1/jobs. Only failed and cancelled jobs are retryable — queued | assigned | running are still in flight, completed jobs should be re-submitted via POST /v1/jobs.

{
  "job_id": "<new-id>",
  "status": "queued",
  "created_at": "2026-05-04T10:42:00Z",
  "retried_from": "<original-id>",
  "estimated_wait_seconds": 5
}

Errors: 404 not_found, 409 conflict (status not retryable), 403 quota_exceeded, 429 rate_limit_exceeded.

Synchronously POSTs a webhook.test envelope to the URL you specify (or the API key's default_webhook_url when the body is empty), then returns the consumer's response inline. Use it to confirm your endpoint is reachable, your TLS chain works, and your HMAC verifier accepts the Stripe-style signature — without waiting for a real job to finish.

The signing secret is generated fresh per test, returned in the response, and never persisted server-side — safe to call from CI without polluting webhooks_log or your monthly_used counter. Counts against the rate limit (real network call).

POST /v1/webhooks/test HTTP/1.1
Authorization: Bearer sk_live_...
Content-Type: application/json

{ "webhook_url": "https://your-app.example.com/webhook" }

{
  "webhook_url": "https://your-app.example.com/webhook",
  "webhook_secret": "f3a9c1...64-hex-chars",
  "payload": {
    "event": "webhook.test",
    "api_version": "2026-05-04",
    "timestamp": "2026-05-05T17:50:57.300Z",
    "message": "Test delivery from Scrappy. ..."
  },
  "delivered": true,
  "response_status": 200,
  "response_body": "ok",
  "duration_ms": 117,
  "error": null
}

event === "webhook.test" is distinct from job.completed / job.failed — your handler can early-return on it without running real job processing.

Single-call snapshot for SDKs and dashboards. Returns the calling key, the plan in effect, and the current monthly usage. Counts against the rate limit but is cheap (one indexed select).

{
  "api_key": {
    "id": "01HZ4...",
    "prefix": "sk_live_AB12CD",
    "name": "Production",
    "is_active": true,
    "created_at": "2026-04-30T10:30:00Z",
    "last_used_at": "2026-05-05T07:12:00Z",
    "default_webhook_url": "https://your-app.example.com/api/webhook/scrappy"
  },
  "plan": {
    "name": "Pro",
    "monthly_quota": 10000,
    "rate_limit_rpm": 60,
    "max_api_keys": 5,
    "screenshot_enabled": true
  },
  "usage": {
    "scope": "user",
    "monthly_used": 1234,
    "monthly_quota": 10000,
    "monthly_remaining": 8766,
    "rate_limit_rpm": 60,
    "resets_at": "2026-06-01T00:00:00.000Z"
  }
}

• usage.scope is "user" when a plan is in effect — the counter aggregates across every API key belonging to the user. "key" for legacy orphan keys (per-key counter).
• usage.monthly_quota and usage.monthly_remaining are null on unlimited plans. plan is null for orphan keys.
• usage.resets_at is UTC midnight of the first of next month — when the quota reset scheduler clears monthly_used back to zero.

Public, no auth. Returns dependency-level checks (database, redis, storage) and current queue depths so external monitors can discriminate failure modes. Worker fleet info is intentionally not exposed.

{
  "status": "ok",
  "uptime_seconds": 12345,
  "checks": {
    "database": { "status": "ok", "latency_ms": 3 },
    "redis":    { "status": "ok", "latency_ms": 1 },
    "storage":  { "status": "ok", "latency_ms": 42 }
  },
  "queue": { "scrape_depth": 12, "webhook_depth": 0 }
}

status is ok when every probe succeeds, degraded (200) when an optional dependency like S3 is down but the core path works, unhealthy (503) when DB or Redis is down.

When webhook_url is set, Scrappy POSTs an envelope when the job reaches a terminal state. Two events: job.completed and job.failed. Cancelled jobs do not fire webhooks.

{
  "event": "job.completed",
  "api_version": "2026-05-04",
  "timestamp": "2026-05-05T17:50:57.300Z",
  "job_id": "058a61d5-54e6-45e2-b6f5-30a4d5bf3354",
  "status": "completed",
  "url": "https://example.com/product/42",
  "client_reference_id": "wholesale_product_abc-123",
  "metadata": { "feed_id": "198", "tenant": "xupe" },
  "result": { ... },
  "error": null,
  "attempts": 1,
  "completed_at": "2026-05-05T17:50:57.300Z"
}

Headers:

Content-Type:        application/json
X-Scrappy-Job-Id:    058a61d5-...
X-Scrappy-Event:     job.completed
X-Scrappy-Signature: t=1746381057,v1=<hex hmac-sha256>

Delivery retries with backoff 1m, 5m, 30m, 2h, 6h (5 attempts, ≈9 hours total). Anything in the 2xx range stops retries; everything else (including timeouts at 15s) triggers a retry. After the 5th failure the delivery is dead-lettered — visible via GET /v1/jobs/:id/webhook_attempts.

Verifying the signature (Node.js)

The signed value is {timestamp}.{raw-body} — thet= from the header, dot, then the raw request body. Reject if the timestamp is older than ~5 minutes to defend against replay.

import { createHmac, timingSafeEqual } from 'node:crypto';

function verify(rawBody: string, header: string, secret: string): boolean {
  const parts = Object.fromEntries(header.split(',').map((p) => p.split('=')));
  const t = parts.t;
  const v1 = parts.v1;
  if (!t || !v1) return false;

  // Replay defence: refuse signatures older than 5 minutes.
  const ageS = Math.floor(Date.now() / 1000) - Number(t);
  if (!Number.isFinite(ageS) || ageS > 300 || ageS < -60) return false;

  const expected = createHmac('sha256', secret)
    .update(`${t}.${rawBody}`, 'utf8')
    .digest('hex');
  if (expected.length !== v1.length) return false;
  return timingSafeEqual(Buffer.from(expected, 'hex'), Buffer.from(v1, 'hex'));
}

Important: verify against the raw body before JSON parsing — re-serialisation can change the bytes and break the signature.

Returns the per-attempt log for a job's webhook deliveries — status code, response body (≤ 1 KB), duration, error message. Brutal-useful when a customer says "your webhook never reached me."

{
  "job_id": "058a61d5-...",
  "attempts": [
    {
      "attempt": 1,
      "timestamp": "2026-05-05T17:50:57.300Z",
      "response_status": 500,
      "response_body": "Internal Server Error",
      "duration_ms": 423,
      "error": "HTTP 500"
    }
  ],
  "count": 1
}

Successful and failed jobs are retained for 24 hours after completed_at, then deleted along with their result row. Cancelled jobs follow the same schedule.