Browser Crawl Engine

Quickstart

Four requests: submit, poll, read, download. The script at the bottom does all four and is safe to copy as-is.

1. Get an API key

Register, then wait for your account to be reviewed. On approval you receive a key of the form bce_live_<id>_<secret>.

The key is shown exactly once. Only a hash is stored, so it cannot be recovered or re-displayed. Store it in a secret manager or an environment variable before you close the page.

2. Submit a job

POST /v1/jobs
export API_KEY='bce_live_…'
export API='https://bankworks.info'

curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "urls": ["https://example.com/"],
    "maxAttempts": 3,
    "resultTtlSeconds": 3600,
    "options": {
      "geo": {"country": "de"},
      "enableJavaScript": true,
      "postLoadDelayMs": 5000
    }
  }'
{"ok": true, "jobId": "0199…", "taskCount": 1, "deliveryDeferred": 0}

3. Poll the job

GET /v1/jobs/{jobId}
curl -sS -H "Authorization: Bearer $API_KEY" \
  "$API/v1/jobs/$JOB_ID?limit=100&offset=0"

The response carries job.totalTasks, job.doneTasks and job.failedTasks, plus the task rows. A job is finished when doneTasks + failedTasks === totalTasks.

Poll every 3–5 seconds, not continuously. A full page render takes about 13 seconds, so polling faster only spends your rate limit. Each poll counts against the 10 requests/second your account starts with.

4. Download the HTML

GET /v1/tasks/{taskId}/html
curl -sS -H "Authorization: Bearer $API_KEY" \
  -o page.html -D headers.txt \
  "$API/v1/tasks/$TASK_ID/html"

grep -i x-html-sha256 headers.txt

X-HTML-SHA256 is computed over the stored bytes, so you can verify what you received. The response is an attachment by default; add ?inline=1 to render it in a browser.

The whole thing as one script

#!/usr/bin/env bash
set -euo pipefail

API="${API:-https://bankworks.info}"
: "${API_KEY:?set API_KEY first}"
URL="${1:?usage: crawl.sh <url>}"

job=$(curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"urls\":[\"$URL\"],\"options\":{\"geo\":{\"country\":\"de\"}}}")

job_id=$(printf '%s' "$job" | jq -r '.jobId')
echo "job $job_id"

# Poll until every task has finished. The 5s interval is chosen against a
# ~13s render: anything faster just spends the rate limit.
while :; do
  state=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$job_id")
  done_n=$(printf '%s' "$state" | jq -r '.job.doneTasks')
  fail_n=$(printf '%s' "$state" | jq -r '.job.failedTasks')
  total=$(printf '%s' "$state" | jq -r '.job.totalTasks')
  echo "  $done_n done, $fail_n failed of $total"
  [ "$((done_n + fail_n))" -ge "$total" ] && break
  sleep 5
done

printf '%s' "$state" | jq -r '.tasks[] | select(.status=="DONE") | .taskId' |
while read -r task; do
  curl -sS -H "Authorization: Bearer $API_KEY" \
    -o "$task.html" "$API/v1/tasks/$task/html"
  echo "  wrote $task.html"
done

Next

← Overview Authentication →