Anakin vs ParseHub: scraper definitions belong in code

Production scraping should not depend on invisible state captured by a desktop app. The moment web data feeds another system, the scraper definition becomes source code: reviewable, testable, versioned, and owned by the service that consumes it. A tool that exposes an API only to run a saved visual project is useful, but it is not the same abstraction as an API that defines the scrape at request time.

The hidden state problem

ParseHub is good at what visual scrapers are supposed to do: a person opens a desktop app, clicks through a page, selects fields, configures pagination, and runs the project. For a one-off spreadsheet job, that is often faster than writing code.

The problem starts when the word API enters the conversation. ParseHub's REST API can start a project and retrieve the output, but the project itself still lives in the visual builder. The API does not define the extraction logic. It triggers a saved artifact that someone built somewhere else.

That distinction matters because hidden state breaks normal engineering workflow. A selector change cannot be reviewed in a pull request. A pagination rule cannot be unit tested next to the ingestion code. A production incident cannot be diagnosed from the service repository because the important logic is outside it.

The failure mode is usually not dramatic. The run completes, the HTTP status is fine, and the downstream job receives an empty array. Or worse, a changed table layout shifts values into the wrong columns, so a price becomes a SKU and nobody notices until analytics looks impossible.

Anakin takes the other side of that line: the URL, rendering choice, geo, and structured extraction live in the request, so the job can be generated by application code rather than prepared in a GUI.

Make extraction failure impossible to ignore

A scraping pipeline should treat extraction as a contract. Transport success only proves that a page loaded. It says nothing about whether the page contained the fields the pipeline needs.

A minimal guard looks like this:

const productContract = {
  title: 'string',
  price: 'number',
  inStock: 'boolean'
};

function assertProductScrape(job) {
  if (job.status !== 'completed') {
    throw new Error(`SCRAPE_JOB_FAILED status=${job.status}`);
  }

  const data = job.data ?? {};
  const missing = [];
  const wrongType = [];

  for (const [field, expectedType] of Object.entries(productContract)) {
    if (!(field in data) || data[field] === null) {
      missing.push(field);
      continue;
    }

    if (typeof data[field] !== expectedType) {
      wrongType.push(`${field}: expected ${expectedType}, got ${typeof data[field]}`);
    }
  }

  if (missing.length || wrongType.length) {
    throw new Error(
      `SCRAPE_CONTRACT_VIOLATION missing=[${missing.join(', ')}] wrongType=[${wrongType.join('; ')}]`
    );
  }

  return {
    title: data.title.trim(),
    price: data.price,
    inStock: data.inStock,
    scrapedAt: new Date().toISOString()
  };
}

async function scrapeProduct(client, url) {
  const job = await client.scrape({
    url,
    renderJavascript: true,
    country: 'US',
    extract: {
      title: 'product name',
      price: 'current numeric price',
      inStock: 'whether the product can be purchased'
    }
  });

  return assertProductScrape(job);
}

This pattern turns bad data into an explicit exception. Without it, the first visible error may appear three systems later as Cannot read properties of undefined, a failed warehouse load, or a customer-facing page with a blank price.

Visual projects make this harder because the extraction definition lives outside the validation code. API-native scraping keeps both sides of the contract close enough to change together.

Throughput caps are architecture, not pricing details

Page limits and worker speed look like commercial details until a pipeline has to backfill data.

ParseHub's free tier has a 200-pages-per-run limit, and public documentation describes worker speed around 5 pages per minute per worker. Those numbers can be perfectly acceptable for a small directory. They become architectural constraints for a catalog, monitoring system, or enrichment job.

A 15,000-page backfill under page-per-run caps forces sharding. Sharding saved visual projects creates new failure modes: duplicate pages, missed ranges, inconsistent selector versions, and partial runs that look complete because each run ended normally.

A better model exposes rate limits directly. Application code can batch URLs, retry only failed jobs, preserve idempotency keys, and store per-URL status. The system can say exactly which 37 pages failed instead of asking an operator to inspect a desktop project run.

Credentials should not live in recorded clicks

Scraping often begins with public pages and then drifts into authenticated workflows: vendor portals, dashboards, partner catalogs, invoice systems. At that point, credential handling becomes part of the scraping architecture.

Plain-text credentials inside recorded steps are the wrong primitive. They mix secrets with scraper behavior, make project sharing dangerous, and complicate incident response. ParseHub's own documentation has historically described run data as stored in plain text by default, encryption only on higher plans, and free-plan projects as public. Its login guidance also advises throwaway accounts for some cases, which is a clear signal about the trust boundary.

Production systems should pass credential references, not passwords. The job should say credentialRef: 'vendor_portal_readonly', and the execution layer should resolve that secret at run time. The API should never return the secret, and logs should never contain it.

For authenticated scraping workflows, Wire treats site actions as API-callable units that can use vault-backed credentials without returning the secret to the caller.

Geo and rendering belong in the job request

A scrape without geography is often underspecified. Retail prices change by country. Marketplaces show different sellers by region. Cookie banners and consent flows vary by jurisdiction. Some sites serve a bot wall from one location and a normal page from another.

When geo is not a request parameter, engineers lose repeatability. A product monitor might pass from a US machine and fail from a German one because a consent modal covers the buy button. The symptom looks like a selector bug, but the real bug is that the scrape contract never named the country.

The same applies to JavaScript rendering. Modern pages routinely ship empty HTML shells and hydrate content client-side. Treating rendering as an implementation detail leads to intermittent null fields when the page loads slower than the recorded click path expected.

ParseHub can handle dynamic sites inside its visual workflow and offers IP rotation on paid plans, but country targeting is not the same as rotating IPs. For code-owned pipelines, rendering and region should sit beside the URL as ordinary job inputs.

Visual scrapers still have a place

ParseHub is not the wrong tool for everyone. A non-developer who needs a weekly CSV from a stable public directory can get real value from a visual builder. The ability to click through a site and export data is useful, especially when the job is small and manual maintenance is acceptable.

The wrong move is promoting that workflow into infrastructure without changing abstractions. Once scraped data feeds search, billing, analytics, model evaluation, alerts, or customer-visible features, the scraper needs the same engineering treatment as any other dependency.

That means definitions in code, validation at the boundary, explicit retries, observable partial failures, safe credentials, and request-level control over rendering and geo.

The decision rule

The practical evaluation is simple: write the scrape request and response that the pipeline should own. Include URL, schema, rendering, country, timeout, credential reference, cache behavior, and the exact shape of partial failure.

If a tool can execute that contract directly, it can fit into an engineering system. If it can only start a project that someone already clicked together, it belongs to a different category. Use the visual scraper for bounded manual extraction. Use an API-native scraper when the data pipeline has to be operated like software.