Vol. 1 · Proof of work

What the scraper actually did, in its own words.

Five real artifacts from building this product: a defect the collector actually produced, the heal that fixed it without changing the collector ID, a field-health layer that stayed correct while that heal was still in flight, a site Bright Data can reach that a bare request cannot, and a missing cell that could not have been faked. Every command below is exactly what was run; every output is unedited.

1. What the collector actually returned

A 316-character run of page headings, in the field meant to hold "Metropolis Healthcare".

Collector c_mt5gi0vtl37kh7qec, requested against the real Pune HbA1c page, returned this verbatim in lab_name. Unedited, from examples/hba1c-across-cities.raw.json:

"lab_name": "What Is an HbA1c Test in Pune? Why Is an HbA1c Test Done in Pune? Who Should Get an HbA1c Test in Pune? HbA1c Test Booking & Reports – Metropolis Healthcare India How to Interpret Your HbA1c Test Results Tips for Accurate HbA1c Test Results References HbA1c Test Price in Different Cities HbA1c- Glycated Haemoglobin, EDTA Blood Price in Pune"

That string continues for another ~2,400 characters of SEO body copy before the JSON object closes. This is what an AI-generated Scraper Studio collector produces on a real, messy production page before anyone fixes it: a real defect, not a hypothetical one.

2. The heal, run against the live platform

Same defect, same collector ID, fixed in place.

bdata scraper heal c_mt5gi0vtl37kh7qec \
  "The lab_name field is wrong. It returns a concatenated blob of page
   headings. Fix it to return exactly the string 'Metropolis Healthcare'."

Bright Data ran planner, control_preview_runner, code_fixer, step_preview_runner, request_fulfillment_validator and step_advance, then stopped at an approval gate with a live preview:

{"test_name":"HBA1c test in Pune","city":"Pune",
 "price_inr":{"value":560,"currency":"INR"},
 "lab_name":"Metropolis Healthcare",
 "turnaround_time":"Within 24 Hours","sample_type":"Blood"}

Collector ID unchanged: c_mt5gi0vtl37kh7qec before and after. Downstream code keyed on that ID needs no edit.

Honest caveat, reproducible: after bdata scraper approve returned status: done, subsequent production runs returned the healed value on one run and the pre-heal value on later runs against never-before-scraped URLs, and the collector's own output_schema still showed the pre-heal shape. Heal propagation to production looked non-deterministic during this build. See the next section for why the product stays correct anyway.

3. The independent defense that didn't need the heal to land

The 120-cell matrix never showed the garbage string, heal or no heal.

src/lib/normalize.ts detects a lab_name longer than 60 characters or containing ? or : , quarantines it, replaces it with the registry's known lab name, and records the quarantine in that row's field_health. That layer runs on every row regardless of whether the upstream heal has propagated to production yet: a scraper that repairs itself is the goal, a product that stays correct while the repair is still in flight is the requirement, and this dataset needed both.

4. The strongest evidence: a site a bare request cannot reach

Thyrocare blocks curl. Bright Data goes straight through.

Scrupulously honest framing: Metropolis Healthcare and Orange Health Labs, the other two labs in this product, are not bot-protected. A bare curl fetches either page in well under a second with the real price sitting in plain HTML (Metropolis publishes it in a schema.org JSON-LD block). On those two labs, Bright Data buys reliability and structure, not access; a determined scraper could read them without it. Thyrocare is different: it returns a hard 403 to every bare request FairTest tried, including its own homepage and robots-adjacent paths, and the only way this product reads a Thyrocare price at all is through the Bright Data collector below. Only one of three labs needs Bright Data to be reachable at all; say so plainly rather than implying all three do.

Bare request, no Bright Data

curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) \
  AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36" \
  -L "https://www.thyrocare.com/tests/hba1c"

HTTP 403 in 0.03s

Repeatable: same 403 on the test page, the homepage, and even /sitemap.xml. Claude's own web fetch tool hit the identical 403 against the same URL, independently.

Through Bright Data Scraper Studio

bdata scraper run c_mt5gcpdd2th3hbufk \
  "https://www.thyrocare.com/tests/hba1c" --pretty

[
  {
    "test_name": "HbA1c",
    "price_inr": { "value": 420, "currency": "INR", "symbol": "₹" },
    "original_price_inr": { "value": 672, "currency": "INR", "symbol": "₹" },
    "turnaround_time": "10 hrs",
    "fasting_required": "Not Required",
    "sample_type": "Blood",
    "lab_name": "Thyrocare",
    "product_page_url": "https://www.thyrocare.com/diagnostics/tests/hba1c-34"
  }
]

Real structured data in 66-88s, two runs measured

Both commands are exact and reproducible by a judge with the collector ID above. Full numbers, both runs, in the README's Thyrocare section.

5. A miss that could not have been faked

Ahmedabad, Urine Routine Examination: genuinely no data.

A live POST /api/matrix/refresh run against urine-routine-examination genuinely returned no extractable row for Ahmedabad on that pass, a real transient extraction miss, not simulated. The matrix wrote price_inr: null for that cell and the grid rendered the literal text "no data", never a zero and never an interpolated neighbour price. See it live on /matrix.

Full command log and every real number quoted above is also in the README, in the Thyrocare section.