Middle+ Full-Stack Engineer (Node.js) - Web Crawlers & AI Extraction

About This Role

We are looking for a Middle+ Full-Stack Engineer to build a new Node.js/TypeScript crawler engine and migrate an existing JVM-based crawler system onto it. You'll work across the full lifecycle — from core orchestration and pluggable modules through AI-powered self-healing extraction to production maintenance — with real ownership over the sources you build and support, including on-call.

Responsibilities

• Implement the shared core of the new Node.js/TypeScript crawler engine: run orchestration, module loading, retries and backoff, concurrency and rate control, error handling, run artifacts and logging
• Implement the pluggable modules - browser control, anti-bot routing, extraction, pagination, output/sink - and design the declarative source format (config schema plus output schema contract), kept separate from runtime parameters
• Adopt proven open-source and managed components for commodity layers (browser automation, crawl loop, stealth) rather than rebuilding them; keep in-house effort on what is genuinely specific to this data
• Build the developer workflow around the engine: add a source, test it locally, register it, schedule it, watch it run
• Read existing JVM crawler implementations and port their behavior with no loss of coverage or field-level fidelity, including the per-portal workarounds that are not written down anywhere
• Validate every migration against known-good reference output before switching a source over; prioritize by how often a crawler breaks, not alphabetically; document per-portal quirks during the port
• Keep live crawlers working while the migration runs: triage failures, repair broken selectors, changed navigation flows, moved portals and new pagination behavior - regularly, as a standing part of the job - and take part in the on-call rotation for the sources you own
• Distinguish failure classes correctly - transient error vs. DOM change vs. new anti-bot protection vs. login wall vs. portal permanently gone - and escalate what engineering cannot solve instead of burning days against the wrong wall
• Work through anti-bot protection within legal and ToS boundaries: reverse-engineer portals via network analysis and undocumented JSON endpoints, and pick the right tool per source (plain HTTP, headless browser, managed stealth browser, proxy strategy, CAPTCHA service)
• Build the AI layer: schema-first LLM extraction with deterministic selector fallback, and a self-healing loop (detect, diagnose, repair, verify) under a hard iteration and cost budget, gated on schema verification and a data diff against the last known-good output
• Register sources in the scheduling and monitoring platform, contribute to normalization on the way to the datastore (name matching, deduplication, idempotent writes, partial-run recovery), and extend the Node.js backend services and internal operations UI where crawler status is surfaced

Requirements

• 3+ years in commercial software engineering
• 1.5+ years of production crawler/scraper development in Node.js - crawlers you built, shipped, and then maintained on a schedule against sites you do not control. Hard requirement, verified in the technical interview
• Experience maintaining crawlers in production over time - not one-off scraping scripts, not a single-site project
• Solid Node.js and TypeScript: async patterns, concurrency and rate control, error handling, retries and backoff
• Playwright and/or Puppeteer in production: headless vs. headful trade-offs, browser contexts, storage state, network interception
• Crawling and parsing tooling in the Node ecosystem (Crawlee, Cheerio, or equivalent), plus raw HTTP-level scraping when a browser is unnecessary
• Node backend services and REST APIs; SQL databases (PostgreSQL or equivalent)
• Has written reusable, library-style code that other engineers build on - clean module boundaries, plugin/strategy patterns, typed interfaces, versioned configuration - and can point to a case of generalizing several one-off implementations into one engine
• Hands-on anti-bot experience: Cloudflare Bot Management, Imperva, reCAPTCHA v2/v3, Turnstile, rate limiting; browser and TLS fingerprinting, proxy strategy, session persistence, managed stealth-browser platforms - plus the judgment to recognize when a CAPTCHA solver is irrelevant because the wall is of a different class
• Practical use of LLMs for data extraction: schema-first structured output, prompt design, output validation, token-cost awareness, and judgment about when LLM extraction beats selectors and when it does not - including awareness of confident-but-wrong output and the verification patterns that contain it
• Data quality engineering: schema validation, reference/snapshot testing and data diffs against previous runs, normalization, fuzzy name matching, deduplication, idempotent writes
• Git, code review, meaningful test coverage, CI/CD, Docker, cloud deployment (AWS preferred); able to read JVM code well enough to port it accurately; English at working level for daily technical communication with the client team
• Soft Skills
• Tolerance for recurring maintenance work - the most important non-technical requirement. A meaningful share of the week is unglamorous repeat repair: a portal changed, the crawler broke, find out why, fix it, verify the data, and do it again next week. Candidates looking primarily for greenfield feature work will not enjoy this role
• Investigative persistence - genuinely enjoys reverse-engineering a difficult site and finding the path through it
• Knowing when to stop - recognizes an unwinnable wall early, documents the finding, and escalates with a recommendation instead of grinding on an exhausted approach
• Data-quality conscience - treats a run where the crawler succeeded but the data is subtly wrong as the worst possible outcome, and builds the checks that prevent it
• Ownership - takes end-to-end responsibility for assigned sources: build, register, monitor, repair
• Documentation habit - writes down per-portal quirks instead of keeping them in their head
• Clear communication - can explain to a non-specialist why one portal took an hour and another is impossible

Nice to Have

• Experience migrating a scraping system from one language/runtime to another without downtime
• Experience with config-driven or declarative crawler setups where sources are data rather than code
• Experience with agent frameworks or agentic workflows (tool calling, multi-step loops, human-in-the-loop) - LangGraph, the Claude Agent SDK, or equivalent; agentic browser automation (Browser Use, Stagehand); MCP servers
• Familiarity with managed scraping/extraction platforms (Firecrawl, Kadoa, ScrapeGraphAI, Apify) and with commercial data providers as an alternative to scraping
• Experience with PDF and document data extraction (OCR, table extraction, structured parsing); experience scraping US government or public-registry portals
• Working knowledge of scraping legality and ToS boundaries: robots.txt, rate limiting, authentication walls, redistribution terms. Frontend experience (React) for the internal operations UI is a plus, never a substitute for the crawler requirement

Benefits

• Competitive salary
• Remote work opportunity
• Comfortable work in your local time zone
• Flexible work schedule
• Professional growth and development
• Multicultural working environment