Senior Data Infrastructure Engineer (Web-Scale Crawling Systems)

reflection • United States

Visa Sponsorship

This Job is No Longer Active This position is no longer accepting applications

AI Summary

Design and operate large-scale web crawling infrastructure to collect, process, and optimize internet-scale data for AI model training. Collaborate with researchers to enhance crawl quality, coverage, and efficiency while ensuring scalability, reliability, and compliance with web constraints. Build distributed systems for URL discovery, content extraction, and dataset delivery at petabyte scale.

Key Highlights

Build and maintain web-scale crawling infrastructure for petabyte-scale data collection across billions of URLs

Collaborate with AI researchers to prioritize high-value web content for model performance improvements

Optimize distributed systems for URL discovery, scheduling, crawling, extraction, and dataset delivery

Searching for Development & Programming roles that provide visa sponsorship? Connect with international employers through Development & Programming Jobs with Visa Sponsorship opportunities actively seeking talented professionals.

Key Responsibilities

Design and operate web-scale crawling infrastructure capable of continuously collecting data across billions of URLs

Develop URL discovery, prioritization, scheduling, and crawl orchestration systems for efficient data acquisition

Build distributed crawlers for content extraction, rendering, parsing, and normalization across diverse web formats

Improve crawl coverage, freshness, efficiency, and quality through measurement and experimentation

Design infrastructure for large-scale recrawling, change detection, and incremental updates

Analyze crawl performance and web coverage to identify gaps and optimize system tradeoffs

Build observability, monitoring, and reliability systems for large-scale crawl operations

Technical Skills Required

Distributed Systems Web Crawling & Scaling Large-Scale Data Processing

Explore our comprehensive directory of visa sponsorship jobs from employers worldwide who are ready to sponsor talented international professionals.

Benefits & Perks

Top-tier compensation with salary and equity structured for global retention

Comprehensive health & wellness benefits including medical, dental, vision, and annual wellness allowance

Visa sponsorship for exceptional talent with support for long-term immigration pathways

Interested in opportunities specifically in United State? Discover our dedicated Visa Sponsorship Jobs in United State page featuring roles from top employers in this location.

Nice to Have

Experience building search engines or internet-scale crawling platforms

Familiarity with anti-bot systems, dynamic web content, and large-scale extraction pipelines

Understanding of how web data influences training and evaluating large language models

Job Description

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. We build open models that let anyone control their intelligence and help shape the future of AI. Our mission: make intelligence open and accessible to all.The web is one of the most important sources of information for frontier AI systems. The quality, coverage, freshness, and diversity of web data directly influence model capabilities.

Want the full job description? Read the complete details on LinkedIn, the original posting.

Continue on LinkedIn

This is a short excerpt. All rights to the full description belong to its original publisher.