Web Automation Engineer

About the Role

We are seeking a highly skilled Web Scraping Engineer to design, develop, and maintain scalable data extraction systems that gather structured information from websites and online platforms. The ideal candidate will have strong experience with automation, data pipelines, and web technologies, along with a deep understanding of handling dynamic content, large-scale scraping, and data integrity.

Key Responsibilities

• Design, implement, and optimize web scraping pipelines.
• Develop robust, scalable, and maintainable data extraction systems capable of handling high-volume or frequently changing websites.
• Create data cleaning and normalization workflows to ensure data accuracy and consistency.
• Manage and schedule scraping jobs using tools like Airflow, Celery, or Cron.
• Implement proxy rotation, CAPTCHA solving, and rate limiting to ensure reliable scraping without service disruptions.
• Work closely with the data science and analytics teams to deliver clean and structured datasets for analysis or downstream applications.
• Monitor and maintain scraping infrastructure; proactively handle site structure changes or scraping failures.
• Ensure compliance with legal, ethical, and terms-of-service guidelines related to web data collection.
• Optimize scraping performance, speed, and resource usage across distributed systems or cloud environments.

Required Qualifications

• Proficiency in Python and libraries like Scrapy, BeautifulSoup, Requests, Playwright, or Selenium.
• Experience deploying scrapers on cloud platforms (AWS, GCP, Azure) or using serverless solutions.
• Knowledge of distributed scraping frameworks (e.g., Scrapy Cluster, Frontera).
• Familiarity with data engineering pipelines, message queues (RabbitMQ, Kafka), and containerization (Docker, Kubernetes).
• Understanding of anti-bot detection systems and strategies to mitigate them responsibly.
• Experience integrating scraping systems with ETL workflows or data warehouses.

What We Offer

• Competitive salary and benefits package
• Flexible work hours and remote options
• Yearly overseas offsite
• Visa sponsorship

NOTE: PLEASE SEND YOUR CV, PORTFOLIO AND DESIRED SALARY TO [email protected]. IF YOU DO NOT PROVIDE THIS, YOU WILL NOT BE CONSIDERED.