Web Scraping Engineer

Where You Come In:

• Help to design and implement the data crawling architecture and a large-scale crawling system
• Work on developing not just large-scale scraping tools and APIs but also data integrity, health and monitoring systems.
• Collaborate with our product and business teams to understand / anticipate requirements to strive for greater functionality and impact in our data gathering systems
• Design, implement, and maintain various components of our data infrastructure

What you'll be bringing to the team:

• 2+ Years experience with Python for data wrangling and cleaning
• Advanced experience with SQL
• Proficiency in crawling & scraping using any libraries such as Scrapy, BeautifulSoup, Selenium
• Experience extracting data from multiple disparate sources including Web, PDF, and spreadsheets.
• Productionized experience with techniques and tools for crawling, extracting and processing data
• Sound Knowledge in bypassing Bot Detection Techniques
• Experience using HTTP Proxy techniques to protect web scrapers against site ban, IP leak, browser crash, CAPTCHA and proxy failure.
• Experience with cloud environments like GCP, AWS, as well as cloud solutions like Kubernetes, Docker, etc
• Ability to maintain all aspects of an analytics tech stack (analytics tool implementation, pipeline maintenance, data integrity)
• Expertise in data warehouse maintenance, specifically with Google BigQuery (ETLs, data sourcing, modelling, cleansing, documentation, and maintenance)

Even better if you have, but not necessary:

• Experience with microservices architecture would be a plus.
• Familiarity with message brokers such as Kafka, RabbitMQ, etc
• Experience with DevOps