Data Engineer - Web Scraper

Description

Collectibles.com (Collectbase, Inc.) is building the world’s first Web3 marketplace and community for the industry, integrating blockchain technology to create a new consumer experience and innovative business model.

We are U.S.-based company with offices in Germany, venture-backed by leading Web3 and marketplace investors & advisors.

By offering a powerful asset management system, Collectibles.com will help collectors organize, dynamically value, and trade their physical + digital items — with transparent data, fair fees and more trusted transactions. Powered by the industry’s most comprehensive data and a more efficient marketplace model, Collectibles.com will deliver a superior solution and user experience.

As a seed-stage startup with big vision and huge ambitions, we’re currently a small team of experienced entrepreneurs and passionate collectors working to build a new category-defining business and leading destination.

In growing our team and looking selectively for exceptionally talented, fellow travelers, passionate builders who share our desire to innovate and succeed.

Opportunity

Even beating the S&P, collectibles are becoming a more widely recognized alternative financial asset on a worldwide scale.

Massive potential for disruption exists in the collectibles industry and consumer experience, which has remained unchanged for decades. Currently valued at over $400B and with a projected 6% annual growth rate, the collectibles TAM is anticipated to reach nearly $500 billion by 2027.

Requirements

Your Role

We are seeking a talented Data Engineer to join our company's growing team. As a Data Engineer, you will be responsible for designing, building, and maintaining the infrastructure required for optimal extraction, transformation, and loading (ETL) of data from a variety of sources, including web scraping. You will also be responsible for developing and implementing data pipelines to support our business needs. The ideal candidate should have a strong background in data architecture and engineering principles, as well as experience with web scraping. You should be able to work independently and as part of a team, have excellent communication skills, and possess a strong desire to learn and grow within the company. If you are passionate about data and want to make an impact on our company's success, we would love to hear from you.

Your Role

• Design and implement the architecture of a large-scale crawling system (50+ crawlers)
• Design, implement, and maintain various components of our data acquisition infrastructure (building new crawlers, maintain existing crawlers, data cleaners & loaders)
• Work on developing tools to facilitate the scraping at scale, monitor the health of crawlers and ensure data quality of the scraped items
• Collaborate with our product and business teams to understand / anticipate requirements to strive for greater functionality and impact in our data gathering systems

Your Profile

• 2+ Years experience with Python for data wrangling and cleaning
• 2+ Years experience with data crawling & scraping at scale (50+ spiders at least)
• Productionized experience with Scrapy is mandatory
• Solid understanding of web technologies (HTML, JavaScript, CSS, JSON, Selenium, API´s etc)
• Familiarity with data pipelining to integrate scraped items into existing data pipelines
• Ability to maintain all aspects of a scraping pipeline end to end (building and maintaining spiders, avoiding bot prevention techniques, data cleaning and pipelining, monitoring spider health and performance)
• Experience using techniques to protect web scrapers against site ban, IP leak, browser crash, CAPTCHA and proxy failure
• Knowledge using MongoDB, Postgres and Redis is a big plus

Benefits

• Healthcare (Medical/Dental/Vision) coverage
• Holiday Pay: All regular, full-time employees are eligible for paid holidays
• Flexible Schedule: We provide a working environment where you're in charge of your time and schedule.
• Fully remote culture: Work from home (or wherever!)
• Learning budget — Buy courses and books
• Hardware — Whatever you need to get things done