Data Engineer - job 3 of 4
Job Title: Data Engineer
Job Description:
Seeking a skilled Data Engineer with a robust background in PySpark and extensive experience with AWS services, including Athena and EMR. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing systems, ensuring efficient and reliable data flow and transformation.
Key Responsibilities:
• Data Pipeline Development: Design, develop, and maintain scalable data pipelines using PySpark to process and transform large datasets.
• AWS Integration: Utilize AWS services, including Athena and EMR, to manage and optimize data workflows and storage solutions.
• Data Management: Implement data quality, data governance, and data security best practices to ensure the integrity and confidentiality of data.
• Performance Optimization: Optimize and troubleshoot data processing workflows for performance, reliability, and scalability.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Documentation: Create and maintain comprehensive documentation of data pipelines, ETL processes, and data architecture.
Required Skills and Qualifications:
• Education: Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• Experience: 5+ years of experience as a Data Engineer or in a similar role, with a strong emphasis on PySpark.
• Technical Expertise:
o Proficient in PySpark for data processing and transformation.
o Extensive experience with AWS services, specifically Athena and EMR.
o Strong knowledge of SQL and database technologies.
o Experience with Apache Airflow is a plus
o Familiarity with other AWS services such as S3, Lambda, and Redshift.
• Programming: Proficiency in Python; experience with other programming languages is a plus.
• Problem-Solving: Excellent analytical and problem-solving skills with attention to detail.
• Communication: Strong verbal and written communication skills to effectively collaborate with team members and stakeholders.
• Agility: Ability to work in a fast-paced, dynamic environment and adapt to changing priorities.
Preferred Qualifications:
• Experience with data warehousing solutions and BI tools.
• Knowledge of other big data technologies such as Hadoop, Hive, and Kafka.
• Understanding of data modeling, ETL processes, and data warehousing concepts.
• Experience with DevOps practices and tools for CI/CD.
Average salary estimate
If an employer mentions a salary or salary range on their job, we display it as an "Employer Estimate". If a job has no salary data, Rise displays an estimate if available.
Seeking a seasoned SAP Technical Lead to drive and manage S/4HANA AMS support and technical operations in a dynamic environment.
A remote contract opportunity for a seasoned SAP Fiori Consultant to design and implement innovative SAP Fiori applications aligned with business needs.
Lead and innovate big data solutions as the Director of Data Solutions at AssistRx, driving technical strategy and team performance remotely.
Maven Clinic is looking for a Senior Data Engineer to build and optimize high-performance data pipelines that empower data-driven decision-making for their award-winning healthcare platform.
Lead the design and implementation of AWS-centric data integration solutions as a Solution Lead / Architect in a hybrid working environment.
Lead and mentor a multidisciplinary data engineering team to develop high-performance cloud-native data platforms at General Motors.
Serving Tulsa’s growth, this Data Warehouse Engineer role leverages expertise in ETL, Python, and cloud data platforms to enhance data infrastructure and support impactful community initiatives.
Samba TV seeks a VP of Data Strategy to spearhead data acquisition and partnerships, enhancing its global media intelligence platform.
Data Engineer position at PNC to develop data services and solutions for regulatory reporting and risk analysis within a hybrid work environment.
Lead data sourcing architecture initiatives for regulatory reporting automation at American Express, driving essential financial data solutions and partnerships.
Join Hireframe as a Remote Data Engineer to drive data onboarding, client collaboration, and process enhancement for enterprise customers.
Seeking a Finance Data Governance Manager at American Express to drive metadata solutions and data governance initiatives ensuring trusted financial data for strategic decision-making.
An established U.S.-based insurance group is looking for a skilled Data Engineer to develop and implement complex data products and pipelines within a remote work environment.
Contribute to Navy operations by developing robust, automated database solutions and metrics analysis as a Database Developer with ProSidian Consulting.
Experienced Data Architect with strong SQL and data warehousing skills needed at PNC to drive data integration and analytics for strategic decision-making.
Our IT solutions empower organizations and individuals throughout the world to maximize value and quality to succeed in today's challenging business environment. As a fast-growing new economy company, we focus our strengths to offer world-class so...
268 jobsSubscribe to Rise newsletter
🔮
Hi, I'm Risa! Your AI Career Copilot
you, just ask me below!
