We are seeking a talented and experienced Senior Data Engineer (Quantexa)with expertise in Hadoop, Scala, Spark, Elastic, Open Shift Container Platform (OCP) and DevOps practices. Elasticsearch to join our team. As a Data Engineer, you will play a crucial role in designing, developing, and optimizing big data solutions using Apache Spark, Scala, and Elasticsearch. You will collaborate with cross-functional teams to build scalable and efficient data processing pipelines and search applications. Knowledge and experience in the Compliance / AML domain will be a plus. Working experience with Quantexa tool is a must.
Responsibilities:
· Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives
· Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions
· Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
· Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
· Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
· Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
· Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
· Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
· Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
· Ensure data quality and integrity throughout the data processing lifecycle
· Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques
· Optimize data engineering workflows for containerized deployment and efficient resource utilization
· Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability
· Implement data governance practices, data lineage, and metadata management to ensure data accuracy, traceability, and compliance
· Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements
· Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure
· Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference
If an employer mentions a salary or salary range on their job, we display it as an "Employer Estimate". If a job has no salary data, Rise displays an estimate if available.
Experienced Data Engineer sought to create scalable data solutions for a dynamic, full-service advertising agency operating across multiple locations.
Experienced Snowflake Data Engineer with strong Python and advanced SQL expertise needed to collaborate across teams and deliver data solutions.
Experienced Data Engineers are needed at Reach Security to develop and optimize scalable data pipelines using Apache Airflow or Dagster, contributing directly to mission-critical backend infrastructure.
Senior Data Engineer needed at Spotify to build and maintain scalable data infrastructure for their podcast and advertising products.
InVitro Capital is hiring a Senior Data & ML Engineer to design scalable data systems and ML workflows in a hybrid role based in Irvine, CA.
Vizient is looking for a knowledgeable Data Engineer to develop and optimize data warehousing solutions with a focus on Microsoft Azure technologies.
As a Staff Data Engineer at Bolt, you'll architect and scale data systems that power innovative financial and commerce solutions in a remote-first, inclusive environment.
A Senior Data Engineer role at Appian in McLean, VA, focused on designing and managing robust data pipelines to empower enterprise-wide analytics and decision-making.
Lead the design and implementation of scalable scientific data infrastructure at CarbonCapture to accelerate climate solutions via advanced data engineering.
Experienced Data Engineer needed for a 12-month contract role with Element, building data pipelines and lakes to support healthcare oversight.
Tyndale seeks an experienced Data Engineer to design and optimize data pipelines supporting advanced AI and machine learning projects within a hybrid work setting.
A Senior Data Engineer role at SmarterDx to develop scalable, cloud-based data pipelines and support clinical AI innovations in healthcare.
Redhorse Corporation is recruiting a Mid-Level Data Engineer to enhance data-driven decision-making within the Department of Defense through advanced data pipeline development.
Unison helps you create extraordinary experiences for your employees, your customers, your community, our world.
128 jobsSubscribe to Rise newsletter