Let’s get started
By clicking ‘Next’, I agree to the Terms of Service
and Privacy Policy, and consent to receive emails from Rise
Jobs / Job page
Senior Data Engineer (Quantexa, Spark ,Scala, Elastic Search) image - Rise Careers
Job details

Senior Data Engineer (Quantexa, Spark ,Scala, Elastic Search) - job 1 of 2

We are seeking a talented and experienced Senior Data Engineer (Quantexa)with expertise in Hadoop, Scala, Spark, Elastic, Open Shift Container Platform (OCP) and DevOps practices. Elasticsearch to join our team. As a Data Engineer, you will play a crucial role in designing, developing, and optimizing big data solutions using Apache Spark, Scala, and Elasticsearch. You will collaborate with cross-functional teams to build scalable and efficient data processing pipelines and search applications. Knowledge and experience in the Compliance / AML domain will be a plus. Working experience with Quantexa tool is a must.

Responsibilities:

·        Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives

·        Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions

·        Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data

·        Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data

·        Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch

·        Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch

·        Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards

·        Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration

·        Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components

·        Ensure data quality and integrity throughout the data processing lifecycle

·        Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques

·        Optimize data engineering workflows for containerized deployment and efficient resource utilization

·        Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability

·        Implement data governance practices, data lineage, and metadata management to ensure data accuracy, traceability, and compliance

·        Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements

·        Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure

·        Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference

  1. More than 5 years of experience as a Data Engineer
  2. · Bachelor's or Master's degree in Computer Science, Software Engineering, or a related discipline
  3. · Possession of Quantexa certification as a Data Engineer , with proficiency in the tool
  4. · Demonstrated experience as a Data Engineer, utilizing Hadoop, Spark, and data processing technologies in large-scale environments
  5. · Expertise in the Scala programming language and familiarity with functional programming principles
  6. · Prior experience with the Quantexa tool is highly desirable
  7. · Comprehensive understanding of Apache Spark architecture, including RDDs, DataFrames, and Spark SQL
  8. · Advanced proficiency in designing and developing data infrastructure utilizing Hadoop, Spark, and associated tools (HDFS, Hive, Pig, etc.)
  9. · Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration via Kubernetes
  10. · Proficiency in programming languages commonly employed in data engineering, including Spark, Python, Scala, or Java
  11. · Knowledge of DevOps methodologies, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
  12. · Experience with Graphana, Prometheus, and Splunk will be considered an added advantage
  13. · Background in integrating and utilizing Elasticsearch for data indexing and search applications
  14. · Solid understanding of Elasticsearch data modeling, indexing strategies, and query optimization techniques
  15. · Experience with distributed computing, parallel processing, and handling large datasets
  16. · Proficient in performance tuning and optimization methods for Spark applications and Elasticsearch queries
  17. · Strong problem-solving and analytical capabilities with the capacity to debug and resolve intricate issues
  18. · Familiarity with version control systems (e.g., Git) and collaborative development environments

Average salary estimate

$135000 / YEARLY (est.)
min
max
$120000K
$150000K

If an employer mentions a salary or salary range on their job, we display it as an "Employer Estimate". If a job has no salary data, Rise displays an estimate if available.

Similar Jobs
Mod Op Hybrid Miami, Florida, United States
Posted 7 days ago

Experienced Data Engineer sought to create scalable data solutions for a dynamic, full-service advertising agency operating across multiple locations.

Photo of the Rise User
Axiom Software Solutions Limited Hybrid Sunnyvale, California, United States
Posted 7 days ago

Experienced Snowflake Data Engineer with strong Python and advanced SQL expertise needed to collaborate across teams and deliver data solutions.

Reach Security Hybrid No location specified
Posted 7 days ago

Experienced Data Engineers are needed at Reach Security to develop and optimize scalable data pipelines using Apache Airflow or Dagster, contributing directly to mission-critical backend infrastructure.

Photo of the Rise User
Posted 4 days ago
Inclusive & Diverse
Empathetic
Take Risks
Transparent & Candid
Feedback Forward
Mission Driven
Collaboration over Competition
Work/Life Harmony
Maternity Leave
Paternity Leave
Snacks
Medical Insurance
Dental Insurance
Vision Insurance
Mental Health Resources
Life insurance
401K Matching
Paid Sick Days
Paid Time-Off
Paid Volunteer Time

Senior Data Engineer needed at Spotify to build and maintain scalable data infrastructure for their podcast and advertising products.

Posted 12 days ago

InVitro Capital is hiring a Senior Data & ML Engineer to design scalable data systems and ML workflows in a hybrid role based in Irvine, CA.

Photo of the Rise User
Vizient Hybrid Irving, TX 75062 Vizient Corporate HQ
Posted 6 days ago

Vizient is looking for a knowledgeable Data Engineer to develop and optimize data warehousing solutions with a focus on Microsoft Azure technologies.

Photo of the Rise User
Transparent & Candid
Collaboration over Competition
Growth & Learning
Mission Driven
Inclusive & Diverse

As a Staff Data Engineer at Bolt, you'll architect and scale data systems that power innovative financial and commerce solutions in a remote-first, inclusive environment.

Photo of the Rise User
Appian Corporation Hybrid McLean, Virginia, United States
Posted 8 days ago

A Senior Data Engineer role at Appian in McLean, VA, focused on designing and managing robust data pipelines to empower enterprise-wide analytics and decision-making.

Photo of the Rise User
CarbonCapture Inc. Hybrid Arts District, Los Angeles, CA
Posted 2 days ago

Lead the design and implementation of scalable scientific data infrastructure at CarbonCapture to accelerate climate solutions via advanced data engineering.

Photo of the Rise User
Posted 7 days ago

Experienced Data Engineer needed for a 12-month contract role with Element, building data pipelines and lakes to support healthcare oversight.

Tyndale Hybrid Pipersville
Posted 11 days ago

Tyndale seeks an experienced Data Engineer to design and optimize data pipelines supporting advanced AI and machine learning projects within a hybrid work setting.

Photo of the Rise User
Posted 7 days ago

A Senior Data Engineer role at SmarterDx to develop scalable, cloud-based data pipelines and support clinical AI innovations in healthcare.

Photo of the Rise User
Posted 13 days ago

Redhorse Corporation is recruiting a Mid-Level Data Engineer to enhance data-driven decision-making within the Department of Defense through advanced data pipeline development.

Unison helps you create extraordinary experiences for your employees, your customers, your community, our world.

128 jobs
MATCH
Calculating your matching score...
FUNDING
SENIORITY LEVEL REQUIREMENT
TEAM SIZE
EMPLOYMENT TYPE
Full-time, remote
DATE POSTED
April 18, 2025

Subscribe to Rise newsletter

Risa star 🔮 Hi, I'm Risa! Your AI
Career Copilot
Want to see a list of jobs tailored to
you, just ask me below!