Prompt Engineer Data Science & Quality Analysis

As a Prompt Engineer focused on Data Science and Quality Analysis, you’ll design, test, and evaluate prompts for AI systems that interact with real-world restaurant data. You’ll work cross-functionally to develop AI solutions that drive operational efficiency, improve data interpretation, and support smarter decision-making for restaurant operators.

Your work will directly shape how AI models perform in high-stakes, dynamic environments like order processing, reporting, support automation, and performance analysis.

Essential Job Functions:

• Prompt Design & Evaluation: Develop, test, and refine prompts for tasks such as text generation, question answering, data classification, and structured data extraction to optimize Voice AI performance.
• Data-Driven Analysis & Quality Measurement: Design evaluation frameworks and analyze prompt outputs using quantitative metrics, human-in-the-loop evaluation, and user feedback to identify improvement opportunities.
• Experimentation & Iteration Conduct experiments to test prompt variations, measure their business and operational impact, and iterate to enhance accuracy, consistency, and safety.
• Regression Testing & Compliance Build principled regression test suites using tools like LangFuse and Galileo to ensure prompts remain compliant and high-performing as models and use cases evolve.
• Collaboration Across Teams: Work closely with data science, product, legal, engineering, and operations teams to align prompt designs with business goals, operational workflows, and compliance requirements.
• Model Adaptation & Strategy Development prompts across multiple LLMs (GPT, LLaMA, Gemini, and Checkmate’s fine-tuned models), understanding model differences to optimize outputs effectively.
• Team Leadership & Mentorship Lead a team of analysts focused on prompt evaluation and data quality analysis, guiding prioritization, experimentation, and reporting. Collaborate with ops teams for seamless deployment and feedback loops.
• Research & Continuous Learning Stay up to date on emerging prompting techniques, LLM behaviors, evaluation frameworks, and AI safety practices to keep Checkmate’s AI solutions best-in-class.

• Strong analytical and data science skills, with hands-on experience in Python (pandas, NumPy, scikit-learn)
• Experience designing and conducting experiments and evaluations in applied AI or NLP contexts
• Proficiency in SQL and working with relational databases (e.g. MySQL, PostgreSQL, Oracle, MS SQL)
• Good understanding of data processing, quality measurement, and testing fundamentals
• Experience leading analyst or operations teams, with strong prioritization, mentorship, and collaboration skills
• Strong problem-solving mindset with a drive to explore, optimize, and automate workflows
• Excellent communication skills for presenting insights to technical and non-technical stakeholders
• Bachelor’s degree in Data Science, Computer Science, Statistics, Engineering, or a related field
• Flexible to work US hours until at least 6 pm ET, with a strong remote setup

Preferred Qualifications

• Experience with LLM evaluation and prompt engineering workflows
• Familiarity with tools like LangFuse and Galileo for prompt evaluation and analysis
• Knowledge of cloud platforms (AWS, GCP, Azure) and data pipeline tools
• Familiarity with machine learning concepts and NLP workflows
• Master’s or PhD in Data Science, Statistics, Computer Science, or a related field

• Health Care Plan (Medical, Dental & Vision)
• Retirement Plan (401k)
• Life Insurance (Basic, Voluntary & AD&D)
• Flexible Paid Time Off
• Family Leave (Maternity, Paternity)
• Short Term & Long Term Disability
• Training & Development
• Work From Home
• Stock Option Plan