Senior Data Scientist, Copilot AI
Job Description
- Develop and improve evaluation methodologies to assess model output quality, for both machine eval and human eval metrics and coverage.
- Design and implement scalable data pipelines to extract, transform, and structure product logs for evaluation use cases.
- Synthesize datasets for human or machine evaluation.
- Analyze and interpret results from A/B tests, offline benchmarks, and live experiments to drive actionable recommendations.
- Train ML classifiers to analyze and label user logs (e.g., classify intent, detect quality issues) for evaluation
- Draw insights from eval results and form recommendations, drive different eval experiments to find the most optimal solutions.
- Work closely with product managers, engineers, and researchers to define evaluation criteria aligned with product goals and user value.
- Create and maintain dashboards and reporting tools to monitor eval performance and trends.
- Contribute to the development of custom metrics that go beyond standard benchmarks to capture product-specific nuances.
- Stay current on the latest in LLM research on evaluation and prompting.
- Embody our Culture and Values.
- Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1 year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3 years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
- OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5 years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
- OR equivalent experience.
- 5 years of experience in data science, ML evaluation, or applied research.
- Working knowledge of LLM evaluation methods, including experience conducting both human evaluations and LLM-as-a-judge assessments
- Experience using Python, SQL, and common data analysis libraries for data processing and analysis.
- Ability to analyze complex problems, communicate findings clearly, and translate insights into actionable steps.
- Experience building or evaluating LLM applications in production.
- Product-driven thinking.
- Ability to work in a fast-paced environment, manage multiple priorities, and adapt to changing requirements and deadlines.
Average salary estimate
If an employer mentions a salary or salary range on their job, we display it as an "Employer Estimate". If a job has no salary data, Rise displays an estimate if available.
Join Microsoft Security as a Back-End Principal Software Engineer and lead innovations in secure cloud technologies.
Become a pivotal member of Microsoft's Copilot Monetization Strategy & Planning Team, driving AI monetization across a diverse consumer ecosystem.
Elevate team collaboration as a Senior Machine Learning Engineer at Atlassian, where your expertise will shape cutting-edge software solutions worldwide.
Drive the future of media at Dentsu as a Senior Data Scientist specializing in generative AI and innovation.
Upgrade is looking for a results-driven Lead Data Scientist to lead innovative machine learning projects that influence credit products.
Be a part of Marker Learning's mission to reform educational assessments as a Senior Machine Learning Engineer focused on innovative AI solutions.
Join SandboxAQ as a Machine Learning Engineer to advance AI systems for genomics and drug discovery.
Lead the charge in transforming advertising precision as MNTN's Principal Data Scientist with a focus on advanced algorithms and impactful data methodologies.
Step into a pivotal role as a Data Science Manager at Sia, leading innovative marketing analytics projects and transforming data into actionable insights.
Become a pivotal part of Peraton's mission by leveraging your data science expertise to deliver actionable insights remotely.
Join Microsoft Advertising as a Senior Data Scientist to drive data-driven decision-making through advanced analytics and innovative techniques.
As a Staff Machine Learning Engineer at Scribe, you'll drive AI technical strategies and mentor a team, impacting productivity for millions.
Join Takeda as a Director of Clinical Data Engineering, where you'll shape innovative data solutions in a hybrid working environment.
Join NBCUniversal's Media Group as a Lead for Statistical Inference & Research, where you'll drive innovation in experimentation strategies.
Transform real estate search as a Senior Machine Learning Engineer at Realtor.com®, leveraging cutting-edge AI solutions to improve user interactions.
Microsoft's mission is to empower every individual and organization worldwide to achieve more.
356 jobs
Subscribe to Rise newsletter
🔮
Hi, I'm Risa! Your AI Career Copilot
you, just ask me below!
