LLM Inference Engineer

As an LLM Inference Engineer at Hippocratic AI, your primary responsibilities will include designing and implementing multi-node serving architectures for distributed LLM inference, optimizing multi-LoRA serving systems, and applying advanced quantization techniques to minimize model size without sacrificing performance. You'll also implement latency optimization strategies, such as speculative decoding, and develop disaggregated serving solutions that include optimized caching strategies. Your role will be crucial in continuously benchmarking and improving the performance of our systems across different deployment scenarios and GPU types.

To be considered for the LLM Inference Engineer position at Hippocratic AI, candidates should possess at least 2 years of experience in optimizing LLM inference systems at scale. A proven track record with distributed serving architectures and hands-on experience with quantization techniques for transformer models is essential. You should have a strong grasp of modern inference optimization methods, including speculative decoding techniques and familiarity with CUDA programming and GPU optimization, although expert-level knowledge is not necessary. Proficiency in Python and C++ is also required.

Hippocratic AI fosters innovation in LLM deployment by actively participating in research and development with a world-class team comprised of healthcare professionals and AI researchers. We're dedicated to creating a safe, healthcare-focused language model designed to revolutionize health outcomes. Our co-founders include experienced individuals from top-tier health and tech organizations, which cultivates an environment of collaboration and forward-thinking. Additionally, we're backed by strategic investors who are equally passionate about pushing the boundaries of what's possible in AI technology.

For the LLM Inference Engineer position at Hippocratic AI, proficiency in Python and C++ is fundamental. Understanding CUDA programming and experience with GPU optimization will also significantly enhance your ability to contribute effectively to our LLM serving infrastructure. Familiarity with advanced quantization techniques and modern inference optimization methods will further enhance your skills and effectiveness in the role, allowing you to implement innovative solutions in real-world scenarios.

Hippocratic AI stands out as an attractive employer for LLM Inference Engineers due to its commitment to innovation in healthcare through AI technology. With a strong focus on safety and improving health outcomes, the company offers an exciting opportunity to contribute to meaningful projects that impact real lives. Additionally, working alongside leading experts in the field and having access to impressive funding from key investors provides a solid support structure for pushing technological boundaries. The collaborative in-office environment in Palo Alto also promotes teamwork and idea sharing, creating a motivating workplace.

In your answer, provide specific examples from previous work where you successfully optimized LLM inference systems. Discuss the techniques you used, such as quantization or multi-node architecture implementation, and the results achieved. Highlight any measurable improvements in performance or efficiency that came as a direct consequence of your efforts.

Be prepared to explain your hands-on experience with distributed serving architectures for large language models. Provide details about past projects, the challenges you faced, and how you overcame them. Mention any specific technologies or frameworks that you have used, emphasizing your role in implementing and optimizing these systems.

Discuss the quantization techniques you've utilized, such as FP4 or FP6, and how you've applied them to reduce model footprints while maintaining accuracy. Include examples of how this work positively impacted deployment outcomes in your previous positions.

Explain your knowledge and experience regarding latency optimization strategies like speculative decoding. Provide examples of how you implemented these techniques, the results obtained, and how they contributed to overall system performance. Sharing quantitative results will strengthen your answer.

Describe your proficiency with Python, C++, and CUDA. Discuss how you've used these languages and any relevant libraries in the past. Use specific examples to show your familiarity with LLM optimization and how these tools helped you achieve your goals.

If you have experience with open-source inference frameworks like vLLM, SGLang, or TensorRT-LLM, share your contributions or usage examples. If you haven’t, discuss your willingness to learn about these frameworks and how they could be beneficial in your future role as an LLM Inference Engineer.

Talk about the benchmarks and evaluation metrics you utilize to validate the performance and quality of your LLM systems. Describe your approach to continuous testing and optimization, and share any techniques you use to identify and resolve performance bottlenecks.

Express your passion for making a meaningful impact in healthcare through AI. Discuss any personal or professional experiences that drew you to this field, and how you see your work as an LLM Inference Engineer positively affecting patient outcomes and healthcare providers.

Articulate a specific challenging project related to LLMs or inference optimization. Discuss the obstacles you faced, your thought process in tackling the problems, and the successful outcome of your efforts. This will demonstrate your problem-solving skills and resilience.

Convey your excitement about Hippocratic AI's mission to innovate healthcare through safe and effective LLMs. Mention aspects of the company's vision, team, and culture that align with your values and professional goals. Show that you have done your research on the company and are genuinely interested in contributing to its success.