Software Engineer, SystemML - AI Networking

As a Software Engineer for SystemML at Meta, you'll primarily be responsible for tech-leading the development of the collective communication library tailored to our large-scale GPU training infrastructure. This includes driving improvements in GenAI/LLM scaling, enhancing our software stack's reliability and performance, and facilitating seamless communication on multi-GPU setups. Your role will also involve building customized features, conducting software benchmarks, and tuning performance for a range of distributed machine learning applications.

To qualify for the Software Engineer position at Meta, you should have a Bachelor's degree in Computer Science, Computer Engineering, or a similar technical field, along with solid experience in C/C++ and Python programming. A proven track record of leading successful projects is important, along with specialized knowledge in areas like distributed ML training, GPU architecture, high-performance computing, and machine learning frameworks such as PyTorch. Advanced qualifications such as a PhD in a relevant field would be a plus.

Essential programming skills for the Software Engineer, SystemML role at Meta primarily include proficiency in C/C++ and Python. These languages are critical for performance tuning and communication library development. Experience with machine learning frameworks such as PyTorch and knowledge of GPU architectures and CUDA programming will also greatly enhance your contributions to the team and our projects.

Beneficial experience for a Software Engineer, SystemML position at Meta includes hands-on work with distributed GPU performance analysis, familiarity with the NCCL library, and practice with deep learning frameworks like PyTorch, Caffe2, or TensorFlow. Additionally, understanding data and model parallel training approaches, as well as HPC and parallel computing, will give you a competitive edge in this role.

As a Software Engineer, SystemML, your contributions will be vital to optimizing Meta's AI initiatives by enhancing the performance and reliability of the underlying software stack. By focusing on areas such as GenAI/LLM scaling, you will directly influence the development of efficient machine learning products that leverage our robust GPU infrastructure, thus shaping the future of AI-driven innovations at Meta.

When answering this question, highlight your practical experience with frameworks like PyTorch or TensorFlow. Discuss specific projects where you applied these frameworks, emphasizing your role in developing machine learning models or optimizing existing systems. Be sure to mention any performance tuning you accomplished and the impact it had on the final outcomes.

For this question, describe NCCL's function in facilitating multi-GPU and multi-node communication during distributed training. Explain how it supports efficient data transfer and synchronization across GPUs, thereby enhancing performance in large-scale machine learning tasks. If possible, share any personal experience or insights you've gained working with NCCL.

Detail specific strategies you've employed for performance optimization in machine learning systems, such as profiling tools, analyzing bottlenecks, or implementing techniques like data parallelism. Use examples from previous projects to illustrate your analytical capabilities and how your interventions improved overall system efficiency.

Here, outline your experience leading software projects, focusing on your ability to guide a team through the development cycle. Discuss how you motivated team members, delegated tasks, managed timelines, and ensured that project goals were met or exceeded. Emphasize your effective communication skills and the impact of your leadership on the project's success.

In your answer, describe a systematic approach to debugging that might include understanding the problem domain, reproducing bugs consistently, and using debugging tools effectively. Share personal experiences where your methods led to resolving significant software issues, demonstrating your problem-solving skills under pressure.

Discuss your understanding of different GPU architectures (like NVIDIA CUDA) and their impact on software development for machine learning. Mention how you've optimized algorithms or implemented strategies that leverage GPU capabilities, including parallel processing and memory management, to achieve better performance in ML systems.

This is an opportunity to demonstrate your understanding of distributed training methodologies. Explain that data parallelism involves splitting data across multiple GPUs while each GPU runs the same model, while model parallelism involves distributing distinct parts of the model across multiple GPUs. Use examples to illustrate situations where each would be beneficial.

When answering this, share your experiences working with HPC environments. Discuss any specific projects where you utilized HPC environments for training large models. Mention the tools or libraries you used, such as MPI or NCCL, to ensure your understanding of HPC's role in fast-tracking complex computations.

Explain your method for task prioritization, possibly citing specific frameworks like Agile or Kanban. Emphasize your ability to adapt to changing circumstances while ensuring that critical milestones are met. Detail experiences where you successfully juggled multiple priorities without sacrificing quality.

In your response, express genuine enthusiasm for Meta's focus on innovative AI solutions. Discuss how your skills and experiences align with the team's objectives, particularly your passion for enhancing distributed machine learning performance. Highlight your desire to be part of a company known for driving technological advancements in AI.