Senior Software Development Engineer in Test – AI Cluster

As a Senior Software Development Engineer in Test at Cerebras Systems, you will innovate and execute tests on our cutting-edge AI cluster infrastructure. This role requires defining optimized test strategies and methodologies, emphasizing automation to achieve 100% test coverage across all cluster features. You will also be involved in ensuring cluster security and reliability, testing both software components like Kubernetes and Grafana, as well as hardware components related to our unique ML wafer scale accelerators.

To qualify for the Senior Software Development Engineer in Test role at Cerebras Systems, you should have a bachelor’s or master’s degree in engineering, computer science, AI, or a related field. Additionally, a minimum of 5 years of experience in testing enterprise software, distributed systems, or data center hardware is required. Strong coding skills in languages such as Python, Golang, or C/C++ and a solid understanding of operating systems and cloud technologies are also essential for success in this role.

In the role of Senior Software Development Engineer in Test at Cerebras Systems, automation is crucial. Given the scale at which we operate, an automation-first approach is vital for driving efficiency and scalability. You will aim for full automation of tests, ensuring that all functionalities of the AI cluster, including high availability and performance under stress, are thoroughly tested to maintain a smooth operational environment.

Experience in debugging large distributed systems, cloud technologies like AWS, Kubernetes, and tools like Grafana and Prometheus is highly beneficial for a Senior Software Development Engineer in Test at Cerebras Systems. Familiarity with ML model training and inference, as well as understanding ML hardware accelerators such as GPUs and custom ASICs, will also give you an edge in this exciting role focused on transforming AI applications.

Cerebras Systems fosters a unique workplace culture that emphasizes inclusivity, innovation, and respect for individual beliefs. As a Senior Software Development Engineer in Test, you’ll thrive in a non-corporate ethos that values collaboration and creativity. The culture encourages continuous learning, personal growth, and contributing to groundbreaking advancements in AI technology.

When discussing your experience with automated testing in large-scale systems, highlight specific projects where you've implemented automation frameworks. Discuss the tools you used, the challenges faced in scaling automation, and the impact of your efforts on overall project efficiency and delivery times.

In answering this question, focus on your systematic approach to debugging distributed systems. Share examples of tools you have used and methodologies applied, such as using logs, metrics, and real-time monitoring to isolate issues, and highlight any particular complex problems you've resolved.

To ensure software reliability and security in your tests, discuss the protocols you follow. Mention incorporating security tests in CI/CD pipelines, running stress tests for resilience, and implementing security-focused code review practices that ensure compliance with best practices.

When asked about a challenge faced while testing AI systems, provide a specific example illustrating the problem complexity. Discuss your thought process in addressing the issue, the steps you took, the resolution, and what you learned that improved your future testing methodologies.

When discussing programming language proficiency, share your experiences with languages commonly used in the industry, such as Python, Golang, or C/C++. Provide examples of projects completed in these languages and how they directly relate to your role as a Senior Software Development Engineer in Test.

In explaining how you would test a large-scale ML deployment, outline a comprehensive testing strategy. Discuss performance benchmarks, automated tests for normal and edge cases, and how you would ensure each component is robust and adheres to security and reliability standards.

To address how you stay current with trends in AI and ML testing, mention specific resources you follow such as journals, tech blogs, or participating in relevant forums. Highlight conferences or workshops you’ve attended and emphasize ongoing learning and adoption of new tools and practices.

When discussing your favorite debugging tools, elaborate on tools like GDB, strace, or specific cloud monitoring tools, stating why they are effective in your workflow. Provide examples where these tools significantly aided your testing process or resolved complex issues.

In designing test cases for distributed systems, outline key factors such as scalability, fault tolerance, concurrency, and performance. Discuss how these elements influence the creation of comprehensive test cases that effectively stress the system under varied conditions.

In answering why you want to work at Cerebras Systems, connect your personal and professional values with the company’s mission. Show enthusiasm for their innovative technology and express how your skills and experience align with their vision in advancing the AI field.