Senior QA Engineer - AI & LLM

As a Senior QA Engineer at Matillion, your key responsibilities will include designing and maintaining test automation frameworks, implementing testing for LLM-integrated features, and collaborating with cross-functional teams to ensure software quality at every phase of the development lifecycle. You'll also analyze test results, mentor team members on best practices, and drive innovative testing strategies.

To be considered for the Senior QA Engineer position at Matillion, you should have strong QA expertise, including experience with end-to-end testing of SaaS applications, deep knowledge of testing methodologies, and proficiency in writing automated tests using frameworks like Cypress. Familiarity with GenAI technologies and LLM APIs is essential, along with experience working with cloud platforms.

In the Senior QA Engineer role at Matillion, you'll leverage Generative AI technologies to evaluate and test LLM-integrated features. This includes implementing tools to evaluate prompts and API behavior, as well as utilizing RAG techniques to enrich LLM responses, further pushing the boundaries of software quality assurance.

Matillion utilizes agile methodologies such as Scrum and Kanban for software development, and as a Senior QA Engineer, you'll engage with these processes to ensure that testing is integrated from the outset, promoting a shift-left approach to quality assurance.

Matillion provides a competitive benefits package for the Senior QA Engineer position that includes company equity, 27 days of paid time off, comprehensive health coverage, enhanced parental leave policies, and access to tools for career development. We foster a flexible and hybrid work culture that promotes work-life balance.

To implement a test automation strategy effectively, start by assessing the current testing processes and identifying which areas are suitable for automation. Choose appropriate tools that align with the technologies used in the project, like Cypress for UI testing or API testing tools. Develop a comprehensive test plan that includes both functional and non-functional testing, ensuring thorough coverage. Collaborate with development teams throughout the testing process to promote a quality-first mindset.

In my previous role, I worked extensively with LLM APIs like OpenAI for testing natural language generation models. My approach involved creating detailed test cases that evaluate the quality of generated content, including prompt variations and result accuracy. I would analyze response patterns and use metrics to benchmark model performance, ensuring that the outputs meet our quality standards.

For performance testing in SaaS applications, I employ methods such as load testing and stress testing using tools like JMeter or Gatling. This involves simulating user load to evaluate how the application behaves under different traffic conditions. I also monitor system resource usage during testing to identify bottlenecks and areas needing optimization, ensuring that the application maintains its performance under varying loads.

To ensure effective collaboration with developers and product managers, I prioritize regular communication through stand-up meetings and sprint planning sessions. I engage in early discussions about requirements and provide feedback on design and functionality to guarantee clarity and alignment. By establishing a partnership mindset, I can advocate for quality while also understanding the broader project goals.

I have extensive experience with several testing tools relevant to this role, including Cypress for end-to-end testing, Postman for API testing, and LoadRunner for performance testing. My familiarity with Git for version control and CI/CD integration has allowed me to streamline testing processes effectively and collaborate with the development team seamlessly.

In a previous project, we faced inconsistent results in our automated tests due to a poorly defined API that kept changing. To address this, I initiated a dialogue with the developers to clarify the API behavior, collaborated on static definitions, and re-structured our test cases to accommodate these changes. This proactive approach not only resolved the inconsistencies but also fostered a stronger relationship with the development team.

I make a conscious effort to stay abreast of industry trends by following key Q&A blogs, attending webinars, participating in QA forums, and enrolling in online courses that focus on testing methodologies and AI advancements. Networking with other professionals at conferences also provides insights into best practices and innovative solutions that I can apply in my role.

CI/CD plays a crucial role in my QA strategy as it automates the testing process, facilitating faster feedback loops for developers. I ensure that automated tests are integrated into the CI/CD pipeline to catch issues early in the development cycle, which ultimately leads to better product quality. This practice allows us to maintain a steady pace of releases while ensuring reliable software.

When mentoring junior team members, I focus on sharing knowledge about testing methodologies and frameworks, emphasizing the importance of a quality-first mindset. I encourage hands-on practice through pair-testing sessions and provide constructive feedback on their test cases and strategies. By fostering an open environment for questions and discussions, I ensure they grow in their confidence and capabilities in QA.

I am excited about the opportunity to work at Matillion because of its commitment to innovation in data productivity and the proactive approach to integrating AI into its products. The company's values resonate with me, particularly the emphasis on collaboration and quality. Joining Matillion’s founding team in Hyderabad presents a unique chance to shape testing strategies alongside talented professionals, making a meaningful impact on both the product and the team culture.