Director, Annotation

The Director, Annotation will provide leadership to a team or teams that label enterprise ticket and in-bot written textual utterance data for neural network training. A successful candidate will have high attention to detail, an appreciation for data integrity, and the people management and leadership skills to drive efficiency for large teams of annotators.

Responsibilities

• Own the data annotation pipeline at various stages of lifecycle from presales to maintaining existing customers while ensuring quality of data annotation for the entire team.
• Implement and improve quality assurance processes for increasingly complex data labeling, driving the optimization of annotation processes.
• Organize sequential and parallel data annotation processes and load balance team members across multiple projects of competing priority.
• Ensure the team follows the correct technical procedures and guidelines.
• Own the data annotation guidelines and ensure appropriate change management for modifying the guidelines when appropriate.
• Manage the team projects through development, pilot and implementation phases, and review processes.
• Provide technical training and mentorship. Ensure the team works as a cohesive unit.
• Work directly with engineers and, as needed, with Product, UX, Customer Success, and AI insights leadership. Own annotation policies that require an understanding of both technical and operational constraints.
• Build metrics that identify key performers, enabling better ownership and career trajectory for top annotators.
• Develop and maintain a capacity model for the entire annotation team
• Provide cost-of-task visibility to engineering
• Be the point person to make tradeoffs and prioritization decisions when the resource constraint dictates that some of requested tasks not be fulfilled

Requirements

• Commitment to data accuracy and throughput
• Strong problem solving skills, with an aptitude for quickly learning systems
• Proven track record executing and meeting sensitive deadlines
• Experience managing multiple competing projects with limited resources
• Understand when to delegate and allocate responsibilities to manage projects end-to-end
• Desire to automate out inefficiencies when applicable
• Ability to articulate and justify tooling additions requirements, serving as a liaison between the annotation teams and the tooling engineers
• Ability to match project tasks with opportunities for employee development
• Strong communication and presentation skills - can explain complicated topics and ideas in a clear and concise manner
• Proficiency in standard workflow software (JIRA, Excel, etc.)

Preferred attributes

• Experience managing and motivating large teams that perform manual operations
• Love of and facility with metrics
• Strong desire to develop internally-consistent, growing, and adaptable guidelines and taxonomies
• Super track record in change management
• Experience managing knowledge collections, making information searchable, useful, and accessible
• Strong interviewing skills