Data Scientist – AI Evaluation

About the role

We are looking for a sharp, analytical Data Scientist to own the evaluation and quality loop of our AutoQA product. You will work at the intersection of AI and customer experience, helping enterprise customers measure and improve LLM‑powered quality assurance.

Key responsibilities

• Translate customer quality rubrics into precise AutoQA instructions for LLM scoring.
• Design and run evaluation experiments on large datasets using LangSmith and BigQuery.
• Curate and build golden and synthetic datasets to ensure balanced coverage of rare cases.
• Develop LLM‑as‑a‑judge pipelines to make evaluation repeatable and actionable.
• Collaborate with AI engineers to adjust prompts, tools, and context based on experiment outcomes.
• Validate retrieval, tool‑calling, and speech pipelines (transcription, diarization) across the stack.
• Participate in customer calls to understand quality definitions and feed insights back to the product.

Required profile

• Recent graduate or early‑career professional with strong fundamentals in large language models.
• Analytical mindset with ability to design experiments and interpret results.
• Comfort working in a fast‑growing SaaS environment and with enterprise customers.

Required skills

• Python
• BigQuery
• LangSmith (or similar LLM experiment platforms)
• Kafka
• Kubernetes
• ElasticSearch
• MongoDB

What we offer

• Flexible hours and open holiday policy.
• Hybrid work model with an office in the heart of Amsterdam.
• Visa sponsorship, great gear, workations and team events.
• Clear career path toward Senior Data Scientist, AI Engineer or ML Engineer.