Senior Software Engineer - LLM Ops & Evals

About the role

DataSnipper’s AI gateway handles every model call for its audit and finance products. The team builds and operates the routing layer, ensures reliable inference across multiple providers, and creates the evaluation platform that measures AI quality for product teams. You will own production systems that affect hundreds of thousands of users.

Key responsibilities

• Design, implement and maintain the LLM gateway – routing, provider failover, rate‑limiting, retries and cost attribution across Azure and GCP.
• Deploy, version and deprecate models as infrastructure‑as‑code, including quota planning and capacity management.
• Build and operate the shared evaluation platform with versioned datasets, experiment tracking, result schemas and reporting.
• Provide observability for AI traffic (latency, token usage, cost, errors) and own on‑call rotation, incident response and post‑mortems.
• Implement security and compliance controls such as data retention, access control, RBAC and SSO.

Required profile

• 5+ years of backend or platform engineering experience with production Python code.
• Hands‑on experience building or operating LLM inference infrastructure, including multi‑provider routing and cost attribution.
• Strong background in cloud environments (Azure, GCP) and infrastructure‑as‑code tools like Terraform.
• Proven ability to run a shared service in production, including on‑call duties, SLOs and incident management.

Required skills

• Python
• LLM inference infrastructure
• Azure and Google Cloud Platform
• Terraform
• OpenTelemetry
• Grafana