Staff Software Engineer – Agentic Search Crawling

About the role

Nebius is building an agent‑native search platform for AI systems. As a Staff Software Engineer you will design and operate large‑scale web crawling and ingestion infrastructure that feeds real‑world data to downstream AI models.

Key responsibilities

• Design, implement and run web‑scale crawling systems that discover, fetch and refresh content from the open web and partner data sources.
• Build ingestion pipelines, URL deduplication, content extraction, and orchestration components capable of handling billions of URLs.
• Develop crawl scheduling, prioritisation, recrawl policies and freshness strategies while monitoring coverage, throughput and quality.
• Define observability, metrics and cost‑control for high‑throughput crawling operations.
• Collaborate with indexing and machine‑learning teams to ensure acquired content meets retrieval requirements.

Required profile

• 8+ years of experience building backend or distributed systems.
• Strong expertise in Go or C++.
• Hands‑on experience with large‑scale distributed pipelines (10k+ RPS, billions of URLs).
• Deep understanding of web protocols, crawling, scraping and content extraction.

Required skills

• Go or C++ programming.
• Distributed systems design and fault‑tolerance.
• Message‑queue platforms such as Kafka, Pulsar, NATS or RabbitMQ.
• Streaming and batch processing frameworks like Spark, Flink, Beam or MapReduce.
• Web crawling, URL discovery, deduplication and content extraction techniques.

What we offer

• Competitive compensation.
• Career growth and learning opportunities.
• Flexible ownership and collaborative culture.
• International environment working on impactful AI projects.