Senior Engineer I - Observability - Pipelines & Visualization

About the role:

The Observability Pipelines and Visualization team manages the doors – both entry and exit – to the telemetry databases that are managed by the Observability Department. The Pipelines and Visualization team ensures that CoreWeave engineers can effectively produce and consume telemetry for their services. This involves building pipes to ingest and route data in predictable, composable ways as well as visualizing that data after the fact.

We are seeking a Senior Engineer who has experience across observability stacks, understands the unique problems that come with moving cloud-level volumes of telemetry data at scale, and is excited at the prospect of ensuring both CoreWeavers as well as CoreWeave customers have eyes into the same underlying telemetry data to run their services as efficiently as possible. Knowledge of and experience running Grafana at scale, serving a wide variety of customers with varying degrees of access, is a strong plus. As you work closely with a team of 3-6 engineers focused on pipelines and visualization, you will have the opportunity to:

• Collaborate with Logging, Tracing, and Metrics SMEs at a cloud scale.
• Ensure CoreWeavers from teams across the organization have easy access to the telemetry they need for compliance, security, billing, and myriad other specialized use cases.
• Build low-touch, automated tooling that enables CoreWeave engineers to clearly articulate which pieces of telemetry, visualizations, and dashboards should be presented to customers.
• Develop and enforce best practices on alerting and tracking the health of telemetry ETL pipelines.
• Improve the performance, security, reliability, and scalability of observability services while participating in the team’s on-call rotation.
• Enable and evangelize the best practices around alerting. Collaborate with teams to establish observability standards.
• Work with telemetry data at enormous scale, ingesting data from industry-leading GPU clusters.
• Grow, change, invest in your teammates, be invested-in, share your ideas, listen to others, be curious, have fun, and, above all, be yourself.

Wondering if you’re a good fit? We believe in investing in our people, and value candidates who can bring their own diversified experiences to our teams – even if you aren't a 100% skill or experience match. Here are some qualities we’ve found compatible with our team. If a portion of this resonates with you, we’d love to talk. 

• You have six or more years of experience in a software or infrastructure engineering industry.
• You enjoy helping your colleagues achieve more with less effort.
• You are customer obsessed, ecstatic to provide infrastructure as a service, and default to adopting a product lens when evaluating platform scale problems.
• You have experience with observability platforms in production and at scale, and are versed in reliability engineering concepts such as the different types of testing, progressive deployments, error budgets, the role observability plays, and fault-tolerant design.
• You have experience using Kubernetes, a conceptual understanding of its major components, and/or have operated Kubernetes clusters at scale for both event-driven and stateful orchestration.
• You have experience using Temporal as an event-based workflow management tool.
• You’re familiar with various logging and metrics systems like ClickHouse, Elastic, Loki, Victoria Metrics, Prometheus, Thanos and Grafana. You have experience with designing and operating these systems at scale.
• You are familiar with PromQL, any other querying language and enjoy understanding the data model for observability systems. 
• You are excited at the prospect of zero-downtime, transparent migrations of major production services.
• You’re comfortable with the idea of using Go as your primary programming language, but don’t shy away from templating yaml as circumstances call for it.
• You know your way around a Linux distro, shell scripting, and/or the Linux storage and networking stacks.
• You can transform problems in elastic solutions, decompose them into achievable tasks, and socialize both to your teammates.
• You’re excited about being part of a team of diverse perspectives and backgrounds that believe in tackling challenges, growing hand in hand, and winning together.

Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $175,000-$225,000. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience. This position also includes a discretionary bonus, equity, and a comprehensive benefits package.