Senior Site Reliability Engineer

AI/HPC platform and infrastructure operations Role summary Provides hands-on engineering and third-line technical support for a large-scale AI/HPC platform across Amsterdam and Phoenix. Owns complex faults from diagnosis through recovery and permanent resolution, and builds the automation and platform improvements that make the service reliable. Works across bare-metal Linux, Kubernetes, GPU compute, storage and networking, with the depth to investigate unfamiliar failures and turn findings into tested fixes. Supports research and AI teams in running workloads successfully on the platform. Key responsibilities - Own third-line escalations across the platform, tracing failures through the workload, container, scheduler, operating system, GPU, network and storage layers. Restore service, establish root cause and deliver fixes that prevent recurrence. - Build, operate and improve Kubernetes clusters and shared platform services, including scheduling, registries, persistent storage and observability. Resolve control-plane, operator, networking and service-dependency failures. - Write and maintain production automation using Ansible, Python or Bash, Git and CI/CD or GitOps pipelines. Automate provisioning, configuration, health checks and recovery, with testing, peer review and clear rollback paths. - Carry out bare-metal lifecycle operations from imaging and role assignment through patching, reprovisioning and return to service. Validate host, GPU and fabric health against the supported hardware and software baseline. - Engineer Run: ai project, quota, priority and placement configuration within agreed allocation policies. Resolve pending jobs, GPU allocation failures and inefficient resource use while preserving tenant isolation. - Diagnose failed or slow distributed training and inference workloads with users, using logs, GPU telemetry and targeted benchmarks to isolate runtime, communication, storage and platform bottlenecks. - Maintain reliable container and package delivery through approved build, scan, promotion and site-distribution workflows. Troubleshoot image pulls, dependencies and runtime compatibility across ARM64 and x86 GPU environments. - Plan and execute platform upgrades, maintenance and recovery exercises with the Technical Lead and infrastructure specialists. Test compatibility, backup restoration and rollback, respecting vendor-managed component boundaries. - Apply access, secrets and security controls, and produce change records and validation evidence for regulated workloads. Work within agreed GxP procedures and quality approvals. - Lead technical investigation with OEMs and specialist teams, providing reproducible evidence and verifying recovery. Direct remote hands for physical diagnostics and replacements, working with Partner Operations on the vendor process. - Improve monitoring and diagnostic coverage with Platform & AI Enablement. Convert incident findings into reviewed runbooks, automation and practical coaching for Engineers, and contribute to service handover and readiness testing. - Participate in the rotating support and on-call rota across European and US coverage windows. Provide clear handovers and senior technical leadership during incidents, escalating to the Technical Lead when impact or risk requires it. Essential skills and experience - Substantial hands-on production experience in platform, DevOps, Linux, HPC or GPU infrastructure engineering, including ownership of complex incidents and changes in business-critical environments. Able to explain the diagnosis, decisions and outcomes of work you personally delivered. - Deep Linux systems administration and troubleshooting: boot and service failures, kernel and driver compatibility, CPU and memory pressure, filesystems, networking and performance under load. Confident investigating beyond standard runbooks. - Strong production Kubernetes administration, including control-plane and worker lifecycle, container runtimes, Helm, operators, CNI networking and CSI-backed storage. Able to diagnose cluster failures as well as deploy workloads. - Proven automation engineering using Ansible or a comparable configuration-management tool, plus Python or Bash. Experience maintaining version-controlled, tested automation and CI/CD or GitOps workflows used by other engineers. - Hands-on operation of GPU compute clusters, including NVIDIA drivers, CUDA runtime compatibility, GPU visibility in containers and hardware health. Practical experience supporting workloads across multiple GPU nodes. - Understanding of distributed AI workload execution, including collective communication such as NCCL, GPU memory constraints and data throughput. Able to distinguish application or environment problems from infrastructure faults. - Bare-metal provisioning and lifecycle experience at scale, using a cluster-management platform such as Base Command Manager or equivalent. Comfortable with remote management, node isolation, reprovisioning and validation before return to service. - Strong networking and storage troubleshooting across hosts and clusters: TCP/IP, DNS, routing, load balancing, NFS, permissions and persistent volumes. Practical understanding of RDMA over Ethernet or InfiniBand, fabric topology and congestion effects on GPU workloads. - Container image and software dependency management, including reproducible builds, registries, approved promotion and deployment. Able to resolve image, package and driver/runtime compatibility failures. - Experience operating multi-tenant platforms with role-based access, workload isolation, certificates and secrets. Uses metrics, logs and diagnostic tools to establish causes and validate recovery. - Disciplined production change execution with impact assessment, testing, rollback and auditable evidence. Must complete the required GxP training and competency authorisation before independently undertaking regulated work. - Clear written and spoken English, sound judgement under pressure and the ability to guide less experienced engineers. Willing to complete security vetting and work the agreed rotating shifts, including evening coverage and out-of-hours on-call. Desirable - Direct experience with NVIDIA Mission Control, Base Command Manager, Run:ai, GPU Operator, Network Operator, DCGM or NetQ; or comparable GPU platform tooling. - Exposure to GB300 NVL72 or other dense GPU systems, ARM64 Linux, NVLink, GPUDirect RDMA, Spectrum-X Ethernet, ConnectX or BlueField networking. - Experience with VAST or comparable scale-out storage; ArgoCD; JFrog Artifactory; Dragonfly or other image-distribution tooling; Prometheus, Grafana and Alloy. - Operating stateful Kubernetes services, including PostgreSQL high availability, coordinated application and database backups, failover and tested restoration. - Supporting PyTorch, JAX or TensorFlow environments, NVIDIA NIM or other inference services, and reusable workload templates or onboarding workflows. - Experience in GxP or another validated environment, and awareness of power, thermal and liquid-cooling dependencies in high-density data centres. Certifications Relevant Linux, Kubernetes or NVIDIA infrastructure certifications are desirable; equivalent hands-on experience is equally valued. Required GxP and local operating-procedure training and competency sign-off must be completed during onboarding.