ML Systems Engineer, Infrastructure & Cloud

About the Company

Basis is a nonprofit applied AI research organization focused on understanding and building intelligence while advancing society's ability to solve intractable problems through a new technological foundation and human-centric collaborative organization.

Responsibilities

• Own distributed training infrastructure including job launchers, checkpointing systems, and recovery mechanisms
• Debug and resolve training failures across GPUs, networking, numerics, and data pipelines
• Profile and optimize training performance to improve step time and resource utilization
• Manage cloud infrastructure, capacity planning, and cost optimization strategies
• Implement security, compliance, and access controls for sensitive data
• Build evaluation and benchmarking infrastructure for reproducible model measurement
• Develop monitoring and alerting systems for training metrics and system health
• Maintain development environments using containerization and dependency management
• Document knowledge through runbooks, post-mortems, and training materials
• Collaborate with researchers to align infrastructure with research goals

Requirements

• Expertise in managing distributed training jobs across large GPU clusters
• Deep knowledge of PyTorch/JAX distributed strategies (DDP, FSDP, ZeRO)
• Strong cloud administration skills (AWS/GCP/Azure) and Infrastructure as Code (Terraform)
• Experience with Kubernetes orchestration and cost optimization
• Ability to debug complex failures including GPU/NCCL issues and memory leaks
• Proficiency in managing the full ML stack from hardware to high-level training loops
• Commitment to documentation and "logbook culture"

Preferred Qualifications

• Experience at organizations training large-scale models
• Background in both ML research and production systems
• Contributions to ML frameworks or distributed training libraries
• Experience with on-premise GPU cluster management
• Knowledge of optimization theory and numerical methods