Software Engineer, Data Infrastructure

Responsibilities

• Design and operate a high-throughput, real-time data integration platform
• Architect a scalable open table format layer for reliable data storage at petabyte scale
• Build and optimize distributed data processing pipelines with Apache Spark and streaming technologies
• Drive performance, reliability, and cost efficiency across the full data infrastructure stack
• Collaborate with platform and product engineering teams to define data contracts and schemas
• Establish best practices, tooling, and patterns for data infrastructure

Requirements

• 2-5 years of experience operating large-scale data infrastructure systems in production
• Experience with open table formats, particularly Apache Iceberg
• Extensive hands-on experience with Apache Spark for batch and streaming processing
• Background in real-time data integration and stream processing (Kafka, Flink, etc.)
• Experience with data pipeline orchestration using Airflow
• Strong software engineering fundamentals in Python and/or Scala
• Experience with AWS or comparable cloud platforms and S3-based data lake architectures
• Experience with Kubernetes and containerized deployment
• Degree in Computer Science, Engineering, or equivalent practical experience
• Located in San Francisco and open to working in office

About the Company

Peregrine helps public safety organizations, governments, and private-sector institutions address society’s challenges with an AI-enabled platform that turns disconnected data into operational intelligence.