Software Engineer, Data Acquisition

Responsibilities

• Own and lead engineering projects in data acquisition, including web crawling, data ingestion, and search
• Collaborate with Data Processing, Architecture, and Scaling teams to ensure system operability
• Work with legal teams to handle compliance and data privacy matters
• Develop and deploy highly scalable distributed systems for petabyte-scale data
• Architect and implement algorithms for data indexing and search
• Build and maintain backend services for data storage and key-value databases
• Deploy solutions in a Kubernetes Infrastructure-as-Code environment
• Conduct and analyze experiments to provide insights on system performance

Requirements

• BS/MS/PhD in Computer Science or a related field
• 4+ years of industry experience in software development
• Strong expertise in large stateful distributed systems and data processing
• Proficiency in Kubernetes and Infrastructure-as-Code concepts
• Strong written and verbal communication skills

Preferred Qualifications

• Experience with large web crawlers

About the Company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.