Data Scientist

Responsibilities

• Design and implement automated evaluation pipelines to assess AI output quality, accuracy, and safety at scale
• Develop uncertainty quantification systems where confidence scores correlate with accuracy
• Build evaluation frameworks combining automated assessment with clinician-validated test cases
• Implement feedback loops to continuously improve model outputs based on validation signals
• Establish scalable quality gates to catch errors before they reach end users
• Contribute to model alignment and fine-tuning efforts

Requirements

• Strong foundation in deep learning frameworks (PyTorch) and LLM architectures
• Experience with model evaluation, benchmarking, and quality metrics
• Proficiency in Python and modern ML development tools
• Strong statistical foundations
• Ability to read, implement, and extend research papers
• Excellent communication skills

Preferred Qualifications

• Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)
• Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)
• Experience with RLHF, DPO, or preference optimization techniques
• Background in healthcare AI or regulated industries
• Experience building evaluation systems for production LLM applications