Principal Data Engineer

About the Company

At BayRock Labs, we pioneer innovative tech solutions that drive business transformation. As a leading product engineering firm based in Silicon Valley, we provide full-cycle product development, leveraging cutting-edge technologies in AI, ML, and data analytics.

Responsibilities

• Design and implement enterprise-grade data architectures and Lakehouse implementations
• Develop end-to-end data pipelines using Azure Databricks, PySpark, and Delta Live Tables
• Lead technical roadmap planning and execution for large-scale data platforms
• Implement metadata-driven ingestion frameworks and Medallion architecture
• Optimize cloud costs and performance across Azure services
• Collaborate with cross-functional stakeholders to deliver scalable data products
• Guide data governance frameworks and ensure security through RBAC and Managed Identity

Requirements

• 5–10+ years of hands-on experience in data engineering or data architecture
• Deep expertise in Azure (ADLS Gen2, ADF, Synapse, Azure SQL)
• Advanced proficiency in Databricks, PySpark, Python, and SQL
• Proven experience with Delta Lake, Unity Catalog, and dimensional modeling
• Strong knowledge of CI/CD, Terraform, and DevOps practices
• Experience migrating legacy EDWs to modern Lakehouse architectures

Preferred Qualifications

• Experience with Data Vault 2.0 modeling
• Expertise in Change Data Capture (CDC) implementation
• Knowledge of GDPR and SOX governance frameworks