Senior Data Quality Engineer

Responsibilities

• Lead the design, implementation, and automation of enterprise-scale data quality frameworks within a Databricks environment.
• Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations.
• Develop AI-assisted profiling notebooks using PySpark to establish baseline data quality scores across completeness, uniqueness, validity, consistency, accuracy, and timeliness.
• Design and build a scalable Data Quality Rule Factory using parameterized PySpark functions to enable automated deployment of thousands of rules.
• Integrate data quality controls and quality gates within Bronze, Silver, and Gold Delta Lake layers.
• Build automated data cleansing pipelines for standardization, deduplication, and schema harmonization.
• Deploy MLflow-managed machine learning models for anomaly detection and duplicate identification.
• Design exception management frameworks, including failed-record handling and quarantine Delta tables.
• Build Delta Lake aggregation tables and deliver data quality KPIs to Power BI dashboards.
• Develop predictive models to identify datasets at risk of quality degradation and support AI-assisted Root Cause Analysis.

Requirements

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field.
• 5+ years of experience in Data Engineering or Data Quality Engineering.
• 3+ years of hands-on experience with Databricks and PySpark.
• Strong expertise in Delta Lake architecture and data pipeline development.
• Experience with Unity Catalog implementation and governance.
• Hands-on experience with MLflow and machine learning deployment.
• Strong SQL skills and data modeling expertise.
• Experience integrating Databricks with Power BI and Azure services.
• Strong understanding of data governance, metadata management, and data quality dimensions.

Preferred Qualifications

• Microsoft Azure certifications.
• Databricks Certified Data Engineer Associate or Professional.
• Experience with enterprise data governance programs.
• Experience implementing AI-assisted data quality and remediation solutions.
• Knowledge of Master Data Management (MDM) principles.

About the Company

Robusta specializes in high-scale data engineering and quality solutions, managing complex data environments to ensure enterprise-grade reliability and insight.