Data Engineer

Responsibilities

• Implement a scalable ETL pipeline in Azure Databricks using medallion architecture to transform raw data into standardized schemas.
• Leverage existing email ingestion tools to extract source files from shared mailboxes.
• Develop Databricks notebooks, jobs, and pipelines for efficient ETL orchestration.
• Design and implement robust validation rules, monitoring, and troubleshooting processes with integrated logging and notifications.
• Integrate data workflows with CI/CD processes using Azure DevOps.
• Produce comprehensive technical documentation to ensure long-term solution maintainability.

Requirements

• 2+ years of hands-on experience with Azure Databricks, including data validation and optimization.
• Expertise in PySpark and Spark SQL.
• Proficiency in building end-to-end data pipelines using Delta Lake and Lakehouse architecture.
• Strong knowledge of Unity Catalog and modern Azure Databricks features.
• Experience implementing CI/CD pipelines, specifically with Azure DevOps.

Preferred Qualifications

• Experience supporting BI reporting layers such as Power BI.
• Previous experience working within the insurance industry.

About the Company

Ryan Specialty is a leading provider of innovative solutions for insurance brokers, agents, and carriers. Recognized as one of Newsweek’s Most Loved Workplaces, we pride ourselves on fostering a diverse and inclusive environment where the brightest talent in the insurance industry can thrive.