Senior Developer (Databricks Technologies)

Responsibilities

• Design, develop, and optimize scalable data Lakehouse solutions and ETL workflows using Databricks and Delta Lake.
• Build robust ingestion pipelines using Databricks Autoloader for both real-time and batch data processing.
• Act as a technical lead and mentor for developers, overseeing complex design components and project deliverables.
• Optimize Spark jobs for maximum performance and cost-efficiency while monitoring production pipelines.
• Implement secure data access and governance using Unity Catalog and other access control mechanisms.
• Collaborate with cross-functional teams, including data architects and business stakeholders, to document data models and pipeline designs.
• Lead agile development teams and provide regular status updates within a Scrum framework.

Requirements

• Strong proficiency in Apache Spark, PySpark, and the Databricks platform.
• Hands-on experience with Delta Lake, Autoloader, and Structured Streaming.
• Advanced skills in Python, Scala, SQL, and PL/SQL.
• Extensive experience with Oracle databases and data modeling.
• Proven experience with cloud platforms such as AWS, Azure, or GCP.
• Practical knowledge of CI/CD, Git, and DevOps practices in a data engineering context.
• Experience with real-time data processing and event-driven architectures.
• Expertise in Agile/Scrum methodologies and adaptive software development.

Preferred Qualifications

• Experience working on Capital Markets projects.
• Ability to architect and design complex systems with minimal requirements and supervision.

About the Company

State Street is a leading global provider of financial services to institutional investors. Our Global Technology Services (GTS) function drives digital transformation through advanced technologies like cloud, AI, and robotics to deliver critical data and insights to our clients worldwide.