Lead Cloud Data Engineer (Databricks) - Assistant Vice President

Responsibilities

• Design, develop, and maintain scalable data pipelines and workflows using Databricks and Spark
• Build and optimize ETL/ELT pipelines for both batch and near real-time data processing
• Collaborate with architects and business stakeholders to translate requirements into technical implementations
• Ensure data quality, governance, and consistency across all data platforms
• Perform performance tuning and optimize the cost efficiency of data workloads
• Support production deployments, troubleshoot issues, and ensure the reliability of data systems

Requirements

• 14-18 years of hands-on experience with Databricks and Apache Spark (Python or Scala)
• Strong experience in Data Engineering and Big Data technologies
• Proficiency with cloud platforms such as AWS or Azure
• Strong SQL and Python/Scala programming skills
• Expertise in data pipeline design, ETL/ELT frameworks, and data modeling
• Proven experience in performance optimization and the scalability of data solutions

Preferred Qualifications

• Experience with streaming technologies such as Kafka
• Knowledge of Delta Lake and data lakehouse architecture
• Experience with CI/CD and DevOps practices for data engineering
• Familiarity with containerization tools like Docker and Kubernetes
• Exposure to data governance and regulatory environments

About the Company

State Street is a leading global provider of financial services to institutional investors. Our Global Technology Services (GTS) function drives the company's digital transformation, utilizing advanced technologies like cloud, artificial intelligence, and robotics to deliver critical data and insights to our clients worldwide.