Data Engineer

Responsibilities

• Design, develop, and maintain large-scale Spark applications using Python, Scala, and/or Java
• Build and optimize batch and streaming data pipelines in distributed environments
• Write production-quality Spark code with a focus on performance, scalability, and reliability
• Optimize Spark jobs through partitioning, caching, memory tuning, and execution plan analysis
• Develop reusable Spark frameworks, libraries, and utilities
• Work with structured and semi-structured data formats including Parquet, Delta, CSV, and JSON
• Collaborate with platform, analytics, and data science teams to support downstream use cases
• Debug and troubleshoot Spark job failures and performance issues in production

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field
• 3+ years of experience as a Data Engineer or in a similar role
• Strong hands-on experience with Apache Spark
• Proficiency in one or more of the following languages: Python, Scala, or Java
• Solid experience with SQL for data transformation and analysis
• Experience working with data lakes and columnar formats (e.g., Parquet, Delta Lake)
• Familiarity with Git and standard software engineering practices

Preferred Qualifications

• Experience running Spark on Azure Databricks
• Experience migrating Spark jobs from on-prem Hadoop/Cloudera to cloud platforms
• Familiarity with orchestration tools such as Airflow or Azure Data Factory
• Exposure to cloud storage (ADLS Gen2) and cloud security concepts

Benefits

• Competitive Rewards program including bonus, flexible vacation, personal, and sick days
• Comprehensive benefits starting on day one
• Upskilling opportunities through online courses and tuition assistance
• Inclusive culture with various Employee Resource Groups (ERGs)
• Dynamic workspace with collaborative spaces and amenities

About the Company

Scotiabank is a leading bank in the Americas. Guided by our purpose "for every future", we help our customers, their families, and their communities achieve success through a broad range of advice, products, and services, including personal and commercial banking, wealth management, and capital markets.