Data Engineer

Responsibilities

• Create and manage robust data ingestion pipelines leveraging Databricks, Airflow, Kafka, and Terraform
• Ensure high performance, reliability, and efficiency by optimizing large-scale data pipelines
• Develop data processing workflows using Databricks, Delta Lake, and Spark technologies
• Maintain and improve the Data Lakehouse, utilizing Unity Catalog for efficient data management and discovery
• Construct automation, frameworks, and enhanced tools to streamline data engineering workflows
• Collaborate across teams to facilitate smooth data flow and integration
• Enforce best practices in observability, data governance, security, and regulatory compliance

Requirements

• Minimum 7 years as a Data Engineer or similar role
• Hands-on experience with Databricks, Delta Lake, Spark, and Scala
• Proven ability to design, build, and operate Data Lakes or Data Warehouses
• Proficiency with Data Orchestration tools such as Airflow, Dagster, or Prefect
• Familiarity with Change Data Capture tools like Canal, Debezium, or Maxwell
• Strong command of Scala, Python, or similar primary languages and SQL
• Experience with data catalog and metadata management (Unity Catalog, Lakeformation)
• Experience in Infrastructure as Code (IaC) using Terraform
• Excellent problem-solving and debugging abilities for complex data challenges
• Strong communication and collaboration skills

About the Company

PayPay is a fintech company providing a service enjoyed by 70 million users. The company is home to a very diverse team of members from more than 50 countries, working to create new value and deliver it to society.