Data Engineer

Responsibilities

• Architect, implement, and maintain scalable data pipelines using Apache NiFi for ETL and real-time data integration
• Design, optimize, and manage Snowflake data warehouses, including schema design, performance tuning, and security controls
• Administer Linux-based cloud environments (Azure, AWS, or GCP), including automation of deployments, patching, and monitoring
• Develop and maintain CI/CD workflows for data engineering solutions
• Collaborate with data scientists and analysts to deliver high-quality, production-ready datasets and analytics solutions
• Implement robust data governance, lineage, and auditing practices
• Troubleshoot and resolve issues across the data platform, including connectivity, performance, and security
• Mentor Agency staff and contribute to technical best practices and documentation

Requirements

• Baccalaureate degree in computer science, engineering, or a related field
• Four years of experience in datacenter engineering, cloud engineering, or complex IT infrastructure
• OR eight years of experience if the degree is in an unrelated field
• Experience with Apache NiFi, Snowflake, and Linux-based cloud environments (Azure, AWS, or GCP)

About the Company

The NYC Department of Transportation's IT & Telecom division supports the Enterprise Data Science and Engineering Unit (EDSE), which designs and operates advanced data storage and processing solutions for Big Data to empower leadership and planners with actionable insights.