Job Description

About the Company

We are looking for a strong core Data Engineer with hands-on experience in PySpark, Databricks, and Azure data platforms to design, build, and support end-to-end data pipelines. The ideal candidate will develop and optimize data transformations, build production-grade Python components, and maintain cloud-native Azure environments while collaborating with application teams and ensuring high-quality, reliable data delivery. This role offers the opportunity to work with large-scale datasets, implement ETL/ELT best practices, optimize Databricks clusters, and leverage modern cloud technologies to support AI/ML initiatives.

About the Role

Location: 3 days Hybrid in Chicago, IL

Duration: 6+ Month Contract

Interview: 2 video interview and final onsite

Responsibilities

• Design, build, and support end-to-end data pipelines, including ingestion, transformation, validation, and publishing.
• Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
• Build production-grade Python modules with logging, error handling, testing, and integration with APIs/files.
• Create, maintain, and operate Azure Data Factory (ADF) pipelines, including triggers, parameterization, monitoring, and failure handling.
• Work within Azure environments: ADLS Gen2 (Blob Storage), Azure SQL, Azure App Service, and resource groups.
• Provision and maintain Azure components using Pulumi (Infrastructure as Code).
• Optimize Databricks clusters, workflows, and jobs for performance and reliability.
• Participate in code reviews, documentation, and operational support, including triage and root cause analysis.
• Collaborate with application teams for integration, troubleshooting, and operational coordination.

Qualifications

• Education: Bachelor’s degree in Computer Science, Engineering, or a related technical field (or equivalent experience).

Required Skills

• Experience: 5+ years as a Data Engineer; 3+ years in ETL/ELT concepts, PySpark, and SQL.
• SQL: Advanced querying, CTEs, views, joins, complex transformations, and performance tuning.
• Python: 2+ years building production-quality modules, unit/integration testing, logging, and CI/CD integration.
• Databricks: 1+ year working with notebooks, jobs, workflows, external/managed tables, Delta Lake, and basic cluster configuration.
• Azure Data Factory (ADF): 1+ year creating and maintaining pipelines, including triggers, parameterization, monitoring, and error handling.
• Azure Cloud: Hands-on with ADLS Gen2, Azure SQL, Azure App Service, and general Azure portal/resource group operations.
• Infrastructure as Code: Experience provisioning Azure resources with Pulumi.
• ETL/ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.

Preferred Skills

• Additional Skills (nice-to-have): R for data validation, TypeScript for Pulumi pipelines, Java/.NET for integration, Angular/Spring Boot for minor troubleshooting.

Job Tags

Similar Jobs