Senior Data Engineer – Azure Databricks
We are seeking an experienced Senior Data Engineer to support a major data modernization and transformation initiative. This is a hands-on role focused on building and optimizing scalable data pipelines, data products, and analytics solutions within the Microsoft Azure ecosystem, with a strong emphasis on Azure Databricks and Delta Lake.
The successful candidate will work closely with data engineering, analytics, and business stakeholders to modernize data assets, integrate new data sources, improve data quality, and develop resilient solutions that support enterprise reporting and analytics.
Key Responsibilities
- Design, develop, and optimize robust data ingestion and transformation pipelines using Azure Databricks.
- Build scalable data solutions using Databricks, Delta Lake, Delta Live Tables (DLT), Python, Spark, and SQL.
- Work with analytics and business stakeholders to understand workflows, data requirements, and reporting needs and translate them into technical solutions.
- Support the design and development of master data products and reusable data assets.
- Participate throughout the development lifecycle, including technical scoping, requirements analysis, specifications, development, testing, and implementation.
- Optimize Databricks environments through code optimization, cluster configuration, partitioning, Z-ordering, scaling, and data-consumption strategies.
- Develop and enhance data ingestion, transformation, validation, and consumption processes.
- Work with Azure Data Factory and Azure DevOps to support data integration and development activities.
- Troubleshoot complex implementation and data pipeline issues and provide technical guidance and recommendations.
- Partner with technical teams to establish data engineering best practices and improve platform performance, reliability, and scalability.
- Communicate project progress, technical risks, dependencies, and recommendations to stakeholders and project leadership.
Required Qualifications
- Bachelor’s degree in Computer Science, Data Science, Information Technology, or a related discipline.
- Strong professional experience as a Data Engineer within Microsoft Azure environments.
- Expert-level, recent hands-on experience with Azure Databricks.
- Strong experience with Azure Data Factory and Azure DevOps.
- Advanced proficiency with Python, Spark/PySpark and SQL.
- Hands-on experience with Delta Lake and Delta Live Tables (DLT).
- Strong understanding of data warehousing, data modeling, ETL/ELT and relational databases.
- Experience optimizing Databricks workloads, including clusters, partitioning, Z-ordering and performance.
Preferred Experience
Experience working with healthcare or other complex enterprise data environments is highly desirable. Exposure to clinical or healthcare information systems and their associated data models would be considered a strong asset.
Work Arrangement
This is a hybrid Vancouver-based opportunity. Candidates must be comfortable working onsite at least one day per week, with additional onsite attendance as required for project needs and team collaboration.
