Data Engineering Consultant

Primary Responsibilities:

• Design, develop, and implement scalable data pipelines using Azure Databricks
• Develop PySpark-based data transformations and integrate structured and unstructured data from various sources
• Optimize Databricks clusters for performance, scalability, and cost-efficiency within the Azure ecosystem
• Monitor, troubleshoot, and resolve performance bottlenecks in Databricks workloads
• Manage orchestration and scheduling of end to end data pipeline using tool like Apache airflow, ADF scheduling, logic apps
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to th...