Vacancy Description
Design, develop, and maintain efficient and scalable data pipelines using Python, PySpark, and Databricks on AWS. Work closely with data scientists, analysts, and other engineers to ensure smooth data integration and high-quality data processing. Build and optimize complex ETL (Extract, Transform, Load) workflows to handle large datasets. Implement data ingestion and transformation logic using PySpark on Databricks to improve data processing performance. Ensure data quality, accuracy, and consistency across all data pipelines. Collaborate with cross-functional teams to identify and resolve bottlenecks in data systems. Utilize AWS cloud services (e.g., S3, Redshift, RDS, EMR) for data storage and management. Write SQL queries to extract, manipulate, and transform data stored in relational databases. Develop, deploy, and maintain APIs for integrating data systems and enabling data access. Monito...