Phyton with Spark Developer

Responsibilities

Direct Responsibilities

·         Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.

• Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
• Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
• Unit‑test, integrate‑test, and review code.
• Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs.
• Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
• Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint.

·         Keep up to date with latest technologies and trends and provide input, expertise and recommendations.

Contrib...