Spark Scala Developer

Roles and Responsibilities

• Design, develop, and maintain Spark-based batch processing pipelines using Scala for large datasets.
• Implement efficient transformations, aggregations, and joins, ensuring correctness and scalability.
• Write optimized SQL for data extraction, validation, and reconciliation across sources and targets.

Performance, Quality & Reliability

• Tune Spark jobs (partitioning, caching, shuffles, memory/executor settings) to improve runtime and cost efficiency.
• Build data quality checks and validations to ensure accuracy, completeness, and consistency of outputs.
• Troubleshoot production issues, perform root‑cause analysis, and implement preventive fixes.

Collaboration & Delivery

• Work with stakeholders to understand data requirements and translate them into technical solutions.
• Participate in code reviews, follow engineering best practices, and contribute...