Senior Data Engineer

Streaming & Real-Time Data Processing

• Design and develop real-time streaming pipelines using Databricks Structured Streaming.
• Build and maintain Kafka-based ingestion frameworks.
• Handle late-arriving events using watermarks, event-time processing, and stateful streaming concepts.
• Implement exactly-once processing and checkpointing mechanisms.
• Monitor and optimize streaming workloads for performance and reliability.

Spark Declarative Pipelines (SDP) & Delta Live Tables (DLT):

• Design and implement Spark Declarative Pipelines using Databricks.
• Develop Delta Live Table (DLT) pipelines for scalable data transformations.
• Implement data quality expectations, validations, retries, and failure handling within DLT pipelines.