Hadoop Developer

Responsibilities

• Develop, review and analyze complex, large-scale BigData/Cloud (AWS) solutions for tactical and strategic business objectives, requiring in-depth evaluation of existing software applications.
• Build and maintain code to populate HDFS and Hadoop with log events from real-time data feeds or data loaded from SQL production systems.
• Assess the impact of external production system changes to Big Data systems on Spark and implement changes to the ETL to ensure consistent and accurate data flows.
• Build Hive scripts to ingest millions of records and perform performance tuning of these scripts.
• Use Sqoop to populate data from RDBMS to HDFS and develop Hive/Pig scripts to perform analysis.
• Design and develop UNIX shell scripts as part of the ETL process to automate data loading and pulling.
• Deliver advanced and complex reporting solutions including dashboards and standardized reporting using Power BI.
...