Sr Data Engineer to assist Data engineering and implement changes to all impacted pipelines and systems - 00400

Our public sector client needs a Senior Data Engineer to assist Data engineering and implement changes to all impacted pipelines and systems.

Must-haves:

• Experience and/or understanding of Agile Methodologies
• Advanced Database Skills (Oracle/DB2 & SQL)
• Experience with or exposure to Big Data platforms, ideally exposure to Hadoop ecosystem (HDFS, Apache Hive, Apache Spark, SparkSQL)
• Extensive programming experience in either Java, C++ or preferably Scala,IntelliJ, Jenkins
• Working experience with Apache Spark Data Processing Framework, Parquet data format
• Experience working with Jira

Duties will include:

• Creating mapping documentation of data elements from source to target and liaising with other Data Engineers, Data Architects and SMEs.
• Developing & testing Data Transformation pipelines by leveraging the latest Big Data tools and technologies
• Collaborate with upstream data teams, develop data validation and test plans,
• performing user acceptance testing, and provide feedback to development and sustainment teams
• Conducting analysis for moderate to complex data requests, defining data fields and
• determining data availability, developing information layout, format and interactivity. Presenting findings and providing clarification.
• Act as a lead developer to implement code changes in various environments and platforms leveraging our DevOps & DataOps
• Work closely with the testing team to assist with issues raised in user acceptance testing (UAT) and Business Value Testing (BVT)

Deliverables:

• Fully implement the new data format across all IM supported platforms including Big Data (Hadoop), Enterprise Data Warehouse (EDW) and Operational Data Store (ODS)