Sr Data Engineer to assist Data engineering and implement changes to all impacted pipelines and systems - 00400
Job Type: Contract
Positions to fill: 1
Start Date: Oct 24, 2022
Job End Date: Mar 31, 2023
Pay Rate: Hourly: Negotiable
Job ID: 123615
Our public sector client needs a Senior Data Engineer to assist Data engineering and implement changes to all impacted pipelines and systems.
- Experience and/or understanding of Agile Methodologies
- Advanced Database Skills (Oracle/DB2 & SQL)
- Experience with or exposure to Big Data platforms, ideally exposure to Hadoop ecosystem (HDFS, Apache Hive, Apache Spark, SparkSQL)
- Extensive programming experience in either Java, C++ or preferably Scala,IntelliJ, Jenkins
- Working experience with Apache Spark Data Processing Framework, Parquet data format
- Experience working with Jira
- Creating mapping documentation of data elements from source to target and liaising with other Data Engineers, Data Architects and SMEs.
- Developing & testing Data Transformation pipelines by leveraging the latest Big Data tools and technologies
- Collaborate with upstream data teams, develop data validation and test plans,
- performing user acceptance testing, and provide feedback to development and sustainment teams
- Conducting analysis for moderate to complex data requests, defining data fields and
- determining data availability, developing information layout, format and interactivity. Presenting findings and providing clarification.
- Act as a lead developer to implement code changes in various environments and platforms leveraging our DevOps & DataOps
- Work closely with the testing team to assist with issues raised in user acceptance testing (UAT) and Business Value Testing (BVT)
- Fully implement the new data format across all IM supported platforms including Big Data (Hadoop), Enterprise Data Warehouse (EDW) and Operational Data Store (ODS)