Sr Data Engineer with Python (orchestration), airflow, GCP and Big Query experience to pipe data and build out our clients enterprise data lake in a CI/CD environment.
Job Type: Contract
Positions to fill: 2
Start Date: May 29, 2023
Job End Date: Dec 31, 2023
Pay Rate: Hourly: Negotiable
Job ID: 130096
Our client is Canada's largest retailer. They are looking for a Sr Data Engineer with Python (orchestration), airflow, GCP and Big Query experience to pipe data and build out our clients enterprise data lake in a CI/CD environment.
- Build pipelines through our client's data lake going to our client's marketing insights tool.
- 27 data sources that must land in the data lake - run through remediation and apply the logic
- Work with data analysts - understand how to land the data .. The Data Engineer will build the orchestration and pipe the data through and build out the data lake
- 5+ years of relevant OO programming experience using Java (asset) and Python. Python for Orchestration using Apache Airflow and Java is for data flow to build data pipelines.
- Experience working with Cloud data platforms technologies: Google Cloud Platform (Dataflow, Big Query, Apache, Apache Airflow) Do a lot of work with Big Query - That is how our client sources the data for the analytical pillars.
- Experience using Jenkins to configure and build CI/CD pipelines.
- Advanced knowledge of SQL queries (verify requirements coming from the data analysts) go in and run queries, data quality checks
- Experience building Terraform scripts to provision infrastructure as code.
- 5 years of experience with relational databases (Teradata, Oracle, SQL Server) and big data technologies (Hadoop, Hive)