Sr. Data Engineer to integrate machine learning models into a Hadoop based big data environment using Apache Spark - 858

Our valued client is currently seeking a Sr. Data Engineer to integrate machine learning models into a Hadoop based big data environment using Apache Spark.

Initial 11-month contract that can be worked out of Montreal, Toronto, or Saint John (hybrid remote, 2 days/week onsite). 

Job Description:
Act as ML Data Engineer focused on Spark applications and support and implement code to build up the ML Development and Production process.  You will be working with a team of data scientists and data engineers to support the MLOps process ongoing for a Security Operation Center.  For this reason, you will be asked to assess the current Big Data technology in use and to test new ones to evolve the system, especially in Public Cloud Infrastructure.

Responsibilities:

• Integrating machine learning models onto a Hadoop based big data environment using Apache Spark.
• Working with data scientists to develop part of their code to interface with other components and to improve its efficiency.
• Implementing automation and monitoring tools with a DevSecOps mindset
• Working with the Architecture Team to test new technology and plan the evolution of the Big Data System
• Developing a log enrichment pipeline capable of functioning at a massive scale.
• Configuring Kafka and other components for handling data pipeline.

Must-Have Skills:

• 5+ years experience as a Data Engineer 
• Experience with Apache Spark and Kafka
• Experience deploying code in Apache Hadoop ecosystems.
• Python development experience
• AWS experience

Nice to Have Skills:

• Strong written and verbal communication skills in English and French
• Ability to code in Java and/or Scala
• Experience with Data Science Platform
• Experience with information security-related applications e.g. SIEM