Job Description

Reports To

Reports to the Feature Lead/Project Manager on the program or the lead solution designer or architect. Will take direction from the CG or client project manager for work and technical design from the solution designer or architect.

Job Overview

The role of Data Engineer involves leading complex business reengineering development projects and to solve organizational problems through IT solutions using Big Data concepts, Platform tools and software engineering languages as required.

Key Responsibilities Include
• Designing, developing and implementing solutions to customers organisational problems;
• Providing high quality service delivery on engagements;
• Designing and developing data reconciliation & metadata driven data processing frameworks;

Primary Skills

Key Skills/Experience:
• Prior experience as a data engineer
• Hands-on experience in Cloud Computing – AWS preferably
• Experience in programming and shell scripting eg. Core/Python/Scala/Java
• Experience in Hadoop Stack eg. Hive, Spark (Core and Streaming), Kafka, Flink, NIFI;
• Experience with Github and Jenkins;
• Experience working on stream transform using Spark Streaming or Flink;
• Experience working on change data capture in a big data ecosystem;
• Experience in building real time or batch ingestion and transformation pipelines;
• Experience in data orchestration tools (Airflow is preferred)

Secondary Skills

Additional Skills/responsibilities Of This Role
• Experience supporting a production service in a DevOps friendly environment;
• Experience working in Agile/Scrum teams;
• Hands on experience in API and cloud-native architecture;
• Experience with ‘container’ technology (e.g. Docker, Kubernetes);
• Experience wrangling data using library such as Pandas, Scikit-learn or Numpy;
• Experience using notebook such as Jupyter or Polyglot;
• Experience with test-driven development, test automation & continuous delivery;
• Experience in Java build automation technologies such as Gradle, Maven, Integration with Sonarqube etc