Data Engineer Consultant - Spark/Python

Job Description

Required Skills :

• Must have excellent knowledge in Apache Spark and Python programming experience

• Deep technical understanding of distributed computing and broader awareness of different Spark version

• Strong UNIX operating system concepts and shell scripting knowledge

• Hands-on experience using Spark & Python

• Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.

• Externally certified in one of the cloud services (foundational or advanced)- (AWS, GCP, Azure, Snowflake, Databricks)

• Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred

• Experience in creating visualizations in either of Tableau, PowerBI, Qlik, Looker or any of the other reporting tools

• Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform

• Should have exposure with Jenkins or equivalent CICD tool & Git repository

• Experience handling CDC operations for huge volume of data

• Should understand and have operating experience with Agile delivery model

• Should have experience in Spark related performance tuning

• Should be well versed with understanding of design documents like HLD, TDD etc

• Should be well versed with Data historical load and overall Framework concepts

• Should have participated in different kinds of testing like Unit Testing, System Testing, User Acceptance Testing, etc

Preferred Skills

• Exposure to PySpark, Cloudera/ Hortonworks, Hadoop and Hive.

• Exposure to AWS S3/EC2 and Apache Airflow

(ref:hirist.tech)