Senior Developer | Spark

Senior Developer - Spark

Location: Bangalore/ Permanent WFH

Jobdescription

Owns the end-to-end implementation of the assigned data processing components/product?features i.e.,?design, development, deployment and testing of the data processing components and associated flows conforming to best coding practices

Creation and optimization of data engineering pipelines for analytics projects.

Support data and cloud transformation initiatives

Contribute to our cloud strategy based on prior experience

Independently work with all stakeholders across the organization to deliver enhanced functionalities

Create and maintain automated ETL processes with special focus on data flow, error recovery, and exception handling and reporting

Gather and understand data requirements, work in the team to achieve high quality data ingestion and build systems that can process the data, transform the data

Be able to comprehend application of database index and transactions

Involve in the design and development of a Big Data predictive analytics SaaS-based customer data platform using object-oriented analysis, design and programming skills, and design patterns

Implement ETL workflows for data matching, data cleansing, data integration and management

Maintain existing data pipelines, and develop new data pipeline using big data technologies

Responsible for leading the effort of continuously improving reliability, scalability, and stability of micro services and platform

Skills Required

Experience in advanced Apache Spark processing framework, spark programming with Java with sound knowledge in?RDBMS.

At least 2+ years into Apache SPARK and Any Orchestration (Eg Apache Airflow)

At least 1 Years hands-on experience with Java,?Spring

Strong object-oriented design and development skills, data structures and algorithms, and design patterns

Experience in working with Core Spark, Spark Streaming,?Data frame?API, Data set API, RDD APIs & Spark SQL programming dealing with processing terabytes of data. Specifically, this experience must be in writing Big Data?engineering jobs for large-scale data integration in AWS.

Experience in writing spark?streaming?jobs integrating with streaming frameworks such as Apache Kafka or AWS Kinesis.

Experience in physical table design in a Big Data environment

Experience in working with external job schedulers such as?AWS?data pipeline, Airflow, etc.

Experience working in?Key Value?data stores such as?Arango,?HBase, etc.

Spark Profiles

a) Check if the candidate has the hands on in each of the below skills

ü Apache Spark

ü Experience in Java

ü Experience in Data Transformations using Java

ü Experience in physical DB tables design in Big Data environment

If answer to any of the above 4 skills is NO, we can ignore that profile

b) If the hands on is not in the recent past, i.e., not within previous 12 months, we can ignore such profile

c) Apache with non-Java programming language experience, like SCALA etc can be ignored i.e., don’t need to forward to Anvizent as we need Apache with Java programming experience only