Job Description
Hands-On Development (75%)
• Build, and maintain scalable data platforms and infrastructure on AWS
• Implement end-to-end data pipelines for batch and real-time data processing
• Build robust ETL/ELT workflows to ingest, transform, and load data from diverse sources
• Implement data lake/Lakehouse architectures using AWS S3, Glue, Athena, and Lake Formation
• Design and optimize data warehouse solutions (Redshift, Snowflake) for analytics and reporting
• Establish data quality frameworks and automated monitoring systems
• Write production-quality Python code for data processing, transformation, and automation
• Build scalable data pipelines using Apache Airflow, AWS Step Functions, or similar orchestration tool
• Develop streaming data solutions using Kinesis, Kafka, or AWS MSK
• Optimize SQL queries and database performance for large-scale datasets
• Implement data validation, cleansing, and quality checks
• Build APIs and microservices for data access and integration
• Create monitoring, alerting, and observability solutions for data pipelines
• Debug and resolve data pipeline failures and performance bottlenecks
Technical Leadership & Collaboration (25%)
• Mentor junior and mid-level data engineers through code reviews and technical guidance
• Establish best practices for data engineering, testing, and deployment
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Work with ML engineers to build data pipelines supporting machine learning workflows
• Partner with platform/infrastructure teams on cloud architecture and cost optimization
• Lead technical design discussions and architectural reviews
• Document data architectures, pipelines, and processes
• Evangelize data engineering best practices across the organization
• Build, and maintain scalable data platforms and infrastructure on AWS
• Implement end-to-end data pipelines for batch and real-time data processing
• Build robust ETL/ELT workflows to ingest, transform, and load data from diverse sources
• Implement data lake/Lakehouse architectures using AWS S3, Glue, Athena, and Lake Formation
• Design and optimize data warehouse solutions (Redshift, Snowflake) for analytics and reporting
• Establish data quality frameworks and automated monitoring systems
• Write production-quality Python code for data processing, transformation, and automation
• Build scalable data pipelines using Apache Airflow, AWS Step Functions, or similar orchestration tool
• Develop streaming data solutions using Kinesis, Kafka, or AWS MSK
• Optimize SQL queries and database performance for large-scale datasets
• Implement data validation, cleansing, and quality checks
• Build APIs and microservices for data access and integration
• Create monitoring, alerting, and observability solutions for data pipelines
• Debug and resolve data pipeline failures and performance bottlenecks
Technical Leadership & Collaboration (25%)
• Mentor junior and mid-level data engineers through code reviews and technical guidance
• Establish best practices for data engineering, testing, and deployment
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Work with ML engineers to build data pipelines supporting machine learning workflows
• Partner with platform/infrastructure teams on cloud architecture and cost optimization
• Lead technical design discussions and architectural reviews
• Document data architectures, pipelines, and processes
• Evangelize data engineering best practices across the organization
💡 Quick Summary
Seeking a career-building opportunity? The Sr. Data Engineer/Tech Lead position is now open for candidates interested in the IT Engineer & Developer Jobs sector. This role in Bangalore offers a professional environment and growth potential.
Requirement Snapshot: Candidates should possess basic communication skills, a proactive attitude, and the ability to work in a team. Experience in IT Engineer & Developer Jobs is a plus.