Job Description
We are seeking a skilled Databricks Architect to design, implement, and optimize scalable data solutions within our cloud-based data platform. This role requires extensive knowledge of Databricks (Azure/AWS), data engineering, and a deep understanding of data architecture principles, with the ability to drive strategy, best practices, and hands-on implementation for high-performance data processing and analytics solutions.
Roles and Responsibilities:
• Solution Architecture: Design and architect end-to-end data solutions using Databricks and Azure/AWS, including data ingestion, processing, and storage.
• Delta Lake Implementation: Leverage Delta Lake and Lakehouse architecture to create robust, unified data structures that support advanced analytics and machine learning.
• Data Processing Development: Develop, design, and automate large-scale, high-performance data processing systems (batch and/or streaming) to drive business growth and enhance the product experience.
• Performance Tuning: Ensure optimal performance of data pipelines and workloads by implementing best practices for resource management, auto-scaling, and query optimization in Databricks.
• Engineering Best Practices: Advocate for high-quality software engineering practices in building scalable data infrastructure and pipelines.
• Architecture/Solution Development : Develop Architecture or solution for large data project using Databricks
• Project Leadership: Lead data engineering projects to ensure pipelines are reliable, efficient, testable, and maintainable.
• Data Modeling: Design data models optimized for storage, retrieval, and critical product and business requirements.
• Logging Architecture: Understand and influence logging to support data flow, implementing logging best practices as needed.
• Standardization and Tooling: Contribute to shared data engineering tools and standards to boost productivity and quality for Data Engineers across the company.
• Collaboration: Work closely with leadership, engineers, program managers, and data scientists to understand and meet data needs.
• Partner Education: Use data engineering expertise to identify gaps and improve existing logging and processes for partners.
• Data Governance: Collaborate with stakeholders to build data lineage, data governance, and data cataloging using unity catalog
• Agile Project Management: Lead projects using agile methodologies.
Roles and Responsibilities:
• Solution Architecture: Design and architect end-to-end data solutions using Databricks and Azure/AWS, including data ingestion, processing, and storage.
• Delta Lake Implementation: Leverage Delta Lake and Lakehouse architecture to create robust, unified data structures that support advanced analytics and machine learning.
• Data Processing Development: Develop, design, and automate large-scale, high-performance data processing systems (batch and/or streaming) to drive business growth and enhance the product experience.
• Performance Tuning: Ensure optimal performance of data pipelines and workloads by implementing best practices for resource management, auto-scaling, and query optimization in Databricks.
• Engineering Best Practices: Advocate for high-quality software engineering practices in building scalable data infrastructure and pipelines.
• Architecture/Solution Development : Develop Architecture or solution for large data project using Databricks
• Project Leadership: Lead data engineering projects to ensure pipelines are reliable, efficient, testable, and maintainable.
• Data Modeling: Design data models optimized for storage, retrieval, and critical product and business requirements.
• Logging Architecture: Understand and influence logging to support data flow, implementing logging best practices as needed.
• Standardization and Tooling: Contribute to shared data engineering tools and standards to boost productivity and quality for Data Engineers across the company.
• Collaboration: Work closely with leadership, engineers, program managers, and data scientists to understand and meet data needs.
• Partner Education: Use data engineering expertise to identify gaps and improve existing logging and processes for partners.
• Data Governance: Collaborate with stakeholders to build data lineage, data governance, and data cataloging using unity catalog
• Agile Project Management: Lead projects using agile methodologies.
💡 Quick Summary
Seeking a career-building opportunity? The Lead Databricks - Immediate joiner position is now open for candidates interested in the IT Engineer & Developer Jobs sector. This role in Pune offers a professional environment and growth potential.
Requirement Snapshot: Candidates should possess basic communication skills, a proactive attitude, and the ability to work in a team. Experience in IT Engineer & Developer Jobs is a plus.
