Job Description
Job Title: Platform Data Engineer
Job Location: Houston, TX – 77079 (Hybrid – 2 to 3 days)
Job Duration: 7+ months contract (with highly possibility of extension) (W2 only)
About the Role:
• We are seeking a Platform Engineer with deep expertise in Databricks administration, data governance, and platform‐level engineering standards. This role enables multiple analytics and AI teams to build safely, efficiently, and consistently on a shared Databricks platform by enforcing data quality, ingestion standards, security policies, and cost governance.
• You will be the technical owner of platform guardrails, operational stability, access patterns, and cost controls—ensuring the platform scales reliably across business teams.
Key Responsibilities:
Platform Administration & Governance:
• Administer Databricks workspaces, clusters, jobs, Unity Catalog, compute policies, environment configuration, and platform guardrails.
• Implement and maintain RBAC and ABAC access controls for secure, compliant data access.
• Define and enforce data ingestion standards, naming conventions, schema rules, Delta Lake design patterns, and data quality expectations.
Data Quality & Ingestion Standards:
• Set platform‐wide standards for ingestion pipelines, Delta architecture, lineage, versioning, and validation.
• Review and approve onboarded pipelines for compliance with platform requirements.
• Partner with data engineering teams to uplift patterns and enforce consistency.
Security, Compliance & Access Controls:
• Manage workspace and catalog permissions, row/column‐level policies, attribute‐based filtering, and workspace isolation.
• Collaborate with security teams to maintain compliance and enforce global data protection standards.
Cost Management & Monitoring:
• Implement cost thresholds, alerts, compute policies, and usage dashboards to prevent overspend.
• Monitor job and cluster costs, detect anomalies, and recommend optimization actions.
• Provide visibility into SKU‐level spend and workspace cost patterns.
Operational Stability & Observability:
• Ensure platform reliability through automated testing, CI/CD templates, and code governance.
• Build dashboards to track code compliance, data access, pipeline health, schema drift, and cost thresholds.
• Resolve platform incidents and prevent recurrence by strengthening guardrails and configurations.
Enablement & Best Practices:
• Define “handrails” for building on the platform: ingestion, Delta conventions, CI/CD, observability, and AI/ML patterns.
• Coach data/analytics teams on compliant onboarding and optimal platform usage.
• Maintain internal documentation, patterns, code templates, and guidance.
Required Skills & Experience:
• 5+ years in data engineering or platform engineering, with at least 2–3 years in Databricks administration.
• Expert knowledge of Unity Catalog, cluster policies, Delta Lake, Spark, workspace configuration, and jobs.
• Strong grounding in data governance, data modeling, ingestion frameworks, schema enforcement, versioning, and lineage.
• Proven experience implementing RBAC and ABAC in Databricks or similar platforms.
• Experience with cost optimization, monitoring, billing logs, and compute governance.
• Strong Python/PySpark and SQL skills; familiarity with DLT, Airflow, or Databricks Workflows.
• Strong communication skills with ability to set standards and influence teams diplomatically.
Preferred Qualifications:
• Experience in large-scale enterprise data platforms (Azure/AWS/GCP).
• Familiarity with Trading & Supply or other high‐stakes analytical environments.
• Experience creating dashboards for governance, cost, compliance, and pipeline health.
• Experience with CI/CD, GitHub Actions, Azure DevOps, or similar tools.
Success Indicators:
• Teams consistently follow ingestion and data standards.
• Platform costs stabilized and predictable.
• Strong adoption of platform guardrails, templates, and operational dashboards.
• Reduced incidents related to access, cost, or ingestion quality.
#IRI-IT
Benefits:
• 401(k)
• Dental insurance
• Health insurance
Work Location: Hybrid remote in Houston, TX 77079
Job Location: Houston, TX – 77079 (Hybrid – 2 to 3 days)
Job Duration: 7+ months contract (with highly possibility of extension) (W2 only)
About the Role:
• We are seeking a Platform Engineer with deep expertise in Databricks administration, data governance, and platform‐level engineering standards. This role enables multiple analytics and AI teams to build safely, efficiently, and consistently on a shared Databricks platform by enforcing data quality, ingestion standards, security policies, and cost governance.
• You will be the technical owner of platform guardrails, operational stability, access patterns, and cost controls—ensuring the platform scales reliably across business teams.
Key Responsibilities:
Platform Administration & Governance:
• Administer Databricks workspaces, clusters, jobs, Unity Catalog, compute policies, environment configuration, and platform guardrails.
• Implement and maintain RBAC and ABAC access controls for secure, compliant data access.
• Define and enforce data ingestion standards, naming conventions, schema rules, Delta Lake design patterns, and data quality expectations.
Data Quality & Ingestion Standards:
• Set platform‐wide standards for ingestion pipelines, Delta architecture, lineage, versioning, and validation.
• Review and approve onboarded pipelines for compliance with platform requirements.
• Partner with data engineering teams to uplift patterns and enforce consistency.
Security, Compliance & Access Controls:
• Manage workspace and catalog permissions, row/column‐level policies, attribute‐based filtering, and workspace isolation.
• Collaborate with security teams to maintain compliance and enforce global data protection standards.
Cost Management & Monitoring:
• Implement cost thresholds, alerts, compute policies, and usage dashboards to prevent overspend.
• Monitor job and cluster costs, detect anomalies, and recommend optimization actions.
• Provide visibility into SKU‐level spend and workspace cost patterns.
Operational Stability & Observability:
• Ensure platform reliability through automated testing, CI/CD templates, and code governance.
• Build dashboards to track code compliance, data access, pipeline health, schema drift, and cost thresholds.
• Resolve platform incidents and prevent recurrence by strengthening guardrails and configurations.
Enablement & Best Practices:
• Define “handrails” for building on the platform: ingestion, Delta conventions, CI/CD, observability, and AI/ML patterns.
• Coach data/analytics teams on compliant onboarding and optimal platform usage.
• Maintain internal documentation, patterns, code templates, and guidance.
Required Skills & Experience:
• 5+ years in data engineering or platform engineering, with at least 2–3 years in Databricks administration.
• Expert knowledge of Unity Catalog, cluster policies, Delta Lake, Spark, workspace configuration, and jobs.
• Strong grounding in data governance, data modeling, ingestion frameworks, schema enforcement, versioning, and lineage.
• Proven experience implementing RBAC and ABAC in Databricks or similar platforms.
• Experience with cost optimization, monitoring, billing logs, and compute governance.
• Strong Python/PySpark and SQL skills; familiarity with DLT, Airflow, or Databricks Workflows.
• Strong communication skills with ability to set standards and influence teams diplomatically.
Preferred Qualifications:
• Experience in large-scale enterprise data platforms (Azure/AWS/GCP).
• Familiarity with Trading & Supply or other high‐stakes analytical environments.
• Experience creating dashboards for governance, cost, compliance, and pipeline health.
• Experience with CI/CD, GitHub Actions, Azure DevOps, or similar tools.
Success Indicators:
• Teams consistently follow ingestion and data standards.
• Platform costs stabilized and predictable.
• Strong adoption of platform guardrails, templates, and operational dashboards.
• Reduced incidents related to access, cost, or ingestion quality.
#IRI-IT
Benefits:
• 401(k)
• Dental insurance
• Health insurance
Work Location: Hybrid remote in Houston, TX 77079
💡 Quick Summary
Seeking a career-building opportunity? The Platform Data Engineer position is now open for candidates interested in the IT Engineer & Developer Jobs sector. This role in Houston offers a professional environment and growth potential.
Requirement Snapshot: Candidates should possess basic communication skills, a proactive attitude, and the ability to work in a team. Experience in IT Engineer & Developer Jobs is a plus.
