Role: Azure Platform Engineer – Databricks

We are seeking an experienced Azure Platform Engineer (Databricks) to design, manage, optimize, and secure enterprise-scale Databricks environments on Azure. The ideal candidate will have strong expertise in Azure infrastructure provisioning, Databricks administration, Terraform automation, security compliance, and platform troubleshooting.

Key Responsibilities

Platform Design

Define and implement best practices for end-to-end Databricks platform architecture.
Collaborate with Databricks and internal teams to evaluate new Databricks features, including private and public previews.
Engage with Databricks product teams on feature enhancements and platform improvements.

Platform Infrastructure

Create and manage Databricks workspaces (Premium, Standard, Serverless) and clusters with proper sizing strategies.
Decommission unused workspaces and optimize platform resources.
Support Delta Sharing initiatives and work with enterprise teams on connected data-sharing solutions.

User & Access Management

Create and manage security groups, users, and access controls.
Assign Unity Catalog permissions to respective groups and teams.
Manage Quantum Collaboration platform environments used as sandbox spaces for enterprise ideation and innovation.

Troubleshooting & Performance Optimization

Investigate and diagnose Databricks performance issues and platform errors.
Analyze Databricks logs and error messages to identify root causes.
Resolve issues related to cluster configurations, job failures, and notebook execution.
Optimize Databricks notebooks and jobs for better performance and scalability.
Coordinate with Databricks support teams for complex or unresolved issues.
Document troubleshooting procedures, known issues, and solutions.
Develop and test Databricks clusters to ensure platform stability and scalability.

Monitoring & Dashboarding

Design dashboards tailored for different user personas such as data scientists, engineers, and administrators.
Integrate Databricks APIs and monitoring tools to provide real-time operational metrics.
Configure alerts for performance degradation and unexpected resource utilization.
Ensure dashboards are user-friendly and provide actionable insights.
Continuously improve dashboards based on feedback and evolving business requirements.

Security & Compliance

Analyze findings from Databricks Security and Compliance reports.
Apply necessary configurations and updates to remediate security vulnerabilities.
Track and validate the effectiveness of implemented security controls.
Collaborate with cybersecurity teams on enterprise security changes and governance initiatives.

Azure Infrastructure Provisioning & Automation

Strong hands-on expertise in Terraform for Azure infrastructure provisioning and automation.
Build and manage Infrastructure as Code (IaC) solutions for scalable Azure environments.
Automate deployment and configuration processes across cloud infrastructure.

Required Skills

Azure Databricks Administration
Azure Cloud Infrastructure
Terraform / Infrastructure as Code (IaC)
Unity Catalog & Access Management
Databricks Cluster Management
Delta Sharing
Monitoring & Alerting Tools
Performance Tuning & Troubleshooting
Security & Compliance Management
CI/CD & Automation
Strong problem-solving and communication skills

Preferred Qualifications

Experience with enterprise-scale Databricks implementations.
Knowledge of Azure networking and security best practices.
Familiarity with DevOps and automation frameworks.
Experience working with cross-functional enterprise teams.

Azure platform engineer (Databricks)