 Role Summary

The Data Engineering Support Associate will be responsible for supporting, monitoring, troubleshooting, and enhancing data engineering pipelines built on Azure Data Factory (ADF), Azure Databricks, Python, and Azure Data Lake. The role involves ensuring smooth operation of data ingestion, transformation, validation, and reporting processes, along with supporting Data Platform, Master Data Management (MDM), Data Governance, and Analytics initiatives.

The platform includes Databricks-based data processing, data quality monitoring, lineage, governance, and Lakehouse capabilities.

Key Responsibilities

Production Support

Monitor daily batch and scheduled data pipelines in ADF and Databricks.
Perform incident analysis and resolution for failed jobs and data issues.
Troubleshoot pipeline, notebook, cluster, and data integration failures.
Analyze root causes and implement preventive actions.
Support service requests, defect fixes, and data correction activities.
Meet agreed SLAs and support targets.

Databricks Support

Monitor and support Databricks jobs, workflows, and clusters.
Troubleshoot PySpark notebook failures.
Optimize Spark jobs for better performance and reduced execution time.
Support Unity Catalog, data governance, and access management activities.
Maintain logging, monitoring, and auditability of data platforms.

Python Development & Support

Debug and maintain Python/PySpark scripts.
Develop utility scripts for validation and automation.
Implement exception handling and performance improvements.
Support data quality checks and reconciliation activities.
Manage code changes and deployments.

Azure Data Factory (ADF) Support

Monitor ADF pipelines, triggers, linked services, and datasets.
Resolve pipeline failures and connectivity issues.
Support incremental and full data loads.
Implement parameterized pipelines and workflow enhancements.

Data Quality & Governance Support

Execute and monitor data quality rules and scorecards.
Track completeness, validity, consistency, accuracy, and timeliness metrics.
Support data remediation activities.
Assist business users in investigating data issues.
Maintain governance and compliance standards.

Documentation & Reporting

Maintain support runbooks and operational procedures.
Prepare weekly status reports and incident summaries.
Document recurring issues and knowledge articles.
Participate in customer governance and support review meetings.

Required Technical Skills

Skill Area	Required Expertise
Azure Databricks	PySpark, Notebooks, Workflows, Clusters
Python	Core Python, Data Processing, Automation
Azure Data Factory	Pipelines, Triggers, Datasets, Linked Services
SQL	Complex Queries, Performance Tuning
Azure Data Lake	ADLS Gen2, Data Management
Data Engineering	ETL/ELT Processing
Monitoring	Job Monitoring & Alerting
Source Control	Git / Azure DevOps
Incident Management	RCA, Ticket Resolution
Data Quality	Validation & Reconciliation

Preferred Skills

Unity Catalog and Data Governance concepts.
Lakehouse Architecture.
Delta Tables and Delta Lake.
Azure Synapse Analytics.

Senior Engineer (databricks and ADF)