Detailed JD

AWS Data Engineer – Data Pipeline & Modelling

Roles and Responsibilities

Design, develop, and maintain scalable data pipelines using AWS Glue (Spark, Python) and Delta Lake for efficient ETL processing.
Model and transform data across different layers (raw, staging, processed) to support analytics and reporting requirements.
Orchestrate complex data workflows using AWS Step Functions, coordinating Glue jobs and Lambda functions.
Manage data storage solutions, primarily using Amazon S3 and relational databases such as Oracle.
Expose structured and curated data for analytics and reporting via AWS Athena.
Implement and maintain CI/CD processes for ETL jobs and data pipeline deployments using GitHub.
Enable and support data access for business intelligence tools (e.g., Power BI).
Collaborate with cross-functional teams to understand business data needs and deliver effective solutions.
Operate within secure development environments (e.g., Laptop/VDI) to access required data and toolsets.

Skills & Experience

Demonstrated experience building data pipelines on AWS (Glue, S3, Athena, Step Functions).
Strong background in ETL development with PySpark and Python.
Experience with Delta Lake or similar data lake architectures.
Solid understanding of data modelling, data transformation, and performance optimization.
Hands-on experience with version control (GitHub) and CI/CD for data engineering workflows.
Familiarity with business intelligence/reporting tools integration (e.g., Power BI).
Experience working in secure or remote development environments.

Skills
Must Have

Nice to Have

Experience with AWS Lambda for serverless transformation tasks
Exposure to Oracle or other relational databases
Experience with Power BI or similar reporting/visualization tools
Familiarity with customer service or CRM data sources especially Salesforce & IVR
Knowledge of secure workspace access (Laptop/VDI environments)

AWS Data Engineer - Data Pipeline & Modelling