Project Description

We are looking for an experienced Full Stack Data Engineer to join our team for building a next-generation data platform built on Data Mesh architecture/principles. The ideal candidate should have extensive hands-on experience in building a big data platform, Big Data Technologies, Data Pipelines, backend development using Python, BI/Analytics tools as well as experience with DevOps, AWS, and UI Development in Angular JS. You will be responsible for designing, developing, and maintaining our web applications and data pipelines, as well as implementing CI/CD best practices.

Responsibilities

• Design, build, and maintain scalable and efficient data platforms using data engineering technologies such as Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Hive, HDFS, and Trino.
• Build/manage data pipelines, and common data-related cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management
• Develop and maintain web applications using AngularJS and Python.
• Build/manage BI/Analytical dashboard reducing time to insight for the business stakeholders.
• Implement CI/CD pipelines using Terraform, Jenkins, Github actions, and Gitflow.
• Collaborate with cross-functional teams to develop and implement new features.
• Write clean, reusable, and efficient code.
• Participate in code reviews and ensure code quality.
• Develop and maintain APIs using Python and ensure API security and best practices are implemented.
• Implement SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.
• Implement integration with AWS Cognito for user authentication and authorization.
• Ensure the application is optimized for maximum speed and scalability.
• Troubleshoot and debug issues as they arise.
• Implement DevOps best practices to ensure efficient application deployment and management.
• Collaborate with data scientists and analysts to integrate data analytics solutions with web applications.
• Stay up to date with emerging trends and technologies.

Skills

Must have

• 8+ years of experience in similar positions;
• Hands-on Experience with data engineering technologies such as AWS Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Apache Hive, Apache Airflow, S3FS, Apache Hudi, and Trino.
• Extensive experience in building data pipelines using orchestration tools like Apache Airflow. Hands-on experience in building cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management
• Proven experience as a Full Stack Developer with AngularJS and Python.
• Strong understanding of web development technologies including HTML, CSS, and JavaScript.
• Experience working with RESTful APIs and JSON. Familiarity with microservices architecture.
• Experience with core AWS technologies such as EC2, ELB, Auto Scaling, S3, EFS, Lambda, API
Gateway, Step Functions, Cloudwatch, VPC, Route 53, ACM
• Hands on experience with SQL and NoSQL databases.
• Hands experience with BI tools like Tableau, AWS QuickSight
• Experience with Git or other version control systems.
• Understanding of agile development methodologies.
• Strong problem-solving skills.
• Excellent written and verbal communication skills.
• Ability to work independently and collaboratively in a team environment.
• Experience with cloud platforms such as AWS.
• Bachelor's degree in Computer Science, Engineering, or related field

Nice to have

• Agile, Scrum framework 2+ years' experience on past projects

• Knowledge of containerization technologies like Docker or Kubernetes.
• Experience with front-end frameworks like React or Vue.js.
• Experience with DevOps and CI/CD best practices.
• Experience with Terraform and the AWS provider.
• Experience with API development, security, and best practices.
• Experience with SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.
• Experience with integration with AWS Cognito for user authentication and authorization