Data Engineer

Overview

Our client is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines, ensuring efficient data integration and optimizing data storage solutions. This role involves collaborating with data scientists, analysts, and software engineers to build robust data architectures that support business intelligence and analytics initiatives. The ideal candidate thrives in a fast-paced, high-growth environment and enjoys working with cutting-edge technologies.

Challenges you would face

Develop, test, and maintain data architectures, including databases, data lakes, and data warehouses
Design and implement scalable and reliable ETL/ELT pipelines to ingest, transform, and load data from various sources
Optimize and improve data processing workflows for performance, scalability, and cost-effectiveness
Ensure data integrity, consistency, and security across all data platforms
Collaborate with cross-functional teams to understand data needs and provide efficient solutions
Provide actionable insights for improving the performance of relational databases
Monitor and troubleshoot data pipelines, ensuring timely resolution of issues
Implement best practices for data governance, metadata management, and documentation
Work with cloud-based data platforms (AWS, GCP, Azure) and leverage services such as S3, Redshift, BigQuery, Snowflake, and Databricks

The ideal candidate

Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field
5+ years of experience in data engineering or a similar role
Proficiency in SQL and experience working with relational databases such as PostgreSQL, MySQL, or SQL Server
Strong experience with Databricks
Hands-on experience with ETL/ELT tools like Apache Airflow
Proficiency in Python
Experience with cloud platforms such as AWS, GCP, or Azure
Knowledge of data modeling, warehousing concepts, and best practices
Experience integrating data solutions into REST APIs
Understanding of CI/CD pipelines and DevOps practices for data infrastructure
Experience with Snowflake or similar cloud-based data warehouse technologies

Plus

Experience with NoSQL databases like DynamoDB or MongoDB
Familiarity with data streaming technologies such as Apache Kafka or AWS Kinesis
Experience working with containerized applications using Docker or Kubernetes
Knowledge of machine learning model deployment and MLOps concepts
Experience with machine learning