Data Engineer with using Apache Spark, Python, Scala.
Location: Melbourne
Position Type: Contract

Data Engineer
As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.
Key Responsibilities:

Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.
Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.
Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.
Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes.
Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.
Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.
Implement and adhere to best practices for data governance, security, and compliance.
Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for dataworkflows.
Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.

Requirements:

Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.
Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.
Experience with performance tuning and optimization of Spark jobs and SQL queries.
Familiarity with stream processing frameworks like Kafka and messaging systems.
Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills, with the ability to work effectively in a team environment

When you apply, you voluntarily consent to the disclosure, collection and use of your personal data for employment/recruitment and related purposes in accordance with the Tech Aalto Privacy Policy, a copy of which is published at Tech Aalto’s website (https://www.techaalto.com/privacy/)
Confidentiality is assured, and only shortlisted candidates will be notified for interviews.

Data Engineer with using Apache Spark, ...