Key Responsibilities:

· Design and implement real-time data streaming pipelines using Kafka or similar technologies

· Work with cross-functional teams to integrate streaming pipelines into existing systems

· Optimize data streaming pipelines for performance and reliability

· Build and maintain monitoring and alerting systems to ensure data quality and system health

· Collaborate with data scientists, data analysts, and other stakeholders to understand business requirements and design data streaming solutions to meet those requirements

· Conduct thorough testing to ensure the quality of the implemented solutions

· Maintain technical documentation, including design specifications, installation instructions, and system configurations

Requirements:

· Bachelor's or Master's degree in Computer Science, Software Engineering, or related field

· At least 3 years of experience with Kafka or similar data streaming technologies

· Strong software engineering skills, including proficiency in Java or other programming languages

· Experience with stream processing frameworks such as Apache Flink, Spark Streaming, or Storm

· Knowledge of distributed systems, microservices architecture, and containerization (e.g. Docker)

· Familiarity with cloud platforms such as AWS, Google Cloud, or Microsoft Azure

· Strong analytical and problem-solving skills

· Excellent communication and collaboration skills

· Experience with Kubernetes or other container orchestration platforms

· Knowledge of data modeling and database technologies (e.g. SQL, NoSQL, Hadoop)

· Familiarity with machine learning and data analytics frameworks (e.g. TensorFlow, Keras, PyTorch)

· Experience with other messaging systems such as RabbitMQ or ActiveMQ

Good to have: Experience of working with Databricks or knowing how to get Kafka / Denodo connected to Databricks / Delta Lake.

Data Streaming Engineer