Major Challenges -

Integrating with an established, complex Multitenant Hadoop based project working to tight deadline
Refactoring the current technology stack and architecture from on premise Hadoop, to Google Cloud Platform
Working with globally dispersed and diversifies team.
Supporting specific source on-boarding activities in line with project delivery timelines

Skills needed for this role – General

Communication skills (data). You know how to communicate to and between technical and non-technical stakeholders as well as facilitate discussions within a multidisciplinary team, with potentially difficult dynamics. You can advocate for the team externally. You know how to manage different perspectives.
Data analysis and synthesis. You know how to undertake data profiling and source system analysis and can present clear insights to colleagues to support the end use of the data
Data development process. You can design, build and test data products that are complex or large-scale. You know how to build teams to complete data integration services
Data innovation. You understand the impact of emerging trends on the organisation in data tools, analysis techniques and data usage.
Data integration design. You know how to select and implement the appropriate technologies to deliver resilient, scalable and future-proofed data solutions.
Data modelling. You understand the concepts and principles of data modelling and can produce relevant data models across multiple subject areas. You know how to reverse-engineer data models from a live system. You understand industry-recognised data modelling patterns and standards and when to apply them. You can compare and align different data models
Metadata management. You can design an appropriate metadata repository and present changes to existing metadata repositories. You understand a range of tools for storing and working with metadata. You can provide oversight and advice to more inexperienced members of the team
Problem resolution (data). You know how to respond to problems in databases, data processes, data products and services as they occur. You can initiate actions, monitor services and identify trends to resolve problems. You can determine the appropriate remedy and assist with implementation of it as well as preventative measures.
Programming and build (data engineering). You know how to use agreed standards and tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations. You can collaborate with others to review specifications where appropriate.
Technical understanding (data engineering). You understand core technical concepts related to the role and can apply them with guidance.
Testing. You can review requirements, specifications and define test conditions. You can identify issues and risks associated with work while being able to analyse and report test activities and results.

Specialised Skills needed for this role

Data pipeline building technologies.
To include Hadoop, pyspark, scala, Hive, Java
Knowledge and experience of Hadoop eco-system and data management frameworks
Knowledge of HSBC data governance framework.
CI/ CD experience – nice to have.
Optionally, knowledge and experience of BigQuery, DataProc, Python

| | Competencies

Google Technologies and Big Data
Forward thinking, independent, creative, and self-sufficient; who can work with less documentation, has exposure testing complex multi-tiered integrated applications. Ability to work with minimal supervision on own initiative and on multiple tasks simultaneously
Excellent communication, interpersonal, and decision making skills
Strong team-working skills, working in global teams across multiple time zones
Good knowledge on Data warehouse concepts and
Basic knowledge on scheduling tools (Airflow)
Knowledge on Software Development Life Cycle (SDLC), and Methodologies like DevOps, Agile, Scrum, Waterfall, and Iterative process
Willingness to learn and quick to adapt to changing requirements
Identify project issues, communicate them and assist in their resolution.

Assist in continuous improvement efforts in enhancing project team methodology and performance.

Cooperative team focused attitude.
Needs to be a Self-starter, proactive.

Experience

Professional software development experience within data environment/s
Programming experience in Java, Scala, and Spark.
Proficient in SQL and relational database design.
Agile and DevOps experience
Project planning.
Google Cloud Platform, or other cloud vendor.

Data Engineer