Senior Data Engineer – AWS Data Lake & Pipeline Architecture

10 requirements, read out of the advert rather than guessed from the job title:

• AWS
• SQL
• Spark
• Glue
• EMR
• Apache Kafka
• AWS Kinesis
• Java
• Python
• Scala

Also mentioned, not required: Git, Terraform, CI/CD. Worth having, but their absence is not what gets a CV filtered out.

See how often each of these is required across open data roles in Europe.

Headquarters:

Summary We are seeking a Data Engineer to support the development of a Data Intelligence Platform. This role focuses on data modeling, data services, data pipelines, and cloud-based data infrastructure for reporting, analytics, and data science. General information This Data Engineer will support data-related operations across a broader Data Intelligence Platform team. The work includes building and consuming web services, integrating search technologies, supporting personalization and recommendation engines, and applying data engineering best practices across the platform. The role spans cloud-based data architecture, database performance, pipeline automation, and support for data scientists, researchers, and internal business units. The environment includes AWS-based infrastructure and a mix of structured and unstructured data sources. Tasks and Deliverables - Participate in the architecture design and implementation of high-performance, scalable, and optimized data solutions. - Create data models from scratch using strong SQL fundamentals. - Write and optimize in-application SQL statements. - Ensure the performance, security, and availability of databases. - Prepare documentation and technical specifications. - Handle database procedures such as upgrades, backups, recovery, and migration. - Profile server resource usage and optimize configurations as necessary. - Design, build, and automate the deployment of data pipelines and applications. - Integrate data from on-premise databases and external data sources using REST APIs and harvesting tools. - Collaborate with business units and data science teams on data access, transformation, processing, and reporting needs. - Support implementation, technical issues, and training related to the data lake ecosystem. - Work with the team to manage AWS resources, including EMR and ECS clusters. - Support provisioning, monitoring, configuration, and maintenance of AWS tools. - Evaluate and promote new cloud technologies that improve capabilities and lower operating costs. - Support automation efforts using Infrastructure as Code with Terraform and CI/CD tools such as Jenkins. - Work with the team to implement data governance, access control, and security risk reduction. Required experience - 7-9 years of experience designing and developing cloud-based data models, ETL pipelines, and infrastructure. - Experience working with both structured and unstructured data. - Strong proficiency with SQL across popular databases. - Experience optimizing large, complex SQL statements. - Knowledge of best practices for relational databases. - Experience configuring database engines and orchestrating clusters. - Ability to plan resource requirements from high-level specifications. - Ability to troubleshoot common database issues. - Experience with Spark, Glue, EMR, and Apache Kafka or AWS Kinesis. - Experience with version control tools such as Git or Subversion. - Experience using automated build systems and CI/CD workflows. - Experience programming in Java, Python, and Scala. - Knowledge of data structures and algorithms. - Knowledge of relational, NoSQL, graph, document, key-value, and time-series databases. - Knowledge of scalable data model design and management. - Knowledge of ML model deployment. - Knowledge of AWS cloud platforms. - Knowledge of TDD and BDD. - Strong interest in improving software development skills, frameworks, and technologies. Engagement highlights - Opportunity to work across data modeling, data services, and data science within a broader Data Intelligence Platform. - Exposure to a varied technical environment spanning AWS, ETL pipelines, databases, search technologies, and recommendation systems. - Direct collaboration with data science teams and internal stakeholders on reporting, transformation, and platform capabilities.

To apply: https://weworkremotely.com/remote-jobs/toptal-senior-data-engineer-aws-data-lake-pipeline-architecture