Data Engineer – Spark and Scala (Big Data)

Description

Job Summary: We are seeking a Big Data Analyst to design and build large-scale data processing solutions, with a focus on high-availability architecture and real-time processing. Key Highlights: 1. Design and development of ETL/ELT pipelines in Big Data environments. 2. Implementation of distributed processing with Spark and Scala (critical). 3. Collaboration in the design of scalable architectures. A leading company is looking for a Big Data Analyst responsible for designing and building large-scale data processing solutions, with a focus on high-availability architecture and real-time processing. Process managed by Kibernum – direct hiring by client Key Responsibilities: • Design and develop ETL/ELT pipelines in Big Data environments (Cloudera and cloud). • Implement distributed processing using Spark and Scala (critical). • Optimize SQL database queries and structures (Oracle and PostgreSQL). • Develop real-time data streams using Kafka. • Automate processes with Unix/Shell scripting. • Manage version control and deployments using Git and CI/CD. • Collaborate on the design of scalable architectures (APIs and microservices). • Ensure quality, security, and performance standards. Requirements: Mandatory • Professional engineering degree (minimum 8 semesters). • At least 3 years of experience in Big Data. • Solid experience with Spark and Scala. • Proficiency in SQL. • Experience with Oracle and PostgreSQL. Desirable • Big Data ecosystem: Hive, Impala, Kudu, Kafka. • Experience with Cloudera and cloud environments (basic level). • DevOps tools: Git, Maven, CI/CD. • Unix/Linux scripting (Shell). • Development in Java, Python, Spring Boot, microservices. Conditions: • Contract: Permanent (direct with client) • Work mode: Hybrid 3x2 – Santiago Center • Working hours: Monday–Thursday 09:00–18:00 | Friday until 17:00 About the Challenge: You will be part of building a robust architecture capable of integrating and processing large volumes of data in real time, transforming information into strategic business assets.-Requirements- Minimum education: University / Technical Institute / Vocational Training Center. 3 years of experience Keywords: data, data, engineer, engineers, female engineer, eng, engineer