Data Engineer – Spark and Scala (Big Data)

Description

Job Summary: We are seeking a Big Data Analyst to design and build large-scale data processing solutions, with a focus on high-availability and real-time architecture. Key Highlights: 1. Design and development of ETL/ELT pipelines in Big Data environments. 2. Implementation of distributed processing with Spark and Scala. 3. Collaboration in the design of scalable architectures. A prominent company is looking for a Big Data Analyst responsible for designing and building large-scale data processing solutions, with a focus on high-availability architecture and real-time processing. Process managed by Kibernum – direct hiring by client Key Responsibilities: • Design and develop ETL/ELT pipelines in Big Data environments (Cloudera and cloud). • Implement distributed processing using Spark and Scala (critical). • Optimize SQL database queries and structures (Oracle and PostgreSQL). • Develop real-time data streams using Kafka. • Automate processes with Unix/Shell scripting. • Manage versioning and deployments with Git and CI/CD. • Collaborate in designing scalable architectures (APIs and microservices). • Ensure quality, security, and performance standards. Requirements: Mandatory • Professional engineering degree (minimum 8 semesters). • At least 3 years of experience in Big Data. • Solid experience with Spark and Scala. • Proficiency in SQL. • Experience with Oracle and PostgreSQL. Desirable • Big Data ecosystem: Hive, Impala, Kudu, Kafka. • Experience with Cloudera and cloud environments (basic level). • DevOps tools: Git, Maven, CI/CD. • Unix/Linux scripting (Shell). • Development in Java, Python, Spring Boot, microservices. Conditions: • Contract type: Indefinite-term (direct with client) • Work mode: Hybrid 3x2 – Santiago Center • Working hours: Monday–Thursday 09:00–18:00 | Friday until 17:00 About the Challenge: You will be part of building a robust architecture capable of integrating and processing large volumes of data in real time, transforming information into strategic business assets.-Requirements- Minimum education: University / Professional Institute / Technical Training Center. 3 years of experience Keywords: data, data, engineer, engineers, female engineer, eng, engineer