Job Location: Mumbai
Job Description
We are looking for a Spark developer who knows how to fully exploit the potential of Spark cluster.
You will be responsible to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to feature developers and business analysts.
This involves both ad-hoc requests as well as data pipelines that are embedded in production environment.
Responsibilities
- Create Scala/Spark jobs for data transformation and aggregation
- Produce unit tests for Spark transformations and helper methods
- Write Scaladoc-style documentation with all code
- Design data processing pipelines
Skills
- Scala (with a focus on the functional programming paradigm)
- Scalatest, JUnit, Mockito
- Apache Spark 2.x
- Spark query tuning and performance optimization
- Experience working with Hive, SQL and HDFS
- Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)
- Good to have experience in Java
- Experience working with tools like Jenkins, Jira, Bitbucket and Git
- Experience in writing shell scripts and working with Linux platforms.
Job Segment: Database, Engineer, Developer, Java, SQL, Technology, Engineering
Larsen & Toubro Infotech Limited | Hiring | Senior Data Engineer | BigDataKB.com | 1/12/2022
Submit CV To All Data Science Job Consultants Across India For Free
