Job Location: Mumbai
Job Description
Role: Data Engineer – Big Data
Experience Level: 4 to 6 Years
Work location: India
What youll do:
As a Big Data Engineer, you will build utilities that would help design and develop ETL pipelines. You would build
data parsing scripts and pipelines that serve several BI reports and adhoc queries per day. The services you
build will integrate directly with cloud services, opening the door to new and cutting-edge re-usable solutions.
You will work with engineering teams, co workers, and customers to gain new insights and dream of new
possibilities.
Role & Responsibilities:
Work with cloud engineers and clients to solve big data problems by developing utilities for migration,
storage and processing on AWS Cloud
Understand requirements and contribute to planning the development
Design and build ETL pipelines
Build tools to ingest and jobs to process several terabytes per day
Design and develop next-gen storage and compute solutions for several large customers
Build advanced tooling for automation, testing, monitoring, administration, and data operations across
multiple cloud clusters
Use Continuous Integration and Continuous deployment pipeline for deployment of code to higher
environment
Communicate with a wide set of teams, including Infrastructure, DevOps and cloud customers
Required Skills:
3+ years experience of Hands-on in data structures, spark, SQL and NoSQL Databases
Hands-on experience in AWS Cloud with services such as AWS Glue, AWS Redshift, EC2, Lambda, S3
Strong software development skills in Python and Pyspark
Strong software development skills in azure, azure data factory, azure data bricks.
Experience building and deploying cloud-based solutions at scale
Experience in developing Big Data solutions (migration, storage, processing)
BS, MS or PhD degree in Computer Science or Engineering, and 4+ years of relevant work experience
in Big Data and cloud systems
Experience in SQL and Query optimisation
Ability to clearly communicate technical roadmap, challenges and mitigation
Experience building and supporting large-scale systems in a production environment
Technology Stack:
Cloud Platforms AWS
Mandatory High programming skill in Python and Pyspark, Hands-on experience with the AWS
services such as AWS Glue, EC2, Lambda, S3
Nice to have – Experience in Big Data Technologies such as Hive, Spark, Lambda, AWS Cloud with focus
on AWS Glue to create ETL pipelines
Design Dimensional Models for Data Warehouse and Data Marts
Knowledge of version control systems, build automation and CI/CD tools and frameworks
Submit CV To All Data Science Job Consultants Across India For Free
