• Create and maintain optimal Big data pipeline architecture; assemble large, complex data sets that meet functional / non-functional requirements

• Design and build production data pipelines from ingestion to consumption within a big data architecture

• Build the necessary datamarts, data warehouse required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.

• Create necessary preprocessing and postprocessing for various forms of data for training/ retraining and inference ingestions as required

• Good Knowledge of Data Modeling, Data Architecture and Enterprise Data Warehouse concepts

• Good knowledge of ETL Tools

• Identify, design, and implement internal process improvements: automating manual data processes, optimizing data delivery, etc.

Requirements and Skills

• You should have a bachelors or master s degree in computer science, Information Technology or other quantitative fields

• You should have 5-10 years working as a Big data engineer / Data Architect in supporting large Big data transformation initiatives related to machine learning, with experience in building and optimizing big data pipelines and data sets

• Strong analytic skills related to working with unstructured datasets.

• Experience with big data tools: Hadoop, Spark, Kafka, etc.

• Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.

• Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.

• Experience with AWS cloud services: EC2, EMR, RDS, Redshift and familiarity with various log formats from AWS.

• Experience with stream-processing systems: Storm, Spark-Streaming, etc.

• Experience with object-oriented/object function scripting languages: Python, Java, C++, etc.

• Experience with ETL