Project Scope:

Data science project. Data pipeline, multiple database system, non/un-structure data. Technical, Databricks, Python, Pysparks, Jenkins, Github, SQL, Data Warehousing concepts

Requirements:

• 7 to 9 years working experience in data integration and pipeline development with data warehousing .

• Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems

• Strong real-life experience in python development especially in pySpark in AWS Cloud environment.

• Design, develop test, deploy, maintain and improve data integration pipeline.

• Experience in Python and common python libraries.

• Strong analytical experience with database in writing complex queries, query optimization, debugging, user defined functions, views, indexes etc.

• Strong experience with source control systems such as Git, Bitbucket, and Jenkins build and continuous integration tools.

• Experience with continuous deployment(CI/CD)

• Databricks, Airflow and Apache Spark Experience is a plus.

• Experience with databases (Oracle, SQL Server, PostgreSQL, Redshift, MySQL, or similar)

• Strong experience with performance tuning, analytical understanding with business and program.

• Exposure to ETL tools including Informatica and any other .

• BS/MS degree in CS, CE or EE.