Data Engineer

Experience: 5 - 8 Years

KEY ACCOUNTABILITY

• The Azure data engineer focuses on data-related tasks in Azure.
• Primary responsibilities include using services and tools to ingest, egress, and transform data from multiple sources.
• Azure data engineers collaborate with business stakeholders to identify and meet data requirements.
• Design and implement solutions. Manage, monitor, and ensure the security and privacy of data to satisfy business needs.
• Design and implement data ingestion techniques for real time and batch processes for structured and unstructured data sources.
• Apply cutting-edge software design and cloud migration expertise.
• Responsible for design, development, and architecture of Microsoft Azure Cloud solutions.
• Experience with architecting and designing technical solutions using combination of Azure IaaS and PaaS 

JOB CONTEXT

• Responsible for implementing robust data pipeline using Microsoft Stack.
• Responsible for creating reusable and scalable data pipelines.
• Responsible for development and deployment of new data platforms.
• Experience with Cloud Platforms such as Azure, AWS , Google Cloud etc.
• Experience on Migrating SQL database to Azure data Lake, Azure SQL Database, Azure Synapse, Data Bricks and Talend.
• Experience in Deploying Distributed Messaging Systems using KAFKA, or other messaging System such Azure IoT / Event Hub, or CDC mechanisms like Striim or HVR.
• Experience building rest API pipeline, and parsing complex nested json into relational format
• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analysing & transforming the data to uncover insights into the customer usage patterns.
• Good understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, Driver Node, Worker Node, Stages, Executors and Tasks.
• Good understanding of Big Data Hadoop and Yarn architecture along with various Hadoop Demons such as Job Tracker, Task Tracker, Name Node, Data Node, Resource/Cluster Manager, and Kafka (distributed stream-processing) .
• Experience in Database Design and development with Business Intelligence using SQL Server 2014/2016, Integration Services (SSIS), DTS Packages.
• Expertise in various phases of project life cycles (Design, Analysis, Implementation and testing).
• Analyse, design and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes.
• Create Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Primary responsibilities include using services and tools to ingest, egress, and transform data from multiple sources.
• Create Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.

Skills:

• Azure Data lake, Data factory
• Azure Databricks, Azure SQL database, Azure Synapse Database.
• Hadoop ecosystem.
• Programming Scala, Python, R, Spark SQL
• KAFKA, Azure MQ etc.
• MSBI (SSIS, SSAS, SSRS), Oracle, SQL, PL/SQL
• Data Migration
• SQL Server programming
• Analytic Problem-Solving

Qualification: 

• Bachelor/master’s in computer science/IT or equivalent.
• Azure certifications will be added advantage (Certification in Az-900 and/or AZ-204, AZ-303, AZ-304 or AZ-400).