Data Engineering on Microsoft Azure

DP-203: Data Engineering on Microsoft Azure, you help stakeholders understand the data through exploration, and build and maintain secure and compliant data processing pipelines by using different tools and techniques. You use various Azure data services and frameworks to store and produce cleansed and enhanced datasets for analysis. This data store can be designed with different architecture patterns based on business requirements, including: Modern data warehouse (MDW), Big data, Lakehouse architecture.

As an Azure data engineer you also help to ensure that the operationalization of data pipelines and data stores are high-performing, efficient, organized, and reliable, given a set of business requirements and constraints. You help to identify and troubleshoot operational and data quality issues. You also design, implement, monitor, and optimize data platforms to meet the data pipelines.

1. Implement a partition strategy

Implement a partition strategy for files
Implement a partition strategy for analytical workloads
Implement a partition strategy for streaming workloads
Implement a partition strategy for Azure Synapse Analytics
Identify when partitioning is needed in Azure Data Lake Storage Gen2

2. Design and implement the data exploration layer

Create and execute queries by using a compute solution that leverages SQL serverless and Spark cluster
Recommend and implement Azure Synapse Analytics database templates
Push new or updated data lineage to Microsoft Purview
Browse and search metadata in Microsoft Purview Data Catalog

3. Ingest and Transform data

Design and implement incremental loads
Transform data by using Apache Spark
Transform data by using Transact-SQL (T-SQL) in Azure Synapse Analytics
Ingest and transform data by using Azure Synapse Pipelines or Azure Data Factory
Transform data by using Azure Stream Analytics
Cleanse data
Handle duplicate data
Avoiding duplicate data by using Azure Stream Analytics Exactly Once Delivery
Handle missing data
Handle late-arriving data
Split data
Shred JSON
Encode and decode data
Configure error handling for a transformation
Normalize and denormalize data
Perform data exploratory analysis

4. Develop a batch Processing Solution

Develop batch processing solutions by using Azure Data Lake Storage, Azure Databricks, Azure Synapse Analytics, and Azure Data Factory
Use PolyBase to load data to a SQL pool
Implement Azure Synapse Link and query the replicated data
Create data pipelines
Scale resources
Configure the batch size
Create tests for data pipelines
Integrate Jupyter or Python notebooks into a data pipeline
Upsert data
Revert data to a previous state
Configure exception handling
Configure batch retention
Read from and write to a delta lake

5. Develop a Stream Processing Solution

Create a stream processing solution by using Stream Analytics and Azure Event Hubs
Process data by using Spark structured streaming
Create windowed aggregates
Handle schema drift
Process time series data
Process data across partitions
Process within one partition
Configure checkpoints and watermarking during processing
Scale resources
Create tests for data pipelines
Optimize pipelines for analytical or transactional purposes
Handle interruptions
Configure exception handling
Upsert data
Replay archived stream data

6. Manage batches and pipelines

Trigger batches
Handle failed batch loads
Validate batch loads
Manage data pipelines in Azure Data Factory or Azure Synapse Pipelines
Schedule data pipelines in Data Factory or Azure Synapse Pipelines
Implement version control for pipeline artifacts
Manage Spark jobs in a pipeline

7. Implement Data Security

Implement data masking
Encrypt data at rest and in motion
Implement row-level and column-level security
Implement Azure role-based access control (RBAC)
Implement POSIX-like access control lists (ACLs) for Data Lake Storage Gen2
Implement a data retention policy
Implement secure endpoints (private and public)
Implement resource tokens in Azure Databricks
Load a DataFrame with sensitive information
Write encrypted data to tables or Parquet files
Manage sensitive information

8. Monitor data storage and data processing

Implement logging used by Azure Monitor
Configure monitoring services
Monitor stream processing
Measure performance of data movement
Monitor and update statistics about data across a system
Monitor data pipeline performance
Measure query performance
Schedule and monitor pipeline tests
Interpret Azure Monitor metrics and logs
Implement a pipeline alert strategy

9. Optimize and troubleshoot data storage and data processing

Compact small files
Handle skew in data
Handle data spill
Optimize resource management
Tune queries by using indexers
Tune queries by using cache
Troubleshoot a failed Spark job
Troubleshoot a failed pipeline run, including activities executed in external services

Fees Structure :	12500 INR / 152 USD
Total No of Class :	90 Video Class
Class Duration :	42:30 Working Hours
Download Feature :	Download Avalable
Technical Support :	Call / Whatsapp : +91 8680961847
Working Hours :	Monday to Firday 9 AM to 6 PM
Payment Mode :	Credit Card / Debit Card / NetBanking / Wallet (Gpay/Phonepay/Paytm/WhatsApp Pay)

Brochure Buy Now Sample Demo

Fees Structure :	30000 INR / 365 USD
Class Duration :	60 Days
Class Recording :	Live Class Recording available
Class Time :	Monday to Firday 1.5 hours per day / Weekend 3 Hours per day
Technical Support :	Call / Whatsapp : +91 8680961847
Working Hours :	Monday to Firday 9 AM to 6 PM
Payment Mode :	Credit Card / Debit Card / NetBanking / Wallet (Gpay/Phonepay/Paytm/WhatsApp Pay)

Download Brochure Pay Online