Dell Data Engineering Optimize D-DS-OP-23

Optimize

Master advanced data engineering skills — warehousing with SQL and NoSQL, ETL/ELT with Hadoop and Spark, data governance, real-time IoT stream processing, and building data pipelines with Python and Apache Airflow.

Apache Spark Apache Kafka Apache Airflow ETL / ELT Data Warehousing NoSQL Data Governance IoT Streaming Python Pipelines Apache Flink

What You Will Learn

Six Data Engineering Modules

Master the end-to-end data engineering stack — from warehousing and ETL through real-time streaming, governance, and building production Python pipelines.

Module 01

The Role of the Data Engineer

›Skills of a data engineer — programming (Python/Scala/SQL), data modelling, and pipeline design
›Role in an analytics project — bridging raw data sources and analysis-ready datasets
›Data engineer vs data scientist vs data analyst — responsibilities and collaboration model
›Career paths — analytics engineer, platform engineer, ML engineer, and data architect

Module 02

Data Warehousing with SQL and NoSQL

›Relational database characteristics — ACID properties, indexing strategies, and query performance
›Relational database schemas — star schema, snowflake schema, and normalisation techniques
›NoSQL database types — key-value (Redis), document (MongoDB), column-family (HBase/Cassandra), graph (Neo4j)
›Use case selection — when to use NoSQL vs RDBMS based on access patterns and scalability needs

Module 03

ETL Offload with Hadoop and Spark

›ETL vs ELT — transformation-first vs load-first approaches and scheduling tools (Oozie, Airflow)
›Hadoop ecosystem for ETL — HDFS ingestion with Flume and Sqoop, Hive for transformations
›Apache Spark architecture — Driver, Executors, RDDs, DataFrames, Datasets, and lazy evaluation
›Spark performance tuning — partitioning, caching, broadcast joins, and Spark UI profiling

Module 04

Data Governance, Security, and Privacy

›Data governance definition — roles (data owner, steward, custodian) and governance models (federated, centralised)
›Metadata and Master Data Management (MDM) — data catalogue, lineage tracking, and golden records
›Security considerations with Hadoop — Kerberos authentication, wire encryption, and perimeter security
›Apache Atlas, Ranger, and Knox — metadata governance, policy-based access control, and API gateway
›Privacy regulations — GDPR, CCPA, and HIPAA — data minimisation, consent, and right to erasure

Module 05

Processing Streaming and IoT Data

›IoT tools and applications — MQTT, device management, and edge-to-cloud data flows
›Apache Storm — topology design, spouts, bolts, and stream processing guarantees
›Apache Kafka — producers, consumers, consumer groups, partitions, and exactly-once semantics
›Spark Streaming — DStreams and Structured Streaming micro-batch processing architecture
›Apache Flink — event time processing, watermarks, stateful stream processing, and savepoints
›Pravega — segmented stream storage architecture and EdgeX Foundry for IoT edge computing

Module 06

Building Data Pipelines with Python

›Python for data engineering — why Python over Java/Scala for pipeline scripting and ecosystem breadth
›Data structures — lists, dictionaries, tuples, sets, and string manipulation for data transformation
›Python libraries — pandas for DataFrame manipulation, SQLAlchemy for database connections, requests for APIs
›Apache Airflow — DAGs, operators (PythonOperator, BashOperator, SQLOperator), sensors, and XComs
›Pipeline best practices — idempotency, retry logic, dead letter queues, observability, and testing

Full Curriculum

6-Module D-DS-OP-23 Programme

Complete data engineering curriculum from data warehousing and ETL through streaming, governance, and production Python pipeline orchestration with Apache Airflow.

›Data engineer skills — Python/Scala/SQL programming, data modelling, pipeline design, and cloud platform knowledge
›Role in analytics projects — raw data ingestion, transformation, quality, and serving to consumers
›Collaboration with data scientists — providing cleaned feature-engineered datasets for model training
›Collaboration with data analysts — building reliable aggregation layers and semantic models
›Career path — analytics engineer, data platform engineer, MLOps engineer, and data architect trajectories

›Relational DB performance — indexing strategies (B-tree, hash, covering), query plans, and partition pruning
›Schema design — 3NF normalisation for OLTP, star schema and snowflake schema for OLAP workloads
›NoSQL paradigms — key-value stores (Redis caching), document stores (MongoDB flexible schema), column-family (HBase/Cassandra wide rows), graph databases (Neo4j traversal)
›Polyglot persistence — selecting the right store for each access pattern in a microservices architecture
›Data lake vs data warehouse vs data lakehouse — medallion architecture (bronze/silver/gold layers)

›ETL vs ELT trade-offs — when to transform in the pipeline vs in the warehouse (ELT with Spark/Snowflake)
›Hadoop ingestion tools — Sqoop for RDBMS bulk import, Flume for log streaming, and Kafka Connect for CDC
›HDFS deep dive — block sizing, replication factor, rack-aware placement, and NameNode HA configuration
›Apache Spark architecture — SparkContext, DAG scheduler, stage/task breakdown, and shuffle operations
›Spark optimisation — narrow vs wide transformations, partition tuning, broadcast joins, AQE, and persist/cache strategies

›Data governance programme — policy definition, data stewardship, ownership assignment, and accountability matrix
›Metadata management — business vs technical metadata, Apache Atlas entity types, and data lineage graphs
›Master Data Management (MDM) — golden record creation, deduplication, and master data hub patterns
›Hadoop security — Kerberos TGT/TGS flow, HDFS permission model, and wire encryption with TLS/SSL
›Apache Ranger — resource-level and row/column-level access policies, audit logging, and dynamic data masking
›Apache Knox — REST API gateway, topology-based routing, and SSO integration for Hadoop services
›Privacy regulations — GDPR data subject rights, CCPA opt-out flows, and HIPAA minimum necessary principle

›IoT data flows — edge sensor → gateway → broker → processing → storage → analytics architecture
›Apache Kafka deep dive — topic partitioning, offset management, consumer lag monitoring, and Kafka Streams
›Apache Storm — spout/bolt topology, stream groupings (shuffle, field, all), and at-least-once/exactly-once modes
›Spark Structured Streaming — watermarking for late data, output modes (append, complete, update), and stateful aggregations
›Apache Flink — event time vs processing time, watermarks and lateness handling, keyed state, and checkpointing
›Pravega — segmented stream storage, auto-scaling segments, and transactions for exactly-once guarantees
›EdgeX Foundry — device services, core services, export services, and edge-to-cloud data pipeline integration

›Python data engineering libraries — pandas, numpy, SQLAlchemy, Pydantic, Great Expectations, and Arrow
›Data structures for pipelines — iterators, generators, and lazy evaluation for memory-efficient processing
›Apache Airflow architecture — Scheduler, Executor (LocalExecutor, CeleryExecutor, KubernetesExecutor), Metadata DB, and Web UI
›DAG design patterns — fan-out/fan-in, dynamic DAG generation, task groups, and dataset-driven scheduling
›Pipeline best practices — idempotency by design, retry with exponential backoff, dead letter queues, data quality checks, and observability (OpenTelemetry, DataDog)

Live Training

Live Instructor

Mon–Fri · Weekend batches

Recording

Live class recording included

30 Days

1.5 hrs/day weekday · 3 hrs weekend

Course Materials

Study notes and resources

Tech Support

Call / WhatsApp support

Working Hours

Mon–Fri · 9 AM – 6 PM

Enroll Now Download Brochure

Have Questions?

Chat with our Dell-certified data engineering trainers instantly.

WhatsApp Us

Dell Data Engineering

Optimize

Six Data Engineering Modules

6-Module D-DS-OP-23 Programme

Ready to Excel as a Data Engineer?