00:00
00:00
Master Data Engineering with Prophecy: Spark & Databricks Pro
Focused View
4:51:26
001 Welcome to Prophecy for Data Engineering on Databricks and Spark.mp4
04:59
001 Whats the future of data transformation.mp4
00:56
002 The evolution of data transformation.mp4
02:05
003 Ideal data transformation solution for the cloud.mp4
05:50
004 Prophecy and the future of data transformation.mp4
02:50
005 How to build the ideal data transformation in the cloud.mp4
01:07
001 What is a data lake and the difference between a data lake and data warehouses.mp4
04:00
002 Introducing data lakehouse and why its the perfect solution.mp4
02:01
001 Meet your instructor and module overview.mp4
03:54
002 Apache Spark architecture and concepts.mp4
09:22
003 Spark language and tooling.mp4
07:44
004 From Apache Spark to Databricks - why are they different.mp4
13:06
005 Data lakehouse, unity catalog, optimization and security.mp4
08:44
006 Working with Spark best practices.mp4
05:49
007 Spark and Databricks tips and tricks.mp4
05:18
001 Prophecy Overview - lets learn together!.mp4
01:44
002 Setting up a Databricks Fabric to execute our Pipelines.mp4
03:53
003 Create a Prophecy Project to manage our Spark code.mp4
03:17
004 Getting started with the Pipeline canvas.mp4
05:40
005 Explore code view and perform simple aggregations.mp4
01:58
006 Join accounts and opportunities data and write results to a delta table.mp4
02:33
007 Create a Pipeline and read from Data Sources to start building our Pipeline.mp4
05:40
008 Deploying Pipelines to production to run our scheduled Pipelines.mp4
03:46
009 Introduction to Prophecy Users and Teams.mp4
02:05
001 Data Sources and Targets overview.mp4
00:38
002 Parse and read raw data from object store with best practices.mp4
02:32
003 Prophecy built-in Data Sources and Data Sets.mp4
02:11
004 Explore Data Source default options.mp4
02:13
005 Read and parse source parquet data and merge schema.mp4
04:20
006 Handle corrupt and malformed records when reading from object stores.mp4
02:51
007 Additional options to handle corrupt and malformed reocrds.mp4
02:41
008 Work with source data schema and delimiters.mp4
02:38
009 Read from delta tables as sources.mp4
01:05
010 Write data to a delta table using a target Gem.mp4
01:55
011 Partition data when writing to a delta table for optimal performance.mp4
02:09
012 What weve learned in this module.mp4
01:51
001 Data lakehouse and the medallion architecture module overview.mp4
02:09
002 Medallion architecture - bronze, silver, and gold layer characteristics.mp4
03:05
003 Read and write data by partition - daily load from object storage.mp4
02:34
004 Additional data load by partition - daily load from object storage.mp4
01:17
005 Introduction to data models in a data lakehouse.mp4
04:08
006 Write the bronze layer data to delta tables.mp4
02:00
007 Introduction to Slowly Changing Dimensions (SCD).mp4
01:43
008 Implement simple SCD2 for bronze layer table.mp4
06:50
009 Bulk load read and write options.mp4
00:57
010 Bulk load historical data with SCD2.mp4
05:41
011 Delta table data versioning.mp4
05:28
012 Work with incompatible schemas.mp4
04:23
013 Recover data from a previous version.mp4
02:07
014 A summary of what weve learned in this module.mp4
00:33
001 Building the Silver and Gold layers - Overview.mp4
03:25
002 Data integration and cleaning in the Silver layer.mp4
02:05
003 Build a data model and integrate data in the Silver layer.mp4
03:17
004 Implement SCD2 in the silver layer.mp4
04:16
005 Generating unique IDs and write data to delta tables.mp4
02:52
006 Business requirements for the Gold layer.mp4
01:27
007 Perform analytics in the Gold layer to build business reports.mp4
03:09
008 Using subgraphs for reusability to simplify Pipelines.mp4
01:54
009 A summary of what weve learned in this module.mp4
00:48
001 Pipeline deployment overview.mp4
00:49
002 Ways to orchestrate workflows to automate jobs.mp4
01:50
003 Configure incremental Pipeline to prepare for scheduled runs.mp4
02:03
004 Create a Prophecy Job to schedule the Pipelines to run daily.mp4
04:04
005 What is CICD and how to deploy Pipelines to production.mp4
02:42
006 Advanced use cases integrate with external CICD process using PBT.mp4
04:01
007 A summary of what weve learned in this module.mp4
00:26
external-links.txt
001 Version management and change control overview.mp4
00:40
002 Prophecy Projects and the git process.mp4
02:20
003 Collaborating on a Pipeline - catching dev branch to the main branch.mp4
01:34
004 Reverting changes when developing a Pipeline before committing.mp4
01:11
005 Reverting back to a prior commit after committing by using rollback.mp4
00:50
006 Merging changes and switching between branches.mp4
01:48
007 Resolving code conflicts with multiple team members are making commits.mp4
02:18
008 Cloning an exiting Prophecy Project to a new repository.mp4
02:11
009 Reusing an existing Prophecy Project by importing the Project.mp4
01:19
010 Creating pull requests and handling commit conflicts.mp4
03:49
011 A summary of what weve learned in this module.mp4
00:32
001 Reusability and extensibility overview.mp4
01:34
002 The importance of setting data engineering standards - reuse and extend.mp4
03:16
003 Convert a script to a customized Gem to share and reuse.mp4
02:03
004 Create a new Gem for multi-dimensional cube using the specified express.mp4
02:57
005 Create an UI for the cube Gem for users to define the cube.mp4
02:01
006 Adding additional features to make the customized Gem UI intuitive.mp4
01:27
007 Error handling with adding validations and customized error messages.mp4
01:59
008 Testing customized cube Gem and publishing the Gem to share with others.mp4
01:57
009 Assigning proper access to share the newly built cube Gem.mp4
01:41
010 Use the newly created cube Gem by adding it a dependency.mp4
05:50
011 A summary of what weve learned in this module.mp4
00:30
external-links.txt
001 Data quality and unit testing overview.mp4
01:25
002 Medallion architecture and data quality.mp4
02:32
003 Data quality Pipeline walkthrough - how to populate data quality log.mp4
03:40
004 Silver layer data quality checks, define errors, and write to delta table.mp4
03:52
005 Data integration quality checks with joins - check if customer IDs are missing.mp4
01:18
006 Performing data reconciliation checks - identify mismatching column values.mp4
04:20
007 Identifying and tracking data quality issues by drilling down to a specific ID.mp4
01:07
008 Executing data quality checks in phases - stop the pipeline if error exists.mp4
02:35
009 Unit testing options - testing expressions using output equality.mp4
03:09
010 Explore code view of the unit test.mp4
01:12
011 Running the unit tests.mp4
01:32
012 Unit testing expressions using output predicates.mp4
02:51
013 A summary of what weve learned in this module.mp4
00:38
More details
Course Overview
Transform your data engineering skills with this comprehensive course on Prophecy, Databricks, and Spark. Learn to build efficient data pipelines, implement lakehouse architectures, and deploy production-ready solutions using low-code tools.
What You'll Learn
- Implement medallion architecture for e-commerce data solutions
- Build and deploy Pipelines with Prophecy's low-code interface
- Master Spark best practices and Databricks optimization
Who This Is For
- Data engineers looking to streamline pipeline development
- Data architects designing lakehouse solutions
- Analysts transitioning to engineering roles
Key Benefits
- Hands-on labs with real-world e-commerce use cases
- Learn CI/CD deployment and version control for data engineering
- Create reusable components to accelerate team productivity
Curriculum Highlights
- Data lakehouse architecture and medallion implementation
- Building silver/gold layers with SCD2 and delta tables
- Production deployment and data quality testing
Focused display
Category
- language english
- Training sessions 101
- duration 4:51:26
- English subtitles has
- Release Date 2025/05/25