Home Page Icon
Home Page
Table of Contents for
Data Engineering on Azure
Close
Data Engineering on Azure
by
Data Engineering on Azure
inside front cover
Data Platform Architecture
Data Engineering on Azure
Copyright
dedication
brief contents
contents
front matter
preface
acknowledgments
about this book
about the author
about the cover illustration
1 Introduction
1.1 What is data engineering?
1.2 Who this book is for
1.3 What is a data platform?
1.3.1 Anatomy of a data platform
1.3.2 Infrastructure as code, codeless infrastructure
1.4 Building in the cloud
1.4.1 IaaS, PaaS, SaaS
1.4.2 Network, storage, compute
1.4.3 Getting started with Azure
1.4.4 Interacting with Azure
1.5 Implementing an Azure data platform
Summary
Part 1 Infrastructure
2 Storage
2.1 Storing data in a data platform
2.1.1 Storing data across multiple data fabrics
2.1.2 Having a single source of truth
2.2 Introducing Azure Data Explorer
2.2.1 Deploying an Azure Data Explorer cluster
2.2.2 Using Azure Data Explorer
2.2.3 Working around query limits
2.3 Introducing Azure Data Lake Storage
2.3.1 Creating an Azure Data Lake Storage account
2.3.2 Using Azure Data Lake Storage
2.3.3 Integrating with Azure Data Explorer
2.4 Ingesting data
2.4.1 Ingestion frequency
2.4.2 Load type
2.4.3 Restatements and reloads
Summary
3 DevOps
3.1 What is DevOps?
3.1.1 DevOps in data engineering
3.2 Introducing Azure DevOps
3.2.1 Using the az azure-devops extension
3.3 Deploying infrastructure
3.3.1 Exporting an Azure Resource Manager template
3.3.2 Creating Azure DevOps service connections
3.3.3 Deploying Azure Resource Manager templates
3.3.4 Understanding Azure Pipelines
3.4 Deploying analytics
3.4.1 Using Azure DevOps marketplace extensions
3.4.2 Storing everything in Git; deploying everything automatically
Summary
4 Orchestration
4.1 Ingesting the Bing COVID-19 open dataset
4.2 Introducing Azure Data Factory
4.2.1 Setting up the data source
4.2.2 Setting up the data sink
4.2.3 Setting up the pipeline
4.2.4 Setting up a trigger
4.2.5 Orchestrating with Azure Data Factory
4.3 DevOps for Azure Data Factory
4.3.1 Deploying Azure Data Factory from Git
4.3.2 Setting up access control
4.3.3 Deploying the production data factory
4.3.4 DevOps for the Azure Data Factory recap
4.4 Monitoring with Azure Monitor
Summary
Part 2 Workloads
5 Processing
5.1 Data modeling techniques
5.1.1 Normalization and denormalization
5.1.2 Data warehousing
5.1.3 Semistructured data
5.1.4 Data modeling recap
5.2 Identity keyrings
5.2.1 Building an identity keyring
5.2.2 Understanding keyrings
5.3 Timelines
5.3.1 Building a timeline view
5.3.2 Using timelines
5.4 Continuous data processing
5.4.1 Tracking processing functions in Git
5.4.2 Keyring building in Azure Data Factory
5.4.3 Scaling out
Summary
6 Analytics
6.1 Structuring storage
6.1.1 Providing development data
6.1.2 Replicating production data
6.1.3 Providing read-only access to the production data
6.1.4 Storage structure recap
6.2 Analytics workflow
6.2.1 Prototyping
6.2.2 Development and user acceptance testing
6.2.3 Production
6.2.4 Analytics workflow recap
6.3 Self-serve data movement
6.3.1 Support model
6.3.2 Data contracts
6.3.3 Pipeline validation
6.3.4 Postmortems
6.3.5 Self-serve data movement recap
Summary
7 Machine learning
7.1 Training a machine learning model
7.1.1 Training a model using scikit-learn
7.1.2 High spender model implementation
7.2 Introducing Azure Machine Learning
7.2.1 Creating a workspace
7.2.2 Creating an Azure Machine Learning compute target
7.2.3 Setting up Azure Machine Learning storage
7.2.4 Running ML in the cloud
7.2.5 Azure Machine Learning recap
7.3 MLOps
7.3.1 Deploying from Git
7.3.2 Storing pipeline IDs
7.3.3 DevOps for Azure Machine Learning recap
7.4 Orchestrating machine learning
7.4.1 Connecting Azure Data Factory with Azure Machine Learning
7.4.2 Machine learning orchestration
7.4.3 Orchestrating recap
Summary
Part 3 Governance
8 Metadata
8.1 Making sense of the data
8.2 Introducing Azure Purview
8.3 Maintaining a data inventory
8.3.1 Setting up a scan
8.3.2 Browsing the data dictionary
8.3.3 Data dictionary recap
8.4 Managing a data glossary
8.4.1 Adding a new glossary term
8.4.2 Curating terms
8.4.3 Custom templates and bulk import
8.4.4 Data glossary recap
8.5 Understanding Azure Purview's advanced features
8.5.1 Tracking lineage
8.5.2 Classification rules
8.5.3 REST API
8.5.4 Advanced features recap
Summary
9 Data quality
9.1 Testing data
9.1.1 Availability tests
9.1.2 Correctness tests
9.1.3 Completeness tests
9.1.4 Detecting anomalies
9.1.5 Testing data recap
9.2 Running data quality checks
9.2.1 Testing using Azure Data Factory
9.2.2 Executing tests
9.2.3 Creating and using a template
9.2.4 Running data quality checks recap
9.3 Scaling out data testing
9.3.1 Supporting multiple data fabrics
9.3.2 Testing at rest and during movement
9.3.3 Authoring tests
9.3.4 Storing tests and results
Summary
10 Compliance
10.1 Data classification
10.1.1 Feature data
10.1.2 Telemetry
10.1.3 User data
10.1.4 User-owned data
10.1.5 Business data
10.1.6 Data classification recap
10.2 Changing classification through processing
10.2.1 Aggregation
10.2.2 Anonymization
10.2.3 Pseudonymization
10.2.4 Masking
10.2.5 Processing classification changes recap
10.3 Implementing an access model
10.3.1 Security groups
10.3.2 Securing Azure Data Explorer
10.3.3 Access model recap
10.4 Complying with GDPR and other considerations
10.4.1 Data handling
10.4.2 Data subject requests
10.4.3 Other considerations
Summary
11 Distributing data
11.1 Data distribution overview
11.2 Building a data API
11.2.1 Introducing Azure Cosmos DB
11.2.2 Populating the Cosmos DB collection
11.2.3 Retrieving data
11.2.4 Data API recap
11.3 Serving machine learning
11.4 Sharing data for bulk copy
11.4.1 Separating compute resources
11.4.2 Introducing Azure Data Share
11.4.3 Sharing data for bulk copy recap
11.5 Data sharing best practices
Summary
Appendix A. Azure services
Azure Storage
Azure SQL
Azure Synapse Analytics
Azure Data Explorer
Azure Databricks
Azure Cosmos DB
Appendix B. KQL quick reference
Common query reference
SQL to KQL
Appendix C. Running code samples
index
inside back cover
MLOps
Search in book...
Toggle Font Controls
Playlists
Add To
Create new playlist
Name your new playlist
Playlist description (optional)
Cancel
Create playlist
Sign In
Email address
Password
Forgot Password?
Create account
Login
or
Continue with Facebook
Continue with Google
Sign Up
Full Name
Email address
Confirm Email Address
Password
Login
Create account
or
Continue with Facebook
Continue with Google
Next
Next Chapter
inside front cover
Add Highlight
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click
here
login for view all page.
Day Mode
Cloud Mode
Night Mode
Reset