Examples

Examples use cases solved with Versatile Data Kit:

Data Engineering

• Hello World Data Job

Ingestion examples

Ingestion using Flow API (Alpha)

Simple Flow based API where you can ingest using only configuration

• VDK Ingest Tutorial Notebook

Ingestion using Python Ingester send APIs

This is a more "low" level and advanced API but provides more power

• Ingesting data from REST API into Database
• Ingesting data from DB into Database
• Ingesting data from S3 into Database
• Ingesting local CSV file into Database
• Incremental ingestion using Job Properties
• Ingesting data from a authenticated REST API using Job Secrets

Data processing examples

• Data Modeling Guide: Treating Data as a Product
• Processing data using SQL and local database
• Processing data using Kimball warehousing templates and a Big Data database (Presto/Trino)

Deployment examples

• Scheduling a Data Job for automatic execution
• Deploying using different python versions

Orchestration examples

• Using DAGs to orchestrate Data Jobs
• Using DAGs to orchestrate Data Jobs with arguments
• Workflow of Data Jobs using Airflow

Job Operations

• Installation and Getting Started with Operations UI
• VDK Operations User Interface Walkthrough

Scenarios

• Ingest, process, and publish using VDK & Streamlit Vincent Van Gogh's paintings
• Ingest, process, and publish using VDK & Streamlit American states life expectancy
• Ingest, process, and publish using VDK & Streamlit analysis of relationship between Natural Gas prices and the temperature

Job development examples

• Debug Data Job in IDE
• Using Data Job Properties vs Secrets

Jupyter tutorials

• Create a Data Job through the Jupyter UI
• Develop a Data Job through the Jupyter UI
• Deploy Data Job through the Jupyter UI
• Convert Data Job to Jupyter Notebook

Infrastructure operators

• Install VDK Control Service with custom SDK
• Data anonymization ingestion plugin
• Supported Python Versions - Control Service Setup
• Configuring Hashicorp Vault Instance for storing Secrets

The source code for the examples can be found in git at https://github.com/vmware/versatile-data-kit/tree/main/examples