Information Density: Delta Lake – Signal Evidence & AI Readability

[IMG: The Linux Foundation Projects]
[IMG: Delta Lake]
Search ctrl K Announcing Delta Lake 4.2.0 on Apache Spark™ 4.1.0: Try out the latest release today!
[H1] Build Lakehouses with Delta Lake
Delta Lake
is an open-source storage framework that enables building a format agnostic
Lakehouse architecture
with compute engines including Spark, PrestoDB, Flink, Trino, Hive, Snowflake,
Google BigQuery, Athena, Redshift, Databricks, Azure Fabric and APIs for
Scala, Java, Rust, and Python. With Delta Universal Format
aka UniForm, you can read now Delta tables with Iceberg and Hudi clients.
Get Started GitHub Releases Roadmap
[IMG: Delta Lake Integrations]
[IMG: Open]
[H4] Open
Community driven, rapidly expanding integration ecosystem
[IMG: Simple]
[H4] Simple
One format to unify your ETL, Data warehouse, ML in your lakehouse
[IMG: UniForm]
[H4] UniForm
A universal format for lakehouse interoperability
[IMG: Production Ready]
[H4] Production Ready
Battle tested in over 10,000+ production environments ​​
[IMG: Platform Agnostic]
[H4] Platform Agnostic
Use with any query engine on any cloud, on-prem, or locally
[H2] The Latest
Integrating the Rust Delta Kernel into ClickHouse Delta Grows Up: Writes, Unity Catalog and Time Travel Delta 4.2.0 Released Delta Lake 4.1.0 Released The next evolution of Delta - Catalog-Managed Tables
[IMG: logo]
[H2] Key Features
[H4] ACID Transactions
Protect your data with serializability, the strongest level of isolation
[H4] Scalable Metadata
Handle petabyte-scale tables with billions of partitions and files with ease
[H4] Time Travel
Access/revert to earlier versions of data for audits, rollbacks, or reproduce
[H4] Open Source
Community driven, open standards, open protocol, open discussions
[H4] Unified Batch/Streaming
Exactly once semantics ingestion to backfill to interactive queries
[H4] Schema Evolution / Enforcement
Prevent bad data from causing data corruption
[H4] Audit History
Delta Lake log all change details providing a fill audit trail
[H4] DML Operations
SQL, Scala/Java and Python APIs to merge, update and delete datasets
[IMG: The Definitive Guide]
[H3] Delta Lake: The Definitive Guide
Building modern data lakehouse architectures with Delta Lake with
forewords by Michael Armbrust and Dominique Brezinski.
Download
[H2] Read the Lakehouse Storage Systems Whitepapers
These whitepapers dive into the features of Lakehouse storage systems and
compare Delta Lake, Apache Hudi, and Apache Iceberg. They also explain the
benefits of Lakehouse storage systems and show key performance benchmarks.
Read the whitepaper
Analyzing and Comparing Lakehouse Storage Systems
Lakehouse: A New Generation of Open Platforms that Unify Data
Warehousing and Advanced Analytics
Read the article
[H2] Organizations that have contributed to Delta Lake
Together we have made Delta Lake the most widely used lakehouse format in
the world!
[IMG: adobe]
[IMG: atlassian]
[IMG: apple]
[IMG: amazon]
[IMG: alibaba]
[IMG: backMarket]
[IMG: byteDance]
[IMG: canva]
[IMG: cockroachdb]
[IMG: comcast]
[IMG: databricks]
[IMG: disney]
[IMG: ebay]
[IMG: grafana]
[IMG: ibm]
[IMG: microsoft]
[IMG: safeGraph]
[IMG: sambaTv]
[IMG: scribd]
[IMG: starburst]
[IMG: td]
[IMG: talend]
[IMG: twilio]
[IMG: ucDavis]
[IMG: kubit]
[H2] Join the Delta Lake Community
Delta Lake is supported by more than 190 developers from over 70
organizations across multiple repositories.
Chat with fellow Delta Lake users and contributors, ask questions and share
tips.
Slack Google Groups LinkedIn YouTube D3L2 | Spotify Check out Last Week in a Byte newsletter: for the latest Delta events...a week late!
[IMG: The Linux Foundation]
[H5] Project Governance
Delta Lake is an independent open-source project and not controlled by
any single company. To emphasize this we joined the Delta Lake Project
in 2019, which is a sub-project of the Linux Foundation Projects. Within
the project, we make decisions based on these rules.

[IMG: The Linux Foundation Projects]
[IMG: Delta Lake]
Search ctrl K
[H1] Delta Lake Blogs
RSS Feed
[IMG: Thumbnail for Integrating the Rust Delta Kernel into ClickHouse]
Latest Post
[H3] Integrating the Rust Delta Kernel into ClickHouse
By Melvyn Peignon , Kseniia Sumarokova , Raul Marin Integrating the Rust Delta Kernel into ClickHouse provides a maintained, consistent interface between the table format and query engine, exposing more features while cutting integration complexity.
[IMG: Thumbnail for Delta Grows Up: Writes, Unity Catalog and Time Travel]
[H3] Delta Grows Up: Writes, Unity Catalog and Time Travel
By Ben Fleis DuckDB's Delta and Unity Catalog extensions shed their experimental tags — now with writes, Unity Catalog and time travel support.
[IMG: Thumbnail for Delta 4.2.0 Released]
[H3] Delta 4.2.0 Released
By Scott Haines , Zheng Hu , Alex Jiang The Delta Kernel continues to drive the future of Delta Lake. With the 4.2 release, we've further eliminated engine inconsistencies and feature fragmentation by embedding the kernel where it matters most - right in the heart of our engine connectors. The kernel ensures strict transactional consistency and peak performance regardless of which engine reads or writes your data. Along that line, this release includes a brand new experimental Apache Flink connector built on the Delta Kernel aimed at providing expanded support for catalog-managed tables. This new connector supports reading and writing catalog-managed tables via Flink SQL...
[IMG: Thumbnail for Delta Lake 4.1.0 Released]
[H3] Delta Lake 4.1.0 Released
By Zheng Hu , Scott Haines We’re excited to announce the release of Delta Lake 4.1.0. This release introduces significant new features, performance improvements, and critical platform upgrades, including full support for Apache Spark 4.1.0 and enhanced storage management in Unity Catalog.
[IMG: Thumbnail for The next evolution of Delta - Catalog-Managed Tables]
[H3] The next evolution of Delta - Catalog-Managed Tables
By Benjamin Mathew , Scott Sandre , Scott Haines This article explains how to enable catalog-managed commits, a Delta table feature that shifts transaction coordination from the filesystem to Unity Catalog, making the catalog the single source of truth for table state.
[IMG: Thumbnail for Delta Lake 4.0.1 Release]
[H3] Delta Lake 4.0.1 Release
By Robert Pack , Timothy Wang We're excited to announce the release of Delta Lake 4.0.1. This release brings important bug fixes and improvements to Delta Lake 4.0, with a focus on Unity Catalog integration, authentication, and compatibility.
[IMG: Thumbnail for Delta Lake 4.0]
[H3] Delta Lake 4.0
By Carly Akerly , Robert Pack We’re thrilled to announce the release of Delta Lake 4.0, a milestone release packed with powerful new features, performance optimizations, and foundational enhancements for the future of open data lakehouses. With new catalog integration, enhanced support for semi-structured data, smarter change tracking, and improved performance, this release delivers practical solutions to everyday data challenges.
[IMG: Thumbnail for Delta Kernel: A Game-Changer for Customer-Facing Analytics]
[H3] Delta Kernel: A Game-Changer for Customer-Facing Analytics
By Sida Shen , Wenbo Yang How Delta kernel enables StarRocks to run customer-facing analytics directly on open table formats
[IMG: Thumbnail for Working with Apache Sedona]
[H3] Working with Apache Sedona
By Avril Aysha Learn how to use Apache Sedona with Delta Lake
[IMG: Thumbnail for Delta Lake 3.3]
[H3] Delta Lake 3.3
By Allison Portis , Susan Pierce Announcing Delta Lake 3.3 on Apache Spark 3.5, with features that improve the performance and interoperability of Delta Lake.
[IMG: Thumbnail for Understanding Open Table Formats]
[H3] Understanding Open Table Formats
By Avril Aysha Learn about open table formats
[IMG: Thumbnail for Delta Lake Liquid Clustering]
[H3] Delta Lake Liquid Clustering
By Avril Aysha Learn how to use Delta Lake Liquid Clustering feature
[IMG: Thumbnail for Delta Lake on Azure Data Lake Storage]
[H3] Delta Lake on Azure Data Lake Storage
By Avril Aysha Learn how to use Delta Lake on Azure Data Lake Storage

[IMG: The Linux Foundation Projects]
[IMG: Delta Lake]
Search ctrl K
[H1] Getting Started with Delta Lake
This guide helps you quickly explore the main features of Delta Lake. It provides code snippets that show how to read from and write to Delta tables from interactive, batch, and streaming queries. It also demonstrates table updates and time travel.
[H2] Set up Apache Spark with Delta Lake
Follow these instructions to set up Delta Lake with Spark. You can run the steps in this guide on your local machine in the following two ways:
Run interactively: Start the Spark shell (Scala or Python) with Delta Lake and run the code snippets interactively in the shell.
Run as a project: Set up a Maven or SBT project (Scala or Java) with Delta Lake, copy the code snippets into a source file, and run the project. Alternatively, you can use the examples provided in the GitHub repository.
[H3] Set up interactive shell
To use Delta Lake interactively within the Spark Scala or Python shell, you need a local installation of Apache Spark. Depending on whether you want to use Python or Scala, you can set up either PySpark or the Spark shell, respectively. For all the instructions below make sure you install the correct version of Spark or PySpark that is compatible with Delta Lake 2.1.0. See the release compatibility matrix for details.
[H3] PySpark shell
Install the PySpark version that is compatible with the Delta Lake version by running the following:
pip install pyspark==<compatible-spark-version>
Run PySpark with the Delta Lake package and additional configurations:
pyspark --packages io.delta:delta-core_2.12:2.1.0 \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
[H3] Spark Scala shell
Download the compatible version of Apache Spark by following instructions from Downloading Spark, either using pip or by downloading and extracting the archive and running spark-shell in the extracted directory.
bin/spark-shell --packages io.delta:delta-core_2.12:2.1.0 \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
[H2] Set up project
If you want to build a project using Delta Lake binaries from Maven Central Repository, you can use the following Maven coordinates.
[H3] Maven
You include Delta Lake in your Maven project by adding it as a dependency in your POM file. Delta Lake compiled with Scala 2.12.
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_2.12</artifactId>
<version>2.1.0</version>
</dependency>
[H3] SBT
You include Delta Lake in your SBT project by adding the following line to your build.sbt file:
libraryDependencies += "io.delta" %% "delta-core" % "2.1.0"
[H3] Python
To set up a Python project (for example, for unit testing), you can install Delta Lake using pip install delta-spark and then configure the SparkSession with the configure_spark_with_delta_pip() utility function in Delta Lake:
from delta import *
builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
spark = configure_spark_with_delta_pip(builder).getOrCreate()
[H2] Create a table
To create a Delta table, write a DataFrame out in the delta format. You can use existing Spark SQL code and change the format from parquet, csv, json, and so on, to delta.
data = spark.range(0, 5)
data.write.format("delta").save("/tmp/delta-table")
These operations create a new Delta table using the schema that was inferred from your DataFrame. For the full set of options available when you create a new Delta table, see the Create a table and Write to a table documentation articles.
This guide uses local paths for Delta table locations. For configuring HDFS or cloud storage for Delta tables, see the Storage configuration Documentation article.
[H2] Read data
You read data in your Delta table by specifying the path to the files "/tmp/delta-table":
df = spark.read.format("delta").load("/tmp/delta-table")
df.show()
[H2] Update table data
Delta Lake supports several operations to modify tables using standard DataFrame APIs. This example runs a batch job to overwrite the data in the table:
[H3] Overwrite
data = spark.range(5, 10)
data.write.format("delta").mode("overwrite").save("/tmp/delta-table")
If you read this table again, you should see only the values 5-9 you have added because you overwrote the previous data.
[H3] Conditional update without overwrite
Delta Lake provides programmatic APIs to conditional update, delete, and merge (upsert) data into tables. Here are a few examples:
from delta.tables import *
from pyspark.sql.functions import *
deltaTable = DeltaTable.forPath(spark, "/tmp/delta-table")
# Update every even value by adding 100 to it
deltaTable.update(
condition = expr("id % 2 == 0"),
set = { "id": expr("id + 100") })
# Delete every even value
deltaTable.delete(condition = expr("id % 2 == 0"))
# Upsert (merge) new data
newData = spark.range(0, 20)
deltaTable.alias("oldData") \
.merge(
newData.alias("newData"),
"oldData.id = newData.id") \
.whenMatchedUpdate(set = { "id": col("newData.id") }) \
.whenNotMatchedInsert(values = { "id": col("newData.id") }) \
.execute()
deltaTable.toDF().show()
You should see that some of the existing rows have been updated and new rows have been inserted.
For more information on these operations, see the Table deletes, updates, and merges documentation article.
[H2] Read older versions of data using time travel
You can query previous snapshots of your Delta table by using time travel. If you want to access the data that you overwrote, you can query a snapshot of the table before you overwrote the first set of data using the versionAsOf option.
df = spark.read.format("delta") \
.option("versionAsOf", 0) \
.load("/tmp/delta-table")
df.show()
You should see the first set of data, from before you overwrote it. Time travel takes advantage of the power of the Delta Lake transaction log to access data that is no longer in the table. Removing the version 0 option (or specifying version 1) would let you see the newer data again. For more information, see the Query an older snapshot of a table (time travel) documentation article.
[H2] Write a stream of data to a table
You can also write to a Delta table using Structured Streaming. The Delta Lake transaction log guarantees exactly-once processing, even when there are other streams or batch queries running concurrently against the table. By default, streams run in append mode, which adds new records to the table:
streamingDf = spark.readStream.format("rate").load()
stream = streamingDf \
.selectExpr("value as id") \
.writeStream.format("delta") \
.option("checkpointLocation", "/tmp/checkpoint") \
.start("/tmp/delta-table")
While the stream is running, you can read the table using the earlier commands.
If you’re running this in a shell, you may see the streaming task progress, which make it hard to type commands in that shell. It may be useful to start another shell in a new terminal for querying the table.
You can stop the stream by running stream.stop() in the same terminal that started the stream.
For more information about Delta Lake integration with Structured Streaming, see the Table streaming reads and writes documentation article.
[H2] Read a stream of changes from a table
While the stream is writing to the Delta table, you can also read from that table as streaming source. For example, you can start another streaming query that prints all the changes made to the Delta table. You can specify which version Structured Streaming should start from by providing the startingVersion or startingTimestamp option to get changes from that point onwards. See the Structured Streaming documentation article for details.
stream2 = spark.readStream.format("delta") \
.load("/tmp/delta-table") \
.writeStream.format("console") \
.start()
[H2] Next Steps
Here are some further resources for learning more about Delta Lake:
Delta FAQ for answers to common questions about Delta Lake
Tutorials for video tutorials about Delta Lake
Documentation—Batch Reading and Writing for detailed documentation on table batch reads and writes
Examples for Delta Lake usage examples

[IMG: The Linux Foundation Projects]
[IMG: Delta Lake]
Search ctrl K
[IMG: logo]
[H1] Delta Lake Integrations
Use the following frameworks, Delta Sharing clients, managed services,
and/or community integrations for Delta Lake and Delta Sharing.
[IMG: UniForm]
[H3]
Delta Universal Format (UniForm)
allows you to read Delta tables with Iceberg and Hudi clients
Requirements Enable UniForm
[H2] Frameworks
Use the following frameworks and languages including but not limited to Apache Flink, Apache Spark, Trino, and Rust.
[H4] Apache Druid
docs
source code
DruidThis connector allows Apache Druid to read from Delta Lake.
[H4] Apache Flink
docs
source code
FlinkstandaloneThis connector allows Apache Flink to write to Delta Lake.
[H4] Apache Hive
docs
source code
HivestandaloneThis connector allows Apache Hive to read from Delta Lake.
[H4] Apache Pulsar
docs
source code
PulsarcommunityThis connector allows Apache Pulsar to read from and write to Delta Lake.
[H4] Apache Spark™
docs
source code
SparkThis connector allows Apache Spark™ to read from and write to Delta Lake.
[H4] ClickHouse
docs
source code
ClickHouseClickHouse is a column-oriented database that allows users to run SQL queries on Delta Lake tables. This provides a read-only integration with existing Delta Lake tables in Amazon S3.
[H4] Dagster
docs
source code
DagsterPythonUse the Delta Lake IO Manager to read from and write to Delta Lake in your Dagster orchestration pipelines.
[H4] Delta Rust API
docs
source code
RustPythonThis library allows Rust (with Python bindings) low level access to Delta tables and is intended to be used with data processing frameworks like datafusion, ballista, rust-dataframe, vega, etc.
[H4] Delta Standalone
docs
source code
ScalaJavastandaloneThis library allows Scala and Java-based projects (including Apache Flink, Apache Hive, Apache Beam, and PrestoDB) to read from and write to Delta Lake.
[H4] FINOS Legend
docs
source code
FINOSLegendAn extension to the FINOS Legend framework for Apache Spark™ / Delta Lake based environment, combining best of open data standards with open source technologiesThis connector allows Trino to read from and write to Delta Lake.
[H4] Hopsworks
docs
source code
HopsworksPythonHopsworks Feature Store stores, manages, and serves feature data in Delta Lake.
[H4] Kafka Delta Ingest
docs
source code
KafkaRustThis project builds a highly efficient daemon for streaming data through Apache Kafka into Delta Lake.
[H4] PrestoDB
docs
source code
PrestoDBstandaloneThis connector allows PrestoDB to read from Delta Lake.
[H4] RisingWave
docs
source code
RisingWavestandaloneThis connector allows RisingWave to write to Delta Lake.
[H4] SQL Delta Import
docs
source code
SQLJDBCThis utility is for importing data from a JDBC source into a Delta Lake table.
[H4] StarRocks
docs
source code
StarRocksStarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. StarRocks has the ability to read from Delta Lake.
[H4] Trino
docs
source code
TrinoThis connector allows Trino to read from and write to Delta Lake.
[H2] Sharing
Use the following clients that integrate with Delta Sharing from C++ to Rust.
[H4] C++
docs
source code
C++Delta SharingcommunityThis connector allows a C++ client to read from Delta Sharing endpoint.
[H4] Excel
docs
ExcelDelta SharingThis connector allows a Excel client to read from Delta Sharing endpoint.
[H4] Go
source code
GoDelta SharingcommunityThis connector allows a Go client to read from Delta Sharing endpoint.
[H4] Java
source code
JavaDelta SharingcommunityThis connector allows a Java client to read from Delta Sharing endpoint.
[H4] Kotosiro Sharing
source code
KotosiroDelta SharingRustA Minimalistic Rust Implementation of Delta Sharing Server.
[H4] MLflow
docs
source code
MLflowDelta SharingcommunityPerform model exchange via Delta Sharing and MLflow
[H4] node.js
docs
source code
node.jsDelta SharingcommunityThis connector allows node.js to read from Delta Sharing endpoint.
[H4] Oracle
OracleDelta SharingThis connector allows for Delta Sharing with Oracle Autonomous Database Data Studio.
[H4] Power BI
docs
PowerBIDelta SharingThis connector allows Power BI to read from Delta Sharing endpoint.
[H4] R
source code
RDelta SharingcommunityThis connector allows a R client to read from Delta Sharing endpoint.
[H4] Rust
docs
source code
RustDelta SharingcommunityThis connector allows a Rust client to read from Delta Sharing endpoint.
[H4] Terminal
source code
TerminalDelta SharingcommunityTerminal application for browsing Delta Sharing Metadata.
[H2] Services
Use the managed services of your choice that integrate with Delta Lake.
[H4] Athena
docs
AthenaAWSThis utility allows Athena to natively read from Delta Lake starting with Athena SQL 3.0
[H4] AWS EMR
docs
AWSEMRStarting with Amazon EMR release 6.9.0, you can use Apache Spark 3.x on Amazon EMR clusters with Delta Lake tables.
[H4] AWS Glue
docs
AWSGlueAWS Glue 3.0 and later supports the Linux Foundation Delta Lake framework
[H4] aws-pandas-sdk
docs
source code
pandasAWSawswrangleraws-pandas-sdkpandas on AWS - Easy integration with AWS services including optional dependency with Delta Lake
[H4] Azure Stream Analytics
docs
AzureASAStream AnalyticsAzure Stream Analytics provides native write support for Delta Lake
[H4] BigQuery
docs
BigQuery’s native Delta Lake support enables seamless delivery of data for downstream applications.
[H4] Databricks
docs
DatabricksAzureGCPAWSDelta Lake is included within Databricks allowing it to read from and write to Delta Lake.
[H4] Microsoft Fabric
docs
In order to achieve seamless data access across all compute engines in Microsoft Fabric, Delta Lake is chosen as the unified table format.
[H4] Power BI
docs
source code
PowerBIcommunityThis connector allows Power BI to read from Delta Lake.
[H4] Redshift
docs
source code
RedshiftAWSmanifestThis utility allows AWS Redshift to read from Delta Lake using a manifest file.
[H4] Snowflake (Beta)
docs
SnowflakeThis preview allows Snowflake to read from Delta Lake via an external table.
[H4] Starburst
docs
StarburstAzureGCPAWSThe Starburst Delta Lake connector is an extended version of the Trino/Delta Lake connector with configuration and usage identical.
[H4] StarTree
docs
StarTreePinotStarTree Cloud includes the Apache Pinot / Delta Lake connector.
[H2] Community
Try out the following community integrations with Delta Lake.
[H4] Apache Beam
docs
source code
BeamstandalonecommunityThis connector allows Apache Beam to read from Delta Lake.
[H4] Athena Query Federation (Beta)
docs
source code
AWSAthenastandalonecommunityThis connector allows AWS Athena to read from Delta Lake.
[H4] Beam Delta Lake
docs
source code
BeamstandalonecommunityWith DataLakeIO, data from Apache Beam's pipelines can be read from and written to Delta Lake
[H4] Ceph
source code
CephcommunityThis connector allows you to read and write from Delta tables on Ceph storage.
[H4] dlt | SparkR
docs
source code
SparkRcommunityThis package allows SparkR to read from and write to Delta Lake.
[H4] DataHub
source code
DataHubcommunityThis connector allows DataHub to extract Delta Lake metadata.
[H4] Datastream Connector
source code
GCSDatastreambadal.iocommunityAs Datastream streams changes to files to Google Cloud Storage, this connector streams these files and writes the changes to Delta Lake.
[H4] MinIO
docs
source code
MinIOcommunityThis connector allows you to read and write from Delta tables on MinIO storage.