The Linux Foundation | 06 June 2024
First-party support for Delta Lake and BigQuery boosts interoperability and diversifies data ecosystems, marking a significant advancement in open source collaboration.
SAN FRANCISCO – June 6, 2024– Delta Lake, hosted by LF AI & Data, is proud to announce that Google Cloud’s BigQuery now offers first party support for Delta Lake. This development further enhances the interoperability and openness of the data ecosystem.
Delta Lake is an optimized storage layer for lake houses, enhancing the reliability and performance of data lakes. Positioned atop existing data lake storage solutions, such as cloud object stores, Delta Lake introduces a scalable metadata layer to data stored in the Parquet format, significantly boosting efficiency and scalability. Known for its interoperability, Delta Lake is used by over 11,000 companies with 20 million downloads per month. Delta Lake is committed to accessibility and ease of integration within diverse data environments.
BigQuery, Google Cloud’s fully managed, gen-AI-ready data platform, enables users to extract insights across various data types, formats, and applications. BigQuery helps tens of thousands of customers maximize value from data, and is designed to be multi-engine, multi-format, and multi-cloud. Customers use BigQuery to manage all data types, structured and unstructured, with fine-grained access controls and built-in ML capabilities. And built-in data governance capabilities provide customers with a unified metadata catalog, data quality, lineage, and profiling.
"We're excited to see BigQuery join the ranks of platforms that support Delta Lake. Native support in BigQuery emphasizes the growing adoption of Delta Lake across diverse ecosystems and highlights the critical role of open collaboration in enhancing data management technologies." - Ibrahim Haddad, Executive Director, LF AI & Data Foundation
The integration of Delta Lake and BigQuery supports sophisticated functionalities, such as Deletion Vectors and "merge on read" capabilities, enabling the efficient execution of INSERT, UPDATE, MERGE, and DELETE operations. This capability ensures that enterprises leveraging both Delta Lake and BigQuery experience a seamless, integrated approach to data management. Organizations can now utilize the combined strengths of both platforms to handle large-scale data workloads with unprecedented efficiency and flexibility.
“Google Cloud has long advocated for customer choice and interoperability with data, including support for open source formats,” said Ritika Suri, Director of Data and AI Partnerships at Google Cloud. “Customers can now use BigQuery to easily manage data stored on Delta Lake, enabling them to optimize gen AI projects, extract business insights, and better unify their data foundations.”
LF AI & Data’s collaboration with Google Cloud signifies its dedication to improving the practicality and accessibility of data technologies, paving the way for future advancements and setting new standards in the industry.
For more information on Delta Lake and its native support in Google BigQuery, visit https://delta.io.
***
About LF AI & Data
LF AI & Data is an umbrella foundation of the Linux Foundation that supports open source innovation in artificial intelligence (AI) and data. LF AI & Data was created to support open source AI and data, and to create a sustainable open source AI and data ecosystem that makes it easy to create AI and data products and services using open source technologies. We foster collaboration under a neutral environment with an open governance in support of the harmonization and acceleration of open source technical projects.
Media Contact
Carly Akerly
The Linux Foundation
cakerly@contraactor.linuxfoundation.org