Apache Iceberg

Apache Iceberg is an open data format for big analytic datasets. Developed by Netflix, Iceberg was designed to be an open community standard and a table format specification that allows compatibility across multiple languages and implementations. After being made open source, organizations like Apple have actively helped in its development.

Between 2016 and 2018, Iceberg, alongside Delta Tables and Apache Hudi emerged to challenge Apache Hive's table format used since 2010. Besides working as a query engine for large batch jobs, Hive works as a metadata catalog and table format used by query engines such as Spark and Presto. The main issue with Hive was handling data changes over large datasets while coordinating multiple applications and not corrupting the data. To solve this atomic transactions were required.

According to Iceberg creators, the project brings the reliability and simplicity of SQL tables to big data while making it possible for engines like Spark, Trino, Flink, Presto, and Hive to work with the same tables simultaneously and safely. It is written in Java and offers a Scala API. The center of its architectural design contains a catalog that supports operations for updating the current metadata pointer, allowing for atomic transactions.

Iceberg is still in active development and has started to be integrated and implemented by multiple organizations like AWS, Adobe, Apple, Netflix, Dremio, Linkedin, Expedia.

Related articles

Data platform requirements and expectations

Data platform requirements and expectations

Categories: Big Data, Infrastructure | Tags: Data Engineering, Data Governance, Data Analytics, Data Hub, Data Lake, Data lakehouse, Data Science

A big data platform is a complex and sophisticated system that enables organizations to store, process, and analyze large volumes of data from a variety of sources. It is composed of several…

David WORMS

By David WORMS

Mar 23, 2023

Adaltas Summit 2022 Morzine

Adaltas Summit 2022 Morzine

Categories: Big Data, Adaltas Summit 2022 | Tags: Data Engineering, Infrastructure, Iceberg, Container, Data lakehouse, Docker, Kubernetes

For its third edition, the whole Adaltas crew is gathering in Morzine for a whole week with 2 days dedicated to technology the 15th and the 16Th of september 2022. The speakers choose one of the…

David WORMS

By David WORMS

Jan 13, 2023

Comparison of database architectures: data warehouse, data lake and data lakehouse

Comparison of database architectures: data warehouse, data lake and data lakehouse

Categories: Big Data, Data Engineering | Tags: Data Governance, Infrastructure, Iceberg, Parquet, Spark, Data Lake, Data lakehouse, Data Warehouse, File Format

Database architectures have experienced constant innovation, evolving with the appearence of new use cases, technical constraints, and requirements. From the three database structures we are comparing…

Gonzalo ETSE

By Gonzalo ETSE

May 17, 2022

Canada - Morocco - France

We are a team of Open Source enthusiasts doing consulting in Big Data, Cloud, DevOps, Data Engineering, Data Science…

We provide our customers with accurate insights on how to leverage technologies to convert their use cases to projects in production, how to reduce their costs and increase the time to market.

If you enjoy reading our publications and have an interest in what we do, contact us and we will be thrilled to cooperate with you.

Support Ukrain