Semantic Coherence: Apache Iceberg – Signal Evidence & AI Readability

[H1] Home

Back to top

[H1] Welcome!?
We're glad that you're interested in joining the Apache Iceberg community! Read along to learn how to connect with existing community members and where you can get involved with the project, both technically and socially.
[H2] Join the Discussion?
Community discussions happen across various mailing lists, on the apache-iceberg Slack workspace, and on specific GitHub issues.
[H3] Mailing Lists?
Apache Iceberg mailing lists:
Developers: dev@iceberg.apache.org -- Iceberg community discussions
Archive
Subscribe
Unsubscribe
Commits: commits@iceberg.apache.org -- GitHub commit notifications
Archive
Subscribe
Unsubscribe
Issues: issues@iceberg.apache.org -- GitHub issue tracking
Archive
Subscribe
Unsubscribe
CI Jobs: ci-jobs@iceberg.apache.org -- GitHub Actions notifications
Archive
Subscribe
Unsubscribe
Private: private@iceberg.apache.org -- private mailing list for the PMC to discuss sensitive issues related to the health of the project
Archive (PMC-only access)
[H3] Slack?
We use the Apache Iceberg workspace on Slack. To be invited, follow this invite link.
Please note that this link may occasionally break when Slack does an upgrade. If you encounter problems using it, please let us know by sending an email to dev@iceberg.apache.org.
[H4] Requesting Slack Integrations?
To request a new app or integration for the Apache Iceberg Slack workspace, send an email to the dev mailing list with the following information:
What you want to add — The name and description of the app or integration
Why you want to add it — The benefit it provides to the community
What permissions does it need — The access levels and permissions required by the app
This allows the community to do a quick consensus check before the app is installed.
[H3] Issues?
Apache Iceberg tracks issues in GitHub and prefers to receive contributions as pull requests.
Issues are tracked in GitHub:
View open issues
Open a new issue
Looking to contribute to Apache Iceberg? See Contributing for more details on how to get started.
[H2] Connect with Community Events?
The Apache Iceberg community regularly meets for technical discussion both in-person and virtually.
[H3] Apache Iceberg Community Calendar?
The community maintains two calendar feeds:
Iceberg Dev Events: A shared calendar for all Iceberg development syncs, including the triweekly community sync, subproject syncs for Python, Rust, and Go, the Iceberg Catalog Community Sync, and other ad hoc syncs the community may want to schedule or discuss.
Iceberg Community Events: Events such as conferences and meetups, aimed to educate and inspire Iceberg users.
[H3] Hosting an Apache Iceberg Meetup?
Meetups related to the Apache Iceberg project are held around the globe thanks to community volunteers. Interested individuals are encouraged to host community meetups using the name Apache Iceberg Meetup <geographical location> in
compliance with Apache Software Foundation's branding and trademarks guidelines. No explicit PMC approval is required.
Hosts are required to ensure that:
The Apache Iceberg ecosystem should be championed in every meetup and technical session
All talks should be vendor-neutral and not sales pitches
Each meetup should have at least two talks with speakers representing different companies/organizations
Planned meetups ought to be brought to the attention of the dev list
All Community Guidelines must be respected
Meetups must be small events under the ASF branding guidelines and are typically small, informal gatherings. If you're unsure whether an event is a meetup, meetups usually:
Rely on a curated selection of talks, where organizers work with the community to include a diverse representation of speakers, topics, and companies. Due to the small number of submissions, meetups might use, but don't require, a CfP.
Are single-tracked and accommodate only a handful of sessions.
Have at most 2-3 hours of content, plus networking time.
Are sponsored by 1-3 companies providing food, drinks, or meeting space; not by selling booth space or marketing opportunities. They don't require substantial financial support.
If you don't know whether an event qualifies as a meetup, please ask the PMC through the private mailing list! (Be sure to do this before using the Apache Iceberg brand or trademark.)
[H2] Community Guidelines?
[H3] Apache Iceberg Community Guidelines?
The Apache Iceberg community is built on the principles described in the Apache Way
and all who engage with the community are expected to be respectful, open, come with the best interests of the community in mind,
and abide by the Apache Software Foundation Code of Conduct.
More information specific to the Apache Iceberg community is in the next section, the Path from Contributor to Committer.
[H3] Participants with Corporate Interests?
A wide range of corporate entities have interests that overlap in both features and frameworks related to Iceberg and while we
encourage engagement and contributions, the community is not a venue for marketing, solicitation, or recruitment.
Any vendor who wants to participate in the Apache Iceberg community Slack workspace should create a dedicated vendor channel
for their organization prefixed by vendor-.
This space can be used to discuss features and integration with Iceberg related to the vendor offering. This space should not
be used to promote competing vendor products/services or disparage other vendor offerings. Discussion should be focused on
questions asked by the community and not to expand/introduce/redirect users to alternate offerings.
[H3] Marketing / Solicitation / Recruiting?
The Apache Iceberg community is a space for everyone to operate free of influence. The development lists, slack workspace,
and github should not be used to market products or services. Solicitation or overt promotion should not be performed in common
channels or through direct messages.
Recruitment of community members should not be conducted through direct messages or community channels, but opportunities
related to contributing to or using Iceberg can be posted to the #jobs channel.
For questions regarding any of the guidelines above, please contact a PMC member.
[H2] The Path from Contributor to Committer?
Many contributors have questions about how to become a committer. This section outlines what committers do and how they are invited.
[H3] What are the responsibilities of a committer??
In the Iceberg project, committers are community members that can review and commit changes to Iceberg repositories. Reviewing is the primary responsibility of committers.
[H3] How are new committers added??
Starting from the foundation guidelines, committers are nominated and discussed by the PMC, which uses a consensus vote to confirm a new committer. This vote is the only formal requirement in the Iceberg community — there are no other requirements, such as a minimum period of time or a minimum number of contributions. Similarly, there is no length of time or number of commits that automatically qualify someone to be a committer.
Committers are added when someone has built trust with PMC members that they have good judgment and are a reliable reviewer.
[H3] What does the PMC look for??
PMC members typically look for candidates to have demonstrated a few qualities:
Conduct — Committers are representatives of the project and are expected to follow the ASF Code of Conduct.
Judgment — Committers should know the areas where they are qualified to evaluate a change and when to bring in other opinions.
Quality — Personal contributions are a strong signal. Contributions that don’t require major help demonstrate the context and understanding needed to reliably review changes from others. If a contributor often needs guidance, they are probably not ready to guide others.
Consistency — Reviewing is the primary responsibility of a committer. A committer should demonstrate they will consistently apply their context and understanding to help contributors get changes in and ensure those changes are high quality.
[H3] How do I demonstrate those qualities??
To be a committer, a candidate should act like a committer so that PMC members can evaluate the qualities above. PMC members will ask questions like these:
Has the candidate been a good representative of the project in mailing lists, Slack, github, and other discussion forums?
Has the candidate followed the ASF Code of Conduct when working with others?
Has the candidate made independent material contributions to the community that show expertise?
Have the candidate’s contributions been stable and maintainable?
Has the candidate’s work required extensive review or significant refactoring due to misunderstandings of the project’s objectives?
Does the candidate apply the standards and conventions of the project by following existing patterns and using already included libraries?
Has the candidate participated in design discussions for new features?
Has the candidate asked for help when reviewing changes outside their area of expertise?
How diverse are the contributors that the candidate reviewed?
Does the candidate raise potentially problematic changes to the dev list?
[H3] How can I be a committer??
You can always reach out to PMC members for feedback and guidance if you have questions.
There is no single path to becoming a committer. For example, people contributing to Python are often implicitly trusted not to start reviewing changes to other languages. Similarly, some areas of a project require more context than others.
Keep in mind that it’s best not to compare your contributions to others. Instead, focus on demonstrating quality and judgment.
[H3] How many contributions does it take to become a committer??
The number of contributions is not what matters — the quality of those contributions (including reviews!) is what demonstrates that a contributor is ready to be a committer.
You can always reach out to PMC members directly or using private@iceberg.apache.org for feedback and guidance if you have questions.

Back to top

[H1] Evolution?
Iceberg supports in-place table evolution. You can evolve a table schema just like SQL -- even in nested structures -- or change partition layout when data volume changes. Iceberg does not require costly distractions, like rewriting table data or migrating to a new table.
For example, Hive table partitioning cannot change so moving from a daily partition layout to an hourly partition layout requires a new table. And because queries are dependent on partitions, queries must be rewritten for the new table. In some cases, even changes as simple as renaming a column are either not supported, or can cause data correctness problems.
[H2] Schema evolution?
Iceberg supports the following schema evolution changes:
Add -- add a new column to the table or to a nested struct
Drop -- remove an existing column from the table or a nested struct
Rename -- rename an existing column or field in a nested struct
Update -- widen the type of a column, struct field, map key, map value, or list element
Reorder -- change the order of columns or fields in a nested struct
Iceberg schema updates are metadata changes, so no data files need to be rewritten to perform the update.
Note that map keys do not support adding or dropping struct fields that would change equality.
[H3] Correctness?
Iceberg guarantees that schema evolution changes are independent and free of side-effects, without rewriting files:
Added columns never read existing values from another column.
Dropping a column or field does not change the values in any other column.
Updating a column or field does not change values in any other column.
Changing the order of columns or fields in a struct does not change the values associated with a column or field name.
Iceberg uses unique IDs to track each column in a table. When you add a column, it is assigned a new ID so existing data is never used by mistake.
Formats that track columns by name can inadvertently un-delete a column if a name is reused, which violates #1.
Formats that track columns by position cannot delete columns without changing the names that are used for each column, which violates #2.
[H2] Partition evolution?
Iceberg table partitioning can be updated in an existing table because queries do not reference partition values directly.
When you evolve a partition spec, the old data written with an earlier spec remains unchanged. New data is written using the new spec in a new layout. Metadata for each of the partition versions is kept separately. Because of this, when you start writing queries, you get split planning. This is where each partition layout plans files separately using the filter it derives for that specific partition layout. Here's a visual representation of a contrived example:
[IMG: Partition evolution diagram]
The data for 2008 is partitioned by month. Starting from 2009 the table is updated so that the data is instead partitioned by day. Both partitioning layouts are able to coexist in the same table.
Iceberg uses hidden partitioning, so you don't need to write queries for a specific partition layout to be fast. Instead, you can write queries that select the data you need, and Iceberg automatically prunes out files that don't contain matching data.
Partition evolution is a metadata operation and does not eagerly rewrite files.
Iceberg's Java table API provides updateSpec API to update partition spec.
For example, the following code could be used to update the partition spec to add a new partition field that places id column values into 8 buckets and remove an existing partition field category:
Table sampleTable = ...;
sampleTable.updateSpec()
.addField(bucket("id", 8))
.removeField("category")
.commit();
Spark supports updating partition spec through its ALTER TABLE SQL statement, see more details in Spark SQL.
[H2] Sort order evolution?
Similar to partition spec, Iceberg sort order can also be updated in an existing table.
When you evolve a sort order, the old data written with an earlier order remains unchanged.
Engines can always choose to write data in the latest sort order or unsorted when sorting is prohibitively expensive.
Iceberg's Java table API provides replaceSortOrder API to update sort order.
For example, the following code could be used to create a new sort order
with id column sorted in ascending order with nulls last,
and category column sorted in descending order with nulls first:
Table sampleTable = ...;
sampleTable.replaceSortOrder()
.asc("id", NullOrder.NULLS_LAST)
.dec("category", NullOrder.NULL_FIRST)
.commit();
Spark supports updating sort order through its ALTER TABLE SQL statement, see more details in Spark SQL.

Back to top

[H1] Reliability?
Iceberg was designed to solve correctness problems that affect Hive tables running in S3.
Hive tables track data files using both a central metastore for partitions and a file system for individual files. This makes atomic changes to a table's contents impossible, and eventually consistent stores like S3 may return incorrect results due to the use of listing files to reconstruct the state of a table. It also requires job planning to make many slow listing calls: O(n) with the number of partitions.
Iceberg tracks the complete list of data files in each snapshot using a persistent tree structure. Every write or delete produces a new snapshot that reuses as much of the previous snapshot's metadata tree as possible to avoid high write volumes.
Valid snapshots in an Iceberg table are stored in the table metadata file, along with a reference to the current snapshot. Commits replace the path of the current table metadata file using an atomic operation. This ensures that all updates to table data and metadata are atomic, and is the basis for serializable isolation.
This results in improved reliability guarantees:
Serializable isolation: All table changes occur in a linear history of atomic table updates
Reliable reads: Readers always use a consistent snapshot of the table without holding a lock
Version history and rollback: Table snapshots are kept as history and tables can roll back if a job produces bad data
Safe file-level operations. By supporting atomic changes, Iceberg enables new use cases, like safely compacting small files and safely appending late data to tables
This design also has performance benefits:
O(1) RPCs to plan: Instead of listing O(n) directories in a table to plan a job, reading a snapshot requires O(1) RPC calls
Distributed planning: File pruning and predicate push-down is distributed to jobs, removing the metastore as a bottleneck
Finer granularity partitioning: Distributed planning and O(1) RPC calls remove the current barriers to finer-grained partitioning
[H2] Concurrent write operations?
Iceberg supports multiple concurrent writes using optimistic concurrency.
Each writer assumes that no other writers are operating and writes out new table metadata for an operation. Then, the writer attempts to commit by atomically swapping the new table metadata file for the existing metadata file.
If the atomic swap fails because another writer has committed, the failed writer retries by writing a new metadata tree based on the new current table state.
[H3] Cost of retries?
Writers avoid expensive retry operations by structuring changes so that work can be reused across retries.
For example, appends usually create a new manifest file for the appended data files, which can be added to the table without rewriting the manifest on every attempt.
[H3] Retry validation?
Commits are structured as assumptions and actions. After a conflict, a writer checks that the assumptions are met by the current table state. If the assumptions are met, then it is safe to re-apply the actions and commit.
For example, a compaction might rewrite file_a.avro and file_b.avro as merged.parquet. This is safe to commit as long as the table still contains both file_a.avro and file_b.avro. If either file was deleted by a conflicting commit, then the operation must fail. Otherwise, it is safe to remove the source files and add the merged file.
[H2] Compatibility?
By avoiding file listing and rename operations, Iceberg tables are compatible with any object store. No consistent listing is required.

Back to top