Everyone knows they should be doing a better job monitoring AWS costs. Almost nobody actually does. It’s not for lack of interest:even basic tools like AWS Cost Explorer make it surprisingly hard to understand what’s going on, especially once usage grows beyond a small setup.

At Twing Data, set out to build something dead simple: point a dashboard at S3, automatically ingest AWS Cost and Usage Report (CUR) files, and instantly get a useful set of dashboards. We wanted to avoid complex modeling, configuration, various ingestions and transformations, etc. The goal was to keep it as simple as possible so you can quickly see where your money is going, how it’s trending, and where to dig deeper.

We’ve gotten to know the Rill team from working with them on some projects and thought they’d be a great fit for this.

What you get out of the box

With this setup, you get a working cost dashboard almost immediately:

• Automatic ingestion and partitioning of AWS Cost and Usage Report (CUR) data from S3

• Default dashboards showing cost trends and breakdowns

• Fast slice-and-dice without writing SQL

• An AI chat layer for ad-hoc questions and analysis

The goal is to give a V1 of a robust cost monitoring system that gives you a clear, usable snapshot of costs with minimal effort and then lets you go deeper if and when you need to.

Why Rill works well here

A few core Rill features do most of the heavy lifting:

1. Automatic ingestion from S3: Point Rill at your CUR bucket and it handles loading new files as they arrive.

2. Partition-aware updates: New cost files are ingested incrementally on a schedule, so the dashboard stays current without reprocessing everything.

3. ClickHouse under the hood: This scales far beyond “toy demo” size. You can throw very large CUR datasets at it without worrying about performance. Rill also provides a nice way to test changes on “dev” before pushing them to production.

Getting started

Note: If you do not already export your Cost and Usage Reports to a bucket follow this quick setup.

1. In console search for `Billing and Cost Management` > `Data Exports`

2. Select Create

   1. Legacy CUR export

   2. Include resource IDs

   3. Refresh automatically with your expected cadence Hourly, Daily, Monthly

3. Configure the bucket to share with Rill

The repo lives here: https://github.com/Twing-Data/aws-cost-dashboard

The setup is deliberately minimal:

1. Clone the repo

2. Configure your .env file with AWS and S3 settings

3. Run rill start

Or in code:

Install rill if you don’t have it:

curl https://rill.sh | sh
git clone https://github.com/Twing-Data/aws-cost-dashboard.git
cd aws-cost-dashboard
rill start

That’s it. Once Rill starts, it ingests the data and serves the dashboard locally.

Defaults first, simple customization

The dashboards are built on top of standard AWS CUR fields so they’re useful right away. In practice, most teams tag their resources to provide more visibility and measurement across applications, teams, environments, and projects.

Rill makes this easy to extend: you modify the YAML files in the sources, metrics, and dashboards folders:

• Define new fields in the source

• Reference them in metrics

• Use them in dashboards

Once added, those fields are immediately available everywhere. Rill avoids the complex overhead of complex modeling or orchestration and quickly gives you a useful dashboard. You can read more docs here to see it in action and all the options Rill offers: https://docs.rilldata.com/build/dashboards

Augmenting with AI

Everyone is incorporating AI into their products and Rill is no exception. Rill includes an AI chat layer that lets you ask questions directly against the dataset. The nice thing is that because it’s tightly integrated with the rest of the platform you get dashboards that link back to the original data sources so you can verify the numbers, a good way to handle the hallucination risk and AI audit trail.

You can also do things like select a portion of the chart that has noisy or interesting data and ask Rill to do a root cause analysis.

This project was about minimizing setup cost while still delivering something useful. You can be up and running quickly, get useful visibility into AWS costs, and still have a path to deeper analysis and customization as needs grow.

If AWS cost monitoring has felt heavier than it should be, this is a much lighter starting point. Let us know if you end up giving it a try.

Apache Iceberg has rapidly become one of the most talked-about technologies in data architecture over the past few years. As organizations shift towards data lakehouse designs that combine the scalability of data lakes with the reliability of data warehouses, open table formats like Iceberg play a pivotal role. Apache Iceberg is an open-source table format that brings SQL table-like functionality (transactions, schema management, etc.) to large analytic datasets on inexpensive storage (e.g. cloud object stores). Its rise in popularity is driven by the promise of high-performance analytics without vendor lock-in, allowing multiple processing engines to work reliably on the same data. This report delves into what Iceberg is and why it’s in high demand, examines the latest integration with DuckDB (as of April 2025), explores ideal technology combinations for lakehouse, streaming, and analytics use cases (in cloud and hybrid environments), and surveys other key technologies in the Iceberg ecosystem (Spark, Trino, Flink, Snowflake, BigQuery, etc.). We also highlight tooling and adoption by major organizations (especially for AI/ML workflows), and provide summary-level insights into Iceberg’s architecture and performance that explain how it all works together.

What is Apache Iceberg and Why Is It in High Demand?

Apache Iceberg is a high-performance open table format for huge analytic tables, originally developed at Netflix and now an Apache Software Foundation project. In essence, Iceberg is a “smart” layer between raw storage (files in formats like Parquet or ORC) and compute engines. It tracks table data and metadata to ensure that multiple analytics applications can all see a consistent view of data and perform ACID transactions on a data lake. This effectively brings the reliability and simplicity of traditional SQL tables to big data stored in distributed files.

Core Features of Apache Iceberg:

• ACID Transactions & Snapshot Isolation: Iceberg allows atomic adds, deletes, and updates to table data, even with concurrent writers, ensuring readers never see partial updates. Each commit creates a new snapshot of the table, and readers can isolate to a specific snapshot, so queries are always consistent and never see “in-progress” data. This means multiple engines or jobs can write to the same table safely, and readers won’t get corrupted or “zombie” data.
  

• Schema Evolution: Iceberg natively supports evolving the table schema over time (adding, dropping, or renaming columns) without cumbersome rebuilds. It tracks schema changes as part of the metadata. This reliable schema evolution avoids the headaches of older Hive-style tables where schema changes could be error-prone.
  

• Time Travel & Rollbacks: Every snapshot of the table is retained, enabling time travel queries – you can query data as of a particular snapshot or time in the past. This is invaluable for auditing and reproducing experiments. It’s also possible to rollback a table to a previous state instantly if a bad data load occurs.
  

• Hidden Partitioning & Evolution: Iceberg manages partition information internally rather than relying on physical folder structures. This means users get automatic partition pruning for queries, and partition schemes can be changed as data and query patterns evolve. There’s no need for manual “repair” or handling of Hive partitions – no more expensive directory listings or strict naming conventions. This dramatically reduces the load on metastores and speeds up query planning for large tables.
  

• Metadata for Performance: Iceberg maintains detailed metadata (in manifest files and manifest lists) about the locations and statistics of data files. Query engines leverage this for advanced planning and filtering, skipping over unrelated files and only reading the necessary subsets of data. For example, Iceberg can avoid scanning thousands of file paths on cloud storage and instead quickly identify which files contain relevant data, eliminating the costly “full directory scan” that plagued older data lakes.
  

• Scalability to Petabytes: Iceberg is designed for petabyte-scale datasets that are “slow-moving” (very large but append/modify occasionally). By tracking individual data files and optimizing how new data is added, Iceberg can handle tables with millions of files and enormous throughput. It also supports features like compaction (merging small files) and split planning to maintain performance at scale.

These features make Iceberg extremely attractive to data engineering teams. It directly addresses the limitations of earlier “open table” approaches (e.g. Hive tables or home-grown metadata), delivering database-like reliability on a data lake. As a result, demand for Iceberg has surged. Organizations want the flexibility of keeping data in open file formats on cheap storage, but with the manageability and trust of a warehouse. Iceberg provides this balance in a fully open standard, avoiding lock-in to any single vendor or engine.

Why the High Demand? A few key reasons stand out:

• Multi-Engine Interoperability: Iceberg was built to be engine-agnostic – many different processing engines can read and write Iceberg tables and all see the same consistent data. This “define once, use anywhere” approach means companies can adopt best-of-breed tools for each job (Spark for ETL, Trino for SQL, etc.) without duplicating data. It unlocks the elusive goal of a true open lakehouse. The excitement around Iceberg has grown as major engines and platforms announced support, confirming it’s not a niche technology. By 2025, virtually every big data platform either supports Iceberg or has plans to: Databricks, Snowflake, Google BigQuery, Amazon Athena/Glue, Apache Spark, Flink, Trino, Presto, and more have all added Iceberg compatibility. This broad adoption across the industry has accelerated interest in Iceberg as a new standard.
  

• Reliability and Performance Improvements: Early adopters like Netflix, Airbnb, and LinkedIn report significant improvements in data reliability and query performance after migrating to Iceberg. For example, Airbnb eliminated heavy Hive Metastore load and latency by replacing Hive tables with Iceberg (no more listing thousands of S3 partitions). Netflix was able to build a streamlined incremental ETL pipeline with Iceberg that improved data freshness and accuracy for its analysts and data scientists. These success stories have made Iceberg a go-to solution for organizations struggling with “data swamp” issues in their data lakes.
  

• Scale for Modern Analytics and AI: Iceberg’s ability to efficiently manage petabyte-scale datasets and keep query speeds high is crucial for modern analytics, including AI/ML workloads. Machine learning models often need access to large historical datasets for training; Iceberg’s time travel and optimized storage layout make those accesses both faster and cheaper. The format ensures data consistency and quality (no partial updates, no corruption), which is critical for reproducible ML experiments. As one analysis notes, Iceberg’s robust framework for managing large datasets leads to faster and more accurate model training and inference, and this is a big reason it’s becoming a preferred choice for data lakes aimed at AI. Companies like Apple, Stripe, Expedia, Adobe, and LinkedIn have all embraced Iceberg in parts of their data platforms, underlining its effectiveness and versatility.
  

In short, Apache Iceberg brings order to the chaos of big data file lakes. It provides warehouse-grade features in an open format, enabling a new generation of data architecture where various tools can share a single source of truth. This unique value proposition explains the intense demand and rapid growth in adoption we’re witnessing.

DuckDB Meets Iceberg: New Integration in April 2025

One of the most exciting recent developments is the integration of DuckDB with Apache Iceberg. DuckDB is an in-process analytical SQL database (often called “SQLite for analytics”) known for its speed and ease of use, especially in data science workflows. As of early 2025, DuckDB gained the ability to read Iceberg tables directly, including those stored remotely on cloud object storage. This new feature, introduced as a preview in DuckDB version 1.2.x, allows users to attach Iceberg catalogs and query them with DuckDB’s SQL engine.

Technical Capabilities: DuckDB’s Iceberg integration is provided via an iceberg extension. Initially, DuckDB could read Iceberg tables (since late 2023) if pointed to local metadata. The April 2025 update adds support for Iceberg REST Catalogs, which is a big leap forward. An Iceberg REST catalog is a service (defined by the Iceberg community) that exposes table metadata through a REST API. DuckDB can now ATTACH to such a catalog with a single command, even if the data and metadata reside in cloud storage. In practical terms, this means DuckDB can connect to Amazon S3-hosted Iceberg tables or those managed via AWS Glue Catalog (which is used by AWS’s “S3 Tables” and SageMaker Lakehouse) as if they were local databases. For example, a user can run:

ATTACH 'arn:aws:s3tables:us-east-1:123456789012:bucket/my-iceberg-bucket' AS mydb

(TYPE iceberg, ENDPOINT_TYPE s3_tables);

and immediately query mydb.my_table in DuckDB, pulling data from S3. DuckDB handles the Iceberg metadata and reads the Parquet files over HTTP/S3. It even supports Iceberg’s schema evolution – if the table’s schema is altered by another engine, DuckDB will pick up the change on next query, as demonstrated in tests with adding new columns.

This integration was developed in collaboration with AWS and coincided with AWS’s own announcements of Iceberg support. In March 2025, AWS introduced Amazon S3 Tables (an Iceberg-backed table service on S3) and its integration with SageMaker Lakehouse (AWS Glue as the metadata catalog). DuckDB’s team worked with AWS to ensure DuckDB could serve as an “end-to-end solution” for reading those Iceberg tables. The result is that DuckDB can query Iceberg data natively, without any Spark or Hive in the middle, bringing truly democratized access to lakehouse data.

Practical Implications: This development has several important implications:

• Lightweight Analytics on Data Lakes: Now a data analyst with a Jupyter notebook can connect to a massive Iceberg table on S3 using DuckDB on their laptop. There’s no need to spin up a heavy Spark cluster or Trino service for interactive analysis. This lowers the barrier to entry for ad-hoc analytics on the data lake. DuckDB’s efficient vectorized engine can handle surprisingly large queries, especially with its ability to push down filters to the Iceberg metadata (e.g. it will only fetch relevant Parquet files). In short, cheap, user-friendly SQL analytics directly on the data lake becomes possible.
  

• Integration with Python/AI Workflows: DuckDB is often embedded in Python environments. With Iceberg support, Python data science workflows (Pandas, Polars, etc.) can pull in fresh data from the lakehouse on demand via DuckDB, all while respecting snapshot consistency. For AI/ML experiments, this means you can directly query feature data or training data that’s stored in Iceberg format, using simple SQL, and get the benefit of Iceberg’s versioning (e.g. load the exact snapshot of data that a model was trained on).
  

• Cloud/On-Prem Hybrid Access: Because DuckDB is just a library, you could run it anywhere – on your local machine, on an edge server, or within a cloud function – and attach to the same Iceberg table in the cloud. This is a form of “serverless” or on-the-fly data access. For instance, a nightly on-prem job could use DuckDB to pull some metrics from a cloud data lake table. The Iceberg abstraction makes the remote data appear as a regular SQL table.
  

• Still Read-Only (for Now): The current DuckDB Iceberg integration is focused on reading tables. The preview is experimental and does not yet support writing back to Iceberg. In practice, this is not a huge limitation – writes are typically handled by heavier ETL processes (Spark/Flink) – and DuckDB can be used for fast querying and prototyping. The DuckDB team has indicated that a fully stable release will come later in 2025, likely with more features.

Overall, the DuckDB-Iceberg integration exemplifies the openness of Iceberg: even an in-process database can join the party of engines working on the same data. It underscores the trend of convergence between analytics tools – cloud data in Iceberg format can be accessed by cloud data warehouses, big data engines, and now even local analytics libraries. This flexibility is exactly what modern data architectures are aiming for.

Building a Lakehouse with Iceberg: Cloud and Hybrid Patterns

Apache Iceberg is a cornerstone of the data lakehouse architecture. A lakehouse typically consists of a unified storage layer (like a data lake on cloud storage or HDFS) with an open table format, and multiple compute engines for different workloads (ETL, ad-hoc SQL, machine learning, etc.). Iceberg enables this by acting as the table format and governance layer on the storage, ensuring all engines see consistent data. Let’s explore some ideal technology combinations for lakehouse and analytics use cases, in both cloud and hybrid environments:

Storage Layer: In a modern lakehouse, the storage is often cheap, scalable cloud object storage (e.g. Amazon S3, Google Cloud Storage, Azure Data Lake Storage) or a distributed filesystem (HDFS) for on-prem. Iceberg sits on top of this storage, managing files (typically in Parquet/ORC format) and tracking metadata. The Iceberg catalog (backed by Hive Metastore, AWS Glue, a REST service, etc.) serves as the source of truth for table definitions and snapshots.

Compute Engines: With Iceberg in place, you can mix and match engines, each specialized for a purpose:

• Batch ETL and Processing: Apache Spark is commonly used for heavy batch transformations. For example, Spark jobs can read raw data, join and aggregate, and then write the results into an Iceberg table (on S3 or HDFS) as the sink. Iceberg’s format ensures each Spark job’s output is atomically added as a new snapshot. Other engines like Flink or SQL-based ELT tools can also produce data into Iceberg. The key is that the ingest engine can be chosen freely – as long as it speaks Iceberg, it can write to the lakehouse.
  

• Interactive SQL and BI: For analytics and BI queries, a SQL query engine like Trino (Presto) is often layered on top. Trino has a native Iceberg connector, so it can query the tables Spark produced, with full support for partition pruning, filtering, even updates and deletes. Because Iceberg tracks precise data file stats, engines like Trino can perform interactive queries efficiently on huge datasets. Even cloud data warehouses like Snowflake or BigQuery can play this role by querying the Iceberg tables (more on that later). The benefit is that business analysts or tools (Tableau, etc.) can run fast queries on the latest data in the lake without moving it to a separate warehouse.
  

• Data Science/ML Access: For machine learning model training or data exploration, teams might use Python or R environments. Instead of extracting data from a warehouse, they can directly load features or training sets from the Iceberg tables – either via DuckDB (as discussed) or Spark/PySpark, etc. This ensures that everyone (analytics or ML) is working off the same single source of truth on the lake.

Cloud-Native Lakehouse Example (AWS): Imagine a stack on AWS – data is stored in Amazon S3, and Iceberg tables are tracked in AWS Glue Catalog (or the new AWS “S3 Tables” service). Apache Spark on EMR might perform nightly batch ETL, writing data to these Iceberg tables. Analysts then use Trino (Athena) to run SQL queries directly on those Iceberg tables for interactive analysis. Because Iceberg is ACID, the Athena users never see partial writes; they only query committed snapshots. If needed, a Redshift Spectrum or Snowflake external table could also query the same data in S3. In this setup, S3 + Iceberg acts like the “data lakehouse” storage, and multiple query engines coexist on it. This provides warehouse-like capabilities (SQL, transactions) without all data having to reside inside a single proprietary warehouse, offering flexibility and cost advantages.

Hybrid Scenario: One of Iceberg’s strengths is in hybrid cloud environments. For instance, consider a company that has on-premises data centers and also uses cloud analytics services. They could keep an Iceberg table’s data in an on-prem HDFS cluster, but use a shared catalog so that cloud engines can also access it. Tools like Apache Hive or Spark on-prem might produce the data, but then a cloud service like Google BigQuery (via BigLake) or Snowflake can query that same table remotely. Because the Iceberg catalog is the authority, even across environments, everyone sees the same schema and snapshots. LinkedIn provides a real-world example: they ingest streams from Kafka and Oracle into Hadoop, writing data into HDFS in ORC format managed by Iceberg. Iceberg’s metadata gives them snapshot isolation and incremental processing for downstream consumers on that data. This means LinkedIn’s various pipelines (even outside of Hadoop) can consume the data gradually as new snapshots appear, without confusion or conflict. In general, Iceberg’s open format and catalog options (Hive, REST, etc.) enable data sharing across on-prem and cloud. A team can migrate or burst to cloud engines step by step, without migrating the entire dataset at once, since cloud and on-prem can both operate on the Iceberg table in parallel.

To summarize these patterns, we present a few typical Iceberg-based stack combinations and their benefits:

In all these scenarios, Iceberg provides the common table layer that knits the stack together. Whether it’s batch or streaming, cloud or on-prem, the decoupling of storage and compute that Iceberg enables is a game-changer. Data resides once (in open files), and many tools can use it in their own way – yet data governance, schema, and consistency are centrally maintained by Iceberg. This architecture is a key reason organizations are moving towards Iceberg-based lakehouses as a future-proof, flexible data infrastructure.

Streaming Data and Real-Time Analytics with Iceberg

Traditional data lakes were often batch-oriented, but Apache Iceberg also supports modern streaming and real-time data pipelines. By leveraging Iceberg’s design, we can build streaming systems that land data into analytical tables continuously, enabling up-to-date analytics without sacrificing consistency. There are a few dimensions to how Iceberg is used in streaming contexts:

Streaming Ingest (Write) into Iceberg: Tools like Apache Flink and Spark Structured Streaming can write to Iceberg tables in a streaming fashion. Rather than huge batch dumps, data can be ingested continuously into the Iceberg table as small batches or micro-streams. For example, Flink has an Iceberg sink connector that will periodically commit new data to the table (each commit becomes a new snapshot). This allows near real-time data to accumulate in the lakehouse. An important advantage of using Iceberg here is that even though data arrives continuously, each Flink commit is atomic – if you query the table, you either see the new batch entirely or not at all. Downstream jobs or analysts don’t see half-written data. This is crucial for data quality in streaming. It effectively brings the reliability of a database to a data stream landing in cloud storage.

• Comparison to Direct Kafka Usage: Often, streaming architectures pipe data into Kafka for real-time and then later batch it into a lake. With Iceberg, some use cases can eliminate the extra queue: you can stream directly into an Iceberg table and treat that as both the source of truth and the serving layer for analytics. Steven Wu (an engineer at Apple) discussed how an “Iceberg streaming source” can power many common stream processing use cases, comparing it to using Kafka as a source. The takeaway is that while Kafka is great for ultra-low latency and pub/sub, Iceberg can serve in scenarios where a few seconds or minutes latency is acceptable in exchange for far simpler architecture and lower cost (since everything lands in the analytic store directly). Flink’s Iceberg integration is designed to allow incremental commits to Iceberg, meaning you can get latency in the order of seconds to a minute, yet have all the data neatly stored in Parquet files and immediately queryable by SQL engines.
  

• Incremental Consumption (Read) from Iceberg: The flip side is reading Iceberg in a streaming manner. Iceberg’s table format supports incremental scan, where a processing engine can ask “give me all new data since snapshot X”. Apache Flink leverages this to create Iceberg source streams. In other words, Flink can treat an Iceberg table as if it were a continually updating source – each new snapshot in the table is like an event that Flink can capture and process further. This is a form of change data capture (CDC) or incremental ETL. For example, if one job is writing to an Iceberg table, another Flink job could monitor that table and immediately process new rows for downstream systems, achieving an end-to-end streaming pipeline entirely on Iceberg. This capability shows how Iceberg isn’t just for static datasets; it can be part of streaming architectures, acting almost like a message bus with storage.
  

Event Streaming + Iceberg in Tandem: Many real-world architectures combine event streams (Kafka, Kinesis) with Iceberg. A common pattern is: Kafka for immediate processing and low-latency data (seconds), and Iceberg for durable, queryable storage of events (minutes and beyond). Tools like Flink can read from Kafka, do some transformations, and continuously upsert into an Iceberg table. The Iceberg table then becomes a queryable history of the stream. This is very useful for late arrivals and backfills – things that are hard to manage in a pure event stream can be naturally handled in Iceberg (since you can always write a correction record or compact data in Iceberg). Netflix’s use case aligns with this: they built an incremental processing solution where data flows through their stream processing (Maestro workflow) into Iceberg, which provides a scalable way to do backfills and guarantee accuracy without reprocessing everything from scratch.

Real-Time Analytics: Once streaming data lands in Iceberg, it opens the door for real-time analytics on the lakehouse. For instance, you might have a dashboard running Presto/Trino queries on an Iceberg table that is updated by Flink every few minutes. You get “near-real-time BI” without a separate real-time database. Iceberg ensures that each query sees a consistent snapshot. If the dashboard is set to refresh every 5 minutes, and Flink commits new data every minute, the viewers will always see a point-in-time consistent view. They might be, say, one minute behind the latest data – often an acceptable trade-off.

Another emerging concept is using Iceberg as a materialized view or serving layer for streams. Because Iceberg can organize data by time and partition, a well-designed Iceberg table can serve both recent data fast and also provide the historical depth. Companies like Confluent (with Kafka and Flink) talk about leveraging Iceberg to unify streaming and batch views of data. Instead of maintaining both a real-time store and a separate batch store, Iceberg can act as the singular store that is continuously updated (sometimes this approach is called a “unified log” or Lakehouse with streaming).

Latency and Considerations: It’s important to note that Iceberg-based streaming typically has slightly higher latency than pure event streaming (due to the time to buffer and create file commits). However, it greatly simplifies architectures for many analytics use cases that don’t require sub-second latency. It’s a more cost-efficient approach for high throughput streams where storing long histories in Kafka would be expensive. By using Iceberg, all that data is immediately available for SQL queries and ML, which is not true for raw event logs.

In summary, Iceberg extends its usefulness to streaming scenarios by providing a framework to ingest, store, and consume continuous data with the same robustness as batch data. This convergence of streaming and batch in the lakehouse is a powerful concept: one platform for both real-time and historical analytics. As tooling (like Flink, Spark streaming, Kafka Connect, etc.) continues to improve around Iceberg, we can expect more and more “real-time lakehouse” deployments in production.

Integration with Other Key Technologies and Engines

Apache Iceberg’s success is largely due to its broad ecosystem integration. It is not a standalone system; it works in concert with many engines and platforms. Here we provide in-depth information on how several key technologies integrate with Iceberg and what capabilities each combination offers:

Apache Spark

Apache Spark was one of the first engines to integrate with Iceberg (indeed, Iceberg was born out of Netflix’s use of Spark with petabyte datasets). Spark provides an Iceberg connector (available as a package iceberg-spark for Spark 3.x) that allows Spark SQL and DataFrame APIs to read and write Iceberg tables natively. This means you can do spark.read.format("iceberg").load("db.table") to load an Iceberg table into a DataFrame, or write DataFrames out to Iceberg with full support for partitioning, schema, etc. Spark + Iceberg offers a powerful combination:

• ETL and Batch Writes: Spark is commonly used to perform large-scale ETL into Iceberg. It can create or replace entire partitions, or append new data, using Iceberg’s API to commit changes. Because Iceberg ensures atomic commits, multiple Spark jobs can safely write to different tables (or even the same table in some cases with proper isolation) without interfering.
  

• SQL Support: Spark SQL understands Iceberg tables as first-class citizens. One can write SQL to query Iceberg tables (Spark will push down predicates to Iceberg for file pruning). Spark 3 even supports DML operations on Iceberg tables: you can MERGE INTO, UPDATE or DELETE from an Iceberg table using standard ANSI SQL syntax, and Spark will translate that into Iceberg’s procedures (creating new data files and delete files under the hood). This brings data warehouse-like data management directly to your data lake.
  

• Streaming Reads/Writes: Spark Structured Streaming can read from an Iceberg table as a source (treating new snapshots as increments) and also write to Iceberg as a sink. This isn’t as real-time as Flink typically, but it allows Spark to be part of streaming pipelines with Iceberg.
  

• Use in Databricks: Notably, Databricks (the company behind Delta Lake) has also added support for Iceberg in its platform. This means on a Databricks cluster you could choose to use Iceberg instead of Delta for certain tables. Databricks has even open-sourced some adaptors to ensure their runtime works efficiently with Iceberg. This is a testament to Iceberg’s popularity – even a would-be competitor like Databricks is embracing it because customers use it.
  

• Migration of Hive Tables: A big use case for Spark + Iceberg is migrating legacy Hive tables into Iceberg format. Spark has utilities to snapshot existing Parquet datasets into Iceberg without data copy. Once under Iceberg, the old Hive table’s issues (like too many partitions) disappear. Many companies (as noted earlier) have done exactly this to solve Hive metastore bottlenecks.

In short, Spark serves as both a producer and consumer in the Iceberg ecosystem. It brings the muscle for heavy data lifting, and Iceberg ensures those outputs are accessible beyond Spark.

Trino / Presto

Trino (formerly PrestoSQL) is a distributed SQL query engine often used for interactive analytics and BI dashboards. Trino has a well-developed Iceberg connector which allows it to query (and modify) Iceberg tables as if they were tables in a database. Key points about Trino + Iceberg:

• Interactive Analytics: Trino is designed for low-latency SQL on large data. With Iceberg, Trino can perform interactive queries even on huge datasets because Iceberg’s metadata helps Trino skip reading a lot of unnecessary data. For example, Trino will use Iceberg’s partition stats and file stats to only read relevant files for a given query, making scans much faster than if it had to brute-force read all files. One blog noted that many engines including Trino have done significant work to optimize Iceberg queries, indicating the performance can be very high.
  

• DML and Transactions: Trino’s Iceberg connector supports not just SELECTs but also INSERT, UPDATE, DELETE, and MERGE operations on Iceberg tables (assuming the Trino cluster has the correct privileges and is configured with an Iceberg catalog). Underneath, each such operation becomes a transactional commit in Iceberg. This means you could use Trino to do data cleaning (e.g. “DELETE FROM table WHERE bad_field IS NULL”) and it would actually rewrite or remove data files in the Iceberg table, following Iceberg’s format. Such capabilities were not possible with older Hive tables in Presto – it’s the Iceberg layer that makes it possible to mutate data reliably.
  

• Multiple Catalogs: Trino can be configured to use various Iceberg catalogs. For example, you might configure one catalog that points to an HDFS-based warehouse (with a Hive Metastore), and another that points to an S3-based Iceberg warehouse (with Glue). This flexibility allows querying Iceberg tables across environments in one Trino query. Trino doesn’t care where the data is – it only needs the Iceberg API and the underlying file access.
  

• Concurrent Use with Other Engines: Because Trino is often a read-heavy engine and Spark is write-heavy, a common pattern is Spark produces data into Iceberg, and Trino serves it to analysts. They coordinate through the Iceberg catalog. As soon as Spark commits a snapshot, Trino queries will see the new data on the next snapshot refresh. If a query is mid-flight, it uses the last snapshot; the next query will use the new one. This ensures readers (Trino) and writers (Spark) don’t conflict – a form of multi-engine concurrency which Iceberg enables by design.

Trino is increasingly used as the SQL interface for the lakehouse in many companies, and Iceberg is a natural fit for it. The combination provides a feel of a “database” (Trino + Iceberg) that is actually running over open files.

(Note: PrestoDB, the Facebook-led fork of Presto, also has an Iceberg connector. However, most of the community and new features are in Trino, so we focus on Trino.)

Apache Flink

Apache Flink is a powerful stream processing engine, and its integration with Iceberg positions Iceberg as a bridge between streaming data and batch analytics. Flink’s Iceberg integration includes:

• Iceberg Sink: Flink can write data into Iceberg tables in real-time. This sink is designed to batch up small events into parquet files and commit them on the fly. It uses the Iceberg API to commit snapshots, so each Flink checkpoint can correspond to an Iceberg commit. This allows exactly-once delivery semantics from Flink into Iceberg (with careful configuration). Many folks use this to ingest data from Kafka into Iceberg continuously, as mentioned earlier. A nice feature is that Flink can also handle UPSERT streams into Iceberg (for tables with primary keys) by writing equality delete files and new rows, enabling slowly-changing data to be stored in Iceberg.
  

• Iceberg Source: Flink can treat an Iceberg table as a source of streaming data. Internally, this means Flink will periodically check the table for new snapshots and emit new rows or changes. This effectively makes the Iceberg table like a change log that Flink can further process. For example, if you have an Iceberg table that is being updated by some batch jobs, a Flink job could monitor it and index those updates into Elasticsearch, etc. This source is a newer capability (supported since Iceberg 0.11+ and Flink 1.14+); it uses Iceberg’s snapshot diff feature to only read the deltas.
  

• Unified Pipeline (Batch + Stream): Flink is unique in that it can do batch and stream in one. With Iceberg, a Flink job could run in micro-batch mode, reading an Iceberg table, doing some aggregation, and writing the result out to another Iceberg table, all continuously. Because both source and sink are Iceberg, the entire pipeline benefits from schema evolution, time travel (for reprocessing old data if needed), and no separate staging storage. This greatly simplifies architectures for complex data transformations that need to run 24/7.
  

• Integration in Confluent Platform: Confluent (known for Kafka) has been investing in Flink and mentioned “Tableflow” in some contexts – essentially using Flink+Iceberg to integrate streaming with table storage. This indicates how streaming vendors see Iceberg as important. By using Flink and Iceberg together, one can implement the concept of data as a continually updating table rather than an unbounded event log, which is a more query-friendly format for many users.

In practice, Flink + Iceberg is a popular choice for real-time data lakes. Companies like Apple, for example, have engineers (like Steven Wu in the talk) working on these integrations for their AI/ML platforms. Alibaba contributed heavily to Flink-Iceberg integration as well (for their cloud). If Spark is the king of batch on Iceberg, Flink is becoming the king of streaming on Iceberg.

Snowflake

Snowflake is a cloud data warehouse that traditionally kept data in its own proprietary storage format. But Snowflake now supports Apache Iceberg tables in a feature they sometimes call “Iceberg Tables” in Snowflake. This integration is significant because it blends the lines between data warehouse and data lake. Here’s how it works:

• External Iceberg Tables: Snowflake can query Iceberg tables that live outside of Snowflake. The data (Parquet files and Iceberg metadata) might be in an S3 bucket that Snowflake has access to. Using an External Volume (Snowflake object) to connect to the cloud storage, Snowflake can treat an Iceberg table almost like one of its own. The Iceberg catalog can be external (like AWS Glue) or Snowflake can be configured to manage the Iceberg metadata itself. In both cases, Snowflake’s compute engine will read the Parquet files directly and use Iceberg metadata for partition pruning, etc. The benefit here is that Snowflake users can join Iceberg data with their internal tables in one query. And importantly, Snowflake does not charge storage for Iceberg tables (since Snowflake isn’t storing the data) – you only pay Snowflake for compute when querying, making it an attractive option to bring your own Iceberg data.
  

• Snowflake-managed Iceberg Tables: Snowflake also introduced the ability to create a table in Snowflake that is actually stored as Iceberg in your cloud storage. In this scenario, Snowflake itself can act as the Iceberg catalog. So when you do CREATE TABLE ... USING ICEBERG in Snowflake, it might create the metadata in its own services but the data files in your S3. This gives you the best of both worlds – Snowflake’s high-performance engine and services like automatic clustering, plus data that’s in open format you can also access outside of Snowflake if needed.
  

• Functionality: Snowflake supports schema evolution and other Iceberg features on these tables. You can use Snowflake SQL to insert or update Iceberg tables. Under the hood Snowflake will generate Parquet files and update the Iceberg metadata (it’s effectively acting as an Iceberg-compatible writer). Snowflake ensures these operations are transactionally consistent via Iceberg’s atomic commit design. One limitation is that currently Snowflake’s Iceberg tables are often read-only to other engines while Snowflake is writing unless using an external catalog integration. Snowflake has something called Snowflake Open Catalog which can sync Snowflake-managed Iceberg metadata out so that outside engines (like Spark) could see it. This is evolving, but it highlights a key point: Snowflake is treating Iceberg as a first-class citizen.
  

• Cross-cloud and Hybrid: A really interesting aspect is cross-cloud querying. Because Iceberg keeps data in e.g. S3, Snowflake (which runs in AWS, Azure, GCP) can attach to an Iceberg table even if the table is stored in another cloud or region, as long as network access is in place. This decoupling can help avoid silos. For example, a Snowflake user in Azure can query an Iceberg table stored on AWS S3. Traditionally that’s hard to do without moving data. Iceberg (plus Snowflake’s external volume concept) makes it feasible.

Why use Snowflake with Iceberg? One might do this to use Snowflake’s smooth managed service and powerful optimizer on existing data lake files. It can be a bridge for organizations that have a foot in the open data lake world and a foot in the warehouse world. It also shows the confidence Snowflake has in Iceberg as a stable format. Snowflake’s documentation and updates indicate they are regularly adding new Iceberg features, meaning over time the gap between what Snowflake can do with its internal tables versus Iceberg tables is closing.

Google BigQuery

Google BigQuery has similarly embraced Iceberg, primarily through its BigLake initiative. BigLake is a capability that allows BigQuery (and other Google Cloud services) to treat external data on cloud storage with governance and performance optimizations. Iceberg is one of the formats supported:

• BigLake External Tables for Iceberg: BigQuery can create an external table pointing to an Iceberg table on Google Cloud Storage (GCS). This is read-only access – BigQuery will query the data via the Iceberg metadata, but not write to it. BigQuery applies its security and fine-grained access control on these external tables, integrating with GCP’s unified controls. The performance is improved over naive external queries because BigQuery can use Iceberg’s partition pruning and maybe pushdown filters (though BigQuery may still need to spin up some execution to read the Parquet files). This allows federated querying, where BigQuery SQL can combine data from Iceberg tables and native BigQuery tables.
  

• BigQuery Iceberg Tables (Managed): In 2025, Google announced BigQuery support for Iceberg Metastore functionality. Essentially, Google introduced a BigQuery Metastore that is Iceberg-compatible, aiming to unify metadata across engines. BigQuery can actually manage Iceberg tables natively – sometimes referred to as “BigQuery managed Iceberg tables”. These are similar to Snowflake’s managed approach: data in GCS, but BigQuery can update it and track it. The BigQuery metastore is a fully managed service where you can register Iceberg tables, and it ensures that engines like Spark or Flink can interoperate with BigQuery on the same data. The motivation, as Google described, is to avoid the scenario of multiple metastores (Hive, etc.) with inconsistent definitions when you use multiple engines. With a unified Iceberg metastore, BigQuery wants to let you query one copy of data with a single schema, whether using BigQuery or Spark or Flink.
  

• BigQuery + Dataproc (Spark): Google’s Dataproc is basically managed Spark/Flint/Hive on GCP. Dataproc has Iceberg packages too. With the new BigQuery metastore, a Spark running on Dataproc can use the same Iceberg table as a BigQuery SQL query, and the metadata doesn’t need duplication – they both talk to the BigQuery Iceberg catalog. This scenario is a classic lakehouse: Spark might write data, BigQuery reads it, or vice versa, with full consistency. It’s essentially Google’s version of the lakehouse using Iceberg under the hood.

The bottom line is Google acknowledges the need for open formats. Rather than only using BigQuery’s native storage (which is proprietary), they’re giving customers the option to use Iceberg to enable multi-engine flexibility. This is attractive for hybrid scenarios too: e.g., on-prem Spark jobs could produce Iceberg data, which BigQuery in GCP can then analyze, facilitating cloud adoption.

Other Engines and Tools

Beyond the above, there are other notable integrations:

• Apache Hive: Iceberg has a Hive runtime module that lets Hive read Iceberg tables (Hive treats it kind of like an external table). However, Hive’s slow development and the rise of newer engines mean this is less commonly used except for legacy compatibility.
  

• Apache Impala / Cloudera: Cloudera’s platform (CDP) has added support for Iceberg in Impala and Hive LLAP. Cloudera, as noted, embraced Iceberg (as well as Hudi) to give its customers modern table format options. So Impala, a traditional MPP SQL-on-Hadoop engine, can query Iceberg tables, use Iceberg’s metadata for performance, etc. This allows Cloudera users (on-prem enterprise Hadoop) to also enjoy Iceberg’s benefits without bringing in new engines.
  

• Dremio: Dremio is a SQL engine and data platform that heavily uses Apache Arrow. Dremio has been a big proponent of Iceberg – they even created a managed Iceberg catalog service called Arctic. Dremio’s query engine can read Iceberg tables and they integrate features like reflection (caching) with Iceberg metadata. Essentially, Dremio provides a BI-friendly layer over Iceberg and has contributed improvements to the project.
  

• AWS Athena: Athena is Amazon’s Trino-based serverless query service. Athena added support for Iceberg as one of its table formats (in addition to Hive, Hudi, etc.). Athena uses the AWS Glue Data Catalog, which now can store Iceberg table metadata. This means Athena users can create Iceberg tables and query them with Athena’s SQL. The benefit is more functionality (updates, time travel via Athena console by specifying a snapshot ID, etc.) over plain Hive tables, and better performance on partition-heavy data.
  

• Redshift Spectrum: Redshift’s external querying feature likely can work with Iceberg tables registered in Glue as well, though Amazon more actively pushes Athena or their new S3 Tables approach for Iceberg.
  

• Query Federation Tools: There are tools like Starburst Galaxy (Trino SaaS) which provide an out-of-the-box Iceberg catalog and maintenance automation, and open source projects like Nessie which serve as an advanced Iceberg catalog with Git-like version control (we’ll discuss in ecosystem). These are part of the integration landscape making Iceberg easier to use in various contexts.

The common theme is ubiquity: virtually any tool that deals with analytic data either already supports Iceberg or is in the process of doing so. This broad compatibility is what makes Iceberg so powerful in a modern stack – you aren’t limited to one framework’s capabilities. Each engine integrates in its own way but adheres to the Iceberg spec, allowing a seamless flow of data across them.

Tooling Ecosystem and Adoption in AI/ML Workflows

Beyond engines and query platforms, Apache Iceberg benefits from a growing ecosystem of catalogs, management tools, and adoption by forward-looking organizations. This ecosystem is crucial, especially as companies deploy Iceberg in support of AI and machine learning workflows that demand both flexibility and control over data.

Catalog and Metadata Tools

The Iceberg catalog is a pluggable component – it stores the mapping of table names to metadata files and coordinates atomic changes. Several options exist:

• Hive Metastore: Many use Hive Metastore as a basic catalog (especially in on-prem or older environments). It stores the pointer to the Iceberg table’s latest metadata file. This works but Hive Metastore has scaling limits and lacks advanced features.
  

• AWS Glue Catalog: On AWS, Glue is a drop-in replacement for Hive Metastore (fully managed). Glue now supports Iceberg table types. It is commonly used for Iceberg catalogs in AWS (Athena, EMR, etc.). Glue is more scalable and integrated with AWS IAM security.
  

• Nessie: Project Nessie is an open source metadata service designed for Iceberg (and Delta/Hudi). It introduces Git-like version control for data. With Nessie, you can have multiple named branches of an Iceberg table, tag snapshots, and merge changes. This is incredibly useful for ML and data experimentation – e.g., a data scientist can create a branch of the table, do some test writes or backfills, and if all looks good, merge to main, or discard if not. Nessie essentially serves as a modern catalog “control plane” for Iceberg. Dremio’s Arctic offering is built on Nessie, providing a cloud service where you get those capabilities plus nice UI. This kind of tooling is important for AI/ML, where you might want to maintain training datasets in different branches (say “production data” vs “experimentation data”) and ensure reproducibility.
  

• Iceberg REST Catalog: The Iceberg community defined a REST API for catalogs, which allows a lightweight service to host table metadata. Companies like Tabular (founded by Iceberg’s creators) offer a hosted REST catalog. AWS’s new S3 Table service also implements the REST catalog API. The advantage is simplicity and uniform access (the client doesn’t need a heavy Hive/Glue dependency, just HTTP). DuckDB’s integration, as mentioned, uses the REST catalog API to attach to AWS Glue or S3 Tables endpoints. This trend points to more cloud-native ways to manage Iceberg metadata that fit well with serverless and polyglot environments.
  

• Polaris/Atlan: Some data catalog companies (like Atlan with their Polaris) are integrating with Iceberg to provide governance, search, and lineage on Iceberg tables. Because Iceberg keeps a history of snapshots, it is fertile ground for building lineage tools and audit trails. We see cloud providers and startups building catalogs that treat Iceberg as the heart of the data lakehouse, adding on governance features like data masking, role-based access, and search over schema/data.

In essence, a rich catalog layer is emerging around Iceberg, which is extremely important for AI/ML as these fields demand data versioning, reproducibility, and experiment tracking. The fact that Iceberg has built-in versioning and tracks all changes makes it much easier to integrate with experiment tracking systems (like one could store the snapshot ID used for a particular ML model training run, and later reconciling what data was used is straightforward).

Data Ingestion and Transformation Tools

Various tools in the data pipeline space have added support for Iceberg:

• Kafka Connect: There are connectors that can sink data from Kafka topics into Iceberg tables (Confluent and others have released such connectors). This automates streaming ingest without needing a full Flink deployment.
  

• Apache NiFi / StreamSets: Some ETL tools can now write to Iceberg as a destination, making it easier to onboard data.
  

• dbt (Data Build Tool): dbt, a popular analytics engineering tool, doesn’t directly interact with file tables, but it can run SQL against engines that do. So if one uses Trino or SparkSQL with Iceberg, dbt models can be built on Iceberg tables just like they would on a Snowflake table. This allows analytics teams to use familiar workflows (version-controlled SQL transformations) directly on the lakehouse data.
  

• Compaction and Maintenance: Over time, Iceberg tables may need maintenance (expiring old snapshots to free space, compacting small files, etc.). Tools to do this include built-in Spark actions (the Iceberg API provides procedures for expiring snapshots, rolling up manifests, etc.), standalone utilities, and features in managed offerings (e.g., Starburst’s platform can automatically compact Iceberg small files in the background). Proper tooling here is important to keep performance optimal (particularly for streaming ingest use cases that create many small files). As the ecosystem matures, we see more automated solutions to handle this, reducing the manual burden on data engineers.
  

Adoption by Major Organizations (especially in AI/ML contexts)

Apache Iceberg has seen widespread adoption at some of the largest data-driven companies, many of whom explicitly mention its use in AI and ML workflows:

• Netflix – Netflix created Iceberg to solve their big data challenges. It remains core to their data platform. Netflix has an end-to-end pipeline platform (called Maestro) and they combined it with Iceberg to handle incremental data processing with guaranteed accuracy. For ML, this means new training data can be backfilled or updated incrementally without confusion. Thousands of Netflix data users (analysts, data scientists) rely on Iceberg daily as the storage layer for data ranging from streaming metrics to content personalization data.
  

• Apple – Apple’s AIML (AI & Machine Learning) data platform team is heavily invested in Iceberg. Apple has spoken about building streaming pipelines with Flink and Iceberg to support things like Siri and iCloud analytics. When a tech giant like Apple with massive ML workloads (think Siri’s natural language models, personalization across devices, etc.) uses Iceberg, it indicates that Iceberg can handle extreme scale and strict data quality requirements.
  

• LinkedIn – LinkedIn uses Iceberg in their Hadoop-based data lake to reduce latency and improve reliability of ingestion for their analytical datasets. One specific use: Gobblin, LinkedIn’s ingestion framework, writes to Iceberg to ensure that downstream jobs see a consistent view of new data. By using Iceberg, LinkedIn achieved lower latency from Kafka into HDFS (because they no longer needed to create ad-hoc Hive partitions or many small files). For ML, LinkedIn can incrementally update feature tables and immediately have those features available for model scoring or training in a consistent way.
  

• Adobe – Adobe Analytics and the Adobe Experience Cloud handle billions of web events and user data points for personalization and marketing analytics. Adobe adopted Iceberg to manage this firehose of event data and make it queryable for personalization algorithms. In practice, Iceberg helped Adobe ensure data consistency at scale and simplified their data engineering for user profiles (which power ML-driven personalization). Adobe has demonstrated how they can replay and time-travel on user event data to test new ML models against historical data, thanks to Iceberg’s snapshot abilities.
  

• Airbnb – Airbnb moved from a heavy Hive-on-HDFS infrastructure to Iceberg on S3 to support their data warehousing needs. They faced Hive Metastore overload and slow partitioning, which Iceberg solved\. Now, Airbnb analysts and ML pipelines (e.g. price optimization models, search ranking models) operate on Iceberg tables in S3. They have noted significantly reduced engineering overhead once on Iceberg – no more maintaining two sets of tables for different time granularities, etc., because Iceberg handles it gracefully.
  

• Stripe – Stripe processes financial transaction data and needs strong consistency. Stripe has been cited as a contributor to Iceberg. It’s likely used in their analytical platform for fraud detection features, which are ML-driven and require combining historical and real-time data under stringent correctness.
  

• Expedia – Expedia contributed to Iceberg as well. They likely use it for their travel data platform – imagine training ML models to recommend hotels or optimize pricing, on top of an Iceberg-backed feature store that contains years of travel booking data combined with fresh user search events.

• TikTok (ByteDance) – Although not listed in sources above, there was a mention of a YouTube talk about building an exabyte-scale feature store at ByteDance with Iceberg. ByteDance (TikTok’s parent) handles enormous volumes of video engagement data and uses ML for recommendation. An Iceberg-based feature store at exabyte scale underscores how well Iceberg can scale and serve ML: it can deliver the throughput needed and maintain data versions for model training at one of the world’s largest social media data scales.

This is just a sample; the list of companies using Iceberg reads like a who’s who of data innovation. Importantly, many of these organizations explicitly mention AI/ML use cases alongside analytics. The ability to share one data foundation between AI and BI is a huge win. Iceberg’s features like time travel allow ML engineers to train models on yesterday’s data and then easily compare with models trained on today’s data. Its schema evolution means adding a new feature column is straightforward and doesn’t break older pipelines – vital for iterative ML experimentation. Also, consistency guarantees mean that a model never trains on partial data from a broken pipeline; it either gets a full snapshot or none.

Community and Contributors: Companies such as Netflix and Apple contribute code; tabular.io (the startup by creators) drives community efforts; Cloudera, Dremio, AWS, and others contribute as well. This broad contributor base (Netflix, Apple, Airbnb, Expedia, LinkedIn, Alibaba, Tencent, AWS, etc.) means Iceberg is not controlled by a single vendor, unlike some alternatives. That has reassured many users and spurred adoption – they see Iceberg as a truly open project with a long future.

AI/ML Workflow Benefits Recap

To highlight why Iceberg is great for AI/ML workflows (tying together the tools and adoption):

• Feature Store Offload: Many feature stores (like SageMaker Feature Store, Feast, etc.) use Iceberg as the offline store. As mentioned, SageMaker saw up to 100x speedups on feature queries with Iceberg due to file compaction and efficient pruning. This leads to faster model training iterations.

• Experiment Reproducibility: By tagging snapshots or using branches, data scientists can recall exactly what data a model was trained on (fixing the “versioning the data” problem in ML). Iceberg’s snapshot IDs act as data version identifiers.

• Data Lineage for ML: Tools like Nessie allow branching, which can isolate “experimental data” from “production data” for ML. Also, if a new data processing logic is tested, you can write to a branch, validate model results, then merge.

• High Throughput for Big Data: AI often means big data (e.g., training a deep learning model on a huge dataset). Iceberg can feed these pipelines with throughput comparable to raw Parquet, because under the hood it is Parquet/ORC. The overhead of Iceberg’s metadata is negligible during large sequential reads – and it may even make things faster by skipping files. Thus ML pipelines (e.g., Spark ML jobs or TensorFlow reading from Petastorm which could use Iceberg) get both performance and reliability.

• Unified Batch and Real-Time Features: An emerging ML challenge is unifying batch and real-time feature engineering. Iceberg can store both historical features and be updated in near-real-time with new feature values (via streaming ingest). So one system can serve for training (which needs full history) and for near-real-time scoring (where new data comes in continually). This unity prevents training/serving skew.

All these benefits are driving organizations with heavy ML investments to adopt Iceberg as a foundational layer. It’s not just a better “data table” – it’s enabling new, efficient ways to do data science on lakes

Architecture and Performance: How Iceberg and Its Tools Work Together

Apache Iceberg’s architecture underpins everything discussed so far, so we conclude with a summary of how Iceberg achieves these capabilities and integrates all the tools:

Layered Architecture: Iceberg’s table format is built in layers – catalog, metadata, and data layers – which cleanly separate concerns:

• The Catalog layer stores the table’s current metadata pointer (and namespace of tables). This is like a lightweight metastore or “name service.” When a writer commits a new snapshot, it atomically updates this pointer in the catalog. When a reader opens the table, it retrieves the pointer to find the latest metadata. This design allows multiple engines to share the table easily: Spark, Flink, Trino, etc. all talk to the same catalog and thus agree on what the “current” snapshot of the table is. The catalog ensures operations like table creation, drop, rename, snapshot switch are atomic and consistent across all clients.

• The Metadata layer is where Iceberg really innovates for performance. The main object is the metadata file (JSON format) which lists table schema, partition spec, and references to snapshot information. Each time data changes, a new metadata file is written. The metadata file contains pointers to one or more manifest lists (one per snapshot). A manifest list is essentially an index of all manifests that comprise that snapshot. Each manifest file in turn contains a list of data files and associated stats (min/max values, record count, etc.). This hierarchical design means that to plan a query, an engine can read just a few files: the metadata file (small JSON), one manifest list (small), and then only the manifest files relevant to the query (Iceberg will let the engine filter manifests by partition or other stats before reading them). This avoids ever having to touch the raw file system or list directories. For example, a table with 10,000 data files might be tracked by 10 manifest files (each listing ~1000 files). To find the subset for a query on one partition, maybe only 1 of those manifest files needs to be read – result: only 1000 file entries are scanned instead of 10,000, a big win. This is how Iceberg achieves advanced pruning and planning and makes queries on large data more efficient. The metadata layer also stores extra goodies like statistics and bloom filters that engines can use to skip even inside files.

• The Data layer is the actual data files – usually Parquet, ORC, or Avro files with the table’s columns. These are stored in the lake (S3, HDFS, etc.). Iceberg does not change the file format; it leverages the highly optimized columnar formats. Thus, once an engine decides to read a particular data file, it uses the normal Parquet/ORC reader – benefiting from vectorized IO, predicate pushdown, column pruning, compression, etc. This means that Iceberg’s presence adds essentially zero overhead to the actual scan – if anything, it reduced overhead by eliminating needless file opens. In practice, query performance on an Iceberg table vs. the same data in raw Parquet files is equal or better, because Iceberg optimizes the planning phase heavily.
  

ACID and Multi-Engine Operation: When a writer (say Spark) wants to commit data:

1. It writes new data files (e.g., Parquet files) to the storage.

2. It creates new manifest files that list those data files (and perhaps mark some old ones for deletion if it’s doing an update).

3. It writes a new manifest list for the snapshot, and a new metadata JSON pointing to that snapshot.

4. Finally, it asks the catalog to atomically update the table’s pointer to the new metadata file.

This final step is where ACID magic happens: Iceberg uses atomic operations (like Glue’s transaction, Hive metastore’s lock, or a lightweight DB transaction in case of a REST catalog) to ensure only one writer’s metadata becomes “current”. If two jobs try to commit, one will succeed, the other will detect the conflict and can retry (using snapshot conflict detection). This optimistic concurrency model is similar to Git – if there’s a conflict, you rebase and retry commit. The outcome is multiple writers across different engines can reliably commit to the same table without corrupting it. This was impossible in the old Hive world. Iceberg’s design guarantees consistency: readers either see the old snapshot or the new one, never a mix. So, engines like Trino or Flink polling the catalog will just start reading the new snapshot once available. They don’t need a complex locking coordination with Spark – Iceberg did the heavy lifting via the metadata atomic swap.

From an architecture integration standpoint, this means each engine (Spark, Flink, etc.) just needs to implement the Iceberg API to read/write those metadata and data files. They don’t implement ACID themselves; they delegate to Iceberg. This made it relatively straightforward to add Iceberg support to engines – which is why we saw so many integrate it quickly. All engines speak the “Iceberg language” in metadata, so they naturally interoperate.

Performance Considerations: A few points:

• Eliminating “Small Files” Problem: By not exposing file count to users and handling merges via metadata, Iceberg encourages practices like compacting small files and writing larger files, which dramatically improves scan performance on cloud storage. Many engines have built-in actions or queries to compact Iceberg table files (e.g., a Spark SQL procedure OPTIMIZE table or an Iceberg API action can rewrite small files into bigger ones). The result is faster reads and fewer I/O operations. Organizations have reported order-of-magnitude improvements in certain queries after migrating to Iceberg simply because they got rid of overhead like millions of tiny files or overloaded metastores.
  

• Reduced Metadata Overhead: In large datasets, metadata operations (listing directories, opening thousands of file footers) can dominate query time. Iceberg’s manifest approach shrinks this overhead. For example, Netflix mentioned that with Iceberg, their metadata read time became a small fraction of total query time, whereas previously it was a big chunk (with Hive metastores struggling). As a quantitative example, if a query needs to scan 1 out of 1000 partitions, Iceberg will open maybe a couple of small files to find which data files to read, whereas a Hive approach might have to list and open hundreds of file system directories. This is a big win in cloud object stores which have non-trivial latency for list operations.
  

• Predicate Pushdown and File Skipping: Iceberg stores min/max statistics for columns at the file level in manifests. So if a query has a filter like WHERE date = '2025-04-18', Iceberg can avoid any manifest (hence data files) that don’t include that date in their stats. This is on top of Parquet’s page-level stats – it’s a higher-level pruning. Engines like Spark and Trino leverage this for extra efficiency. It’s like partition pruning on steroids (since it’s not limited to just partition columns, it can use any column stats present).
  

• Parallelism and Scale-Out: Iceberg is designed to work with distributed engines. The planning (reading manifest files) can itself be done in parallel by the engine (many engines will read manifests using multiple threads or tasks). Once the list of data files is known, they’re distributed among worker nodes to read. There is no single choke point in Iceberg – the metadata is stored distributed (many manifest files) so planning can scale. Some very large tables might have hundreds of manifest files per snapshot; reading them is embarrassingly parallel. This ensures that as your data grows, query planning doesn’t become a bottleneck.
  

• Resource Footprint: The metadata files are typically small JSON and Avro files; memory overhead to hold the file list is moderate and controllable (Iceberg will split manifests to keep each a reasonable size). This means an engine can plan a huge table in constant memory by streaming through manifests. It won’t blow the driver heap, etc. It’s a subtle but important architectural point for stability.
  

Ecosystem Synergy: Because Iceberg handles the hard parts of metadata and atomicity, each tool in the ecosystem can focus on its strength (fast SQL execution, or stream processing, or UI, etc.) and rely on Iceberg for data correctness. This synergy is what we observed:

• DuckDB could quickly add remote data querying because Iceberg presented a stable REST interface.
  

• Snowflake and BigQuery could relatively easily incorporate Iceberg because they just consume the metadata and attach their compute; they didn’t have to reinvent the wheel for how to track table versions.
  

• Tools like SageMaker Feature Store gained huge performance boosts by using Iceberg’s compaction and file tracking instead of a naive “dump files to S3” approach.
  

• Streaming engines and batch engines coordinate implicitly: Flink writes to table -> Spark reads it -> Trino serves it, all orchestrated by Iceberg’s snapshot mechanism.
  

In effect, Apache Iceberg provides a common language of data that all these different systems speak. It abstracts the physical data layout and provides a consistent view and set of operations. This is why we often call Iceberg the data platform or data layer in a modern architecture.

To conclude, Iceberg’s architecture delivers reliability, openness, and performance at scale. Its careful design (inspired by lessons from big data’s first decade) removes previous inefficiencies (like the old Hive approach) and replaces them with robust, performant techniques. The result is a virtuous cycle: as more engines adopt Iceberg, it becomes more valuable, and as more organizations use it in novel ways, the community improves it further (for example, upcoming Iceberg spec versions will add new features like row-level column encryption, finer-grained change tracking, etc., driven by real-world needs).

Apache Iceberg, backed by this solid architecture and thriving ecosystem, truly realizes the vision of a unified, high-performance data lakehouse. It empowers organizations to build data platforms that cater to BI analytics and AI/ML alike, in cloud, on-prem, or hybrid settings, all on top of open and vendor-neutral infrastructure. With the integration of engines like DuckDB and the support from cloud giants, Iceberg’s role in modern data stacks is set only to expand in the coming years – bringing ever more flexibility and power to data professionals.

We put the following table together as a quick reference when architecting a data workload. The reference instance in each case is a 64 vCPU AWS Graviton instance suited for a typical workload and shows the key metrics and notes the first thing that will cause problems. It’s meant for a quick back of the napkin sizing rather than a full capacity plan.

This was a balancing act as there are so many dimensions at play: the details of the workload, the gamut of available instance types, each with their own EBS/NVMe configurations. Trying to capture it all in a single table would be impossible. The goal is to treat this as a starting point, then benchmark and test your workloads before committing to an approach.

A common challenge every company has to deal with is personal data. As a consumer, you want to make sure companies are protecting your data, or ideally not even storing it in the first place. Various regions, countries, states, etc have their own regulations. Europe has GDPR, California has the CCPA, Canada has PIPEDA, Brazil has LGPD, and so on.

Adtech data tends to not be as sensitive as others — we’re not dealing with government identities or payment information, after all — but there is the matter of user browsing data. For example, if you’re using a browser that supports third party cookies (like Chrome), your behavior can be tracked from site to site, and companies pay to generate a targeting profile for you. If you’re using a browser that only supports first party cookies (like Safari) you can’t be tracked as easily across sites, but your browsing behavior is still captured within the same site.

For this reason, the aforementioned regulations often have a requirement around data deletion requests. If you’re storing this personal data, you have an obligation to delete it when asked. The types of data that qualifies as personal data varies between companies and industries, but for adtech it’s typically the user id, IP address, user agent — anything that allows you to identify and track individual users.

Going back to deletions, in a typical database, the deletion would be a simple “DELETE FROM TABLE WHERE user_id = ‘abc’” — but adtech data isn’t stored in a simple database. Instead, we have hundreds of billions of records that have some personal data captured each day and for cost reasons usually stored on S3 in some compressed and analytics-optimized format, such as Parquet.

So what do you do when a user wants to delete their data? The cookie that’s stored in the browser is easy to delete, but what do you do with the trillions of rows scattered across these Parquet files going back for months? You can run the “DELETE FROM TABLE .. “ command above, but what it will do is read the terabytes/petabytes of data into memory, filter and remove the relevant rows, then write it back to S3. It’s doable, but it’s slow and incredibly expensive. Think on the order of thousands or tens of thousands of dollars for a single user deletion request. It’s not obvious how to do it.

Four Approaches To Data Deletion

Don’t store the data at all

What’s not stored doesn’t need to be removed. But that comes at a cost: the data is needed to run a variety of analyses. Being able to count the distinct numbers of users, calculate trends, or use the data to cluster users to create these advertiser audiences in order to improve your ad targeting and hopefully improve the return of your ad spend.

Best for: Strict compliance and lowest storage and compute cost.

Tradeoff: The data is not available for analysis.

Store the data for a limited time

Alternatively, store the data for the minimum time you need, and then purge the rest. There’s still a cost to the removal, but if you only retain this data for a week then your deletion query only needs to run for a week’s worth of data.

We can extend the above to be even better by storing the personal data separately. Instead of storing the personal data in the same files as the rest we can split it into multiple file types with their own retention policies. That way the personal data can have a shorter retention window and you avoid having to run the deletion request on the full dataset. This allows the non-personal data to still be used for analysis and investigations without incurring the cost of storing the personal data.

Best for: Having some personal data available with low operational costs.

Tradeoff: Deletion still costs money, but the cost and likelihood of needing to run a deletion is low.

Store hashed values instead of original values

With this approach, the data isn’t tied back to the original user, but you are still able to run the bulk of the analysis you did originally. You still delete the cookie in the browser, but at that point the link is broken and you can let the data naturally phase out based on your retention policies. The key here is that if a deletion request comes in, you only have to remove it from the browser. No need to worry about the data stored in S3.

Best for: Having long term pseudonymized personal data available for a subset of analyses (counts, discount counts, various grouping).

Tradeoff: You cannot easily map this data back to the original user id, which limits the ability to associate properties with users — for example, associating users with a specific advertising audience.

Use a mapping table

The challenge with the one way hash approach is that it’s still deterministic. If you know what the user’s cookie id was and the hashing approach, you can rederive the hashed value. But we can make the association truly random by generating a completely random id, then using a mapping table to connect the cookie id to a random id that will be what’s stored in the S3 logs. When a user submits a deletion request, we purge the mapping entry, breaking the link similar to the prior approach, also letting it phase out naturally with the retention policy. Similar to the solution above, we remove it from a much cheaper and quicker data store — the mapping table — without having to touch anything on S3. As part of this step, we can also do full anonymization on the fields we don’t need details for; for example, just mapping user agent to a browser version, and stripping the last octet of the IP address.

Best for: Being able to do advanced and robust analysis and audience creation.

Tradeoff: This is more operationally complex and costly than the others.

For each of these strategies, it’s all about evaluating tradeoffs. Product and data science teams typically want as much data as they can get, since it’s what they use to innovate. On the other hand, legal and compliance teams want to limit the risk. And the data engineering teams are charged with walking that tightrope, all while keeping costs down so they can work on other problems. It’s no easy feat, but we hope these approaches help you think through the best approach for your team.

Last month in Oakland, Data Council 2025 felt like an early unveiling of the next gen, AI‑native data stack. Over three packed days, database legends, AI researchers, and bleeding-edge startup founders converged on a single storyline: our monolithic warehouses are being deconstructed into Lego bricks, and those bricks are being animated by self-driving logic. In this new era, sub-second interactivity/responsiveness has gone from luxury to table stakes. Soon, agents won’t only patrol BI dashboards, but run entire data pipelines, with ironclad observability and self-healing capabilities. For those that missed this awesome conference, here are some of my major takeaways.

The Future Isn’t Just Real Time, It’s Responsive

There was a big turnout by companies that provide low-latency data infrastructure. As I talked to their reps about features like Firebolt’s sub-second queries, ClickHouse’s focus on real-time analytics, MotherDuck’s instant SQL previews, and local-first DuckDB in the browser, it became clear that users can now expect interactive, immediate results, whether from dashboards, apps, or AI assistants. And not just over analytics data – as Aaron Katz (CEO @ Clickhouse) noted in Day 2’s keynote fireside chat, OLAP will remain core, but OLTP and OLAP are unbundling and blending. While Hybrid Transactional/Analytical Processing databases (HTAP) are not a new concept, advancements in technology and architecture have made them increasingly compelling.

Still, the perception of latency is relative. In one talk by Ethan Brown (Twitch), their new internal analytics slackbot handles masses of natural language analytics requests from several teams, at all hours of day and night. These requests used to be handled manually by analysts, becoming an unsustainable drain on resources. Even with 60 seconds of latency from the AI slackbot, it still feels real time. Teams who were used to waiting hours are thrilled, and now analysts can finally sleep through the night. To hammer this point home, Nikunj Handa (OpenAI) spoke on OAI’s evolving Responses API, citing an observed preference of trading latency for quality as the decision to make the API agentic by default. A single API call triggers an agentic loop that plans, iterates, and uses provided (and preloaded) tools to produce the best possible results.

Overall, developer platforms can now prioritize asynchronous, near-real-time architectures. Whether it’s supporting WebSocket query streams, or building UIs that update incrementally (like streaming SQL results), the tools that make data work feel instantaneous will set the new standard. This also means data engineers should get comfortable with event-driven patterns and local caching.

The Semantic Layer is Back: AI and Self Service Demand It

In the last decade, the idea of a separate semantic abstraction faded as the industry chased ELT and “schema on read” in data lakes. Sure, tools like Looker had some popularity, but most teams leaned into dbt/Jinja for transformations, treating “metrics” as just another SQL recipe.

(As you can imagine, this resurgence of interest has Looker’s creator, Lloyd Tabb, hyped! Infectiously enthusiastic, he was there to show off his new project, Malloy, that takes the semantic layer to the next level by embedding reusable, object‑oriented query definitions)

I promise I won’t bring up my company, Twing Data, repeatedly, but it’s important to support the point. From what we see, even with DBT, data fragmentation happens easily and is extremely difficult to pinpoint and fix. As data consumption diversifies (from humans in a notebook, to AI agents in Slack, and well beyond), the risk of fragmentation increases. So does the need for a unified semantic layer to minimize that risk.

I saw great presentations from the aforementioned Lloyd Tabb (Malloy) and Michael Driscoll (Rill Data), representing the next-gen semantic layer and metric store, respectively.

Semantic Layer: Malloy

When every dashboard or AI bot defines “revenue” in its own SQL, you get drift. Malloy instead lets you declare “revenue” once, with explicit grain and business logic built in, so a report in Looker, a DuckDB notebook, or an LLM‑powered Slackbot all compute the identical number

Metric Store: Rill

Bridging OLTP, OLAP, and AI, Rill Data’s SQL‑based metrics layer similarly centralizes KPI definitions, then automatically monitors them for anomalies and powers real‑time dashboards. That same layer can be queried by AI agents or BI UIs, ensuring every consumer sees the same semantics and alert rules, all without custom code per tool.

Also worth mentioning, and as a segue to the next takeaway, after hearing Julien Le Dem’s (co-creator of Parquet and Arrow) talk on open and composable data systems, it seems plausible to contain semantic definitions within an Arrow-based in-memory layer, or simply an Iceberg table representing the single source of truth. With emerging deep interoperability, these can work with a variety of query engines as needed.

In this AI-native era, semantic layers/metric stores are the glue that holds together various databases, self-service analytics and autonomous agents. Define your metrics once, govern them centrally, and watch as every front‑end, whether human or machine, speaks the same language.

Interoperability, Modularity, and the Deconstructed Stack

Thanks to talks by Julien Le Dem and others, it became clear that the modern data stack is comparable to Lego bricks. Want a new ML feature store? Plug it into your lake via Parquet. Need a faster query engine? Swap it in! Your data is in Iceberg.

This open source modularity is enabling small team to do big things. It’s also shifting value away from proprietary systems/storage formats and towards workflow orchestrated and user experience on top of common building blocks. Here are some examples:

• DuckDB is being embedded everywhere (Malloy, Hex, and Rill use it under the hood, Rill uses it in-browser for local development, and MotherDuck does cloud DuckDB). DuckDB + Parquet/Arrow is like the SQLite of data.

• Iceberg, Arrow and Parquet have hit escape velocity. From python and Spark to BigQuery and Snowflake, these formats tie every engine and language together. Support for Iceberg isn’t universal quite yet, but cloud providers are racing to bake this in natively. As Julien Le Dem noted, “blob storage is becoming table‑aware.” With transactions, schema evolution, time travel and catalog services, Iceberg turns S3/GCS into a fully-featured table store.

• The stack is becoming syntax-agnostic, mixing SQL, Python, and more. With Bauplan, you can define models and pipelines in python. On the analytics side, Hex’s multi-modal workbooks support a combination of languages in a single workflow.

Apache Arrow was the unequivocal hero of nearly every session on modular data architectures. I learned it underpins the data-movement for products at every scale. On the larger end, Ganesh from Hex did a medium-depth dive on how its DAG‑based “tablestore” persists results as Arrow tables so SQL and Python cells can execute in parallel. Elias from SDF (recently acquired by DBT) showed how Arrow enables the generated query plan to execute against any engine that supports it, so a model validated on Snowflake can be test-run locally in DuckDB, for instance. Earlier in its journey, Okta uses arrow to transfer data from shards to s3/Iceberg for analytics. And the aforementioned Bauplan also uses Arrow when moving Iceberg tables in and out of a lambda.

Disclaimer: seizing this new composability comes with a caveat: ripping out a legacy warehouse overnight is probably suicide. Migrating decades of ETL, dashboards, and access controls into Iceberg tables or a DuckDB‑backed semantic layer takes planning and incremental roll‑outs. But for any new initiatives, it certainly seems like the ideal bandwagon to hop on.

Agents All The Way Down: AI Across the Stack

The term “agent” was everywhere. From building reliable agentic systems, to OpenAI’s agentic API, to RAG/testing/benchmarking frameworks, to infrastructure applications, the entire community is clearly excited about the potential, and also sober about the challenges.

AI as the Data Consumption Layer

As a baseline, RAG is the standard for agents. Nearly every AI use case used RAG to ground the model with knowledge. The unanimous advice among Tengu Ma (Voyage AI), Skylar Thomas (Cake.ai), a panel of AI leaders, and several other talks, was to skip naive vector search entirely, and to go straight to hybrid (embeddings + keywords), and to use rerankers. An agent is only as good as its knowledge base, and building that base is a first-class engineering task. Lots of great techniques for data processing/retrieval were discussed. Skylar from Cake.ai gave an excellent (and fast, and dense) talk on RAG at scale, where he suggested beefing up retrieval with LiteRAG and knowledge graphs, suggesting > 2 types of retrieval across multiple databases. And Tengu from Voyage AI suggested various ways to process long documents with metadata like neighboring chunk summaries to increase retrieval quality.

In the spectrum of chains/loops vs end to end models, it seems the trend is moving away from fully-scripted chains of LLM calls toward giving the model more autonomy to plan, use tools, evaluate, and iterate dynamically. For instance, the internal Twitch slackbot used a selector agent to pick from several relatively deterministic chains. Meanwhile, OpenAI is rolling out the Responses API that allows dynamic loops with tools from a single call. And, somewhere in between, the presentation on Harvey AI, a specialized agentic platform for legal work, demonstrated a delicate balance between humans in the loop with dynamic agents.

The strongest point of consensus, as ever, was observability and evals. An example of a simple check was to run EXPLAIN on a query before running it within a text to sql system. Deeper, more elaborate systems were a major topic of Skylar’s talk, like running regression tests against eval datasets, with the idea being that the qualifying metric for “good” results must be defined and tested against to measure progress and alert to issues. Lastly, human review is an inevitability, especially for new or domain-specific systems.

AI as System Architecture

Considering the catastrophic risk of mangled data, the idea of agents building and running pipelines might sound like sci‑fi. After all, AI still hallucinates now and then, right? Believe it or not, we’re closer to solving that than most realize. If you’ve built a RAG system, you know the power of well-targeted context. The challenge has always been acquiring that context.

Historically, your data “context” – the rich metadata, semantic rules, transformation logic and business definitions that make raw tables meaningful – has been scattered across silos: pipeline orchestration tools like Airflow only capture task dependencies and schedules, while transformation logic lives as standalone SQL files in dbt or custom scripts without any machine‑readable declaration of input/output schemas or business intent; semantic layers in BI platforms such as LookML, Power BI datasets or Tableau models define dimensions and measures behind proprietary GUIs with no unified registry; cloud warehouses (Redshift, Snowflake, BigQuery) expose schema and lineage via vendor‑specific system tables and APIs that know nothing of downstream semantics; and any hand‑written documentation – in Confluence pages, data dictionaries or spreadsheets – quickly grows stale and disconnected from the code that actually runs. Because all these pieces live in different systems, in different formats, and often behind closed‑source UIs or proprietary APIs, an AI agent can’t simply “look up” the context it needs.

Today, though, that context can all exist as normal code – version‑controlled, semantically defined, and instantly retrievable by an AI agent. This is because end to end data systems are being fully redesigned as composable code artifacts by several cutting edge companies. At Rill Data, DuckDB and ClickHouse power instant, streaming dashboards, in‑browser, that reflect metrics defined both semantically and as code. Further, Bauplan takes the version‑controlled, data‑as‑code paradigm beyond modeling and metrics to the entire pipeline infrastructure.

Pair this with self-healing over deep observability, along with the ever-increasing model quality and tool usage abilities, and the future is unmistakable: the tools of tomorrow won’t just store or query data—they’ll reason over it, monitor themselves, and act in real time. Considering the unparalleled exponentiality of growth we’ve seen so far, it’s clear to me that tomorrow is right around the corner.

The Future is What We Make It

In his talk, Julian Le Dem reminisced about the past excitement around the potential of native data applications as the next wave of value creation. He pointed out that AI’s emergence as the new consumption layer tempered that wave. Whether native data apps are swallowed completely by ubiquitously embedded AI or not is too early to tell. But the industry isn’t holding their breath to find out. The trend towards modularity, a unified semantic layer, hyper-fast databases and responsive front ends all help enable AI to more deeply integrate throughout and across systems.

Of course, many see AI as a threat – to their jobs, to humanity even – but at Data Council 2025, the vibe was infectious, energetic excitement. Perhaps the biggest takeaway of all is that it’s still so early that no one really knows what AI’s role in our world will look like, in any of its applications. I can certainly see the negative viewpoint, but Data Council 2025 was an affirmation that there’s nowhere I’d rather be than on the bleeding edge, helping to shape what this awesome technology becomes.

One of our recent blog posts detailed how and why we built a Snowflake marketplace application — but that wasn’t the only marketplace we’ve been building for. Twing Data has also made its way into the AWS Marketplace! This article outlines our experience with getting this integration set up, the technical approach we took, and technical challenges that we came across.

What is the AWS Marketplace?

Like the Apple Store or Google Play Store, The AWS Marketplace is a curated digital marketplace that allows third-party developers to sell different types of software and services such as interesting data sets, development containers, machine learning models, and in our case — SaaS.

The SaaS Marketplace operates similarly to other ecosystems that enable third-party developers to integrate payments and authentication through the platform, while increasing product visibility with a page highlighting the features of the product.

For the buyer, AWS Marketplace offers important advantages, like flexible pricing and contract options that allow for pre-defined terms (with different pricing models) to avoid surprises. There are also options to create private contracts that can be fine-tuned further for specific buyers. Another bonus is that billing is consolidated through AWS. Finally, there’s a lot of value in knowing that the application has been through AWS’s curation and quality assurance process, and abides by their terms. Different organizations have different compliance and billing processes, so being vetted by and billed through AWS helps streamline adoption.

Twing Data AWS Marketplace App: Technical Details

For those interested in the technical implementation of this integration, here’s an overview of how it works.

Our SaaS offering is classified as “Contract with Consumption” meaning that AWS facilitates a contract between Twing Data and a buyer, and Twing Data is responsible for tracking usage (specifically, number of Redshift queries analyzed) and reporting it to AWS for billing, in case of overages. A buyer can update their contract to increase their usage caps (but not decrease).

The first step to get started is to create the listing, following the official “Lab: Create a SaaS listing” guide.

The "Lab: Integrate your SaaS" guide was an invaluable technical resource when designing and building this out, but assumes your application runs on AWS Lambda. In our use case we had to adapt the guide to work with our Remix application (soon to be React Router) and asynchronous analysis (in Python).

The linked guide is organized into three parts:

1. Integrate with buyer onboarding

2. Get entitlement updates

3. Get subscription updates and report usage

Let’s go into some details on each one.

Integrate with buyer onboarding

The first part of the guide acted as a template for linking the buyer’s AWS account and their Twing Data account. More directly, AWS redirects the buyer to the landing page URL which makes a POST request to a new Twing Data app route. This request includes a “x-amzn-marketplace-token” which we can use to get the buyer’s AWS account information for later.

In classic frontend development fashion, there are a few factors to consider: Does the buyer already have a Twing Data account? Is the buyer logged in? If they’re not logged in, how do we make this link?

These questions are answered by using the token header to get the AWS information, and store it in our own database. If the user is logged in, we can link their AWS account to their Twing Data account immediately, otherwise we redirect the user login/signup and make the link after. In each case, we also initialize information about their contracts’ limitations and usages so far.

Get entitlement updates

For the Contract with Consumption pricing model, it’s necessary to listen for events from AWS about contract terms changing; for example when a subscription is created or updated with increased limits.

The guide walks you through setting up an email subscription for the SNS topic, but the production implementation is effectively left as an exercise for the reader. It can be difficult to debug so the less unknowns there are, the better. We created a new route in our Remix app to handle this event and set the new URL as the subscription endpoint (making sure to use the correct protocol and “www” subdomain if required – otherwise your endpoint won’t get messages!)

HTTPS subscriptions need to be “Confirmed” which is a one-time process that tells AWS the endpoint is set up properly. Since it’s a one-time thing, we simply did the “Manual confirmation” that the SNS documentation describes.

Ultimately, this endpoint receives the “entitlement-updated” notification, and uses the provided customer-identifier and product-code to see what entitlements were created or changed. Again, the SaaS guide walks through a manual workflow using the AWS cli, but it was straightforward to adapt to use the @aws-sdk/client-marketplace-entitlement-service nodejs package.

Get subscription updates

The last part of the SaaS guide, “Report usage to charge the buyer” also walks you through creating another subscription, for listening to the aws-mp-subscription-notification. It’s important enough on its own so I decided to treat it as its own section.

For the consumption-based pricing models (“Contract with Consumption” and “Subscription (pay-as-you-go)”, consumption is tracked while the subscription is active, so we need to track not only usage (which we’ll go through below) but also changes to the subscription status. For this, we created a new route for the aws-mp-subscription-notification topic, and followed a similar process as the previous section to update an AWS user’s subscription on our side.

Report usage

Lastly, we need to track usage (number of queries analyzed) so we can report usage to AWS for billing.

Since usage reports are expected to be no more frequent than hourly, (“Timestamps that fall into the same hour are truncated to that hour.”), the logic between “tracking” and “reporting” are decoupled – we track number of queries as they’re analyzed, and then send consumption reports as needed to make sure they’re accurate and avoid any getting dropped.

End-to-end testing

Testing this flow turned out to be a bit cumbersome due to having to “test in production” – the seller and buyer account is the same during testing so to test a new contract, for example, the previous one has to be canceled. As of this writing, this is only done through contacting the AWS Marketplace support team.

To alleviate this friction, the different flows are covered by our automated testing suite that runs on each pull request to ensure there are no regressions to this critical feature. Of course, automated testing is a good practice regardless, but having to wait for canceling test contracts takes time so this is practically a requirement to shorten this feedback loop (and as a bonus, alleviate the AWS Marketplace support team’s workload).

Since you’re “testing in prod”, you end up being billed, but the price can be set low (and is said to be refundable):

Give our App a Try

Similar to our case for building a Snowflake Native application, we want to meet our users where they are — Twing Data already supports Redshift, so the next step is to get that integration set up as easily and securely as possible.

Being on the AWS Marketplace simplifies procurement, accelerates deployment, and aligns with the compliance, billing, and security standards you already trust. If your team is looking for a faster, easier way to integrate Twing Data into your Redshift warehouse, the AWS Marketplace is a direct path forward.

Helpful Links

What is AWS Marketplace? (AWS article)

What is AWS Marketplace? (video)

"Lab: Integrate your SaaS" guide (primary guide referenced above)

AWS Marketplace - Twing Data for Redshift: Query Analysis & Optimization (our live marketplace listing)

In 2007, Apple simultaneously shattered the computer and mobile phone industries with the introduction of the iphone. In 2008, they followed up with the app store, shattering the software distribution model and catalyzing a new mobile-first economy. Seventeen years later, what started as a mobile revolution has become a full-blown cross-platform app economy, spanning not just B2C but even B2B. With products in categories like CRM (Salesforce), collaboration (Slack, Zoom, etc), project management (Trello), and analytics (Tableau), decision-makers expect to manage ops from phones, tablets, and even smartwatches.

Today, the entire app lifecycle lives on the cloud, from development and deployment, to data pipelines, payment, compliance, and even user interaction. Snowflake’s relatively new app marketplace is a refreshingly unique and opinionated twist on the AWS approach that focuses on low ops integrations for data sharing and represents the expansion from warehousing to the OS of the enterprise data stack.

Enter: Snowflake

Snowflake launched onto the scene as a product sitting atop one particular cloud vertical, the data warehouse. Now it’s expanding horizontally with some depth in all the major data categories, like lakehouses, OLAP, data exchanges, BI/ML/AI, and more. In tandem with their product development, the user base continues to grow at a steady pace. It’s no wonder - aside from the novel warehouse UX, the prospect of doing ML, cross-cloud data sharing, etc with none of the complexity/latency/risks of transferring data to external tools is a great UVP.

Their Native App Marketplace represents Snowflake’s expansion from mere networked warehousing to being the OS of the enterprise data stack. A departure from the AWS/GCP approach to apps, its highly opinionated, deeply integrated experience is focused solely on data and ai applications that run securely within the user’s Snowflake environment.

As a user, the accessibility of downloading an app and running it on your data with zero configuration is quite an experience. Also, because the app runs entirely within your own Snowflake account, it enables the use of external tools that would otherwise be impossible on tables with sensitive or valuable data. Combined with Snowflake’s strict permission model, object isolation, and transparent resource usage, this architecture offers a level of control and trust that’s difficult to match — even with custom, self-hosted solutions.

As a developer, backend development has a new paradigm with a moderate learning curve (as expected) and the frontend, if used, is extremely limited and opinionated. While frustrating at times, these constraints also focus the product and force simplicity, which isn’t necessarily a bad thing. On top of that, we get the advantages of increased visibility, streamlined procurement, and the elimination of overhead associated with infra management or custom integrations.

Finally, while monetization is well integrated, reporting and billing aren’t quite up to the standard of AWS.

But who cares? As a data system optimization business, the advantages are obvious.

The Business Case For Building On Snowflake

Procurement

Twing Data’s core strength is its warehouse-agnostic analysis engine. Among the three large warehouses we currently support - BigQuery, Snowflake, and Redshift - there are major differences. We’ve invested deeply, and sometimes painfully, in making our systems flexible to handle each platform’s data and surface the appropriate insights. This gives us access to a wider customer base, and moves us towards our target of full end to end analysis.

The flip side: Depending on the industry/client, an external data connection comes with security/compliance concerns, and can even be a non-starter altogether. In our experience it can frequently lead to months of legal reviews and contract revisions. This is just another day for large enterprises, but we are a small team, so maintaining these ongoing open cases is not ideal for us.

This is simply not an issue if the data never leaves the client’s system. Native apps run directly in the user’s warehouse environment. A marketplace install and quick permissions agreement in the UI replace tens of hours of work by lawyers and engineers. Importantly, creators’ IP is also protected, as the queries run are not exposed to the user.

Finally, aside from the security and compliance hurdles, budget approval for data tooling is a hard sell. Executives in charge of allocating budgets typically prioritize investments that directly impact sales, and saving on data spend is typically far down the list. Billing on Native Apps contributes to existing Snowflake spend, eliminating the need for data teams to request an additional separate budget.

Visibility and Accessibility

The platform power of app stores has been proven time and time again. Users know to look there, and apps want to be there. The Snowflake Marketplace is just a baby, with some large limitations and hurdles on the development side, but the visibility is massive. Look at these numbers:

• Snowflake Marketplace: 38 data products per monthly active user

• AWS Marketplace: 7.8 apps per user

(based on “active customers” who transact on a regular basis. See source below)

Now, consider that AWS hosts a vast range of apps across every imaginable use case. Snowflake, on the other hand, is focused tightly on data and analytics. That means a much higher percentage of Snowflake users are in a Native App’s direct target market.

Summary of Market Opportunity

While mobile app stores are at maturity, now growing at 15-20% YoY, B2B marketplaces are earlier in the curve, similar to an earlier phase of B2C adoption, growing at 50-100% YoY.

That’s great for the marketplaces that capitalize on volume alone, but does that trickle down to the creators? The earlier one is on the platform, the more this is true. It’s still early days on the Snowflake Marketplace. The advantages are potentially enormous, IF you can capitalize on them.

One advantage is that it’s possible to develop a closer relationship with the platform. Snowflake has a “co-sell with partners” culture, and are actively looking to showcase Native Apps in their marketplace. Other benefits could be roadmap previews, design partner programs, or inclusion in investor/demo events.

Another is that there are so few listings currently (~3000). Being early on a platform means there’s high visibility and the opportunity for category leadership, or even to define a new category. With over 100,000 monthly active users, that means there are, as previously mentioned, 38 users for every app. In a straight comparison to AWS and GCP (and any other market), these are great numbers. And given the narrow market scope and the frictionless integrations, the likelihood to connect to users is much higher on Snowflake.

Give Our App a Try

Procurement struggles that involve hurdles like regulatory compliance keep companies from using industry-leading tools that could potentially help unlock massive value. For the developers of those products, like us, it can be equally stifling.

So far, Snowflake Native has been a simple way to promote our features to clients natively, while protecting our IP. We're continually iterating and delivering more sophisticated features as we validate our hypotheses. If you’re a Snowflake user and are interested in deeper observability to connect how your system is actually used with Snowflake’s execution and pricing behaviors, give Twing Pulse a download. It’s free, and all we ask is that you send us some feedback. We can then prioritize the highest-impact improvements and feature migrations.

For an idea of what’s to come to Twing Pulse, reach out to us for demo links to our full platform at twingdata.com.

Have you developed for the Snowflake Native Marketplace? What was your experience? Let us know below!

We just published our first Snowflake Native app — Twing Pulse — to the Snowflake Marketplace! It features a few of the most useful queries from our original much-loved product, Twing Data. Now Snowflake users can access a portion of our useful warehouse and workload metrics instantly, with none of the friction or security concerns that come with external integrations. It’s also free for now, so check it out and let us know what you think!

Fresh after the experience from 0 to 1, we thought it would be helpful to condense the mental model, development processes we used, and submission/review processes into a guide for those who are also interested in building their own Snowflake app.

Mental Model

Published Native Apps are run directly on the consumer account. When the app code is ready to publish, the developer generates or updates an application package based on that code. Then that package is published either privately or to the marketplace. Publishing requires three review processes, which we’ll cover later on.

In development, it’s possible to create an application for testing directly from the application package, without publishing. Here is the flow I have been using:

Initial Setup

The file tree below includes everything you need to get a repo started:

On the Snowflake side, it goes without saying you’ll need a Snowflake account and, depending on the type of app you’re building, some data to test with.

Because testing within Snowflake with data was such an important part of my development workflow, I set up an integration with GitHub from the outset. Here are the queries you’ll need to set up a Snowflake secret, use it to create an API integration and a Snowflake Git Repository:

Next, pull your repo in from GitHub and verify:

Finally, build your application package and application from the git repo. Make sure to build using the path to your app if not at the root of the repo:

Required Components

setup_script.sql: A script that creates objects within the application object that are required by the app. It runs when a consumer installs or upgrades the app, or when the provider creates/upgrades the application object for testing the package.

manifest.yml: Contains configuration information and the setup_script.sql path, to be used by the application package to create and manage a Snowflake app.

environment.yml (required if using Streamlit): Lists dependencies and versions used within the app. The channel must be set to Snowflake, and it’s recommended to specify Streamlit versions here.

Optionally, you might also consider a marketplace.yml file to prevent the user from ending up with an unusable app by validating region resources before install/upgrade. It also adds resource requirements to the marketplace listing.

Permission Management

“As per our enforced requirements, all account-level privileges and references listed in the application package manifest file must be requested from the consumer through Snowsight or the Python Permission SDK. The account level privileges listed in the manifest should be requested via the Permissions SDK in a Streamlit. You can do this by following these steps. ”

• Snowflake Marketplace Operations Rep (via submission review email)

Certain app functionality requires explicit privileges to be granted to the app by the consumer, such as reading or modifying consumer data. These privileges must be listed in the manifest, and also be requested by the app. The Permissions SDK is used to easily embed permission requests in the Streamlit UI. Here’s a function you can call directly within your Streamlit entrypoint file. Just be sure to import Snowflake.permissions as permissions first.

Application Versioning

Snowflake Native Apps support versioning through application packages, allowing developers to maintain and upgrade apps safely. Best practices include keeping at most two active versions at a time, using versioned schemas for stateless code and regular schemas for persistent data, and ensuring backward compatibility between versions. Release directives can be used to control which version consumers install, and upgrades should be coordinated carefully to avoid disrupting active users.

Frontend Development: Streamlit

While it’s not technically required for a Native App to have a UI, we opted for one. In Snowflake Native, that means Streamlit. Now, Streamlit is great for what it is, allowing devs to quickly spin up simple frontends for demoing or internal uses. When the goal is to expose a feature-rich product for a marketplace, Streamlit is not the tool I’d usually reach for. Additionally, Snowflake Native overrides or simply does not support many of Streamlit’s features. Some are documented, and some aren’t. The documented list is linked at the end of the article. It’s worth reviewing before writing any code, as it will likely change the approach to the application or even affect its feasibility entirely.

A few of the undocumented issues I encountered:

• Background and element styling: Background styling just does not work. I assume this is due to Snowflake’s own internal light/dark theming. Styling on other elements may or may not stick. Lastly, text color is something that shouldn’t be touched because Snowflake’s light/dark themes include appropriate contrasting text colors. If they’re changed, those colors will be static and must be tested on both light and dark backgrounds for visibility.

• Streamlit’s st.navigation doesn’t work, so for a multipage app the old /pages approach must be used. This means nested, folder-based navigation is not possible.

Backend Development

The current backend for Twing Pulse is a set of queries set within the app/code_artifacts/Streamlit/libraries directory. To leverage these queries to return data, we use Snowpark sessions, which can be created/retrieved like this:

The session returned from this function is used throughout the app to run queries and return data in pandas dataframes like this:

The current version of Twing Pulse gets slightly more complex from here with query parsing and recursive traversal logic used for query pattern detection, for example. But much deeper complexity is available, from native user defined functions (UDFs) and stored procedures, all the way to external functions, which are developed, maintained, stored, and executed outside of Snowflake’s environment. We’re early in our journey in Snowflake Native, and our exploration is ongoing. Perhaps as Native is developed further and we continue to explore, we will someday use external functions to natively connect our fully-featured Twing Data product for select clients.

Publishing

Once the app’s ready for the world, it’s time to publish. If the app is for internal use only, publishing offers systematic versioning and streamlined distribution throughout the org, as well as simpler access control. And, obviously, if it’s destined for the public marketplace, publishing is required. Public publishing has some additional steps, so I’ll cover this pathway here.

If you recall from the Mental Model section, it was shown that the application package, and not the application itself, is what is distributed to consumers. The downstream application that was used for development is just for that, and has nothing to do with the publishing process. From here we’ll only be referring to the application package itself.

The next step is to set package distribution to external:

‘ALTER APPLICATION PACKAGE <package_name> SET DISTRIBUTION = EXTERNAL;’

This will likely prompt you to accept some terms of service. Going forward, with this setting, every time the application package is updated, an automatic security scan kicks off. Completion on our app typically took several seconds to several minutes. Security scan status is located in the review_status column in the results from the query:
‘SHOW VERSIONS IN APPLICATION PACKAGE <package_name>;’

If the current version/patch shows ‘REJECTED’, you have some work to do. Ours failed many times during development, and it was never clear why or where the error logs were located. Luckily removing a few lines from the startup_script.sql file corrected the issue for us. Worst case scenario, the docs say that upon failure of the automatic review, the package goes into manual review, which can take days to complete.

When the review_status column starts saying, ‘ACCEPTED’, you’re in the clear to proceed.

Next, you must set default and, if necessary, custom release directives for the app package. A release directive represents the version of the application that’s available to consumers. Custom release directives can be assigned per consumer if desired. You’ll find those queries at the bottom of this useful worksheet screenshot showing all the queries you’ll need work through this process:

All versions to publish, defined by release directives, must pass the entire review process. In addition to the security scan, this involves an in-depth manual review upon publishing.

Now that we’re ready to submit for manual review, in the Snowsight UI, in the lefthand navbar, go to Data Products -> Provider Studio. In the top right you’ll see a blue button that says ‘Create Listing’. Here you can select whether to publish to the Internal or Public Marketplaces or for another specific consumer. Then name your app and select the type and availability:

Next, follow the prompts, assigning the intended application package from the list of packages with release directives, then filling in the details in the UI as required.

When all required sections and fields are filled in, the publish button at the top will become activated. Clicking that button will turn it yellow with a message signifying that it is under review.

Congratulations! The app is submitted!

The Review Process

The initial feedback took approximately 5 days to arrive. The review made clear that Snowflake is taking the process seriously. The issues they surfaced covered code, privilege permissions, and UI. While the message included many directives, there were also questions that indicated that they respected that the context affects the appropriateness of certain decisions.

Snowflake is prompt with replies after the process is started, and requires by policy that the publisher is also. I found the Snowflake contact to be warm, engaging, informative, and collaborative. A great first experience, to be sure!

That’s a Wrap

Developing for the Snowflake Native ecosystem makes a whole lot of sense for the right application. For instance, building a product destined to run within the isolated consumer account alleviates the need to worry about infrastructure. On the other hand, its intentionally-restrictive nature greatly limits the types of solutions possible, trading instead for greater security/governance.

We have a ways to go in exploring the full capabilities of the framework. While we’ve found it takes some rethinking and compromising to fit our existing approaches into this new model, we believe Snowflake Native is likely a simple way to natively expose our features to clients while protecting our IP. Twing Pulse in its current state is merely a test. As this thesis is validated, much more of our full product features will be on the way. If you’re a Snowflake user and are interested in deeper observability to connect how your system is actually used with Snowflake’s execution and pricing behaviors, give Twing Pulse a download. It’s free! All we ask is that you send some feedback to us so we can better prioritize the highest-impact improvements and feature migrations.

For an idea of what’s to come to Twing Pulse, reach out to us for demo links to our full platform at twingdata.com.

Have you developed for the Snowflake Native Marketplace? What was your experience? Let us know below!

Some Useful Docs Sources:

Snowflake unsupported Streamlit features: https://docs.Snowflake.com/en/developer-guide/native-apps/adding-Streamlit

https://docs.Snowflake.com/en/developer-guide/Streamlit/limitations#unsupported-Streamlit-features

Privileges:

https://docs.Snowflake.com/en/user-guide/security-access-control-privileges

Functions and Procedures:

https://docs.Snowflake.com/en/developer-guide/extensibility

External Functions:

https://docs.Snowflake.com/en/sql-reference/external-functions-introduction

These days so much data engineering work is paging through an API, parsing the responses, storing them in a database, and then joining them with other data to build an actually useful product.

It’s not glamorous work. In fact, it’s a grind dealing with the various APIs and figuring out the right incantation of parameters to get the data you want. Not to mention dealing with failures, rate limiting, undocumented “features,” and everything else under the sun.

And after you have the data, you have to parse the results and transform it iso it’s easy to work with, and join with the rest of the data you have.

What’s worked for us is integrating DuckDB into our analysis workflows. For those unfamiliar, it’s akin to running a small data warehouse within your script. It has a standalone mode, but you can also simply “import” it into a Python script and interact with it with SQL just like you would with any database. It’s basically an OLAP version of sqlite.

A sample project we’ve done is to fetch some data from a customer’s database and then hit the vendor API with those ids to enrich the data. This vexing workflow becomes a lot simpler (and maybe even fun) with DuckDB.

The basic idea is to just dump the API responses as JSON into a directory and then use DuckDB to quickly query the semi-structured data.

Here’s the basic folder structure:

/script.py
/data/data_type1/file1.json
/data/data_type1/file2.json

Steps:

1. Pull the ids you want to enrich

This can be from the database or another API. The key thing is that this is just a simple list without the details since those need to be retrieved via a separate API call.

2. Pull the ids you already pulled from the API

We want to be efficient and not refetch data we already have. In this case we can use DuckDB and do something like SELECT distinct(id) from read_json_auto(“/data/data_type1/*json”); to get the unique set of values we already have data for.

3. Compare the two and fetch the new data

Generate the set difference between steps 1 and 2 above to identify the new data we need to fetch and then run the script to fetch the data from the APIs. The result of this can be dumped into the data folder above as simple JSON without worrying about parsing or any transformations. Just dump it into the folder. For larger tasks you can also parallelize the calls to run through the fetches quicker.

4. Use DuckDB to bulk read the JSON data

Now this is where DuckDB shines. We can use the “read_json_auto” function to convert the semi-structured JSON data into a proper table we can query via SQL. If it’s a small project you can just query it directly. Otherwise, you can persist it and make it joinable with other data sets.

5. Profit

It’s an incredibly powerful tool and worth considering as part of any data transformation workflow. Dealing with APIs and polling and transformations is never fun – but DuckDB makes these workflows much better.

Full code here.

This became clear to us when we spoke to someone who used Microstrategy (it’s not just Bitcoin) as their BI tool. It’s not the hippest tool, but what it does well is a concept called “extracts” – a way of caching the underlying data. Most enterprise-focused tools offer the same functionality:

• Tableau and Power BI (traditional BI apps that offer a desktop client) store some data on the client, which avoids the need to hit a database on every request

• Looker offers similar functionality via their PDTs, but the PDTs are stored within the database rather than on the client

• Newer tools, such as Omni and Rill, improve on this functionality by embedding DuckDB in their webclient, or in a dedicated proxy layer

These sorts of decisions end up affecting the overall cost and performance of the BI system, especially consumption based warehouses like Snowflake.

One dimension to think about here is how intelligent the caching is, and where the cache is stored. The more intelligent the caching, the fewer requests your system will need to make back to the data warehouse – but there’s a limit based on the memory available. An aggressive caching strategy will take a superset of dimensions and metrics and store them client side in case they need to be pulled. This will increase the hit rate of subsequent requests, but requires more memory. A less aggressive caching strategy will reduce the memory footprint but reduce the hit rate. The ideal solution here finds the optimal balance based on predicted query behavior.

The other dimension is how the extract or cached data is stored. As mentioned, Looker will keep it in the underlying data warehouse while others may keep it in an intermediate cache while others may keep it on the client. Tools like Greybeam can intercept your queries and route them to an internal cache and compute layer if they deem it will provide a cost-effective performance boost.

You can imagine an optimal DW/BI system looking like this:

Your data warehouse and BI tools aren’t standalone components. The most effective implementations treat the data warehouse and BI tool as a unified system, where each component's strengths amplify the other's performance.

Imagine a company that’s collecting tons of data points per day. They start with a traditional OLTP database, but inevitably realize they need OLAP for their analytics workloads – so they start using a data warehouse.

That means they inevitably start building more and more workloads on top of the data warehouse:

• The product and application teams are running queries to incorporate reporting data into the customer facing dashboards.

• Data science is querying raw events to feed into their data science models.

• The analytics team is using dbt to do additional transformation and Looker to build dashboards for the executives and internal teams.

• The data team is using it to do complex data transformations to support the use cases above.

• The finance team is extracting data using some arcane Looker report into PowerBI

All teams and stakeholders are using the same data tool – that’s the dream, right? Well, in our experience, this setup will only take you so far. At a certain point, you’ll be better served by picking the right tool(s) for the job.

Snowflake, or basically any other data warehouse, acts as a pretty good initial data transformation layer – they do make it dead easy to get up and running with a large data warehouse. But the tradeoff is that it will be expensive unless you finely tune it, and it’ll be quite complex, since you’ll likely have multiple stakeholders trying to achieve different ends. It’s akin to using the same car to cart a family to weekend sports, then go offroading in the Sierra Nevada, to then race Formula One.

Additionally, it won’t give you the same low-level control that another tool would. Spark, for example, tends to be better for a very large data task that you really do need to optimize at a low level, or working with data scientists that do want to bring in a lot more custom logic or custom libraries. Clickhouse, on the other hand, is designed for this very fast querying, but it's limited because it's designed not for joins or any complex SQL queries, but for very fast queries against time series data.

For the example above, the solution may be to keep Snowflake for the last mile of reports that don’t have subsecond performance requirements. And introduce Spark to handle the initial expensive data transformations and data science modeling tasks. And if you’re feeling good, introduce Clickhouse and a tool such as Rill for the interactive internal analytics.

There’s no one size fits all solution in big data, as much as we may want it.

That’s why we’re always reminding clients to be intentional about what they’re using the data for, in order to implement the stack that works best for their needs. We’re also excited about tools like Apache Iceberg and the trend to “bring your own compute” to the data at hand. Using Iceberg allows you to keep the data neutral, and not locked into a single warehouse. Instead, you're able to pick whatever warehouse and compute layer you need to do the data processing.

If you’re beginning to feel pain from a single-warehouse data stack, start with an audit of workloads, and analyze each function's cost-performance dynamics. What are the actual use cases and workloads that your warehouse is powering? For example, maybe data transformation is responsible for 40% of your data budget. Maybe BI accounts for another 20%. Make sure to weigh the costs, in terms of both infrastructure and people, against the relative value of the output. After you have a good sense of that break, you’ll be better positioned to look into more specialized solutions that can efficiently handle the specific workflows. And that means getting to work with the precision of a scalpel, rather than blindly hammering away.

Unsure of where to start? We’re here to help – just hit reply to get in touch.

[Note: this post corrects a previous version that had a few placeholders instead of links]

Previously, we explored the pricing models of BigQuery and Redshift (and Serverless Redshift). In this article, we'll look at Snowflake's pricing models and optimizations that can be made to both reduce cost (which can be significant!) and improve performance.

Snowflake offers a wide range of features which gives us a lot of options when it comes to reducing cost and improving performance. However, many of these options come with pros and cons that vary from use case to use case, so we’ll also look at ways to ensure these tradeoffs are well-informed.

Since these tradeoffs can be complex, a good way to get started is with some proven best practices. We’ve compiled some below, tried and tested with our client base, to help you get started.

Understanding Snowflake’s pricing model

Snowflake’s flexible pricing model is mainly based on three components: Compute, Storage, and Cloud Services. Snowflake offers contracts with reduced pricing, or “Capacity pricing”. Some more advanced features may incur additional costs.

The prices below are as of March 1, 2025 and more details can be found on Snowflake’s Service Consumption Table (pdf). More general pricing information can be found on their Pricing Options page.

Compute

Snowflake offers a credit-based pricing model in which different sized warehouses cost different amounts of credits:

• X-Small: 1 credit/hour

• Small: 2 credits/hour

• Medium: 4 credits/hour

• Large: 8 credits/hour

• … and so on up to 6XL (scales exponentially)

The cost per credit varies between about $2 and over $6 depending on the plan, region, and platform (AWS, Azure, and GCP are currently supported).

To save compute resources, warehouses auto-suspend when not in use.

Storage

Storage is charged separately from Compute and is billed by TB per month. This cost varies by plan and region but ranges between $23 and $46 per TB/month.

Data is compressed to reduce actual storage cost.

Cloud Services

Snowflake's cloud services are priced using a fair-use model. Some features incur additional cost such as SSO and query optimization. These fees are typically small and included in compute costs, as long as cloud services usage doesn't exceed 10% of compute costs.

Tiers

• On-Demand

  • Pay-as-you-go model

• Capacity

  • Pre-purchased credits: per-credit rates may be reduced by 30+%

  • Storage costs can also be reduced

Other Costs

• Charges apply for transferring data to external services and between regions.

• Snowflake doesn’t charge an ingress fee, but other services might have their own egress fees.

• Some other features are not included in the “fair-use” model: Snowpipe, Materialized View Maintenance, Search Optimization.

Realistic pricing breakdown example

These rates are reasonable but they can add up quickly. Consider this real-world example: BigDataCo, an Ad Tech company, needs to process real-time ad impressions, clickstream data, and audience segmentation. BigDataCo’s employees regularly run complex queries on historical data to optimize ad targeting, measure campaign performance, and refine audience segmentation strategies.

Data Volume: ~300 TB stored

Users: 25 data engineers, 100 analysts, automated workloads

Compute Needs: Multiple warehouses (Small & Large), Snowpipe for streaming data ingestion

• Storage

  • Let’s say we want to store 300TB (and this doesn’t change). At $23/TB/month, this comes out to $6,900/month.

• Compute

  • Let’s assume ETL runs a large warehouse for 240 hours/month, analysts run 240 hours worth of medium warehouse compute, and automated workloads run small warehouses for a monthly total of 120 hours.

  • Adding this all up, we get (240 * 8) + (240 * 4) + (120 * 2) for a total of 3,120 credits.

  • Billed at $3/credit, this comes out to $9,360/month.

• Cloud Services - Data transfer

  • Snowflake does not provide specific rates but for this example we can assume it is negligible.

• Total

  • $6,900 + $9,360 = $16,260/month

Reducing cost and improving performance

There are plenty of ways to optimize warehouse cost and performance and this is by no means an exhaustive list. These are some ways we’ve reduced cost and improved efficiency for our customers.

Capacity Pricing

Using Capacity pricing can reduce the token costs by up to 30%. If we can accurately predict our spend for the example above, we can reduce our monthly spend to $13,452 – an overall reduction of 17%! At larger spend, Capacity pricing can save on storage costs as well.

However, accurately predicting future usage (to avoid paying for a larger commitment than needed) requires understanding historical usage, workload patterns, and expected growth.

Breaking down previous usage patterns can give more accurate results. For example, if we expect our above ETL workloads to increase by 20%, analysis workloads to increase by 10%, and automated workloads to stay constant, we can calculate that our credit usage will increase from 3,120 to 3,600 (assuming the warehouse sizes don’t change). This can give a more accurate prediction than a more holistic approach would.

After receiving reduced rates, it’s important to set up regular reviews to compare expected and actual usage and adjust warehouse sizes, schedules, or commitment levels as needed.

Optimizations

Snowflake operates on a few principles that are important to consider holistically. Unlike many other databases, Snowflake is highly optimized for analytics and uses a combination of micro-partitions and columnar storage to improve query performance and storage efficiency.

Micro-partitions are immutable storage units of about 50MB to 500MB (when uncompressed). Within each micro-partition, data is stored by column rather than by row, and each column (within each micro-partition) is stored together physically, which allows for better compression and improved read performance.

Understanding how the data is physically organized helps in identifying high-level performance bottlenecks by allowing us to visualize how a query might run: does it need to get data from lots of different columns and partitions (slow) or can we reasonably expect data to be co-located (fast)?

Data Ingestion

To get the most out of Snowflake’s compression, be mindful of how it will be fed into Snowflake storage from other sources.

If ingesting data from S3/cloud storage be thoughtful around the format. We found that zstd Parquet generates both the smallest files (cheap storage) and fastest ingestion (when using vectorized load).

See which columns are actually used and consider moving less frequently used columns to other tables – especially if that data has sensitive information that should have different retention rules. This way you can avoid unnecessarily reprocessing data.

Be careful of array types. They are often easier to store and query but query performance suffers. If there's a way of "exploding" the data to be in rows or lookup tables, you might be better off.

Lastly, take a close look at platform- and region-specific transfer pricing. Moving data across regions and between different platforms incurs additional costs compared to data moving within the same region and platform.

Optimizing Columnar Data Storage

Since Snowflake stores data by column, structural decisions go a long way. Here are some high-level improvements that can be done to keep Snowflake’s compression space-efficient:

Replace long strings (e.g. “United States”) with a numeric ID and lookup table (a “country_id” value of “US” maps to a lookup table “name” value of “United States”). This helps with query execution speed as well.

The bottom line is that Snowflake’s columnar storage benefits most from efficient table structuring and applying these strategies improves both compression efficiency and query performance, thereby reducing the overall cost regardless of plan, region, or whether it’s prepaid or pay-as-you-go.

Warehouse Sizes

Warehouses can be set to different sizes depending on the tasks they are responsible for. In our example above, we’re spending 1,920 credits/month on the Large ETL warehouse. Maybe there's an opportunity to reduce the warehouse during off-peak times when a large warehouse is over-sized. Alternatively, maybe queries will run slower but still be acceptable given the cost savings.

Snowflake clusters offer auto-scaling capabilities but they can be adjusted manually (or programmatically) if more optimization is needed. Adjusting warehouse sizes can be a quick way to save credits but making it too small can lead to slow or queued queries, so care should be taken.

Data Retention

A seemingly obvious improvement is to reduce how much is being stored - this might mean reducing duplication (e.g. with a lookup table) or clearing out old data or moving old data to a cheaper storage service like AWS S3, Iceberg, or Google Cloud Storage. That data can then be queried from Snowflake without having to import it. Snowflake also offers an “external tables” feature that allows you to store external storage metadata within Snowflake but the data itself in S3 or elsewhere.

A well-structured data retention policy balances cost, performance, and accessibility. The key is automating archival, using external tables for historical data, and monitoring storage usage to optimize costs.

These methods are simple on the surface but require careful implementation to ensure there are no side effects and any performance trade-offs are acceptable. Any long-term change like this should be modeled and their costs estimated before implementation.

Idle Time

There’s a startup and shutdown costs associated with warehouse usage. Idle warehouses are shut down automatically after a period of inactivity but they can also be shut down manually if they’re not expected to be used again for some time. One of our clients does exactly this because they have a predictable ETL workload, so turning the warehouse off after their tasks are done is an easy way for them to save on their bill.

Query Batching

Conversely to managing idle time, queries can be batched to minimize idle time. Since there’s a hidden cost associated with warehouses starting up and shutting down, queries can be batched to eliminate idle time, especially if warehouses are idle frequently without shutting down. It’s cheaper to run a warehouse at 100% capacity for an hour than 50% capacity for two hours.

Similarly, partitions are cached on the warehouse level, so if we have queries that read the same partitions, we can automatically speed queries up by running them on the same warehouse (assuming it’s not shutdown and spun back up in between queries).

Query Optimizations

Many of the topics covered above (especially in “Optimizing Columnar Data Storage”) improve query performance (and therefore reduce credit usage). Snowflake offers some additional features that can speed things up.

Materialized views are pre-computed results that speed queries up, but are re-calculated when data changes. There’s a clear tradeoff between speed and storage cost and cost of re-calculation (or “refresh cost”), so this approach is best when there are frequent reads that depend on seldom-changing data. MVs can be thought of as memoization.

Dynamic tables are a newer feature with some key differences. Instead of refreshing when data changes, they refresh at a given interval. Additionally, instead of storing the query result, dynamic tables store a full snapshot of the data.

In our work, however, we’ve found that these features are better on paper than in practice. As always, the best way to get meaningful insights is to test, test, and test.

For a more in depth comparison, take a look at the breakdown below:

Clustering keys allow you to physically partition rows of data, for example clustering by “date” improves performance when retrieving data in chronological order. By default, Snowflake inserts data randomly (due to how partitioning works without clustering keys), so this approach is great for large tables where queries scan too many partitions. Data can be re-clustered in the future as data is added, though doing so uses Compute credits, so make sure clustering is overall favorable beforehand!

As an example, consider a real-world example of a large table (let’s say it’s 500TB and contains 5 billion rows) that’s frequently filtered by order_date. Clustering by order_date can lead to dramatic results:

That’s over $90,000, or almost 85%, in annual savings –- more than enough for a few pizza parties! And remember, Capacity pricing can reduce compute costs further. Our friends at select.dev have written an article dedicated to clustering that has more details on the topic.

Summary

As you can see, there are a lot of levers to pull, each with its own pros and cons – as with any decision in software engineering. Previous history provides invaluable insight to these big decisions, and Twing Data can help you on your quest by letting you visualize historical usage and patterns.

In our experience, and as mentioned throughout this article, the best way to quantitatively measure any changes is to test and inspect the results. Twing Data would be happy to work with you to identify opportunities to optimize your datamart so you can have more pizza parties.

We’ve been talking about some shortcomings of Snowflake recently (see this post about query hashing blind spots), so today we’re changing tacks to give some credit where credit is due. Snowflake actually does a great job capturing all sorts of query metadata that you can use to dig in to explore your warehouse usage. Below, we’re sharing three particular queries that have been incredibly helpful in helping us analyze our customers’ data warehouses and identify optimization opportunities.

Table size vs Table usage

with usage as (
   select
   obj.value:objectId as table_id,
   obj.value:objectName::string table_name,
   APPROX_COUNT_DISTINCT(query_id) as num_queries,
   APPROX_COUNT_DISTINCT(case when array_size(objects_modified) = 0 then query_id end) as num_select_queries,
   min(query_start_time) as first_usage,
   max(query_start_time) as last_usage
   from snowflake.account_usage.access_history,
   table(flatten(base_objects_accessed)) obj
   where query_start_time >= '2025-02-01'
   group by all
)
select at.table_id, at.table_catalog, at.table_schema, at.table_name, at.table_owner, at.table_type, at.is_transient, at.clustering_key,
at.row_count, at.bytes / 1e6 as megabytes, at.created, at.last_altered, at.last_ddl,
at.table_catalog || '.' || at.table_schema || '.' || at.table_name as fully_qualified_name,
u.num_queries, u.num_select_queries, u.first_usage, u.last_usage
from (select *
     from snowflake.account_usage.TABLES
     where deleted is null
) at
left join usage u on at.table_id = u.table_id
order by at.bytes desc;

Snowflake provides a neat view SNOWFLAKE.ACCOUNT_USAGE.TABLES that has all sorts of metadata about each table, ranging from size and number of rows, to the most recent time both the table structure and contents have been modified. But what makes this even more powerful is joining it against the SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY. This lets you see how often this table is actually being queried, and you can also see whether it’s the target of a changing operation (eg UPDATE, INSERT, etc) or not (eg SELECT).

You will likely discover that you have tables that haven’t been touched in years that you can safely remove. You will also notice that some tables have a disconnect between the number of times they’re written vs read — another optimization candidate. And finally, you take the additional step of looking at how the largest tables are used to see if you actually need them to be as big as they are. Two examples here are fields that are rarely queried, as well as the lookback period for time series data — you may have a table that goes back multiple years in history and discover that no one has written a query looking back more than 6 months.

Usage of columns for a single table

WITH parsed_access_history AS (
 SELECT
   QUERY_ID,
   query_start_time,
   OBJECT_NAME,
   COLUMN_NAME
 FROM (
   SELECT
     QUERY_ID,
     query_start_time,
     OBJECTS.VALUE:objectName::STRING AS OBJECT_NAME,
     COLUMNS.VALUE:columnName::STRING AS COLUMN_NAME
   FROM
     snowflake.account_usage.access_history,
     TABLE(FLATTEN(INPUT => BASE_OBJECTS_ACCESSED)) OBJECTS,
     TABLE(FLATTEN(INPUT => OBJECTS.VALUE:columns)) COLUMNS
   WHERE query_start_time >= '2025-02-01'
 )
)
SELECT
 column_name,
 count(1) as cnt
FROM
 parsed_access_history
WHERE
 OBJECT_NAME = 'DB.SCHEMA.TABLE'
GROUP BY all;

Another use of the SNOWFLAKE.ACCOUNT_USAGE.TABLES view is to look at the usage of a table by column. Snowflake also has the BASE_OBJECTS_ACCESSED column as well as the DIRECT_OBJECTS_ACCESSED column, which allows you to see both the views (direct) as well as the underlying table (base) data being accessed. You can also join these against the SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY view to pull the query type and the query text to dig in more.

This is useful to see if some columns have drastically different usage patterns which can signal that it may be worth creating other versions of the table, especially if they are high cardinality.

Inefficient warehouse idle settings

WITH warehouse_events AS (
 SELECT *
 FROM SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_EVENTS_HISTORY e
 WHERE event_name IN ('RESUME_WAREHOUSE', 'SUSPEND_WAREHOUSE')
   AND event_state = 'STARTED'
   AND e.warehouse_name = 'WH_NAME'
   AND e.timestamp >= '2025-02-01'
   AND e.timestamp < '2025-03-01'
),
minute_bounds AS (
 SELECT
   MIN(DATE_TRUNC('MINUTE', CAST(timestamp AS timestamp))) as min_ts,
   MAX(DATE_TRUNC('MINUTE', CAST(timestamp AS timestamp))) as max_ts
 FROM warehouse_events
),
minute_sequence AS (
 SELECT
   DATEADD(minute, seq4(), (SELECT min_ts FROM minute_bounds))::timestamp as minute_start
 FROM TABLE(GENERATOR(ROWCOUNT => 50000))
 WHERE minute_start <= (SELECT max_ts FROM minute_bounds)
),
active_periods AS (
 SELECT
   warehouse_id,
   warehouse_name,
   event_name,
   CAST(timestamp AS timestamp) as start_time,
   LEAD(CAST(timestamp AS timestamp)) OVER (PARTITION BY warehouse_id ORDER BY timestamp) as end_time
 FROM warehouse_events
),
minute_activity AS (
 SELECT
   m.minute_start,
   DATEADD(minute, 1, m.minute_start) as minute_end,
   a.warehouse_id,
   a.warehouse_name,
   GREATEST(a.start_time, m.minute_start) as period_start,
   LEAST(COALESCE(a.end_time, CURRENT_TIMESTAMP()), DATEADD(minute, 1, m.minute_start)) as period_end
 FROM minute_sequence m
 LEFT JOIN active_periods a
   ON a.event_name = 'RESUME_WAREHOUSE'
   AND m.minute_start < COALESCE(a.end_time, CURRENT_TIMESTAMP())
   AND DATEADD(minute, 1, m.minute_start) > a.start_time
),
base_activity AS (
 SELECT
   COALESCE(warehouse_id, (SELECT warehouse_id FROM warehouse_events LIMIT 1)) as warehouse_id,
   COALESCE(warehouse_name, 'WH_NAME') as warehouse_name,
   minute_start as timestamp,
   period_end > period_start as is_active,
   SUM(COALESCE(
     DATEDIFF('microsecond', period_start, period_end) / 60000000.0,
0
   )) as pct_active
 FROM minute_activity
 GROUP BY ALL
 HAVING pct_active > 0 AND pct_active < 1
),
grouped_activity AS (
 SELECT
   *,
   DATEDIFF('minute', LAG(timestamp) OVER (ORDER BY timestamp), timestamp) != 1 AS new_group
 FROM base_activity
),
numbered_groups AS (
 SELECT
   *,
   SUM(CASE WHEN new_group THEN 1 ELSE 0 END) OVER (ORDER BY timestamp) as group_id
 FROM grouped_activity
),
counts_for_min AS (
 SELECT
   warehouse_id,
   warehouse_name,
   MIN(timestamp) as start_timestamp,
   COUNT(*) as minutes_active,
   AVG(pct_active) as avg_pct_active
 FROM numbered_groups
 GROUP BY warehouse_id, warehouse_name, group_id
 ORDER BY start_timestamp
)
SELECT
 DATE(start_timestamp) as ymd,
 minutes_active,
 COUNT(1) as cnt
FROM counts_for_min
GROUP BY ALL
ORDER BY ymd, minutes_active;

This one is long, but does something simple: it computes the consecutive minutes where a warehouse has been going through a resume/suspend loop. You probably already know that you can reduce your Snowflake cost by coming up with a more intelligent idle timeout, versus the default which is almost always set too high at 5 minutes. What you may not know — hidden in the Snowflake docs — is that you are going to be charged for at least 60 seconds, even if your warehouse runs for 10 seconds. That means if your warehouse goes through multiple resume/suspend sessions during a minute, you will be charged for multiple minutes of activity. The query above is meant to identify these cases where a warehouse resumes, runs a query, then suspends, only to be resumed again within the same minute. This is a sign that the idle timeout may be too low. That doesn’t mean increasing the timeout will be better, since during other times a lower timeout may be ideal, but it usually warrants another look.

In an ideal world, you’d have the following flow for each warehouse session: start the warehouse, run whatever queries you need, then explicitly suspend the warehouse without needing to rely on the auto idle settings. This is only possible in well defined and controlled workloads, and any sort of BI tool usage likely means you’re dealing with limited ability to predict usage so your warehouse will inevitably have periods where it fluctuates between being on and off.

These are just a few example queries to explore what’s happening under the hood. Snowflake captures rich metadata, though its dashboards don’t always make it easily accessible. With some effort, you really can answer an unlimited number of questions — writing the query is the easy part. The real challenge is knowing what to ask.

Why Serverless? Why BigQuery?

In 2017, The Economist published an article titled “The World’s Most Valuable Resource is No Longer Oil, But Data”.

As both the fuel that drives AI progress, as well as the core business of the largest companies in the world, it’s no wonder modern companies of all sizes, stages and industries implement a data strategy with a bias towards storing as much as possible, just in case. Since the hardware overhead and technical requirements of implementing and managing these systems in the traditional way put data systems out of the reach of small or agile businesses, cloud providers have emerged to fill the gap. These providers are able to leverage their scale to offer data systems affordable and accessible enough for businesses of any scale. At the pinnacle of accessibility sits the serverless model, where the provider securely distributes resources across accounts dynamically based on demand, serving them within managed environments that, to varying degrees and ways, abstract much or all of the complexity involved. In exchange, the user gets a simple way to reliably store and access their data at relatively transparent pricing.

Google Cloud’s implementation, BigQuery (which we'll shorthand as BQ), is a shining example of serverless infra. Google’s BQ documentation is robust, and while there will be some information overlap, the goal here is to add real-world usage context.

Getting Into the The Guts of BigQuery

High level, BigQuery (BQ) is a managed system that abstracts and integrates several proprietary elements, as well as many independent Google Cloud Platform (GCP) services. These elements populate, support or interact within BQ’s 2 main segments: the Data Plane and Control Plane.

• Data Plane: Encompases the storage (Colossus), compute/execution (Dremel), and networking (Jupiter) layers, which contain logic and resources for storing and reading data as well as query processing. It executes queries by reading, processing and returning data from Colossus via Dremel using allocated slots.

• Control Plane: Responsible for metadata tracking, high-level resource allocation, query parsing/validation, and job orchestration.

Here is the 50,000 ft view of the architecture at large:

It’s a complex system, but the result is the orchestration of several highly-optimized components working in sequence and parallel to dynamically handle a wide variety of conditions and applications. As this is a general overview and not a deep dive into one particular component, we’ll stick to examining 2 significant elements in the Data Plane at mid-depth.

Colossus, the underlying storage layer for several GCP services, including BQ and Google Cloud Storage (GCS), is fundamentally aligned with serverless principles; BQ’s integration with Colossus stands out by fully managing data-loss prevention with automatic multi-region data replication, at no additional cost. Within Colossus, BQ stores data in Capacitor format, their proprietary, highly-compressed columnar storage structure. This formatting is also fully managed, with no configuration options available. Additionally, a storage optimizer periodically rewrites files in a format that is more efficient for querying. The primary 2 manual storage optimizations, partitioning and clustering, which we’ll touch on later, offer some user control, but even clustering on changing data requires a layer of automated re-clustering that’s managed internally.

The pattern of highly abstracted complexity doesn’t stop at Colossus. Dremel, the query compute engine, rewrites or breaks down executed queries into stages, then enables the dynamic provisioning of very small resource units, called “slots”, in parallel for running workload stages across all existing orgs. The exact algorithm governing resource provisioning is, of course, not publicly available, but generally it’s based on a range of factors like query complexity and overall system load, and even down to a query’s particular execution stage. The goal of optimizing efficiency—and by extension, minimizing waste—is balanced against the need to ensure that some level of compute capacity remains available, even for inactive organizations (they call this “Fair Scheduling”).

With so many layers of management systems, the natural question is, 'How well does all this work?' While answering in full is beyond the scope of this section, it's largely an academic question. Users have no direct control over these internal mechanisms, aside from workload-level optimizations or switching providers. Given the sheer scale and automation of BigQuery, its performance remains a modern engineering feat, even if there are some inefficiencies.

Breaking Down BigQuery Costs: Storage and Compute

While storage and compute resources are largely automated, there are yet many ways to configure when and how they’re used.

Storage

BigQuery offers two primary pricing models, Logical and Physical Bytes Storage Billing (LBSB and PBSB respectively). Each model has a half-price long-term version as well, which is automatically set per table after 90 days without modification to the table.

Metadata storage has quite the elaborate pricing model. That’s for good reason: the metadata is stored differently than user data because it’s accessed frequently by the system and crucial for warehouse functionality. If metadata storage begins to surpass the 2% + 10 GiB free tier threshold, it is likely an indication that there is a path to reducing metadata generation (we’ll touch on optimizations in the next section).

While not technically part of BigQuery, it’s also worth mentioning that strategically integrating GCS into BigQuery operations can enable other ways to save money on storage. Aside from handling unstructured and semistructured data storage, GCS is commonly used for storing archived data at a fraction of the cost. Most practically, GCS is ideal for staging tables due to its capacity for large-scale data ingestion, and broad support of data and compression formats. Also, hybrid pipelines can use GCS as the data bridge that connects to other storage systems.

Compute

Fully separated from storage pricing, compute pricing options range from fully on-demand to reserved capacity, or even a blend of both.

• On demand: You’re charged based on the amount of data scanned. There is no minimum usage, and the first 1 TiB per month is free. Fair scheduling reasonably ensures that each query has equal access to available slots, whether the org is highly active or mostly inactive. The resource limit for on-demand usage is 100 concurrent queries and/or 2000 slots at a time (per project per org), with 2 exceptions: the slot limit is subject to availability, and if even more are available they might be used to push through small workloads (this they call transient burst capability)

• Capacity based: In this scheme, charging is based on the amount of slots “reserved”. This reservation can be a static amount, or, optionally, a min/max range can be set to enable autoscaling for highly dynamic or unpredictable requirements. An optional 1-3 year commitment is available for Enterprise/Enterprise Plus tiers, offering discounts for committing to continuous billing for an amount of persistent compute capacity. While this ensures resource availability at all times, any autoscale compute used in addition to commitment slots are charged at the un-discounted rate. At higher volumes and commitment levels, it’s also possible to negotiate additional discounts. Finally, the pay-as-you-go option, available on all capacity-based tiers, is similar to on-demand in that charges only occur while resources are in use. One caveat here is that the set minimum reserved slots are the minimum to be used, no matter how small the workload. Another is that resources made available for a workload stay up for 60 seconds after use. With certain cadences of usage, this can be a factor worth investigating.

These pricing options can be pieced together like legos to fit your needs. Slots can be reserved per team, project, folder, or even workload type. Scaling on reserved slots can use capacity-based autoscaling or on-demand slots.

Finally, it’s worth mentioning that when these capacity-based “editions” were introduced a few years ago, they discontinued the flat-rate + flex slots model, which allowed a more deliberate or scheduled approach to reserved resource scaling. Now, even capacity-based scaling is 100% automated, further leaning away from traditional provisional models and into a fully-managed experience.

Managing a Managed System

The fully-managed services within BigQuery are highly nuanced and opinionated. Understanding how they work and what they prefer are crucial for optimizing usage. Levers like query composition, partitioning/clustering tables, metadata management and more make massive impacts on efficiency and costs. As usual, the official documentation covers optimizations fairly well, but we’ve found there’s so much nuance that even talented teams can overlook major points. I’ll drill into one peculiarity not addressed by BigQuery’s query optimizer.

Consider the following query and execution graph:

Since cte1 is referenced twice, BigQuery executed the CTE both times. Under the hood, BQ tends to treat non-recursive CTEs by inlining them in the final query plan as part of the plan creation process.

A possible optimization is to materialize the data that will be used multiple times:

Of course, this scenario is a bit contrived. And, in this simple example, while slot seconds are saved, billed bytes are equivalent. That means we get increased efficiency, but with on-demand pricing the cost doesn’t change. With increased CTE complexity, though, all metrics, including bytes processed/billed are dramatically affected in expected and unexpected ways. While it might appear obvious that reducing compute and data scanned by caching partial data as an initial step is beneficial, consolidating the entire query can help the optimizer in other ways that can offset the CTE reruns.

The high-level takeaway is that it might be worth testing significant workloads that fit this profile with temp tables.

But wait, there’s more! Temp tables in BigQuery are session-based. As long as the session is live, any temp tables created within persist. In the console, a session in this context really only means per tab, per single or multi-statement execution. But the real value is in its API usage, through what they creatively call BigQuery Sessions. A session can be generated during an API call, then referenced in subsequent API calls to associate with the same session. That means an early API call can generate temp tables and later calls can use them. Essentially, it amounts to a caching layer for larger datasets, where intermediate results remain in BigQuery instead of overloading front or back ends.

An example of this put to great use is for dashboards with multiple levels of drill downs, where complex aggregations are done at each level. Within a session, a temp table can store the data for each level, so the user can navigate down, back out, then down again with no need to wait for redundant heavy computation or rely on client-side storage.

As I said before, temp tables persist for the duration of the session. A session, if not manually terminated, will deactivate automatically after 24 hours of inactivity or 7 days after creation. Depending on the application, this can have a major impact on a system’s performance, cost efficiency, and user experience.

The TL;DR on BigQuery

BigQuery and its surrounding ecosystem is an incredible complex of intricate, interlocking systems operating at enormous scale. Every piece contains so many layers of automated management based on unknowable conditions. With almost no barriers, it really just works.

Having said that… (Curb, anyone?), to work with the system instead of against it requires a relatively deep understanding of proprietary behaviors. In a traditional warehouse, user management means to manage resources in a cluster, tuning query execution, and provisioning capacity. In BigQuery, control shifts from explicit resource management to understanding the inner workings of BigQuery’s execution model, slot allocation, caching mechanisms, and query optimizations.

We put together a similar guide for Redshift last month – you can find it here

At Twing Data, we specialize in saving clients time and money on their data systems. By analyzing warehouse metadata, our stack-agnostic product builds end to end system models based on how they’re actually used, surfacing underused resources, overlooked workload inefficiencies, suboptimal warehouse scaling, and much more.

Curious if there’s room to optimize your data spend? We’d love to take a look and share what we find—no strings attached.

Stay tuned for a deeper dive into optimizing within BigQuery.

Data warehouse optimization tools connect to platforms such as Snowflake, Amazon Redshift, and Google BigQuery to analyze usage and improve performance and cost. Since these tools interface with sensitive enterprise environments, robust security practices are critical.

Below, we’ll dive into security aspects that you should consider when choosing warehouse optimization and observability tools for your organization. More generally, these guidelines can apply to other tools that access your infrastructure or data.

Data Access

How can you be confident in the safety of the tool that's analyzing your warehouse? That's a tough question. By knowing what it can access and by what means, you can begin to form an answer.

Least Privilege

An important aspect of security is the “principle of least privilege”, or making sure that different services (especially tools that aren’t developed internally) can only access what they need to be effective. In a data mart set up, that can be a lot of critical information, so it’s important to minimize exposing anything that doesn’t need to be.

Limiting an integration to a dedicated user is a good first step. Giving this user minimal permissions, such as read-only metadata-only access, also alleviates some risk as this user could be revoked with minimal disruption if needed.

For more complex integrations, it may be necessary to have role-based access controls (RBAC) to manage who has access to what. For example, someone inspecting the metadata may not be the same person setting up billing or data connections, but maybe they should still be able to see what connections are set up. Some tools can go even further and support data-specific privilege and limit who can access different parts of the (meta)data and optimization and observability results themselves.

Twing Data takes this principle seriously by supporting secure connections to your warehouses and limiting access to read-only metadata by design and following best practices internally.

API & Integration Security

Different vendors have different, and usually multiple, ways to connect to their systems such as passwords, key-pair authentication, gateways, or entirely unique service accounts.

It’s important to remember that encryption is only as strong as its implementation, and to verify that a more secure method actually is more secure. For example, simply switching from HTTP to HTTPS doesn’t guarantee security if certificate validation is improperly handled, allowing for man-in-the-middle attacks. Similarly, using OAuth 2.0 for API authentication is a step up from basic authentication, but failing to properly manage token expiration, refresh, and revocation could expose your integration to token hijacking or replay attacks. Always evaluate the entire security chain, not just the algorithm or protocol. We’ll talk more about encryption below.

A quick internet search can also reveal any vulnerabilities that were exposed in the past for a given vendor or user.

Data Management

Whether it’s just metadata like the queries being run or the data they return, whoever is analyzing the data needs to ingest and store it somewhere for analysis, and possibly to surface it back to their users. This section analyzes which aspects may require a closer look.

Data Retention

Since the vendor may have access to business-critical data (or metadata) it’s important to consider their data retention policy. What a “good” policy is will vary from business to business and use case to use case, but typically you don’t want someone to hold on to your data “forever” (especially not past the point of canceling the service). Long-term data retention can introduce unnecessary risks, such as unauthorized access, data breaches, or compliance violations. On the flipside, too short of retention might give less useful insights.

When vetting vendors, ask about their data retention and deletion policies up front and negotiate shorter retention periods or periodic purges, especially for sensitive information. It’s also a good idea to have an offboarding plan that verifies that the third-party has deleted your data after your integration has ended.

While some vendors don’t share this policy, Twing Data retains 90 days of metadata and persists the resulting analysis during the duration of service. This can be adjusted on a company-by-company basis if requested.

Data Isolation

There are also different methods for segmenting or isolating data and access. Separating customers’ data from one another can be done by setting up separate environments for each customer, ensuring that risk from one customer isn’t carried over to another. This can be especially important for services with privileges that extend beyond read-only, such as those that automatically adjust the size, clustering, or resources of your warehouses, or services that rely on data beyond metadata.

Twing Data can safely grant low-level access to our users by only exposing our analyses through company-specific accounts and views, ensuring that users (or threat actors) aren’t able to view anything they’re not supposed to.

Data Anonymization

Besides the tactics listed above, a vendor can, and should, anonymize sensitive data — even metadata. Even seemingly innocuous queries that may be executed can contain personal information that is not necessary for an external tool to see. Consider a query such as “WHERE username = ‘celebrity@personalemail.com’ AND phonenumber = ‘(555) 867-5309’” being exposed. A leaked query such as this not only exposes a high-profile individual’s email address but also ties it to their phone number.

Twing Data offers to redact query text at ingestion, replacing it with a constant placeholder token​. This way, we can still analyze query structures (e.g. to identify patterns or heavy queries) without storing any actual identifiers or literals. Keep in mind, though, some analyses may not be as useful if we can't identify what makes different query pattern executions different from one another. We’re always looking to strike the right balance..

Encryption (Data at Rest & In Transit)

Nowadays, it’s safe to assume that any enterprise integration is going to use a secure protocol, but it’s still good to make sure. Any person or server that is accessing your data (metadata or otherwise) should be doing so securely. If not — run!

The (meta)data itself should also be stored securely. Twing Data leverages Google BigQuery which encrypts data at rest and adds additional access controls and auditing.

App Architecture

The “cloud”, as we know, is generally “someone else’s servers”. It enables distributing and scaling applications across the world while optimizing cost and speed.

However, as infrastructure evolves and becomes more complex, there are more points of failure and more visibility and failsafes are needed. Let’s take a look at what that means for warehouse optimization tools.

Cloud Infrastructure Security

Similar to data isolation, containerization or virtual machine (VM) isolation can ease security concerns, especially for more than read-only access. A setup like this may spin up VMs per tenant, either on-premise or managed by another provider like AWS or GCP.

Some providers, like fly.io (which Twing Data uses for hosting our application), are easy to scale globally while maintaining security practices. Twing Data’s analyses run on demand and on-premises, interfacing securely with our databases.

Logging and Monitoring

For any organization, it’s important to log events to detect, prevent, or alert on security issues or outages. For big data, an outage can mean inaccurate insights and lost revenue, so alerting and monitoring is crucial. Similarly, monitoring and alerting help catch bugs or security vulnerabilities sooner than later (with additional context about what happened and why), making it quicker to resolve them.

For example, if an API integration suddenly starts returning a high volume of 500 errors, alerting can notify the team before customers are impacted. In a security context, logs showing a spike in failed authentication attempts could indicate a brute-force attack, allowing for a proactive response. Additionally, monitoring data pipeline performance can catch bottlenecks – like a slow ETL job – that might otherwise cause reporting delays or incomplete/inaccurate analytics.

Some services (like fly.io) offer built-in logging, monitoring, and alerting (e.g. Grafana and Sentry integrations) that makes it fast, easy, and secure to set up the necessary infrastructure.

Company Practices

Lastly, company practices can give additional insight into what to expect when working with them. Do they take security seriously as part of their culture, or just to check some boxes? If a problem occurs, how likely are they to fix it within a given timeframe?

Compliance & Certification

Some certifications (SOC 2 Type I) evaluate security at a single point in time, and others (SOC 2 Type II) evaluate security over a period of time. These evaluations are based on many factors such as access controls, monitoring & logging, incident response, data encryption & integrity, secure development practices, and others.

Seeing that a vendor has these certifications gives a clear indication of their security practices. It’s something Twing Data is working towards!

Incident Response & Recovery

A company’s processes for detecting, responding to, and recovering from security incidents is just as important as their processes to avoid and alert on such incidents. After all, an alert is only valuable if it cuts through the noise and prompts action – otherwise, it’s just another overlooked warning.

SOC2 certification requires to document related processes thoroughly and test them at least annually. Of course, companies without the certification can still have a plan in place.

These processes cover responsibilities and actions that would be performed internally (e.g. patching a vulnerability) and externally (e.g. notifying affected users) with clear steps for identifying, containing, mitigating, and resolving incidents. In some cases, a service level agreement (SLA) between the vendor and customer may dictate time-to-resolution requirements, ensuring accountability.

For example, if a critical data breach occurs, an engineering team might immediately isolate the affected systems while security and legal teams work together to draft and send a breach notification within the SLA-defined timeframe.

Secure Development Practices

Cutting-edge companies should embed security throughout the development lifecycle. Increasingly, security and code quality checks are happening earlier in the development lifecycle. Tools like static code analysis, linting, code reviews, and automated security scanning help catch issues before they even reach a staging environment.

Security-minded SaaS providers also conduct regular automated and manual penetration testing on their application and infrastructure​, and any critical findings are fixed as part of their vulnerability management process.

By embedding security holistically within the development workflow, companies can better safeguard sensitive data and reduce the overall cost and risk of fixing vulnerabilities later.

Third-Party Risk Management

Third-party risks typically encompass technical topics such as security breaches and downtime, but can include more qualitative risks like reputational risk.

Using reputable third parties like GCP and fly.io that have their own certifications is one way to mitigate risk that comes with outsourcing to external service providers, especially when it comes to some of the topics above around data and cloud security — items that are most commonly outsourced (since maintaining your own hardware comes with its own high costs and risks).

Vendors can also be classified based on risk — a database vendor is of higher risk than an office supplier, for example, and possibly doesn’t need to be reassessed as frequently.

Choosing the right data vendor for your business

As I was writing and researching for this article, the main takeaway became clear: it's essential to choose a vendor that analyzes your company's unique and mission-critical data effectively and securely. I hope these findings help you mitigate that risk by knowing what to look for and avoid when digging into vendors’ security policies.

To continue the Redshift pricing topic introduced in my previous blog, The Real Cost of Redshift Warehousing, I want to zoom in on Redshift Serverless. This product can be a major convenience for the right user, but it’s vitally important to understand the documented, and undocumented, details that govern their pricing policies in order to evaluate if it’s the right choice for your company.

In case you haven’t read my previous blog or are otherwise unfamiliar with Redshift Serverless, let me quickly outline the basics. The term “serverless” means that you don’t need to manage the servers—the provider handles all the heavy lifting behind the scenes (even though, of course, servers are still there). In a serverless data warehouse, the infrastructure is fully managed for you, scaling automatically with usage so that you only pay for the compute you actually use, with storage priced separately. Redshift’s take on a serverless system follows this model, though with some high-impact nuances that are the focus of this post.

Limitations of Redshift Serverless: Looking Under the Hood

When considering whether or not Redshift Serverless is right for you, it’s essential to understand what it actually is in contrast to a true serverless system like BigQuery, and how it can more accurately be thought of as a version of their typical provisioned cluster. To establish a frame of reference, let me explain a bit about both truly serverless systems and the Redshift provisioned cluster:

• As an example of a true serverless system, BigQuery, Google’s cloud data warehouse offering, was built from the ground up to provide all the hallmarks of this class: a shared compute pool dynamically allocates resources to tenants as needed, charged only for use, and storage is handled completely independently, billed per MiB per second. To be sure, BigQuery is a feature-rich solution with many configurations and an option provisioning compute as well, so I am simplifying here.

• A provisioned cluster is Redshift’s standard system option in which the user defines the attributes of a cluster - like node size/type, its execution behavior - like Workload Management (WLM) or Short Query Acceleration (SQA), as well as its scaling boundaries, as in Concurrency Scaling and Elastic/Manual Resize. Storage either falls within Redshift Managed Storage (RMS) or uses external S3 and is accessed at a small premium via Spectrum. RMS capacity is linked to cluster composition, as different node sizes/types offer varying max RMS capacity per node. In this way, while storage is priced separately, it is yet tied to compute. A provisioned cluster offers the cheapest per-unit compute pricing, but not only do charges accrue as long as it’s running, even when idle, but also potentially requires an enormous amount of configuration, monitoring and management across both workloads and storage to achieve reasonable cost and efficiency.

TL;DR: A Redshift cluster is an isolated instance with relative compute and storage limits managed at the individual level and rented by the hour, while a traditional serverless system decouples storage and compute, scaling them independently and pricing on demand.

With these differences in mind, let’s now examine Redshift Serverless. While traditional serverless leverages multi-tenant compute infrastructure, Redshift Serverless, under the hood, is actually running on clusters of nodes identical to those that can be configured and run manually. Rather than accessing a shared, always-on system of clusters for compute, Redshift’s version merely manages an isolated system on the user’s behalf and prices compute in Redshift Processing Units (RPU). It’s important to understand this for a few reasons:

1. RMS storage has limits. Serverless storage limits are addressed in exactly 1 bullet point in the docs, and if you miss it, you could reasonably be under the impression that it scales limitlessly, automatically, and only increases Redshift Managed Storage (RMS) costs. This is not true. RMS has max capacity based on node types and counts in the cluster in use. The minimum RPU settings essentially define the amount of available RMS. If data volume exceeds that max capacity, the minimum RPUs that will be used for any workload will be the number that can handle the needed amount of storage. The way to work around this is to manually place a portion of the data in external S3. While Spectrum is required to scan external S3 data, its use is included in the per RPU pricing.

2. “On-demand compute” is not always available on demand. Even when no compute resources are being used, there are ongoing costs for things like hardware maintenance, power, and cooling. With an on-demand pricing model, customers only pay when compute is in use, but the host still has to handle these ongoing costs. In a truly serverless system, a multi-tenant architecture manages this by balancing resource usage across many customers, which helps reduce waste. Amazon’s version of serverless boasts on-demand pricing, yet manages resources at the level of individual accounts. There’s no mechanism to pass unused resources from one account to another. Do they simply eat the cost of idle resources for their inactive serverless customers? Absolutely …not. After 1 hour of idling, the instance fully deactivates. When it’s inevitably needed again, the entire cluster has to reactivate first, which, in our experience, can take up to around 10 seconds. If the efficiency of that workload was business-critical, RIP. Since the one-hour deactivation threshold isn’t adjustable, your only options are to schedule important queries within an hour of another query or to send periodic pings to keep the system active.

Redshift Serverless Minimum Pricing Nuances: The Risk of Blind Trust

Minimum Charge Policy: “You pay for the workloads you run in RPU-hours on a per-second basis, with a 60-second minimum charge.”

On first pass the policy seems aggressive but pretty straightforward. A naturally-emerging follow-up question, “How many RPUs factor into that minimum?” actually has an innocent answer: your base (minimum) RPUs. Even if we stop here and assume that’s all there is to it, it’s clear that without purposefully managing your workloads around the minimum charge policy, your warehouse bill will likely be many times what it should.

We’ve indeed found Redshift’s minimum pricing policies to have an outsized impact on our clients’ warehouse costs. And not just because the policy is completely overlooked. The docs are actually deceptive on this topic. What constitutes “...workloads you run…” is actually a combination of the queries you execute and system queries related to the environment and metadata tracking processes. Also included are mysterious seconds that I haven’t been able to account for. Consider the following query and results:

Since this is rolled up by minute, you might assume that query_run_seconds should be the same as total_used_seconds. That’s quite a discrepancy! 22 seconds -> 177? There’s even more: in any of these minutes I actually ran between 0 and 2 queries. Queries logged to sys_query_history include those which ran automatically as part of background processes, which may or may not be directly associated with the workload you ran. To understand what’s happening, let’s dig in to examine the queries logged for minute 15:46:00:

The highlighted queries are the ones I wrote. The last one wasn’t even run in that minute, but added to 15:46:00 later. Everything else was often-redundant metadata fetching or tracking/validation behavior related to the environment or systems. 3 queries became 16. At least they’re fast. But, given that the total query_run_seconds for all 16 queries is 3 seconds, yet the calculated total_used_seconds is 24, we can guess that there are some major associated, undocumented processes that accrue to sys_serverless_usage.compute_seconds. Here’s an even more surprising example from a few minutes previous:

I ran 1 query during this minute, and it maybe took a second or two. RS recorded 3 runs of the same query, one taking 22 seconds! Then, the total ~22 query_run_seconds recorded for the entire minute translated to 177 seconds used. A two second query run became 177 seconds “used”!

For this experiment I used the smallest base capacity possible, and didn’t dare go over the minimum charges per minute. In the real world this sort of behavior can lead to massive overspend. I wish I was able to wrap this up with a recommended implementation, and perhaps we’ll discover one and write a blog. The conclusion to make so far is that this behavior must be monitored. At a certain scale of overcharge, managing your own cluster might be worth considering.

Who Benefits From This Setup?

As a rule, abstracting away lower-level complexity introduces various types of costs that are difficult to track and sometimes impossible to correct. Serverless systems have some degree of this issue by nature, prioritizing simplicity and efficiency at a premium price. I would not put Redshift Serverless in this category, though. While it nails the premium price tag, the simplicity and efficiency are not included to the same degree. Where true serverless systems are built from the ground up to benefit from the scale of distributed compute and separate storage, Redshift’s version is merely a repackaging of the same isolated system offering, with some abstracted optionality based on an arbitrary, derivative metric RPU. No shared, always-on compute pool, no care-free limitless storage, but you will get a black box management layer that partially obstructs the visibility into how it operates.

On the positive side, if your stack already includes other AWS products, RS Serverless does earn simplicity points, as AWS products integrate nicely together. And if you’re otherwise a prime candidate for a serverless warehouse (ie: lacking monitoring knowledge/resources or have uneven/uncertain workloads) it might still be the best option for you. Just make sure to deliberately set usage limits and base capacity (default is 128 RPU, which may be too much).

At Twing Data, we use our warehouse-agnostic query parsing and analysis system to help our clients surface and diagnose a variety of inefficiencies, anomalous behaviors, and antipatterns across the data stack. We are always pushing to better understand our clients’ stacks. If you have insights into how to best use or understand Redshift Serverless, please reach out, or, better yet, leave a comment here for others to find!

Teams are storing petabytes of data, yet stakeholders don’t trust the reporting, or even the data points underlying them. Unfortunately, this is a common occurrence when data teams are the recipients of an endless queue of isolated requests, to be completed without further interrogation.

Our best advice on making data teams strategic partners, rather than simply receivers of requirements, might also be counterintuitive: start manually. To understand why, let’s start with the biggest challenges we’ve come across.

Siloed data teams

We see this often in our work: data teams are often too far removed from the product, without strong product management. One team will produce a data point, and then it falls to the data engineering team to transform it into a table or a report that can subsequently inform business decisions. But the feedback loop between getting the data and making it actionable is often broken.

Ramp, recognizing this challenge, has implemented a successful approach that embeds data skills within the business-aligned teams. They allow more contributors to the code base, enabling the organization to tighten the feedback loop between data insights and the business decisions that follow from them.

Requirements without context

Data teams are often on the receiving end of requests from other teams, and are forced to support a growing list of requirements without having the necessary context to understand the business case. As a result, teams can’t make reasonable tradeoffs, and end up developing Frankenstein models that become more costly to maintain.

We’ve seen cases where teams are proud that they have thousands of dbt models, but we think teams should be aiming for the opposite: doing more with less. The benefit is obvious – by simplifying your data, you’ll have less of it to manage. Queries will run faster, insights will be generated faster, and you’ll be able to move faster since you’re dealing with less overhead.

Our approach: Start manually

The best advice we can give is to validate the need and the solution manually at the outset. Data storage is much cheaper than compute – so rely on the fact that you can always rederive the necessary data, rather than automating the end-to-end flow. By starting manually, you will validate that there is a recurring need and only then should you focus on productionalizing the process.

As Kent Beck famously said, “Make it work, make it right, make it fast.” Here’s how to put that into action:

• Make it work: Validate the need manually.

• Make it right: Productionalize and design for maintainability and scalability, and see how it fits in with your other data flows.

• Make it fast: Optimize, reduce costs, and reduce latency after you have real validation that the output is being used.

Despite there being obvious wins in optimizing the data warehouse, the real win comes from getting crystal clear on the requirements and having a deeper understanding of the tradeoffs.

Unlike other software engineering disciplines, data has much more inertia: it’s difficult to remove something once it’s built. If you haven’t gone through the exercise of validating the need first, it may be a lot easier to not build the thing in the first place — especially if there’s a chance it balloons into 10,000 tables.

At Twing Data, we take a holistic view of our clients’ data stacks. This requires the ability to understand, debug, and diagnose many data systems. The data warehouse is a major focus, as it can determine whether a data system meets or falls short of a company’s needs in terms of performance and budget. The pricing models of these modern warehouse solutions can be nuanced, where a misunderstanding in configuration can potentially go forever unnoticed.

In this three-part series, we’ll take a focused look at three major data warehouse solutions, Redshift, BigQuery, and Snowflake. In part one, we’ll focus on Amazon Redshift. As Amazon's flagship warehouse offering, Redshift introduced many to the possibilities of cloud-based data warehousing, but its pricing model and configuration options demand careful consideration for optimal performance and cost management.

Understanding Redshift's Pricing Model

Redshift offers a configurable pricing structure that can be as powerful as it is complex. Pricing is divided into two main components: storage and compute, with options that cater to both smaller datasets and large-scale enterprise-level use cases. Three main system options are available:

DC2 Node Cluster

Storage and compute are coupled, leading to less flexibility in system configuration and higher overhead for resizing.

The case for DC2 Nodes is increasingly harder and harder to make, as reserved RA3 instances and serverless systems better suit modern data warehousing needs (ie: high variability in compute with independently-scalable storage)

RA3 Node Cluster

The most flexible option for both regular and variable workloads, this model offers scaling by adjusting node size (elastic resizing), cluster size (node count), and number of concurrent clusters

Redshift Serverless

This is a fully-managed data system, available on demand, that automatically scales to accommodate current workloads. With no scaling or cluster uptime to manage, Serverless offers the shortest path to reliable results. While compute costs are only incurred while a workload is active, the optionality comes at a premium price not readily detectable in the docs. Lack of scaling control and the minimum charge policy (see Bonus: Death by 1000 Minimum Charges below) can result in significantly higher compute costs overall. It’s best for companies that:

• Are uncertain about the cadence of their workloads

• Have irregular workloads

• Lack the peoplepower, technical acuity, or desire to manage an RA3 system

Coupled with the above configurations, AWS offers the option to reserve capacity for Redshift, committing to usage for a set period (e.g. 1 or 3 years) in exchange for a discounted rate. The discount scales with the length and value of the commitment, making it an attractive option for many organizations. Reserved capacity applies to various Redshift configurations, including RA3 nodes and serverless compute. However, this approach requires accurate forecasting of usage levels to maximize cost savings. For most growing companies, reserving capacity can help mitigate the resources needed for actively managing cluster uptime, while also incentivizing efficient workload distribution to optimize utilization.

Inherent Challenges with RA3

With great flexibility comes great complexity, and oversights can be costly. Redshift RA3 clusters are billed based on node size and uptime. Whether it’s at max capacity or sitting idle, AWS charges for that time. How much? That depends on the node/cluster size, as well as the number of concurrent clusters running. All options remaining equal, this essentially hinges on how fast you need data at times of peak usage. Even when factoring in the discount for reserving nodes, a persistently-running cluster with irregular heavy usage can be expensive if not closely managed. Redshift does offer a few automated or potentially-automated features to help – but be aware of a few gotchas with each:

Short Query Acceleration (SQA) - automated

• Queries are classified as short or long based on a predicted runtime threshold. Short queries are prioritized, ensuring the maximum number of workloads are completed in a given timeframe

SQA Gotchas:

• Prioritizing one group of queries means deprioritizing others. When execution time does not factor into the priority equation, this could lead to longer queries being perpetually deprioritized (ie: thrashing).

• The threshold is dynamic by default - it changes based on machine learning using real usage. Might require experimenting with hard values.

Auto Workload Management (Auto WLM) - automated

• General workload management system dynamically adjusts resource allocations within the cluster depending on current query load

Auto WLM Gotchas:

• Not great for predictable, static workloads where fine-tuning resources is possible.

• Requires an additional layer of management in Query Monitoring Rules (QMR), which govern the thresholds used by Auto WLM.

• Critical queries that require guaranteed resources are likely hard to target using QMR.

Concurrency Scaling - automated

• An additional temporary cluster is automatically provisioned to handle the overflow of queued queries during peak usage. When the overflow is handled, it shuts down. Redshift offers the first hour of concurrency scaling free, per day per cluster (this should be plenty to cover peak usage times).

Concurrency Gotchas:

• If the free hour is exceeded, the concurrency scaling cluster is subject to the same per second charge, with 60-second minimum charge for each activation.

The general theme, as ever, is that the more complexity is hidden, the greater the potential sacrifice in speed and/or cost. Even concurrency scaling, which can be completely free if the system is well-optimized, can end up doubling compute costs if it’s not. Either way, tight monitoring is essential; we’ve seen situations with our customers where concurrency scaling simply stopped working without any changes to the data warehouse and complete silence from AWS.

In addition to concurrency scaling, other scaling options to be aware of involve changing the node type, size or count within the main cluster, and come in two flavors, each to be used in specific contexts:

Elastic/Classic Resize - schedulable, automation possible

• Features used for adjusting node type, size, or count. Out of the box, these are either triggered manually or scheduled, but, using the API, the operation can be integrated with monitoring for full automation. Elastic is faster but more limited in terms of types of scaling possible. If elastic doesn’t serve your purpose, classic can handle more complex changes.

Resizing can be a blog topic on its own, and beyond the scope of this work. A docs link will have to suffice: https://docs.aws.amazon.com/redshift/latest/mgmt/resizing-cluster.html

Saving Data, Losing Dollars

Up to now, we haven’t touched on details around storage management. If you’re already sold on serverless, you’re in luck - storage scales 100% independently of compute and is charged per GB/month. If RA3 is more your flavor, and you have or expect to have significant amounts of data, get ready for more to manage!

RA3 nodes of different sizes have corresponding compute and storage stats. Redshift Managed Storage (RMS) capacity ranges from 1-128 TB depending on the node size. Due to the hard limits on RMS capacity, both compute and storage needs must be factored into the decision around node size/count. While storage is technically billed separately (per GB/month), these storage limits essentially manifest in a partial coupling of compute and storage billing. Maxing out the size and number of nodes in a cluster, the storage limits can reach petabyte scale. But as nodes are added and sized according to max RMS needs, uptime costs increase. Given modern companies collect increasingly more and more data, it’s no wonder solutions that completely decouple compute and storage, like Redshift Serverless or Snowflake, are so popular.

The only independent scaling solution for RA3 cluster users is to store certain data in S3 and access as needed using Redshift Spectrum. While there are cost and efficiency sacrifices compared to RMS, they are typically small if handled correctly. Here are some suggestions:

• Avoid data transfer fees by storing data in the same location as your cluster

• Experiment with format/compression types to determine the most efficient

• Partition data by frequently-filtered columns

At Twing Data, we test your flows with various file/compression formats as part of our system optimization service.

Bonus: Death by 1000 Minimum Charges

Consider two scenarios, in which two companies with unique needs choose their systems wisely:

Scenario 1: Out of uncertainty in data system requirements, you’ve decided not to reserve your new company’s RA3 instance. But you’ve compensated by painstakingly studying, testing and optimizing its configs, and have seen significant cost savings and efficiency gain.

Scenario 2: Given your growing company’s sizable task backlog and fast-changing data needs, you decide to forgo the complexity of managing a cluster and use Redshift Serverless, which has allowed your small team to build a data system quickly while still focusing on business-critical items.

Similar to these scenarios, at Twing Data we frequently see talented teams making sound decisions validated by positive outcomes. Yet they’re still likely overpaying on their data warehouse. One reason we frequently see is Redshift’s minimum charge policy, a nasty little detail that deserves much more consideration than it usually gets. Non-reserved RA3 clusters and Serverless options both have similar per-workload and per activation minimum charge policies. To manage around them, each requires a unique approach.

Policy: 60-second minimum per activation, and charged per second of uptime after that.

RA3: Since charges are based on cluster uptime, if your instance is not reserved, it is likely because you plan on managing cluster uptime based on less-frequent, periodic access. The risk here is triggering deactivation/reactivation within a minute, causing multiple 1-minute charges in that time. If possible, designate downtimes explicitly and orchestrate workloads accordingly.

Serverless: Charges are based on resources in use only. These systems manage uptime automatically, shutting down after 1 hour of idle time. The docs are very unclear about how minimum charges apply to serverless instances, so I’ll clarify based on our experience:

• The 60-second minimum charge applies to every minute in which a workload is run according to the base capacity of the instance

• The per-second charge applies to the amount of seconds within the minute a certain number of RPUs (Redshift Processing Units) were used.

• You are charged for at least the full billable minutes at the selected base (minimum) capacity of your instance for each minute any workload is running. It’s possible to exceed the minimum charge if capacity is scaled to meet higher demand within that minute.

Due to this policy, serverless systems are best suited for data systems that see irregular usage, as the heavy-handed minimum charge policy is offset by the lack of compute overhead when the system is idle.

Another thing to note about the serverless system is that 1 hour of idle time will trigger a deactivation event. If your company frequently goes > 1 hour without a new workload, the serverless instance will timeout. In our experience, when the system reactivates for the next workload, it introduces up to 10 seconds of latency. Remember, idle time is uncharged in a serverless setup, so it’s only a disadvantage for the instance to deactivate. We’ve seen companies avoid instance deactivation by scheduling a simple query, such as “SELECT 1;” every 55 minutes or so; it’s a simple tradeoff to pay for an occasional minute of base execution to avoid latency issues.

Takeaways

Amazon Redshift is a capable and versatile data warehouse, with options for a broad range of requirements. And, it should be noted, if already using other AWS services, the benefits of using Redshift, such as the Aurora/Redshift zero-ETL integration, are significant. However, the complexity involved in selecting, monitoring and maintaining an optimal system is high, with the potential for massive unnecessary expense. In the next part of this series, we’ll explore Google BigQuery, which takes a completely different approach to pricing and resource allocation.

For more on optimizing Redshift, we’ll soon release a breakdown between sort/dist keys and compare them with partitioning and clustering columns used by other modern cloud warehouses. Keep an eye out!

At Twing Data, we understand that no two businesses are alike, and neither are their data warehouse needs. With our expertise in query metadata analysis, organizations gain visibility into their Redshift usage patterns and the knowledge needed to make informed decisions. Reach out to learn how we can help you uncover hidden costs and turn insights into actionable savings.

There’s a hint that they take a more thoughtful approach in their docs, where they mention that “select * from table” and “select id from table” will generate the same hash if that table has a single column. That wouldn’t be possible from looking solely at the query text without a deeper understanding of the table structure.

Now on to the test cases:

Single values comparisons

SELECT a from (select 1 as id, 'a' as a) b where id in (2);
SELECT a from (select 1 as id, 'a' as a) b where id in (1);
SELECT a from (select 1 as id, 'a' as a) b where id = 2;
SELECT a from (select 1 as id, 'a' as a) b where id = 1;

In this case, Redshift correctly recognized that these were the same query, unlike Snowflake which only correctly handled the strict equality condition. But as soon as we expanded it to “where id in (1, 2),” Redshift generated a different query hash..

Table aliasing

SELECT name from (select 1 as id, 'a' as a, 'Test' as name) where id = 1;
SELECT d.name from (select 1 as id, 'a' as a, 'Test' as name) as d where d.id = 1;
SELECT c.name from (select 1 as id, 'a' as a, 'Test' as name) as c where c.id = 1;
SELECT b.name from (select 1 as id, 'a' as a, 'Test' as name) as b where b.id = 1;
SELECT name from (select 1 as id, 'a' as a, 'Test' as name) b where id = 1;

Redshift detected that these were all the same table and gave each the same hash. Snowflake, on the other hand, generated different query hashes for each.

Table qualification

SELECT name from customers where customer_id = 1;
SELECT name from public.customers where customer_id = 1;
SELECT name from testing.public.customers where customer_id = 1;

Unlike Snowflake, Redshift detected that these queries were all using the same table and gave each the same hash.

Column aliasing

select name as other_name from (select 1 as id, 'a' as a, 'Test' as name) where id = 1;
select name from (select 1 as id, 'a' as a, 'Test' as name) where id = 1;

In this case, Redshift generated different query hashes despite the structure of the query being identical.

Column order

select city, name from (select 1 as id, 'a' as city, 'Test' as name) where id = 1;
select name, city from (select 1 as id, 'a' as city, 'Test' as name) where id = 1;

Similarly, here, Redshift didn’t generate the same query hash here despite the same data being returned.

CTE order

with d as (select name from (select 1 as id, 'a' as a, 'Test' as name) where id = 2),
     c as (select name from (select 1 as id, 'a' as a, 'Test' as name) where id = 1)
     select c.name, d.name from c join d on 1=1;

with c as (select name from (select 1 as id, 'a' as a, 'Test' as name) where id = 1),
     d as (select name from (select 1 as id, 'a' as a, 'Test' as name) where id = 2)
     select c.name, d.name from c join d on 1=1;

Once again, Redshift had trouble with the order.

Overall, Redshift did a better job of parsing compared to Snowflake, correctly mapping the cases where the differences were in the table structure (aliasing) – but didn’t do as good of a job for column renaming, or when the the clauses were reordered.

Here’s a quick side by side comparison of Redshift vs Snowflake for all the test cases.

The current crop of cloud infrastructure is really good at a couple things: getting you up and running quickly, and getting you to spend a lot of money.

It’s a great boost to productivity when you can get your AWS or Snowflake instance humming in minutes, but it’s easy to forget that there are actual computers – with nebulous pricing and processes – behind the scenes. The costs of managing data and infrastructure has left many a finance team scratching their heads.

There are horror stories of surprise bill (a $90,000 AWS bill, an $8,000 Snowflake charge) that were dramatically higher than people expected. It’s easy to blame the data engineer or team who incurred the cost, but part of the issue is the design of the tools themselves. They’re not incentivized to safeguard your organization against wasteful spending—but there are a couple best practices you can implement to prevent it.

There are two things you should to get ahead of any unpleasant surprises:

1. Tag everything. What is the environment of this resource? What is the service or application that is consuming the resource? If it’s a query which API or users are triggering the request?

2. Track your unit costs. Most services you run should be correlated to some unit of your business. If you run an ecommerce business, your unit cost may be orders or customers. If it’s video streaming, your costs are going to be correlated with hours streamed and active users. If it’s healthtech, it might be a function of patient records or generated reports.

The key is to understand what’s the primary driver of your service costs and start tracking the unit costs over time. As things get more complex, you’ll also need to have different unit costs depending on the service. You can break this down into units for different systems. The ultimate goal is to get to apples-to-apples comparisons on a recurring basis, to see if there are any significant changes. If anything looks off, you can then use the tagged costs to dig into those items.

At TripleLift, we refined a cost tracking system over multiple years that became increasingly automated over time. Since we were running an ad exchange, we knew that the majority of our infrastructure costs was a function of the number of bid requests we sent when running an ad auction. Our data pipeline costs, on the other hand, were correlated with the number of auctions run. By breaking these down and looking at trends over time, we could track patterns and dig into anomalies.

For example, when we introduced a new type of data job or report, we would see an increase in unit costs for our data pipeline. On the other hand, when we optimized our Spark clusters to use Spot instances, we saw a decrease in unit costs. Without this visibility, it would have been incredibly difficult to understand what was happening.

The benefits of this sort of tracking compound over time. You can start accurately forecasting your infrastructure costs. You’ll instill a culture of discipline and efficiency with your organization. And, no promises, but you may even score some hard-won brownie points with your finance team.

Here’s the LinkedIn thread that inspired this post.