For years, the data lake and the operational database have occupied different architectural worlds until Random Access Parquet.
Data lakes were designed to store enormous volumes of information economically. They became the foundation for analytics, machine learning, experimentation and increasingly artificial intelligence. Operational databases, meanwhile, were designed for a different requirement: retrieve a particular record quickly, often within milliseconds.
That separation has created a familiar enterprise pattern.
Data is stored in a lake for analytics. Selected data is then copied into a key-value store or operational database for applications that require low-latency access.
Spotify is challenging that pattern with Random Access Parquet (RAP).
The architecture adds an external indexing layer over Apache Parquet data. It allows online services and AI applications to locate individual records directly inside the data lake rather than first copying those records into a separate serving database. Spotify says the same Parquet files can continue supporting analytics, machine learning, notebooks and experimentation while also serving interactive point queries.
The significance goes beyond Spotify.
RAP points towards a possible change in how enterprises think about the boundary between analytical storage and online serving.
The fundamental problem: data lakes are excellent at scanning, not looking up
Apache Parquet is a columnar storage format. It is highly effective when an analytical engine needs to process large quantities of data while reading only the columns required for a particular computation.
That is precisely what modern analytical platforms need.
A query calculating listening activity across millions of users can scan large datasets efficiently. A machine-learning pipeline can process enormous collections of records. A business-intelligence system can aggregate billions of events.
But consider a fundamentally different request:
“Show me everything this user listened to last summer.”
The system needs a relatively small amount of information from an enormous dataset.
That is a point-query problem.
Distributed SQL engines such as Trino and BigQuery are optimised for analytical throughput, not necessarily for interactive key-based lookups. Query planning, metadata processing and file discovery can introduce substantial overhead before the system reaches the actual bytes containing the requested record. Spotify describes this as a mismatch between the access pattern and the architecture of the query engines.
Spotify’s scale makes that mismatch particularly important.
The company says petabytes of data reside in Bigtable for online use cases, while exabytes sit in its Google Cloud Storage-based data lake. Maintaining another serving copy of increasingly large datasets creates storage, synchronization and operational costs.
RAP approaches the problem from the opposite direction.
Instead of moving more data out of the lake, it tries to make the lake easier to access.
The architectural shift: know where the record is before reading the file
The core idea behind the Random Access Parquet data lake architecture is deceptively simple.
Create an external index that knows where a particular key exists.
A lookup might begin with a user ID. The index maps that ID to:
- the relevant Parquet file
- the relevant rows
- and, where useful, additional information about the values
The serving layer can then perform targeted ranged reads against cloud object storage.
The query no longer needs to search through thousands of files to discover where the requested information lives.
This distinction is critical.
A conventional approach effectively asks:
“Which file contains this user?”
RAP attempts to answer that question before the data files themselves have to be searched.
Spotify describes the index as a multimap because one key can occur across multiple files and partitions. Its entries can contain the lookup key, a dictionary-encoded file identifier, row numbers and optionally a value count that can support pagination.
The result is a different I/O pattern.
Rather than a chain of dependent reads, the system can identify the required locations and issue precise reads in parallel.
That matters because latency is often determined not merely by how many bytes are read, but by how many dependent operations must happen before the next read can begin.
Why cloud object storage changes the equation
The RAP architecture arrives at an interesting point in the evolution of cloud infrastructure.
Object storage was once regarded primarily as cheap, scalable storage with relatively high access latency.
That assumption is changing.
Spotify notes that Google Cloud Storage can provide approximately 30–100 millisecond request latency, while newer services such as S3 Express One Zone and Google Cloud’s Rapid Storage offerings target much lower latency.
As the underlying storage becomes faster, another layer becomes increasingly important.
The question becomes:
Can the software stack reach the right bytes quickly enough?
If the storage system can retrieve a small range rapidly but the query engine spends seconds planning the query, discovering files and traversing metadata, faster storage alone does not solve the problem.
RAP therefore targets the access path above object storage.
Its external index provides the location information needed to bypass much of that discovery process.
The index does not replace Parquet
One of the most important characteristics of RAP is that it does not require replacing Parquet with a proprietary database format.
Spotify says RAP can operate on existing Parquet files without special preparation. The index builder reads file metadata and page locations, identifies the relevant key columns and constructs the external mapping.
That preserves an important property of the modern data lake.
The same Parquet files can remain available to conventional analytical systems.
This is crucial because enterprises generally do not want to redesign their entire data platform merely to support one new access pattern.
The architecture instead introduces another access path.
The underlying data remains in Parquet.
The index tells a latency-sensitive reader where to find it.
That separation also means the index can evolve independently of the data files.
Append-only indexing fits immutable data lakes
Modern lakehouse architectures commonly rely on immutable or append-oriented data files.
Spotify’s approach fits that model.
As new data arrives in Apache Iceberg tables, the index builder creates corresponding index fragments. Rather than modifying previously written Parquet files, the index grows through appended fragments.
This is an important engineering choice.
Rewriting enormous Parquet datasets every time an index changes would undermine much of the economic advantage of object-based data lakes.
Keeping the index separate allows the data files to retain their analytical characteristics while the serving layer maintains additional access structures.
There is, however, a trade-off.
External indexes introduce another piece of infrastructure that must remain consistent with the underlying data. Apache DataFusion’s research on external Parquet indexes highlights the same issue: external indexes can significantly accelerate access, but they introduce operational overhead because the index must track changes to the underlying files.
RAP therefore does not make data architecture simpler in every respect.
It moves complexity from data duplication towards indexing and serving infrastructure.
The hidden challenge: a correct index is not enough
Finding the correct Parquet file and row is only part of the problem.
Suppose the index identifies the correct row.
A conventional Parquet reader may still need to retrieve an entire data page containing that row. If the desired record is tiny compared with the page, the system could end up reading substantially more data than the application actually needs.
Spotify therefore describes several write-time optimisations designed to make random access more efficient.
This is where the architecture becomes particularly interesting.
Sorting by key
Sorting records by the primary lookup key keeps records belonging to the same key close together.
That can concentrate a user’s records into fewer pages and reduce the amount of data that must be fetched.
Hash bucketing
Hash bucketing can deterministically map a key to a particular file within a partition.
That reduces the number of files that a point query may need to touch.
Coarser partitioning
Partitioning data too finely can increase the number of files associated with a single entity.
Spotify notes that moving from daily to weekly partitioning, for example, can reduce the number of partitions a user’s data spans across a year from 365 to 52.
One page per key
The writer can arrange data so that a key’s records occupy their own page.
That allows the index to point much more precisely to the required data.
ZSTD frame resets
Spotify also describes resetting ZSTD compression frames at key boundaries.
This permits the reader to address compressed data more precisely without requiring the preceding data in the page to be decompressed first.
These optimisations reveal an important principle.
Random access performance is not solely an indexing problem. It is also a data-layout problem.
Reducing the number of storage requests
The number of bytes transferred matters, but the number of remote reads matters too.
Imagine an application needs ten columns from a user’s record.
A conventional columnar layout may require separate reads for multiple columns.
Spotify’s RAP design explores techniques to reduce those requests.
One option is to combine fields needed for random access into a single blob or Variant column. JSON, Protobuf or Parquet Variant can allow an application to retrieve a logical record through fewer storage operations.
Another approach is physical interleaving.
Instead of placing all values from one column together and then all values from another, selected columns can be physically arranged so that the data required for a particular key is adjacent.
The RAP reader can then issue one contiguous ranged read.
This creates an intriguing hybrid.
Parquet remains a columnar analytical format, but selected parts of its physical layout become optimised for row-oriented retrieval.
Spotify describes this as effectively pivoting the layout towards row-major access for selected column groups while retaining compatibility with conventional Parquet readers.
There is a trade-off.
An analytical reader requesting only one interleaved column may encounter additional data that it does not need.
That is why the right layout depends on the workload.
The architecture is not claiming that row-oriented storage is universally better than columnar storage.
It is recognising that one dataset may need to support multiple access patterns.
The most extreme optimisation: don’t read the data at all
RAP also supports the idea of a covering index.
If the requested value is sufficiently small, the index can contain the value itself.
The lookup then becomes:
key → value
rather than:
key → file → row → bytes
That can eliminate the storage read altogether. Spotify describes this as hoisting small values directly into the index.
This is a familiar database principle applied to a data-lake architecture.
A conventional covering index stores enough information to answer a query without visiting the underlying table.
RAP extends that idea to object-backed analytical files.
The broader lesson is significant:
An index does not always need to point to data. Sometimes it can become the data required to answer the query.
Secondary indexes make the architecture more interesting
Real enterprise datasets are rarely queried through only one key.
A transaction dataset might need to be accessed by:
- transaction ID
- customer ID
- buyer ID
- seller ID
- merchant ID
RAP can support multiple lookup dimensions through separate access structures over the same indexed entries.
Hash-based structures can provide exact lookups, while sorted indexes can support range queries.
This creates an interesting separation between storage and serving.
The underlying data pipeline does not necessarily need to be redesigned whenever a new access pattern emerges.
A serving layer can add another index.
That is powerful, but it also reinforces the need for disciplined index governance.
Every additional index consumes storage and introduces maintenance requirements.
At Spotify’s scale, even a compact index becomes substantial. The company gives a rough rule of thumb that indexing terabytes can produce gigabytes of index data, while indexing petabytes can produce terabytes.
The index is therefore not free.
The architectural question becomes whether its cost is lower than maintaining another copy of the data in an operational system.
Why this matters for AI agents
The most forward-looking aspect of RAP is not conventional application serving.
It is AI.
An AI agent frequently needs information that does not fit neatly into a predefined application query.
A user may ask:
“What was I listening to last summer?”
Or:
“Which songs did I repeatedly listen to during my trip?”
The agent needs to retrieve historical information before an LLM can reason over it.
That creates a retrieval requirement that is different from conventional application APIs.
The required information might be:
- old
- rarely accessed
- distributed across large datasets
- specific to one user
- useful only occasionally
Keeping every possible piece of historical context in a high-performance KV store can be expensive.
The Random Access Parquet data lake architecture offers another model: retain the historical dataset in the lake and make selected dimensions directly addressable.
The AI system does not necessarily need a complete operational replica of the data.
It needs an efficient way to reach the relevant records.
That distinction could become increasingly important as enterprise AI agents move beyond retrieval from curated knowledge bases and begin working with operational and historical data.
The data lake becomes both analytical and interactive
This is ultimately the architectural proposition behind RAP.
Traditionally:
Data lake → analytics and ML
while:
Operational database → applications
RAP introduces another possibility:
One dataset → multiple access paths
The same Parquet files can support batch analytics through systems such as BigQuery while an indexed serving layer retrieves selected records for interactive applications or AI agents. Spotify explicitly describes the same files being used for analytics, ML, experimentation and online retrieval.
That could reduce one of the most persistent forms of data-platform duplication.
Organizations often maintain:
system of record → data lake → serving database → cache
Each additional layer creates another synchronization problem.
If historical or low-frequency data can instead remain in the lake while still being retrieved interactively, some of those copies become less necessary.
This does not mean operational databases disappear.
Highly dynamic transactional workloads still require transactional guarantees, predictable write performance and application-specific serving characteristics.
RAP is better understood as expanding the range of workloads that can reasonably remain on the lake.
RAP is not entirely synonymous with external indexing
There is another important technical qualification.
External indexing of Parquet is not an idea that originated exclusively with Spotify.
The Apache DataFusion project published detailed work in August 2025 describing external indexes, metadata stores and caches for accelerating Parquet queries. That work explains how an external index can identify relevant files and then narrow access to specific portions of those files.
Spotify’s contribution is therefore best understood in terms of its particular engineering implementation and workload.
The company has focused the concept on low-latency point queries at very large data-lake scale, combined it with specific Parquet layout techniques and explored its use for online services and AI agents. Spotify’s engineer Will Edwards also shared the work with the Apache Parquet community in July 2026.
That distinction matters.
The story is not that Spotify invented external indexes.
The story is that Spotify is demonstrating how an external-index approach can be engineered into a practical serving architecture for enormous analytical datasets.
What changes for enterprise architecture?
The long-term impact could be less about Parquet itself and more about architectural economics.
Enterprises have historically decided which data deserves online serving based partly on access frequency and infrastructure cost.
Frequently accessed information gets promoted into operational stores.
Rarely accessed historical information stays in the lake.
That creates a hard boundary.
RAP potentially softens that boundary.
Historical information does not necessarily need to be copied into an expensive serving database simply because an application occasionally needs it.
The lake can remain the primary storage layer.
An index can make selected portions accessible.
This could be particularly relevant for:
- customer-service AI
- personalization
- recommendation systems
- fraud investigation
- financial history
- enterprise search
- customer experience platforms
- agentic AI
- long-tail application features
The important phrase is long-tail data.
A record that is accessed once a month may not justify a permanent operational copy.
But if the cost of retrieving it becomes closer to the cost of a targeted object-storage read, the economics change.
The remaining trade-offs
RAP should not be interpreted as a universal replacement for databases.
There are several important constraints.
First, external indexes have to remain synchronized with the data they describe.
Second, indexes consume storage and require infrastructure.
Third, optimising Parquet for random access can create trade-offs for conventional analytical workloads.
Fourth, point-query performance depends heavily on the physical organisation of the data.
Fifth, an architecture designed around immutable or append-oriented data has different characteristics from a transactional system handling frequent updates.
The technology therefore introduces a new point in the design space rather than eliminating existing systems.
That is perhaps the most useful way for enterprise architects to view it.
A new boundary between the lake and the database
Spotify’s RAP work is significant because it attacks a problem that has existed almost since the rise of cloud data lakes.
The lake became the cheapest and most scalable place to keep enormous amounts of information.
But applications still needed databases.
That forced organizations to copy data between systems.
The Random Access Parquet data lake architecture suggests another possibility.
Instead of asking:
“Which data should we copy into the serving database?”
architects can increasingly ask:
“Which data actually needs a separate serving database, and which data can be served directly from the lake?”
That is a more consequential question.
It could influence storage design, data pipelines, AI infrastructure, serving architectures and cloud economics.
RAP does not turn a data lake into a transactional database.
It does something more subtle.
It makes the boundary between analytical storage and interactive retrieval less rigid.
And as AI agents increasingly need access to large amounts of historical enterprise data, that boundary may become one of the most important architectural questions in modern data infrastructure.
The bigger picture
The most important idea in Spotify’s work may ultimately be its insistence on one dataset serving multiple purposes.
The files used for analytical reporting can also become the source for interactive retrieval.
The data used by machine-learning pipelines can also provide context to AI agents.
Historical information can remain in economical object storage while selected records become addressable at interactive latency.
That is a significant change in philosophy.
The future data platform may not be defined by a single system optimised for every workload.
Instead, it may consist of one durable data layer with multiple highly specialised access paths.
Spotify’s Random Access Parquet architecture is an early and technically sophisticated example of that direction.
The data lake may not be replacing the database.
But the database may no longer be the only practical way to get a single record out of a data lake.