Visualizing millions of rows in the browser
Most JavaScript data grids and charting libraries hold rows as JavaScript objects and lay out one DOM or SVG node per datum, which stalls somewhere between ten thousand and a few hundred thousand rows. Perspective takes a different approach at each layer:
- Columnar engine in WebAssembly. Data is stored in typed, columnar buffers inside a C++ query engine compiled to WebAssembly and run in a Web Worker, off the main thread. Strings are dictionary-encoded. Nothing is materialized as JavaScript objects unless you ask for it.
- Apache Arrow in, Apache Arrow out. A
Tableloads Arrow directly into those buffers without per-row parsing, andView.to_arrow()exports the same way. CSV and JSON are also supported. - Queries, not rows, cross the boundary. Grouping, pivoting, filtering and sorting happen inside the engine. The UI requests only the window of the result it is about to draw.
- Virtual-scrolling data grid. The Datagrid renders only visible cells, so scrolling a 10 million row grid costs the same as scrolling a 100 row grid.
- WebGL charts. Scatter, heatmap, line and map plugins draw on the GPU, staying interactive at point counts where SVG and 2D canvas charts do not.
Loading a large file
const worker = await perspective.worker();
const response = await fetch("/data/trips.arrow");
const table = await worker.table(await response.arrayBuffer());
await document.querySelector("perspective-viewer").load(table);
Prefer Arrow (optionally LZ4 or ZSTD compressed) over CSV or JSON for large datasets: it is smaller on the wire, carries its own schema, and skips type inference.
More than 4GB: Memory64
32-bit WebAssembly caps the engine’s heap at 4GB.
@perspective-dev/server also ships a
Memory64 build which raises the ceiling
to 16GB in browsers which support it. Register both binaries and only the one
the browser selects is downloaded:
perspective.init_server({
wasm32: () => fetch(SERVER_WASM),
wasm64: () => fetch(SERVER_WASM64),
});
More than memory: page_to_disk
A Table normally lives in the engine’s memory. Created with
page_to_disk, its columns
are backed by on-disk storage instead — the browser’s Origin Private File
System under WebAssembly, memory-mapped files in Python and Rust — and the
engine evicts the coldest columns to it when it is over its
resident memory budget (1 GiB by default in the browser), reading them back
when a query needs them.
const table = await worker.table(await response.arrayBuffer(), {
page_to_disk: true,
});
Everything else about the Table is unchanged, including streaming updates
and the viewer on top of it. Use it for wide tables where users touch a few
columns at a time, or to hold several large tables in one tab; leave it off
for data which fits, since a query over evicted columns pays to read them
back.
How many rows?
It depends on column count and types more than row count — numeric and
datetime columns are compact, high-cardinality strings are not. Millions of
rows is routine; tens of millions is practical for narrow numeric tables. Free
memory by calling delete() on views and
tables you no longer need, and consider page_to_disk for tables which are
large but only partly in use at any moment.
When the data does not fit in the browser
Keep it on a server and send the browser only what is visible:
- Server-only mode — the same engine, running natively in Python, Node.js or Rust, with the browser connected over WebSocket.
- Virtual servers — no Perspective engine at all. View configurations are translated to SQL and run by DuckDB, ClickHouse, PostgreSQL or Polars, which can be as large as those systems allow.
In both modes the viewer, its configuration and its saved layouts are identical to the in-browser case.