js/data/frame
js/data/frame.ts
fino:data/frame - bounded lazy DataFrame plans over Arrow record batches.
A DataFrame<Row> is an immutable plan, not a table wrapper. Operators add
filter, projection, computed-column, aggregation, join, sort, or limit nodes;
batches() executes the plan lazily and collect() asynchronously gathers
its Arrow batches into a Table. Streaming operators preserve input batch
boundaries. Aggregation, joins, and sort are deliberately blocking because
their bounded first version needs the complete input.
Expressions are reusable and columnar. frame.col('name') preserves the
caller's Row type, while col<T>('name') is convenient for reusable plans.
Comparisons, boolean logic, and arithmetic use SQL-like null semantics:
null propagates, filters retain only literal true, and aggregates ignore
null inputs. Joins never overwrite colliding right fields; they suffix them.
scanParquet() inspects footer metadata at plan time. Named projections are
pushed into selective column decoding, and simple comparisons/null checks
conservatively select row groups from statistics. The filter still executes
after decoding, so missing or unsupported statistics can only reduce
optimization, never change results.
This is intentionally not a general query engine. The operator set serves evaluation, memory ingestion, batch inference, classical ML preprocessing, and display. A future SQL surface should parse into this same plan.
import { DataFrame, col, count } from 'fino:data/frame';
const report = DataFrame.scanParquet<{ team: string; score: number }>(bytes)
.filter(col<number>('score').gte(0.8))
.groupBy('team')
.agg({ rows: count(), meanScore: col<number>('score').mean() })
.sort(col<string>('team').asc());
const table = await report.collect();Types
type DataFrameRow = object
Plain object shape tracked by a typed DataFrame.
type ExpressionSelection = Readonly<Record<string, Expr<unknown>>>
Named scalar expressions accepted by projection and computed-column APIs.
type AggregateSelection = Readonly<Record<string, AggregateExpr<unknown>>>
Named aggregate expressions accepted by aggregate() and GroupBy.agg().
type SelectionRow<S extends ExpressionSelection> = { [K in keyof S]: ExprValue<S[K]> }
Infer a row shape from a named scalar-expression object.
type AggregationRow<S extends AggregateSelection> = { [K in keyof S]: AggValue<S[K]> }
Infer a row shape from a named aggregate-expression object.
type NullPlacement = 'first' | 'last'
Null placement for a sort key.
type DataFrameSource = RecordBatch | Table | Iterable<RecordBatch> | AsyncIterable<RecordBatch>
Sources accepted by DataFrame.from().
type JoinHow = 'inner' | 'left' | 'right' | 'full'
Supported relational join behavior.
type JoinOptions<Left extends object, Right extends object> = { how?: JoinHow; suffix?: string } & (
| {
on: SameJoinKeys<Left, Right> | readonly SameJoinKeys<Left, Right>[];
leftOn?: never;
rightOn?: never;
}
| {
on?: never;
leftOn: RowKey<Left> | readonly RowKey<Left>[];
rightOn: RowKey<Right> | readonly RowKey<Right>[];
}
)
Explicit join keys and collision behavior.
Classes
class Expr<T = unknown> {
Reusable scalar or column expression.
Construct with frame.col(), col(), or lit(). Expression objects are
immutable and may be reused across plans.
Methods
eq<U>(other: Expr<U> | U): Expr<boolean | null>
Equality comparison. Null on either side produces null.
ne<U>(other: Expr<U> | U): Expr<boolean | null>
Inequality comparison. Null on either side produces null.
lt<U>(other: Expr<U> | U): Expr<boolean | null>
Strict less-than comparison with null propagation.
lte<U>(other: Expr<U> | U): Expr<boolean | null>
Less-than-or-equal comparison with null propagation.
gt<U>(other: Expr<U> | U): Expr<boolean | null>
Strict greater-than comparison with null propagation.
gte<U>(other: Expr<U> | U): Expr<boolean | null>
Greater-than-or-equal comparison with null propagation.
and(
this: Expr<boolean | null>,
other: Expr<boolean | null> | boolean | null,
): Expr<boolean | null>
Three-valued boolean AND.
or(
this: Expr<boolean | null>,
other: Expr<boolean | null> | boolean | null,
): Expr<boolean | null>
Three-valued boolean OR.
not(this: Expr<boolean | null>): Expr<boolean | null>
Three-valued boolean negation.
isNull(): Expr<boolean>
True exactly when the value is null or undefined.
isNotNull(): Expr<boolean>
True exactly when the value is present.
add<U extends number | bigint | null>(
this: Expr<number | bigint | null>,
other: Expr<U> | U,
): Expr<number | bigint | null>
Numeric addition with null propagation.
sub<U extends number | bigint | null>(
this: Expr<number | bigint | null>,
other: Expr<U> | U,
): Expr<number | bigint | null>
Numeric subtraction with null propagation.
mul<U extends number | bigint | null>(
this: Expr<number | bigint | null>,
other: Expr<U> | U,
): Expr<number | bigint | null>
Numeric multiplication with null propagation.
div<U extends number | bigint | null>(
this: Expr<number | bigint | null>,
other: Expr<U> | U,
): Expr<number | null>
Numeric division as a JavaScript number, with null propagation.
count(): AggregateExpr<number>
Count non-null values in this expression.
sum(
this: Expr<number | bigint | null | undefined>,
): AggregateExpr<Nullable<Exclude<T, null | undefined>>>
Sum non-null values, returning null when none are present.
mean(this: Expr<number | bigint | null | undefined>): AggregateExpr<number | null>
Average non-null values as a number, returning null when none are present.
min(): AggregateExpr<Nullable<Exclude<T, null | undefined>>>
Minimum non-null value, returning null when none are present.
max(): AggregateExpr<Nullable<Exclude<T, null | undefined>>>
Maximum non-null value, returning null when none are present.
asc(options: SortOptions = {}): SortExpr<T>
Ascending stable sort key. Nulls default to last.
desc(options: SortOptions = {}): SortExpr<T>
Descending stable sort key. Nulls default to last.
class AggregateExpr<T = unknown> {
Aggregate expression produced by count() or an Expr aggregate method.
class SortExpr<T = unknown> {
Immutable expression plus direction/null placement used by DataFrame.sort().
Readonly Properties
readonly expr: Expr<T>
Expression to evaluate.
readonly direction: 'asc' | 'desc'
Sort direction.
readonly nulls: NullPlacement
Null placement.
Constructors
constructor(expr: Expr<T>, direction: 'asc' | 'desc', nulls: NullPlacement)
Create a sort key. Prefer expr.asc() or expr.desc().
class DataFrameError extends Error {
Error thrown for invalid plans, missing columns, or unsupported values.
Properties
name
Error name, always 'DataFrameError'.
class DataFrame<Row extends object = Record<string, unknown>> {
Immutable lazy plan over Arrow record batches.
The Row parameter is compile-time guidance for column names and common
projections; the runtime Arrow schema remains authoritative.
Static Methods
static from<Row extends object = Record<string, unknown>>(
source: DataFrameSource,
options: DataFrameSourceOptions = {},
): DataFrame<Row>
Wrap Arrow batches or a table. Supply schema for a possibly empty stream.
static scanParquet<Row extends object = Record<string, unknown>>(
input: Uint8Array | ArrayBuffer,
): DataFrame<Row>
Plan a selective scan over complete Parquet bytes.
Methods
col<Key extends RowKey<Row>>(name: Key): Expr<Row[Key]>
Build a typed expression for one row key.
filter(predicate: Expr<boolean | null>): DataFrame<Row>
Add a filter. Only values equal to true pass; false and null are removed.
select<Key extends RowKey<Row>>(...columns: Key[]): DataFrame<Pick<Row, Key>>
Select named columns while preserving their TypeScript keys.
select<Selection extends ExpressionSelection>(
selection: Selection,
): DataFrame<SelectionRow<Selection>>
Select and name arbitrary expressions, inferring the output row shape.
withColumns<Selection extends ExpressionSelection>(
selection: Selection,
): DataFrame<Omit<Row, keyof Selection> & SelectionRow<Selection>>
Add or replace named columns.
Expressions are evaluated against the input schema simultaneously; one expression in the same call cannot refer to another newly named column.
groupBy<Key extends RowKey<Row>>(...keys: Key[]): GroupedDataFrame<Row, Key>
Group by one or more existing keys before calling agg().
aggregate<Selection extends AggregateSelection>(
selection: Selection,
): DataFrame<AggregationRow<Selection>>
Aggregate the complete input into one row.
join<Right extends object>(
other: DataFrame<Right>,
options: JoinOptions<Row, Right>,
): DataFrame<Row & Omit<Right, keyof Row>>
Hash-join another frame with explicit keys and collision suffixing.
sort(...keys: Array<RowKey<Row> | SortExpr<unknown>>): DataFrame<Row>
Stable sort by one or more expressions. A bare column name sorts ascending.
limit(count: number): DataFrame<Row>
Keep at most count rows and stop pulling upstream once satisfied.
batches(): AsyncIterableIterator<RecordBatch>
Execute and stream Arrow record batches.
async collect(): Promise<Table>
Execute and collect the result as an Arrow table.
explain(): string
Render the optimized plan, including Parquet projections and row groups.
class GroupedDataFrame<Row extends object, Key extends RowKey<Row>> {
Grouped frame returned by DataFrame.groupBy().
Call agg() to finish the plan; grouped frames do not execute directly.
Methods
agg<Selection extends AggregateSelection>(
selection: Selection,
): DataFrame<Pick<Row, Key> & AggregationRow<Selection>>
Aggregate each distinct key tuple, preserving key fields in the output.
Functions
function col<T = unknown>(name: string): Expr<T>
Create a reusable named-column expression.
function lit<T>(value: T, dataType?: DataType): Expr<T>
Create a scalar literal expression. Pass dataType for a projected null literal.
function count(): AggregateExpr<number>
Count input rows, including rows whose fields are all null.
Interfaces
interface SortOptions {
Sort-key construction options.
Properties
nulls?: NullPlacement
Where nulls sort, independent of direction. Defaults to 'last'.
interface DataFrameSourceOptions {
Source options for streaming inputs whose first batch may never arrive.
Properties
schema?: Schema
Known schema; required to collect an empty arbitrary stream.