hub
js/model/hub.ts
fino:model/hub - reproducible model and dataset resolution from a Hugging Face hub.
Resolves a revision to a commit, fetches files with resumable ranged requests,
verifies sha256 as the bytes arrive, stores them content-addressed, and pins
repository → commit → digests in a project models.lock. The lockfile is the
point of the module: main is a moving target, so a script that loads a model
by revision can silently load different weights tomorrow. Pinned, a later run
resolves from the lock rather than the network and fails loudly if the bytes it
gets back are not the bytes that were pinned.
Everything runs on Fino's own HTTP stack, and every cache path is derived from the repository, commit, and digest rather than invented, so two processes agree on where a file lives without coordinating and a resumed transfer finds its own partial file after a restart.
The same client resolves dataset repositories through the same cache and
lockfile discipline; pass type: 'dataset'.
import { HubClient } from 'fino:model/hub';
const hub = new HubClient({ lockfile: './models.lock' });
const file = await hub.download(
{ repo: 'bert-base-uncased', revision: 'main' },
'tokenizer.json',
);
console.log(file.commit, file.sha256, file.path);Constants
const HUGGING_FACE_ENDPOINT
The default public hub.
const LOCKFILE_VERSION
The format version written into the file.
Interfaces
interface RepoRef {
A repository revision to resolve.
Properties
repo: string
owner/name, or a bare name for a canonical repository.
revision?: string
Branch, tag, or commit. Defaults to main.
type?: RepoType
Repository kind. Defaults to model.
interface HubClientOptions {
Client construction options.
Properties
endpoint?: string
Hub root. Defaults to https://huggingface.co.
token?: string | null
Bearer token. Defaults to HF_TOKEN, then HUGGING_FACE_HUB_TOKEN.
cacheDir?: string
Cache root. Defaults to the resolved model cache directory.
lockfile?: string | null
Lockfile path, or null to run unpinned. Defaults to ./models.lock.
fetch?: FetchLike
Injectable transport, for tests and for self-hosted transports.
attempts?: number
Attempts per transfer, including the first. Defaults to 3.
concurrency?: number
Concurrent transfers in snapshot. Defaults to 4.
offline?: boolean
Never touch the network; only serve what is already cached.
interface FetchOptions {
Options shared by the fetching methods.
Properties
update?: boolean
Re-resolve the revision even when the lockfile pins it.
This is how a pin is moved forward: without it, a pinned revision never consults the network for a new commit.
onProgress?: (progress: TransferProgress & { path: string }) => void
signal?: AbortSignal
interface SnapshotOptions extends FetchOptions {
Options for fetching a whole revision.
Properties
allow?: ReadonlyArray<string | RegExp>
Only fetch files matching one of these predicates or suffixes.
ignore?: ReadonlyArray<string | RegExp>
Skip files matching one of these suffixes or patterns.
interface HubFile {
A file resolved into the cache.
Properties
type: RepoType
repo: string
revision: string
commit: string
name: string
Repository-relative path.
path: string
Absolute path of the cached blob.
sha256: string
size: number
cached: boolean
true when the bytes were already cached and no transfer ran.
interface HubSnapshot {
A whole revision resolved into the cache.
Properties
type: RepoType
repo: string
revision: string
commit: string
files: HubFile[]
interface HubFileInfo {
One entry of a repository listing.
Properties
name: string
Repository-relative path.
size: number | null
Size in bytes, when the hub reported one.
sha256: string | null
Content sha256, when the file is LFS-tracked and the hub reported one.
interface VerifyReport {
The outcome of verifying the cache against the lockfile.
Properties
ok: string[]
Files whose cached bytes match the pinned digest.
missing: string[]
Files pinned but not present in the cache.
corrupt: string[]
Files present with contents that do not match the pin.
interface Manifest {
The recorded contents of one repository commit.
Properties
type: RepoType
repo: string
commit: string
files: Record<string, ManifestEntry>
Repository-relative path to digest, in sorted key order.
interface ManifestEntry {
What the cache knows about one file of one commit.
Properties
size: number
Size in bytes.
sha256: string
Lowercase hex sha256 of the contents.
interface LockEntry {
One pinned repository revision.
Properties
type: RepoType
repo: string
revision: string
The revision that was asked for, such as main or a tag.
commit: string
The commit it resolved to.
files: Record<string, ManifestEntry>
Digests for the files fetched from this commit.
interface Lockfile {
The whole lockfile.
Properties
version: number
entries: Record<string, LockEntry>
interface TransferProgress {
Progress during a transfer.
Properties
transferred: number
Bytes on disk so far, including any resumed from a previous attempt.
total: number | null
Total bytes expected, or null when the server did not say.
Classes
class HubClient {
A Hugging Face-compatible hub client.
One instance owns one cache root and one lockfile. Methods are safe to call
concurrently against distinct paths; snapshot batches its lockfile write so a
multi-file fetch produces one pin rather than a series of partial ones.
Constructors
constructor(options: HubClientOptions = {})
Getters
get cache(): ModelCache
The cache this client reads and writes.
get lockfilePath(): string | null
The lockfile path, or null when running unpinned.
Methods
async lockfile(): Promise<Lockfile>
The current lockfile contents.
async pinned(ref: RepoRef): Promise<LockEntry | null>
The pinned entry for a revision, or null when it is not pinned.
async resolveRevision(ref: RepoRef, options: FetchOptions = {}): Promise<string>
Resolve a revision to a commit.
A pinned revision resolves from the lockfile without a request, which is what
makes a locked project's resolution reproducible and offline-capable. Pass
update to consult the hub instead.
async listFiles(ref: RepoRef, options: FetchOptions = {}): Promise<HubFileInfo[]>
List the files of a revision without downloading them.
Sizes and digests come from the hub's own metadata, so this is enough to decide what is worth fetching before any bytes move.
async download(ref: RepoRef, name: string, options: FetchOptions = {}): Promise<HubFile>
Fetch one file, returning where it landed in the cache.
async load(ref: RepoRef, name: string, options?: FetchOptions): Promise<Uint8Array>
Fetch one file and return its contents.
async loadText(ref: RepoRef, name: string, options?: FetchOptions): Promise<string>
Fetch one file and return its contents decoded as UTF-8.
async snapshot(ref: RepoRef, options: SnapshotOptions = {}): Promise<HubSnapshot>
Fetch a whole revision.
Transfers run concurrently up to the client's limit and the lockfile is written once at the end, so a snapshot either pins the set it fetched or leaves the previous pin alone.
async verify(): Promise<VerifyReport>
Check the cache against the lockfile.
Re-hashes every pinned blob rather than trusting its filename, so a blob that was truncated or edited in place is reported as corrupt instead of served.
fileUrl(type: RepoType, repo: string, commit: string, name: string): string
The URL a file resolves to on this hub.
class IntegrityError extends Error {
Raised when a transfer's bytes do not match what was expected.
Readonly Properties
readonly expected: string
readonly actual: string
Constructors
constructor(what: string, expected: string, actual: string)
class ModelCache {
The cache's on-disk layout.
Readonly Properties
readonly root: string
Cache root directory.
Constructors
constructor(root: string = defaultCacheRoot())
Methods
blobPath(sha256: string): string
Where a blob with this digest lives.
manifestPath(type: RepoType, repo: string, commit: string): string
Where the manifest for one commit lives.
partialPath(type: RepoType, repo: string, commit: string, path: string): string
Where an in-progress download for one file lives.
async hasBlob(sha256: string): Promise<boolean>
true when a blob with this digest is already stored.
async readBlob(sha256: string): Promise<Uint8Array>
Read a stored blob.
async commitBlob(from: string, sha256: string): Promise<string>
Move a completed file into the blob store.
A blob that is already present wins: identical contents make the incoming copy redundant, and keeping the existing one avoids replacing a file another process may be reading.
async writeBlob(bytes: Uint8Array): Promise<{ sha256: string; path: string }>
Store a buffer as a blob, returning its digest.
async readManifest(type: RepoType, repo: string, commit: string): Promise<Manifest | null>
Read a stored manifest, or null when the commit has not been fetched.
async updateManifest(
type: RepoType,
repo: string,
commit: string,
files: Record<string, ManifestEntry>,
): Promise<Manifest>
Merge entries into a commit's manifest.
Manifests accumulate: fetching one file at a time and fetching a whole snapshot converge on the same record, and keys are written sorted so the file is byte-stable across runs.
Types
type RepoType = 'model' | 'dataset'
Which kind of repository a reference names.
Functions
function defaultCacheRoot(): string
Resolve the cache root.
FINO_MODEL_CACHE wins so a project can pin its own location; HF_HOME is
honored next because a machine that already caches Hugging Face artifacts
should not need a second copy of the setting.
async function readLockfile(path: string): Promise<Lockfile>
Read a lockfile.
A missing file is an empty lock, not an error — the first run of a project has nothing pinned yet. A file that exists but cannot be parsed is an error, because silently discarding pins would defeat the point.
async function writeLockfile(path: string, lock: Lockfile): Promise<void>
Write a lockfile with sorted keys.