Quilt T4
  • Introduction
  • Installation
  • Quickstart
  • Walkthrough
    • Creating a Package
    • Distributing a Package
    • Installing a Package
    • Reading from a Package
    • Working with the Catalog
    • Working with a Bucket
  • Advanced Usage
    • Filtering a Package
    • .quiltignore
    • Materialization
    • Working with Manifests
    • S3 Select
  • API Reference
    • t4
    • t4.Package
    • t4.Bucket
  • References
    • Technical Reference
    • Contributing
    • Further Reading
Powered by GitBook
On this page
  • Overview
  • A team data hub for S3
  • Use cases
  • Key features
  • Components
  • Roadmap

Introduction

NextInstallation

Last updated 5 years ago

Overview

.

A team data hub for S3

  • T4 adds search, content preview, versioning, and a Python API to any S3 bucket

  • Every file in T4 is versioned and searchable

  • T4 is for data scientists, data engineers, and data-driven teams

Use cases

  • Collaborate - get everyone on the same page by pointing them all to the same immutable data version

  • Experiment faster - blob storage is schemaless and scalable, so iterations are quick

  • Recover, rollback, and reproduce with immutable packages

  • Understand what's in S3 - plaintext and faceted search over S3

Key features

  • Browse, search any S3 bucket

  • Read/write Python objects to and from S3

  • Immutable versions for objects, immutable packages for collections of objects

Components

  • /catalog (JavaScript) - Search, browse, and preview your data in S3

  • /api/python - Read, write, and annotate Python objects in S3

Roadmap

Preview images, Jupyter notebooks, - without downloading

Vega visualizations
Roadmap
Important Note: this project now lives in the quiltdata/quilt repository.
Rethinking S3: Announcing T4, a team data hub
docs on_gitbook
chat on_slack
codecov
pypi