The LAION2B and projections

About the LAION5B

The LAION5B dataset is an openly available image collection that has been used for learning very large visual and language deep-neural models; for instance, the famed stable diffusion generative model used it as the training set. The collection equips each image with a URL handle, allowing people to showcase demonstrations easily.

A more detailed description can be found here:

The English subset, often called LAION2B, contains over 2 billion objects.

Subset of the challenge

The dataset is divided into parts containing close to 1M vectors. We selected the first 112 parts (0000 to 0111); we used the first part to extract the public query set and the rest to extract the database. The subset use approximately 160GB of space and its associated metadata 20GB (the first 112 parts). Embeddings are distributed using single precision (16bits) floating point vectors bundled in the NumPy data-specific format .npz. They can be loaded on most platforms due to the format's popularity.

The challenge has three subsets:

• 10M subset: concatenation of 1-11 parts.

• 30M subset: concatenation of 1-33 parts.

• 100M subset: concatenation of 1-111 parts.

• public queries: computed from part 0.

All parts should be concatenated in order and also removing NSFW entries (marked in metadata files).

• Note 1: You will get 768 dimensional 16-bit floating point vectors that may be changed to a 32-bit format to get full speed on legacy hardware.

• Note 2: Our gold-standards were computed using $L_2$-normalized vectors (i.e., unitary norms) and the $1-\cos(\cdot, \cdot)$ as distance function.

• Note 3: Our gold-standard .h5 files contain the 100 nearest neighbors of each query using two associated matrices knns and dists, i.e., columns correspond to queries and rows to nearest neighbors for each query.

  • The knns identifiers start indexing on 1.

  • The dists contains raw distance values for each corresponding query and object, i.e., 1-\cos(\cdot, \cdot); please consider that this is not a proper metric distance. People using metric properties can use the angle with minor changes.

Subsets

We provide access to different subsets of the dataset and also created three different lower-dimensional projections that can be used. In particular, we computed two PCA projections using 32 and 96 dimensions and one more projection into binary sketches designed to work with bit-level hamming distance (using 1024 bits). Find below the URLs to download these bundles.

768d clip embeddings (clip768)

32d PCA projections (pca32)

96d PCA projections (pca96)

1024-bit binary sketches (hamming)

Gold standard list (computed with 32-bit floating point arithmetic, 100 nearest neighbors)

Gold standard for public queries (computed with 64-bit IEEE floating point arithmetic, 1000 nearest neighbors)

Gold standard for private queries (computed with 64-bit IEEE floating point arithmetic, 1000 nearest neighbors)

For instance, you can download the 10M subset and the query set using the following commands from a typical linux terminal:

curl -O https://sisap-23-challenge.s3.amazonaws.com/SISAP23-Challenge/laion2B-en-clip768v2-n=10M.h5
curl -O https://sisap-23-challenge.s3.amazonaws.com/SISAP23-Challenge/public-queries-10k-clip768v2.h5

Note that our projection models were trained with our 10M subset. Other approaches may vary the resulting quality.

Note: Projections will reduce the result's quality concerning the original embeddings, but you can use these datasets to fast prototype your solution and for hyperparameter optimization. Please email us if you are interested in the associated metadata (which can also be obtained as described in the rest of the document.)