Submission Archive and Evaluation Workflow for SISAP 2026

The SISAP 2026 challenge is closed. Final results are available in the leaderboard. This page documents the submission workflow and evaluation setup used during the challenge.

Workflow Used During the Challenge

• Pre-register your team via GitHub.

• Register your team in TIRA.

• Run a local dry run with the TIRA CLI.

• Authenticate with TIRA and submit your code.

• Wait for the organizers to trigger the full evaluation.

Challenge Overview

The SISAP 2026 Indexing Challenge features three tasks:

• Task 1: K-nearest neighbor graph construction (metric self-join) on 6.4M Wikipedia vectors (1024-dim, BGE-M3).

• Task 2: Maximum Inner Product Search on LLM attention workloads (256k vectors, 128-dim, Llama-3-8B).

• Task 3: Indexing very sparse high-dimensional vectors using SPLADE-v3 embeddings (~2.68M documents, 30,522-dim).

All tasks are evaluated inside Docker containers with 8 vCPUs and 24 GB RAM. The CPUs used are AMD EPYC 7F72 24-Core Processor. Full task descriptions are available at: sisap-challenges.github.io/2026

Submission via TIRA

For this year, submissions are handled through TIRA (tira.io/task-overview/sisap-2026). TIRA provides a reproducible, containerized evaluation framework. Code submissions for SISAP 2026 were handled only through TIRA. Submissions are now closed. We keep the instructions below as archive material.

How to Submit

Step 1 — Register your team

1. Sign up or log in at tira.io (GitHub login supported).

2. Navigate to tira.io/task-overview/sisap-2026 and click Register.

3. Optionally add team members via tira.io/g?type=my.

Step 2 — Verify locally

Install the TIRA CLI and do a dry run against one of the spot-check datasets:

pip3 install --upgrade tira

tira-cli code-submission \
    --path . \
    --command '{YOUR EXECUTABLE} --input $inputDataset/*.h5 --task-description $inputDataset/config.json --output $outputDir' \
    --task sisap-2026 \
    --dataset task-1-spot-check-20260602-training \
    --dry-run

Use task-2-spot-check-20260602-training or task-3-spot-check-20260529-training if your approach only targets Task 2 or Task 3, respectively.

Step 3 — Authenticate and submit

Retrieve your authentication token from the TIRA task page (Submit → Code Submissions → New Submission → I want to submit from my local machine), then:

tira-cli login --token AUTH-TOKEN
tira-cli verify-installation --task sisap-2026 --team YOUR-TEAM

tira-cli code-submission \
    --path . \
    --command '{YOUR EXECUTABLE} --input $inputDataset/*.h5 --task-description $inputDataset/config.json --output $outputDir' \
    --task sisap-2026 \
    --dataset task-1-spot-check-20260602-training

Your program should write its output files to $outputDir in the format described on the task description page.

Step 4 — Trigger evaluation in the TIRA UI

Navigate to the task page, click Submit → Code Submissions, select your submission, choose a dataset and hardware configuration. The organizers handled execution on all datasets once a submission looked correct. During code submission, TIRA only ran small workloads. These are available in the baseline mentioned below.

Python Baseline

A working Python baseline is available to help you get started: github.com/sisap-challenges/sisap26-python-baseline

It includes Docker support, GitHub Actions CI, evaluation scripts, and plotting utilities. You are encouraged to fork it as the starting point for your own solution.

A Julia example is also available: github.com/sisap-challenges/sisap2026-julia-example

Important Dates

Short paper

Please submit your short paper through the regular EasyChair submission system for SISAP 2026. Pick "indexing challenge" as paper category. Submissions have to follow the rules of short papers described in the call for paper (in particular, at most 8 pages in standard LNCS style), but should be provided non-anonymized. More detailed analysis can be provided through a technical report referenced in the short paper.

Pre-registration Archive

Pre-registration for SISAP 2026 is closed. During the live challenge, teams were asked to open a "Pre-registration request" issue at: github.com/sisap-challenges/challenge2026

Troubleshooting

After registering your team at TIRA, you can reach out to the organizers via the platform. After login, you can use the following direct link, or write a direct message to Maik. We will help you with the submission process.

For questions about results or archival material, contact the organizing committee at sisap-2026-indexing-challenge@googlegroups.com.

The SISAP 2026 Indexing Challenge Organizing Committee

Eric S. Téllez, Martin Aumüller, Maik Fröbe, Vladimír Míč