© sisap challenge committee.
© sisap challenge committee.
The SISAP 2025 Indexing Challenge Committee invites groups of students, researchers, and practitioners worldwide to submit proposals for new tasks to be considered as part of the SISAP Indexing Challenge 2025. The SISAP Indexing Challenge is an event for researchers and practitioners to test new and/or well-tuned and optimised indexing approaches for shared tasks in laboratory and practical conditions.
The proposal must describe the proposed task succinctly: the metric problem being solved, the queries and datasets, gold standard data, score functions, the goals, the ordering definition of participating teams, and participation rules in general. The proposal must list the names of its authors, affiliations, and contact information. The task name must feel to be part of the SISAP 2025 Indexing Challenge. The tasks from previous years are available at https://sisap-challenges.github.io/2024/tasks/ and https://sisap-challenges.github.io/2023/tasks/
A successful proposal nominates a single person to join the organization committee. This person is responsible for carrying out the final evaluation of all solutions from all Challenge participants and for providing the evaluation infrastructure, such as evaluation code and server infrastructure. The server should be limited by the commodity hardware (specified below), and its main parameters should be declared in the proposal. Tasks based on quality measures, i.e., those that do not need to be run in the same infrastructure, can be considered using the Codalab platform. The organizing committee suggests requiring the GitHub Actions scheme or a Dockerfile scheme. If the proposing group is unfamiliar with these schemes, the Organizing committee members are ready to help with the evaluation.
The task proposal should attend to the following deadlines:
Jan. 27th. Task proposal deadline.
Feb. 10th. Task proposal acceptance notification.
Feb. 20th. (tentative) Publishing of the Call for Participation on the Indexing Challenge with the complete list of tasks and task planning dates and details.
Tentative deadlines for the SISAP 2025 Indexing Challenge, i.e., submitted solutions:
June 6th. Submission of solution implementations deadline
June 13th. Short paper descriptions deadline
July 1st. Final ranking announcement
July 11th. Paper notification
July 31st. Participant (short paper) camera ready
Task organizers and participants will also have the opportunity to share results and insights in a special oral session of the SISAP conference https://sisap.org/2025 to be held in Iceland on October 1st-3rd,
The reports will be published as short papers in the SISAP proceedings.
Previous editions used GitHub repositories to declare participation intentions and save the implementations in GitHub. Asking for a public repository with the solution is congruent with the reproducibility spirit of the challenge.
SISAP 2025 Indexing Challenge does not focus on specialised or extremely limiting hardware that is affordable in rather extreme environments. Conversely, the Challenge aims to address tasks on a common infrastructure of commodity hardware, i.e., hardware that is affordable for small and mid-size companies. The indicative parameters of (each) server(s) that should suffice for task evaluations are:
up to 50 CPU cores,
up to 512 GB of RAM
up to 2 TB on NVMe SSD
The proposals for the tasks should be prepared in a shared web document, e.g., a Google document, and the link sent to the e-mail addresses of the committee members (listed below).
Edgar L. Chavez, CICESE, México <elchavez@cicese.edu.mx>
Eric S. Téllez, INFOTEC-CONAHCyT, México <eric.tellez@ieee.org>
Martin Aumüller, ITU Copenhagen, Denmark <maau@itu.dk>
Vladimir Mic, Aarhus University, Denmark <v.mic@cs.au.dk>