Submission deadline - 25 Sept 2023
---
CHiME is a challenge series whose purpose is to produce scientific advances in the field of distant microphone speech processing. The challenge’s six previous iterations from 2011 until 2020 have resulted in major improvements in word error rate and diarisation error rate for single-speaker voice commands in various environments and for multi-speaker conversations in homes. These six previous iterations have been centrally organised.
Starting from last year, we have now introduced a more open model, where the annual CHiME Challenge features independent tasks organised by teams who work within a schedule defined by the CHiME Steering Group. Teams will be responsible for managing the tasks, i.e., providing data and instructions, scoring submissions and results, etc. The CHiME Steering Group will then work to support teams in the development of their task. The Steering Group will also organise a CHiME workshop where the tasks will be presented.
The CHiME Steering group is now inviting task proposals for the CHiME-2024 Challenge. Proposed tasks will be evaluated according to the following criteria:
- relevance to the CHiME goals
- methodology - i.e., likelihood to lead to robust and novel scientific findings
- accessibility - i.e., how easy is it for teams to engage with
- feasibility - likelihood to complete on schedule, risk management
- reproducibility - e.g., openness of data, baseline
- diversity - amongst organisers and expected participants, e.g., academia vs industry, geography, gender etc.
We wish to foster a collaborative proposal process that minimises overlap between proposals. To this end, if you are considering proposing a task, please sign up to the CHiME Google group and feel free to discuss your plans. Or if you wish information to be shared more privately, i.e., amongst other bidders only, then you may share details or initial plans with the CHiME Steering Group who will then share with others whose plans may overlap.
Proposal
Task bidders should submit a task proposal for review by the Steering Group. The proposal should be a maximum of two pages (pdf format preferred) including the following information:
- Task name
- Coordinators and their roles
- Keywords (e.g. enhancement, diarization, speaker recognition, etc)
- Definition (one sentence)
- Short description (incl. research question the task is tackling)
- Dataset description: development, evaluation (short description, how much data is already available and prepared, how long would it take to prepare the rest, mention if you allow external data/transfer learning or not)
- Evaluation method / metric
- Baseline system (brief description, planned method if you do not have one from a previous challenge, current readiness including an expected readiness date)
- List of invited participants who are likely to participate
- Contact person (for main communication, website)
Pitching Session
We will be running an optional “proposal pitching” session at the CHiME-2023 workshop in Dublin, 25th August. This will be an opportunity for teams who are developing a submission to get some feedback from members of the CHiME community. If you are interested in presenting in this session please contact us to discuss.
Submission
The proposal should be sent to the CHiME Steering group by 26th September. The Steering Group will review the proposals and provide feedback by 10th October. In case task amendments are requested, the task acceptance decision will be finalized by 4th November.
If you want any advice on preparing your proposal, please do not hesitate to contact us.
Important dates
The tentative schedule for CHiME-2024 challenge round is as follows.- 25 Aug 2023 - Pitching session at CHiME-2023 workshop
- 25 Sept 2023 - Submission of task proposals
- 9 Oct 2023 - Feedback from Steering Group
- 20 Oct 2023 - Resubmission of conditionally accepted proposals
- 3 Nov 2023 - Notification of acceptance
- 1 Feb 2024 - Challenge opening
- 28 June 2024 - Challenge Submission deadline
- 6 Sep 2024 - Interspeech satellite workshop (date and details TBC)
CHiME Steering Group
- Jon Barker Univ. of Sheffield, UK
- Emmanuel Vincent Inria, France
- Shinji Watanabe Carnegie Mellon Univ., USA
- Michael Mandel CUNY/META, USA
- Marc Delcroix NTT, Japan
- Leibny Paola Garcia Perera Johns Hopkins Univ., USA