NTCIR-10 Core Task: "2nd round of IR for Spoken Documents (SpokenDoc-2)"


The growth of the internet and the decrease of the storage costs are resulting in the rapid increase of multimedia contents today. For retrieving these contents, available text-based tag information is limited. Spoken Document Retrieval (SDR) is a promising technology for retrieving these contents using the speech data included in them. Following the NTCIR-9 SpokenDoc task, we will continue to evaluate the SDR based on a realistic ASR condition, where the target documents are spontaneous speech data with high word error rate and high out-of-vocabulary rate.

Data Set

The new speech data, the recordings of the first to sixth annual Spoken Document Processing Workshop (the SDPWS data set), are going to be used as the target document in SpokenDoc-2. The larger speech data, spoken lectures in Corpus of Spontaneous Japanese (CSJ), are also used as in the last SpokenDoc-1.

Task Overview

Targeting spoken documents, four subtasks of two categories will be conducted.

  • Spoken Term Detection: Within spoken documents, find the occurrence positions of a queried term. The evaluation should be conducted by both the efficiency (search time) and the effectiveness (precision and recall). Two subtasks will be conducted for the STD task. In addition to these tasks, an "inexistent Spoken Term Detection (iSTD) task" will also be conducted. In the iSTD task, task participants inspect whether a queried term is existent or inexistent in a speech data collection.
  • STD large-size task uses the CSJ as the target documents.
  • STD moderate-size task uses the SDPWS data set as the target documents.
  • iSTD task uses the SDPWS data set as the target documents.
  • Spoken Content Retrieval: Among spoken documents, find the segments including the relevant information related to the query, where a segment is either a document (resulting in document retrieval task) or a passage (passage retrieval task). This is like an ad-hoc text retrieval task, except that the target documents are speech data.
  • SCR lecture retrieval task uses the CSJ as the target documents.
  • SCR passage retrieval task uses the SDPWS data set as the target documents.


Standard STD and SDR methods first transcribe the audio signal into its textual representation by using Large Vocabulary Continuous Speech Recognition (LVCSR), followed by text-based retrieval. The participants can use the following three types of transcriptions.

  1. Manual transcription

    It is mainly used for evaluating the upper-bound performance.

  2. Reference Automatic Transcriptions

    The task organizers are going to provide reference automatic transcriptions for the target speech data. These enabled researchers interested in SDR, but without access to their own ASR system to participate in the tasks. They also enabled comparisons of the IR methods based on the same underlying ASR performance.

    The textual representation of them will be both the n-best list of the word or syllable sequence depending on the two background ASR systems, and the lattice representation of them.

    1. Word-based transcription

      Obtained by using a word-based ASR system. In other words, a word n-gram model is used for the language model of the ASR system. With the textual representation, it also provides the vocabulary list used in the ASR, which determines the distinction between the in-vocabulary (IV) query terms and the our-of-vocabulary (OOV) query terms used in our STD subtask.

    2. Syllable-based transcription

      Obtained by using a syllable-based ASR system. The syllable n-gram model is used for the language model, where the vocabulary is the all Japanese syllables. The use of it can avoid the OOV problem of the spoken document retrieval. The participants who want to focus on the open vocabulary STD and SDR can use this transcription.

  3. Participant's own transcription

    The participants can use their own ASR systems for the transcription. In order to enjoy the same IV and OOV condition, their word-based ASR systems are recommended to use the same vocabulary list of our reference transcription, but not necessary. When participating with the own transcription, the participants are encouraged to provide it to the organizers for the future SpokenDoc test collections.

Task Description


2012-06First call for participation
2012-06-30 2012-08-31Task registration due
2012-07 2012-09Document set release
2012-09 2012-12-14 ~ 21Dry run
2012-11 2013-1-15 ~ 22Formal run
2013-02Evaluation results release
2013-05Camera-ready copy of participant paper due
2013-06NTCIR-10 Workshop Meeting


  • Kiyoaki Aikawa (Tokyo University of Technology)
  • Tomoyosi Akiba (Toyohashi University of Technology)
  • Xinhui Hu (National Institute of Information and Communications Technology)
  • Yoshiaki Itoh (Iwate Prefectural University)
  • Tatsuya Kawahara (Kyoto University)
  • Seiichi Nakagawa (Toyohashi University of Technology)
  • Hiroaki Nanjo (Ryukoku University)
  • Hiromitsu Nishizaki (University of Yamanashi)
  • Yoichi Yamashita (Ritsumeikan University)


Registration form is available at the official page of NTCIR-10.


トップ   編集 凍結 差分 バックアップ 添付 複製 名前変更 リロード   新規 一覧 単語検索 最終更新   ヘルプ   最終更新のRSS
Last-modified: 2012-12-13 (木) 19:31:30 (1826d)