* NTCIR-10 Core Task: "2nd round of IR for Spoken Documents (SpokenDoc-2)" [#u37d48a4]

** Introduction [#k4656e87]

The growth of the internet and the decrease of the storage costs are
resulting in the rapid increase of multimedia contents today. For
retrieving these contents, available text-based tag information is
limited. Spoken Document Retrieval (SDR) is a promising technology for
retrieving these contents using the speech data included in them.
Following the NTCIR-9 SpokenDoc task, we will continue to
evaluate the SDR based on a realistic ASR condition, where the target
documents are spontaneous speech data with high word error rate and
high out-of-vocabulary rate.

** Data Set [#ra96cf9b]

The new speech data, the recordings of the first to sixth annual
Spoken Document Processing Workshop (the SDPWS data set), are going to be used as the
target document in SpokenDoc-2. The larger speech data, spoken
lectures in [[Corpus of Spontaneous Japanese (CSJ)>http://www.ninjal.ac.jp/csj/data/]], are also used as in the
last SpokenDoc-1. 

** Task Overview [#p938876b]

Targeting spoken documents, four subtasks of two categories will be conducted.

- ''Spoken Term Detection'': 
Within spoken documents, find the occurrence positions of a queried
term. The evaluation should be conducted by both the efficiency
(search time) and the effectiveness (precision and recall). Two subtasks 
will be conducted for the STD task. In addition to these tasks, an "inexistent Spoken Term Detection (iSTD) task" will also be conducted. In the iSTD task, task participants inspect whether a queried term is existent or inexistent in a speech data collection.

-- ''STD large-size task'' uses the CSJ as the target documents.

-- ''STD moderate-size task'' uses the SDPWS data set as the target documents.

-- ''iSTD task'' uses the SDPWS data set as the target documents.

- ''Spoken Content Retrieval'': 
Among spoken documents, find the segments including the relevant
information related to the query, where a segment is either a
document (resulting in document retrieval task) or a passage
(passage retrieval task). This is like an ad-hoc text retrieval
task, except that the target documents are speech data.

-- ''SCR lecture retrieval task'' uses the CSJ as the target documents.

-- ''SCR passage retrieval task'' uses the SDPWS data set as the target documents.

** Transcription [#z4119923]

Standard STD and SDR methods first transcribe the audio signal into
its textual representation by using Large Vocabulary Continuous Speech
Recognition (LVCSR), followed by text-based retrieval.  The
participants can use the following three types of transcriptions.

+ Manual transcription
~ It is mainly used for evaluating the upper-bound performance.
+ Reference Automatic Transcriptions
~ The task organizers are going to provide reference
automatic transcriptions for the target speech data. These enabled
researchers interested in SDR, but without access to their own ASR
system to participate in the tasks. They also enabled comparisons of
the IR methods based on the same underlying ASR performance.
~ The textual representation of them will be both the n-best list of the
word or syllable sequence depending on the two background ASR systems,
and the lattice representation of them.
++ Word-based transcription
~ Obtained by using a word-based ASR system. In other words, a word
n-gram model is used for the language model of the ASR system. With
the textual representation, it also provides the vocabulary list used
in the ASR, which determines the distinction between the in-vocabulary
(IV) query terms and the our-of-vocabulary (OOV) query terms used in
our STD subtask.
++ Syllable-based transcription
~ Obtained by using a syllable-based ASR system. The syllable n-gram
model is used for the language model, where the vocabulary is the
all Japanese syllables. The use of it can avoid the OOV problem of
the spoken document retrieval. The participants who want to focus on
the open vocabulary STD and SDR can use this transcription.
+ Participant's own transcription
~ The participants can use their own ASR systems for the transcription.
In order to enjoy the same IV and OOV condition, their word-based ASR
systems are recommended to use the same vocabulary list of our
reference transcription, but not necessary. When participating with
the own transcription, the participants are encouraged to provide it
to the organizers for the future SpokenDoc test collections.

** Task Description [#s9276fd1]

- [[STD task ver.0.4 (2012.12.13)>http://www.cl.ics.tut.ac.jp/~sdpwg/ntcir10/std.pdf]] &color(red){(* updated *)};
- [[STD task ver.0.5 (2012.12.13)>http://www.cl.ics.tut.ac.jp/~sdpwg/ntcir10/std.pdf]] &color(red){(* updated *)};

- [[iSTD task ver.0.2 (2012.12.13)>http://www.cl.ics.tut.ac.jp/~sdpwg/ntcir10/istd-e.pdf]] (Japanese version is available [[here>http://www.cl.ics.tut.ac.jp/~sdpwg/ntcir10/istd-j.pdf]]) &color(red){(* updated *)}; 

- [[SCR task ver.0.4 (2012.12.13)>http://www.cl.ics.tut.ac.jp/~sdpwg/ntcir10/scr.pdf]] &color(red){(* updated *)};
- [[SCR task ver.0.5 (2012.12.13)>http://www.cl.ics.tut.ac.jp/~sdpwg/ntcir10/scr.pdf]] &color(red){(* updated *)};

** Schedule [#o19a0028]

| 2012-06 | First call for participation |
| %%2012-06-30%% 2012-08-31 | Task registration due |
| %%2012-07%% 2012-09 | Document set release |
| %%2012-09%% 2012-12-14 ~ 21  | Dry run |
| %%2012-11%% 2013-1-15 ~ 22  | Formal run |
| 2013-02 | Evaluation results release |
| 2013-05 | Camera-ready copy of participant paper due |
| 2013-06 | NTCIR-10 Workshop Meeting |

** Organizers [#mf15a758]

- Kiyoaki Aikawa (Tokyo University of Technology)

- Tomoyosi Akiba (Toyohashi University of Technology)

- Xinhui Hu (National Institute of Information and Communications Technology)

- Yoshiaki Itoh (Iwate Prefectural University)

- Tatsuya Kawahara (Kyoto University)

- Seiichi Nakagawa (Toyohashi University of Technology)

- Hiroaki Nanjo (Ryukoku University)

- Hiromitsu Nishizaki (University of Yamanashi)

- Yoichi Yamashita (Ritsumeikan University)

** Registration [#e79e3272]

Registration form is available at [[the official page of NTCIR-10>http://research.nii.ac.jp/ntcir/ntcir-10/howto.html]].

** Link [#xfec44fe]

- [[NTCIR-10>http://research.nii.ac.jp/ntcir/ntcir-10/index.html]]
- [[NTCIR-9 SpokenDoc>ntcir9]]
- [[NTCIR-9>http://research.nii.ac.jp/ntcir/ntcir-9/index.html]]

トップ   編集 差分 バックアップ 添付 複製 名前変更 リロード   新規 一覧 単語検索 最終更新   ヘルプ   最終更新のRSS