RunID Definition NTCIR GeoTime
Each run has to be associated with a RunID which is an identity for each run. Using NTCIR CLIR format as a base, the RunID is defined as follows:
[Group's ID]-[Topic Language]-[Document Language]-[pp]-[Run Type]
Two character language codes are as follows:
- EN (English)
- JA (Japanese)
Priority Parameter (pp)
The 'pp' is two digits used to represent the priority of the run, taking 01 as the highest and 05 as the lowest value. It will be used as a parameter for pooling. The participants have to decide the priority for each submitted run in the basis of each language pair.
For GeoTime participants, please submit at least one D-run (only DESCRIPTION field is used). You can submit up to 5 runs per language pair, but the at least one run should be the D-run as mandatory.
- D-run: only DESCRIPTION field is used
- DN run: both DESCRIPTION and NARRATIVE field is used
For example, a participating group, CMUJAV, submits 3 runs for English-Japanese. The first is a D run and the second is a DN run and third is a DN run with blind feedback (assumed to be best and ranked as priority 01). Therefore, the Run ID for each run is
CMUJAV-EN-JA-03-D CMUJAV-EN-JA-02-DN CMUJAV-EN-JA-01-DN
Note: Top 'n' documents in each of the submitted runs will be collected and put into the document pool. Only documnets in the pool will be judged by human assessors. If the number of the submitted runs are too large, the runs to be put in the pool may be selected based on the priority that you assign to each of the runs.