Note: the instructions contained on this page were written for use specifically in the CLSP WS02 Lab. They may refer to applications, files or other materials that are not accessible from other locations.

Information Retrieval Evaluation Lab

In this lab assignment, you will install an existing IR system (MG) and run it on a standard English document collection. Then you will do some experimentation with cross-language information retrieval, using the formal evaluation methods of the information retrieval community to assess experimental alternatives.

Getting an IR system running

MG information retrieval software

many other IR systems

Take a look at the introductory MG manual page mgintro.1 (e.g. via the command 'nroff -man mgintro.1' or on the Web), particular the section called "Getting Started with MG". Index the sample collection that comes with the system: if you follow the directions you'll index Alice in Wonderland. This is a good time to set up your executable path to include MG's bin directory. Note that you need the MGSAMPLE and MGDATA environment variables defined and the directory that MGDATA points to created for this to work. Again, here are cookbook-style instructions, assuming you're in your mg directory.
```
        mkdir ~/mgdata
	setenv MGDATA ~/mgdata
	setenv MGSAMPLE ~edrabek/ws02/mg/mg-1.2.1/SampleData
	set path = ($path /home/ws01/edrabek/ws02/mg/bin/)
	rehash
	mgbuild alice
   
```
Try a simple query or two, noting that MG defaults to interactive Boolean retrieval. For example, you could try the query mad AND hatter against the Alice collection, and then try mad WITH hatter to see how that's different. To issue queries against the Alice collection, get into mg's interactive query mode by executing:
```
        mgquery alice
   
```
Now, index a small English document collection of TIME magazine articles, formatted appropriately for use in TREC-style evaluations. First, edit the ~/.mg_getrc file that was automatically created when you ran mgbuild for the first time. Add a line for a new collection, called "trec" (without the quotes) -- for arcane reasons "trec" must be the name. (Yes, it's hard-wired. Ugh.) The file format should be PLAIN and the file containing the documents should be given as /home/ws01/edrabek/ws02/data/time/time_sgml.mgdata.gz. Note that the items on the line must be separated by tabs, not spaces!
Based on these specs, MG assumes all of the documents are in a single file with ctrl-B separating each document. MG strips SGML markup by default, which is appropriate for these documents, but it indexes everything else.
To actually build the index, invoke mgbuild trec.
Try a few queries to this collection. Using MG's default interactive query mode, you'll see sets of documents in response to your queries.
Extract this collection's document identifiers and put them where MG expects to see them, in the form that MG requires. To do this, issue the following commands:
```
          mg_get trec | mg_passes -S -f $MGDATA/trec/trec
          mv $MGDATA/trec/trec.docnums $MGDATA/trec/trec.DOCIDS
       
```
The first of these commands invokes the code in src/text/mg.special.c, which simply writes anything between start and end DOCNO tags to a file.
Ultimately when you do experiments, you'll want to get back lists of document ID's ranked by their scores. In mgquery, to perform ranked retrieval and suppress the display of document text, you issue the commands ".set query ranked" and ".set mode docnums" (without the quotes, but note the leading period). Try doing this and reissuing one or two of the queries you issued before.
Next set up a batch job that will (a) configure MG so that gives TREC-style output (ranked retrieval, no document text, see the .set commands above), and (b) run the query india and pakistan negotiations over kashmir. The easiest way to do this will probably be to create a script containing the relevant lines of input to MG, and then running mgquery with input redirected from that file and output redirected to an output file.
That is, put this into a file:

.set query ranked .set mode docnums india and pakistan negotiations over kashmir .quit
And then run:

cat $myfile | mgquery trec > $myotherfile

Doing IR evaluation

To do evaluation, you will use the trec_eval software, located on the system at /home/ws01/edrabek/ws02/utils/trec_eval. As you know, evaluating an IR system means comparing its output on a set of queries with a set of true relevance judgments. In TREC parlance, each query corresponds to a "topic" (a statement of the information being sought, containing a title, a description, and a longer narrative characterization of relevant documents), each document is identified by a "docno" (document number), and true relevance judgments appear in "qrels" files (qrels is short for "query relevance judgments").

The directory /home/ws01/edrabek/ws02/data/trec_eval_example contains an example of the relevant files.

frDocs.tar.gz - small set of English documents
queries.351-400.zip - small set of English "topics"
qrel.txt - "qrels" file containing manually determined relevance judgments
SampleResults.dat - example output of an IR system for 3 queries
Run trec_eval -q /home/ws01/edrabek/ws02/data/trec_eval_example/qrel.txt SampleResults.dat to evaluate the output in SampleResults.dat against the "ground truth" in qrels.txt.

You will see that the format of the system results (SampleResults.dat) contains 5 columns, for example:

  030 Q0 ZF08-175-870 0 4238 prise1 
  qid iter docno    rank sim run_id

Relevance for each docno to qid is determined from text_qrels_file, which consists of text tuples of the form You will see that the format of the query relevance judgments (qrels.txt) contains 4 columns, for example:

       
   353   0  FR940314-0-00049  1
   qid  iter  docno          rel

  trec_eval -q qrel.txt SampleResults.dat

Evaluating IR runs

Use MG to index the documents in frDocs.tar.Z. (Subtle point: see the note above about MG expecting ctl-B as a document separator for multiple TREC documents appearing in a single file.)
Create queries from the titles of the topics in topics.txt
Write a program to reformat MG's output into the 5-column system output format required by trec_eval.
Write a driver script to run your queries on the document index you've created and produce output suitable for trec_eval.
Run trec_eval to compare MG's output to the qrels in qrel.txt.

A more substantial experiment

The goal of this exercise is to experience the process of trying out several different cross language IR variations and evaluating them formally. The full group should break up into several teams trying different things out. Some obvious approaches to try include:

Monolingual Chinese IR using the C-desc field as the query. This provides a monolingual upper bound for English-Chinese CLIR approaches to aim for.
Dictionary-based query translation of terms from the E-desc field into Chinese.
Dictionary-based document translation of the terms from the Chinese documents into English, using the topic E-desc fields as queries.

Each group should do its development using topics 1-28, and at the end of the lab all groups should show their evaluation results using topics 29-54.

Note that you'll almost certainly need to use Chinese segmentation (see tokenize, below) and you may want to experiment with English stemming. If you're feeling particularly ambitious you might find some interesting way to use the WordNet data discussed below.

Some tools you might want to work with:

utils/segmenter/tokenize infile outfile
Must be run from directory utils/segmenter. This script runs the NMSU Chinese segmenter and does postprocessing in order to bring the SGML tags back in order, put real (ASCII) spaces in as token delimiters, etc. See README for how to use from/to stdin/stdout. You may need to break the input up into smaller pieces in order to get this to work.

~edrabek/ws02/data/dicts/

Berkeley.Chinese.Dictionary. A Chinese wordlist that might be useful for (re-)segmentation.
wordnet_nouns.txt. A list of nouns, where each noun is accompanied by identifiers for all the concepts in WordNet 1.7 that subsume it. Could be useful for term expansion.
EnZh.combined. For each Chinese word in this dictionary, there is an English word selected statistically on the basis of association in the Hong Kong News parallel corpus, together with the two top translations from a non-corpus-based dictionary.

Philip Resnik