JHU Summer School on Human Language Technology

cp -r /export/ws03_mt/lab .

Training EGYPT/GIZA++

1. Modify your PATH environment variable:
• If using tcsh:
  

  setenv PATH ~anoop/tools/bin:$PATH; rehash

• If using bash:
  

  export PATH=~anoop/tools/bin:$PATH

2. Now to create the training data (a parallel corpus of aligned source e and foreign f text) for training the MT system. The particular translation task we will consider is translating from the archaic form of English used in some versions of the Bible to a more modern form of English:
   

   f: Abraham begat Isaac ; and Isaac begat Jacob ; and Jacob begat Judas and his brethren
   

   e: Abraham was the father of Isaac , and Isaac the father of Jacob , and Jacob the father of Judah and his brothers
   

   Other f languages available in the ~/mtrun/corpora directory are French, Spanish, Swedish and Tetun (East Timorese). As shown above, for this lab, we will be using two forms of English as source text and foreign text. This is useful since this will allow you to understand the kinds of decisions and the kinds of errors made by MT systems. Since training is computationally expensive we'll create a subset of our training data (note that system performance will be better with more training data). First, we create the training data for GIZA++:
   

   cd lab/giza-lab/english

   head -3000 bible.train.english > run/bible.e

   head -3000 bible.train.foreign > run/bible.f

3. To train GIZA++ on the data created above, run the training code. The trainer takes three arguments, the source text, the ``foreign'' text and the directory where the output translation model is written. The trainer first runs a tool called whittle which creates the training data for GIZA++. The trainer then runs the iterative EM training procedure described in the tutorial (this will take quite some time to terminate, depending on your machine specs).
   

   cd run

   runGIZA++.pl bible.e bible.f linux     

Viewing word alignments with Cairo

1. cd ~/lab/giza-lab/english/run
   

   
   
2. In this part of the lab, we will visualize the alignments induced by GIZA++ using Cairo (also part of the EGYPT system, written by Mike Jahr and Noah Smith). First we need to ``cairoize'' the statistical MT model (notice the "." after the command)

   sh do_cairoize .

3. Then run the Java based word alignment viewer, Cairo:

   /home/ws02/anoop/tools/cairo/run_cairo

4. You will have to browse (using File|Open) to the location where the alignments were generated (in ~/lab/giza-lab/english/run). The alignments are stored in a filename of the form username.date.pid -- open up the file that corresponds to your username
5. Find at least 5 distinct kinds of alignments that you think are incorrect. Explain for each one why you think it is an incorrect alignment.

Creating a Language Model with the CMU-Cambridge LM Toolkit

1. Now that we have a trained translation model we can try to decode sentences from the foreign text into the source English text. Before we can run a decoder, we need to build a language model for the source English text. We will use the CMU-Cambridge Language Modeling Toolkit to construct a LM.
2. First, we create a vocabulary list which will be used as input to the LM toolkit.
   

   cd ~/lab/giza-lab/english/run/lm

   perl voc.pl < ../../bible.train.english > english.bible.voc

3. Now we create a language model (an n-gram model of the text) using the LM toolkit:
   

   perl ~anoop/tools/bin/runCMUToolkit.pl ../../bible.train.english -d . -v english.bible.voc

Translating with the ISI ReWrite Decoder

1. cd ~/lab/giza-lab/english/run/decode
   

2. The config file isi-decoder.config has to be edited. The top two lines of the config file should be modified as follows (note that you have to supply to full path to your home directory and you cannot use ~  instead):
   

   LanguageModelFile = YOUR_HOME_DIR/lab/giza-lab/english/run/lm/bible.train.english.binlm

   TranslationModelConfigFile = YOUR_HOME_DIR/lab/giza-lab/english/run/linux/tmconfig.cfg

3. Now we can translate from modern English to archaic English. Note that the input to the decoder is the ``foreign'' text (modern English) and the output is archaic English. To run the decoder and save the results in a file output run:
   

   head -100 ../../bible.eval.foreign | isi-decoder.linux --config isi-decoder.config > english.out

4. Examine the input to the translation and the output. Compare the entire process of translation that you have just run to the noisy channel model (for a graphic of the model use acroread ~/lab/noisychannel.pdf) 

 Evaluating Translations with BLEU metric

1. cd ~/lab/giza-lab/english/run/eval

2. Score the baseline - no translation

   perl mteval-v09c.pl -r ref.sgm -s src.sgm -t src1.sgm -b

3. Score your translations

   perl trg2sgml.pl ../decode/english.out > trg.sgm
   

   perl mteval-v09c.pl -r ref.sgm -s src.sgm -t trg.sgm -b
   

 Improving Translation Quality

1. Now your task is to improve translation quality. You will need to create a new training set, retrain GIZA++ models, Language models, decode. Here are some ideas to go on - you are welcome to try your own ideas
• Based on alignments you saw using Cairo, merge certain multi-word Modern English phrases (e.g. "the father of " -> the_father_of ). Before evaluating the BLEU scores, you will need to remove the "_" from the decoder output.
• Lowercase the training data to reduce vocabulary size. Note that your test set has to be lowercased as well.
• Find the Porter stemmer from the web. Apply it to the archaic English. Again note that your test set needs to be consistent.

Word Segmentation and Machine Translation

Chinese Word (pinyin)	English Translation
da4	big
da4jie1	avenue
wo3	I
fang4	place
jie1	avenue
jie3fang4	liberation
bu4	not
liao3jie3	understand
wang4	forget
wang4bu4liao3	unable to forget
fang4da4	enlarge
na3li3	where
zai4	at

•  Extending the FSM (view it using acroread ~/lab/fsm1.pdf) construct a FST T1 that will insert a space after each word.  Denote the space character as    in your FST.
• Use the following sentence as an input to T1 and produce at least two sentences with different word segmentations:
  wo3wang4bu4liao3jie3fang4jie1zai4na3li3
•  Construct a FST T2 that takes as input the output of T1 and produces an output English translation based on the English entries in table shown. Write down the output of T2 for two different word segmentations.
• Which is the most plausible English sentence produced as an output by T2.
• This example illustrates that ambiguity in word segmentation can lead to problems in machine translation. Consider what would happen if all possible word segmentations were treated equally. Come up with a precise method to deal with many possible word segmentations in a Chinese-English MT system.

Automating Word Segmentation using the AT&T FSM Toolkit

• Set your paths for the FSM tools

• If using tcsh:
  

  setenv PATH /export/WS97/riley/tools-99/linux/bin/fsm-3/:$PATH; rehash

• If using bash:
  

  export PATH=/export/WS97/riley/tools-99/linux/bin/fsm-3/:$PATH

• The fsm text file for this FSM is in lab/seg-lab/fsm1.txt

  cd ~/lab/seg-lab

• Create vocabulary file for Chinese characters:

  awk 'NF>2 {print $3}' fsm1.txt | sort -u | perl -e ' print "<epsilon>\t0\n"; print "_\t1\n"; while (<>) { chomp; print "$_\t",$.+1,"\n";}' > char.map
  

•  Compile fsm1.txt into a FSM in AT&T format

  fsmcompile -i char.map fsm1.txt > fsm1.fsa
  

• Generate 5 strings from this FSM

  cat fsm1.fsa | /export/skumar/work/ATT/bin.linux_glibc/fsmmknb_fsa -n 5 -i char.map -l -
  

•  Extending the FSM (view using acroread ~/lab/fsm1.pdf ) construct a FST T1 that will insert a space after each word.  Denote the space character as     in your FST.An incomplete FST file T1.txt is provided for you. Compile this as an FST:

  fsmcompile -t -i char.map -o char.map T1.txt > T1.fst

•  You can generate random strings from fsm1.fsa to test your transducer T1

  fsmrandgen -n 1 fsm1.fsa > string.fsa

• Compose the random strings from fsm1.fsa with T1 as follows:

  cat string.fsa | fsmcompose - T1.fst |  fsmproject -o | fsmrmepsilon | /export/skumar/work/ATT/bin.linux_glibc/fsmmknb_fsa -n 5 -i char.map -l - 

• Use the following sentence as an input to T1 and produce at least two sentences with different word segmentation:  wo3wang4bu4liao3jie3fang4jie1zai4na3li3 -- This sentence is provided as a file string2.txt. Compile the file as an FSA:

  fsmcompile -i char.map string2.txt > string2.fsa

• Try composing string2.fsa with T1 and produce 2 sentences:

  fsmcompose string2.fsa T1.fst | fsmbestpath -n 2 | fsmproject -o | fsmrmepsilon | fsmprint -i char.map
  

• Construct a FST T2 that takes as input the output of T1 and produces an output English translation based on the English entries in table shown. A portion of this FST is given in a txt file T2.txt. 
• Generate the English vocabulary based on this file

  awk 'NF>2 { print $NF}' T2.txt | sort -u | perl -e 'while (<>) { chomp;  print "$_\t$.\n";}' > eng.map

• Compile the T2.txt as an FST

  fsmcompile -t -i char.map -o eng.map T2.txt >  T2.fst

• Compose output of T1 with T2 for two different segmentations

  fsmcompose string2.fsa T1.fst | fsmbestpath -n  2 | fsmproject -o | fsmrmepsilon | fsmcompose - T2.fst | fsmprint -i  char.map -o eng.map

• The output of fsmprint from the previous step contains the two different English translations due to the two word segmentations of the Chinese source input.

References

• Franz Josef Och, Hermann Ney. "Improved Statistical Alignment Models". Proc. of the 38th Annual Meeting of the Association for Computational Linguistics, pp. 440-447, Hongkong, China, October 2000. (Reference for GIZA++)
• The EGYPT toolkit. Developed by the Statistical Machine Translation team, WS'99, CLSP/JHU.
  http://www.clsp.jhu.edu/ws99/projects/mt
• P.R. Clarkson and R. Rosenfeld.Statistical Language Modeling Using the CMU-Cambridge Toolkit From Proceedings ESCA Eurospeech 1997. http://mi.eng.cam.ac.uk/~prc14/toolkit.html
• The ISI ReWrite Decoder by Daniel Marcu and Ulrich Germann. http://www.isi.edu/licensed-sw/rewrite-decoder/ 
• Mehryar Mohri, Fernando C. N. Pereira, Michael D. Riley.  AT&T FSM LibraryTM -- General-Purpose Finite-State Machine Software Tools. http://www.research.att.com/sw/tools/fsm/
  
  

Acknowledgements

• The EGYPT/GIZA++ tutorial above was taken from an earlier Summer Workshop lab conducted by David Yarowsky and Charles Schafer.
• Shankar Kumar (JHU) and Alex Fraser (ISI) helped tremendously to create and run this lab.