../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 14:13:32 1999 Analysis of 200k/200k-train.snt: ================================================================================ CORPUS DATA: 200k/200k-train.snt contains 196000 sentences (180484 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: 200k/200k-train.snt 2877016 32498 (20770) TARGET: 200k/200k-train.snt 3166693 41504 (26293) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- 200k/200k-train.snt 58306717 7089082 (2549932) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- 200k/200k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 14.6786530612245 LENGTH COUNT (* = 158 SENTENCES) 1 : 111 2 : 808 ***** 3 : 1227 ******* 4 : 2589 **************** 5 : 3836 ************************ 6 : 5155 ******************************** 7 : 5896 ************************************* 8 : 6937 ******************************************* 9 : 7398 ********************************************** 10 : 7962 ************************************************** 11 : 8465 ***************************************************** 12 : 8644 ****************************************************** 13 : 8994 ******************************************************** 14 : 9255 ********************************************************** 15 : 9382 *********************************************************** 16 : 9304 ********************************************************** 17 : 9443 *********************************************************** 18 : 9107 ********************************************************* 19 : 9099 ********************************************************* 20 : 8619 ****************************************************** 21 : 8068 *************************************************** 22 : 7713 ************************************************ 23 : 6871 ******************************************* 24 : 6209 *************************************** 25 : 5295 ********************************* 26 : 4405 *************************** 27 : 3571 ********************** 28 : 2812 ***************** 29 : 1966 ************ 30 : 1343 ******** TARGET: AVERAGE SENTENCE LENGTH: 16.1565969387755 LENGTH COUNT (* = 158 SENTENCES) 1 : 83 2 : 743 **** 3 : 934 ***** 4 : 2011 ************ 5 : 2834 ***************** 6 : 3835 ************************ 7 : 4829 ****************************** 8 : 5736 ************************************ 9 : 6354 **************************************** 10 : 6784 ****************************************** 11 : 7247 ********************************************* 12 : 7515 *********************************************** 13 : 7538 *********************************************** 14 : 7941 ************************************************** 15 : 8152 *************************************************** 16 : 8383 ***************************************************** 17 : 8321 **************************************************** 18 : 8244 **************************************************** 19 : 8278 **************************************************** 20 : 8134 *************************************************** 21 : 7898 ************************************************* 22 : 7918 ************************************************** 23 : 7690 ************************************************ 24 : 7438 *********************************************** 25 : 7140 ********************************************* 26 : 6607 ***************************************** 27 : 6258 *************************************** 28 : 5738 ************************************ 29 : 5281 ********************************* 30 : 4620 ***************************** ================================================================================