../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 15:02:59 1999 Analysis of 500k/500k-train.snt: ================================================================================ CORPUS DATA: 500k/500k-train.snt contains 490000 sentences (446659 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: 500k/500k-train.snt 7146630 44604 (30159) TARGET: 500k/500k-train.snt 7864975 56956 (38351) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- 500k/500k-train.snt 145865555 12820966 (4984699) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- 500k/500k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 14.5849591836735 LENGTH COUNT (* = 392 SENTENCES) 1 : 171 2 : 1668 **** 3 : 2775 ******* 4 : 6161 *************** 5 : 9131 *********************** 6 : 12606 ******************************** 7 : 14662 ************************************* 8 : 17214 ******************************************* 9 : 18451 *********************************************** 10 : 19398 ************************************************* 11 : 21001 ***************************************************** 12 : 21461 ****************************************************** 13 : 22113 ******************************************************** 14 : 23008 ********************************************************** 15 : 23231 *********************************************************** 16 : 23518 *********************************************************** 17 : 23268 *********************************************************** 18 : 22610 ********************************************************* 19 : 22330 ******************************************************** 20 : 21474 ****************************************************** 21 : 20162 *************************************************** 22 : 19049 ************************************************ 23 : 17374 ******************************************** 24 : 15145 ************************************** 25 : 13266 ********************************* 26 : 11160 **************************** 27 : 9096 *********************** 28 : 6732 ***************** 29 : 4865 ************ 30 : 3559 ********* TARGET: AVERAGE SENTENCE LENGTH: 16.0509693877551 LENGTH COUNT (* = 392 SENTENCES) 1 : 150 2 : 1499 *** 3 : 2173 ***** 4 : 4613 *********** 5 : 6905 ***************** 6 : 9414 ************************ 7 : 11819 ****************************** 8 : 13952 *********************************** 9 : 15637 *************************************** 10 : 16731 ****************************************** 11 : 17873 ********************************************* 12 : 18561 *********************************************** 13 : 18962 ************************************************ 14 : 19773 ************************************************** 15 : 20079 *************************************************** 16 : 20666 **************************************************** 17 : 20939 ***************************************************** 18 : 20439 **************************************************** 19 : 20382 *************************************************** 20 : 20329 *************************************************** 21 : 19843 ************************************************** 22 : 19474 ************************************************* 23 : 19090 ************************************************ 24 : 18474 *********************************************** 25 : 17696 ********************************************* 26 : 16559 ****************************************** 27 : 15437 *************************************** 28 : 14315 ************************************ 29 : 13132 ********************************* 30 : 11743 ***************************** ================================================================================