../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 14:07:48 1999 Analysis of 100k/100k-train.snt: ================================================================================ CORPUS DATA: 100k/100k-train.snt contains 98000 sentences (90758 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: 100k/100k-train.snt 1446844 25284 (15570) TARGET: 100k/100k-train.snt 1592188 31985 (19378) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- 100k/100k-train.snt 29265894 4436290 (1513139) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- 100k/100k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 14.7637142857143 LENGTH COUNT (* = 80 SENTENCES) 1 : 60 2 : 437 ***** 3 : 623 ******* 4 : 1425 ***************** 5 : 1816 ********************** 6 : 2560 ******************************** 7 : 3046 ************************************** 8 : 3450 ******************************************* 9 : 3758 ********************************************** 10 : 3901 ************************************************ 11 : 4338 ****************************************************** 12 : 4312 ***************************************************** 13 : 4532 ******************************************************** 14 : 4629 ********************************************************* 15 : 4677 ********************************************************** 16 : 4710 ********************************************************** 17 : 4762 *********************************************************** 18 : 4583 ********************************************************* 19 : 4483 ******************************************************** 20 : 4285 ***************************************************** 21 : 4059 ************************************************** 22 : 3794 *********************************************** 23 : 3605 ********************************************* 24 : 3052 ************************************** 25 : 2640 ********************************* 26 : 2288 **************************** 27 : 1806 ********************** 28 : 1393 ***************** 29 : 1021 ************ 30 : 713 ******** TARGET: AVERAGE SENTENCE LENGTH: 16.2468163265306 LENGTH COUNT (* = 80 SENTENCES) 1 : 52 2 : 392 **** 3 : 483 ****** 4 : 1065 ************* 5 : 1471 ****************** 6 : 1946 ************************ 7 : 2404 ****************************** 8 : 2797 ********************************** 9 : 3133 *************************************** 10 : 3405 ****************************************** 11 : 3670 ********************************************* 12 : 3741 ********************************************** 13 : 3812 *********************************************** 14 : 4077 ************************************************** 15 : 4134 *************************************************** 16 : 4210 **************************************************** 17 : 4197 **************************************************** 18 : 4075 ************************************************** 19 : 4244 ***************************************************** 20 : 4064 ************************************************** 21 : 3936 ************************************************* 22 : 3852 ************************************************ 23 : 3868 ************************************************ 24 : 3711 ********************************************** 25 : 3535 ******************************************** 26 : 3254 **************************************** 27 : 3109 ************************************** 28 : 2965 ************************************* 29 : 2770 ********************************** 30 : 2386 ***************************** ================================================================================