../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 14:41:17 1999 Analysis of 400k/400k-train.snt: ================================================================================ CORPUS DATA: 400k/400k-train.snt contains 392000 sentences (358321 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: 400k/400k-train.snt 5730060 41553 (27590) TARGET: 400k/400k-train.snt 6303813 52999 (35013) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- 400k/400k-train.snt 116667544 11143847 (4242291) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- 400k/400k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 14.6175 LENGTH COUNT (* = 314 SENTENCES) 1 : 164 2 : 1423 **** 3 : 2286 ******* 4 : 5099 **************** 5 : 7331 *********************** 6 : 9908 ******************************* 7 : 11737 ************************************* 8 : 13722 ******************************************* 9 : 14879 *********************************************** 10 : 15870 ************************************************** 11 : 16674 ***************************************************** 12 : 17478 ******************************************************* 13 : 17708 ******************************************************** 14 : 18362 ********************************************************** 15 : 18791 *********************************************************** 16 : 18664 *********************************************************** 17 : 18596 *********************************************************** 18 : 18046 ********************************************************* 19 : 17647 ******************************************************** 20 : 17176 ****************************************************** 21 : 16177 *************************************************** 22 : 15281 ************************************************ 23 : 13984 ******************************************** 24 : 12171 ************************************** 25 : 10574 ********************************* 26 : 8991 **************************** 27 : 7350 *********************** 28 : 5363 ***************** 29 : 3972 ************ 30 : 2897 ********* TARGET: AVERAGE SENTENCE LENGTH: 16.0811556122449 LENGTH COUNT (* = 314 SENTENCES) 1 : 140 2 : 1278 **** 3 : 1791 ***** 4 : 3723 *********** 5 : 5532 ***************** 6 : 7652 ************************ 7 : 9326 ***************************** 8 : 11086 *********************************** 9 : 12690 **************************************** 10 : 13422 ****************************************** 11 : 14273 ********************************************* 12 : 15007 *********************************************** 13 : 15316 ************************************************ 14 : 15825 ************************************************** 15 : 16288 *************************************************** 16 : 16580 **************************************************** 17 : 16606 **************************************************** 18 : 16393 **************************************************** 19 : 16447 **************************************************** 20 : 16348 **************************************************** 21 : 15877 ************************************************** 22 : 15441 ************************************************* 23 : 15199 ************************************************ 24 : 14895 *********************************************** 25 : 14225 ********************************************* 26 : 13366 ****************************************** 27 : 12444 *************************************** 28 : 11379 ************************************ 29 : 10371 ********************************* 30 : 9401 ***************************** ================================================================================