../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 14:24:54 1999 Analysis of 300k/300k-train.snt: ================================================================================ CORPUS DATA: 300k/300k-train.snt contains 294000 sentences (269592 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: 300k/300k-train.snt 4314007 37594 (24579) TARGET: 300k/300k-train.snt 4746879 47983 (31135) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- 300k/300k-train.snt 87794356 9258745 (3444114) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- 300k/300k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 14.6734931972789 LENGTH COUNT (* = 236 SENTENCES) 1 : 124 2 : 1114 **** 3 : 1782 ******* 4 : 3797 **************** 5 : 5501 *********************** 6 : 7517 ******************************* 7 : 8787 ************************************* 8 : 10175 ******************************************* 9 : 11217 *********************************************** 10 : 11767 ************************************************* 11 : 12772 ****************************************************** 12 : 12734 ***************************************************** 13 : 13471 ********************************************************* 14 : 13821 ********************************************************** 15 : 13993 *********************************************************** 16 : 13991 *********************************************************** 17 : 14131 *********************************************************** 18 : 13631 ********************************************************* 19 : 13338 ******************************************************** 20 : 12963 ****************************************************** 21 : 12210 *************************************************** 22 : 11428 ************************************************ 23 : 10732 ********************************************* 24 : 9172 ************************************** 25 : 8038 ********************************** 26 : 6756 **************************** 27 : 5365 ********************** 28 : 4078 ***************** 29 : 2998 ************ 30 : 2189 ********* TARGET: AVERAGE SENTENCE LENGTH: 16.1458469387755 LENGTH COUNT (* = 236 SENTENCES) 1 : 98 2 : 1011 **** 3 : 1385 ***** 4 : 2856 ************ 5 : 4288 ****************** 6 : 5704 ************************ 7 : 7071 ***************************** 8 : 8279 *********************************** 9 : 9261 *************************************** 10 : 10148 ******************************************* 11 : 10728 ********************************************* 12 : 11136 *********************************************** 13 : 11495 ************************************************ 14 : 11960 ************************************************** 15 : 12356 **************************************************** 16 : 12348 **************************************************** 17 : 12496 **************************************************** 18 : 12199 *************************************************** 19 : 12401 **************************************************** 20 : 12292 **************************************************** 21 : 11859 ************************************************** 22 : 11603 ************************************************* 23 : 11438 ************************************************ 24 : 11246 *********************************************** 25 : 10699 ********************************************* 26 : 10018 ****************************************** 27 : 9380 *************************************** 28 : 8771 ************************************* 29 : 7992 ********************************* 30 : 7074 ***************************** ================================================================================