../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 15:16:17 1999 Analysis of experiments/40k/40k-train.snt: ================================================================================ CORPUS DATA: experiments/40k/40k-train.snt contains 39200 sentences (39127 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: experiments/40k/40k-train.snt 606642 29497 (17783) TARGET: experiments/40k/40k-train.snt 539070 67114 (30297) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- experiments/40k/40k-train.snt 9700318 3863252 (827800) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- experiments/40k/40k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 15.4755612244898 LENGTH COUNT (* = 35 SENTENCES) 1 : 6 2 : 206 ***** 3 : 563 **************** 4 : 665 ******************* 5 : 897 ************************* 6 : 1226 *********************************** 7 : 1392 *************************************** 8 : 1496 ****************************************** 9 : 1725 ************************************************* 10 : 1885 ***************************************************** 11 : 1978 ******************************************************** 12 : 2001 ********************************************************* 13 : 2008 ********************************************************* 14 : 2039 ********************************************************** 15 : 2040 ********************************************************** 16 : 2041 ********************************************************** 17 : 1965 ******************************************************** 18 : 1863 ***************************************************** 19 : 1757 ************************************************** 20 : 1573 ******************************************** 21 : 1523 ******************************************* 22 : 1407 **************************************** 23 : 1327 ************************************* 24 : 1220 ********************************** 25 : 1015 ***************************** 26 : 1028 ***************************** 27 : 871 ************************ 28 : 759 ********************* 29 : 651 ****************** TARGET: AVERAGE SENTENCE LENGTH: 13.7517857142857 LENGTH COUNT (* = 35 SENTENCES) 1 : 12 2 : 254 ******* 3 : 668 ******************* 4 : 953 *************************** 5 : 1313 ************************************* 6 : 1655 *********************************************** 7 : 1893 ****************************************************** 8 : 2066 *********************************************************** 9 : 2227 *************************************************************** 10 : 2246 **************************************************************** 11 : 2295 ***************************************************************** 12 : 2291 ***************************************************************** 13 : 2274 **************************************************************** 14 : 2163 ************************************************************* 15 : 2134 ************************************************************ 16 : 1981 ******************************************************** 17 : 1804 *************************************************** 18 : 1717 ************************************************* 19 : 1484 ****************************************** 20 : 1388 *************************************** 21 : 1244 *********************************** 22 : 1130 ******************************** 23 : 927 ************************** 24 : 743 ********************* 25 : 705 ******************** 26 : 552 *************** 27 : 448 ************ 28 : 315 ********* 29 : 240 ****** 30 : 5 ================================================================================