../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 15:12:44 1999 Analysis of experiments/20k/20k-train.snt: ================================================================================ CORPUS DATA: experiments/20k/20k-train.snt contains 19600 sentences (19576 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: experiments/20k/20k-train.snt 304900 22160 (12188) TARGET: experiments/20k/20k-train.snt 271375 45556 (18333) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- experiments/20k/20k-train.snt 4900897 2203002 (441777) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- experiments/20k/20k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 15.5561224489796 LENGTH COUNT (* = 18 SENTENCES) 1 : 3 2 : 104 ***** 3 : 273 *************** 4 : 341 ****************** 5 : 424 *********************** 6 : 634 *********************************** 7 : 690 ************************************** 8 : 764 ****************************************** 9 : 861 *********************************************** 10 : 951 **************************************************** 11 : 961 ***************************************************** 12 : 999 ******************************************************* 13 : 947 **************************************************** 14 : 992 ******************************************************* 15 : 1046 ********************************************************** 16 : 1007 ******************************************************* 17 : 978 ****************************************************** 18 : 902 ************************************************** 19 : 875 ************************************************ 20 : 787 ******************************************* 21 : 761 ****************************************** 22 : 704 *************************************** 23 : 707 *************************************** 24 : 646 *********************************** 25 : 535 ***************************** 26 : 529 ***************************** 27 : 446 ************************ 28 : 372 ******************** 29 : 337 ****************** TARGET: AVERAGE SENTENCE LENGTH: 13.8456632653061 LENGTH COUNT (* = 18 SENTENCES) 1 : 9 2 : 126 ******* 3 : 304 **************** 4 : 466 ************************* 5 : 683 ************************************* 6 : 824 ********************************************* 7 : 951 **************************************************** 8 : 989 ****************************************************** 9 : 1098 ************************************************************* 10 : 1116 ************************************************************** 11 : 1109 ************************************************************* 12 : 1129 ************************************************************** 13 : 1139 *************************************************************** 14 : 1080 ************************************************************ 15 : 1089 ************************************************************ 16 : 986 ****************************************************** 17 : 906 ************************************************** 18 : 893 ************************************************* 19 : 789 ******************************************* 20 : 673 ************************************* 21 : 619 ********************************** 22 : 590 ******************************** 23 : 465 ************************* 24 : 371 ******************** 25 : 363 ******************** 26 : 278 *************** 27 : 236 ************* 28 : 154 ******** 29 : 139 ******* 30 : 2 ================================================================================