../../../../tools/bin/corpus_analyze.prl run by knight on Mon Aug 2 13:53:20 1999 Analysis of 50k-train.snt: ================================================================================ CORPUS DATA: 50k-train.snt contains 49000 sentences (45589 unique). -------------------------------------------------------------------------------- TOKENS TYPES (C > 1)* -------------------------------------------------------------------------------- SOURCE: 50k-train.snt 724780 19200 (11384) TARGET: 50k-train.snt 797743 24070 (13872) -------------------------------------------------------------------------------- *The number of types which appear more than once. ================================================================================ CO-OCCURENCE DATA: -------------------------------------------------------------------------------- TOTAL UNIQUE (C > 1)* -------------------------------------------------------------------------------- 50k-train.snt 14632212 2741303 (882685) -------------------------------------------------------------------------------- *The number of co-occurences which appear more than once. ================================================================================ SENTENCE LENGTH DATA: -------------------------------------------------------------------------------- 50k-train.snt SOURCE: AVERAGE SENTENCE LENGTH: 14.7914285714286 LENGTH COUNT (* = 40 SENTENCES) 1 : 34 2 : 257 ****** 3 : 342 ******** 4 : 726 ****************** 5 : 999 ************************ 6 : 1315 ******************************** 7 : 1552 ************************************** 8 : 1740 ******************************************* 9 : 1854 ********************************************** 10 : 1963 ************************************************* 11 : 2088 **************************************************** 12 : 2187 ****************************************************** 13 : 2278 ******************************************************** 14 : 2291 ********************************************************* 15 : 2332 ********************************************************** 16 : 2394 *********************************************************** 17 : 2266 ******************************************************** 18 : 2342 ********************************************************** 19 : 2206 ******************************************************* 20 : 2120 ***************************************************** 21 : 2087 **************************************************** 22 : 1954 ************************************************ 23 : 1756 ******************************************* 24 : 1558 ************************************** 25 : 1333 ********************************* 26 : 1136 **************************** 27 : 917 ********************** 28 : 729 ****************** 29 : 476 *********** 30 : 357 ******** TARGET: AVERAGE SENTENCE LENGTH: 16.2804693877551 LENGTH COUNT (* = 40 SENTENCES) 1 : 25 2 : 208 ***** 3 : 270 ****** 4 : 552 ************* 5 : 785 ******************* 6 : 1021 ************************* 7 : 1183 ***************************** 8 : 1427 *********************************** 9 : 1576 *************************************** 10 : 1790 ******************************************** 11 : 1817 ********************************************* 12 : 1906 *********************************************** 13 : 1953 ************************************************ 14 : 1962 ************************************************* 15 : 2024 ************************************************** 16 : 2083 **************************************************** 17 : 2079 *************************************************** 18 : 2030 ************************************************** 19 : 2047 *************************************************** 20 : 1931 ************************************************ 21 : 2026 ************************************************** 22 : 1960 ************************************************* 23 : 2006 ************************************************** 24 : 1864 ********************************************** 25 : 1812 ********************************************* 26 : 1701 ****************************************** 27 : 1619 **************************************** 28 : 1380 ********************************** 29 : 1327 ********************************* 30 : 1225 ****************************** ================================================================================