EXPONENTIAL RECURRENCE DISTRIBUTION IN THE SIMON-YULE MODEL OF TEXT

ABSTRACT

Exponential recurrence phenomenon has been reported in the study of gaps between repetitions of words in a text. The phenomenon has its applications in several computer–based natural language systems. In this article, four leading statistical models of text generation are evaluated and we identify the Simon Yule model of Zipf's law as a promising approach. A realistic refinement of the Simon–Yule model is made to allow for a decreasing entry rate of new words. Simulation methods are used to show that the exponential recurrence phenomenon is preserved with this change in assumptions. Significant implications of the approach are discussed.