An Information Theoretic Interpretation of the Variable Rule

Deseret Language and Linguistic Society Symposium Volume 11 Issue 1 Article 17 2-15-1985 An Information Theoretic Interpretation of the Variable Rule Matthew R. Sorenson Follow this and additional works at: https://scholarsarchive.byu.edu/dlls BYU ScholarsArchive Citation Sorenson, Matthew R. (1985) "An Information Theoretic Interpretation of the Variable Rule," Deseret Language and Linguistic Society Symposium: Vol. 11 : Iss. 1 , Article 17. Available at: https://scholarsarchive.byu.edu/dlls/vol11/iss1/17 This Article is brought to you for free and open access by the Journals at BYU ScholarsArchive. It has been accepted for inclusion in Deseret Language and Linguistic Society Symposium by an authorized editor of BYU ScholarsArchive. For more information, please contact , . 162 r·nformat ion T1woret_ic rnterpn>tation of tho ViJridble Rule 1\11 Matthew R. Sorenson WICAT Education Institute 'Information' has been used informally and as a technical term in flf'VPUtl di~H~iplin0.q, 1>llt hi"l:'i not- bf!f'n wpll-d.,v01()I,,'d in t!lf' ~j()('i;lI sciences. 'fhis pap(H wi II Luuk at .inLorrnatioli theory "lid it:l r','!f.)Vdll';'! to linguistics, trying in particular to show how it illuminates the nature of the variable rule. I, , , Claude Shannon, John Von Neuman, and Norbert Weiner were the major [igures in the post war interdisciplinary science of systems. Von r:seuman contributed game theory, dealing with strategic decisions. Weiner founded cybernetic control theory and focussed on how systems regulate their own behavior to meet goals. And Shannon established a mathematical definition of information that encompassed both communication and thermodynamics. Although Shannon is the recognized father of information therory, it is the main contention of this paper that the ideas contributed by Weiner and Von Neuman are essential in a correct application of it. It is because decision and control have been neglected by linguists that information theory has not been found fruitful. I: By 1953, when Charles Hockett published a review of Shannon and Weaver's Mathematical Theory of Communication, there had already been some interest by linguists in information theory. Hockett lists three references and adds that 'It is not certain that all these references are based on adequate understanding of the theory.' (Hockett 1953: footnote 2) Hockett's (1953) review of Shannon and Weaver was tentative and ambiguous. He approached the topic with the attitude that it must be valuable to linguists, but he failed to show how. The review's first section describes the essential concepts of the theory. The following are paraphrases of Hockett's main points. Signals are chosen out of a set of possible alternatives. Information is transmitted when the variety in the alternatives is narrowed down by selecting one of the possibilities. If there is no choice, there is no information transmission. If the variety in the set of alternative signals increases, then the system has a greater informational capacity. A system with 'yes', 'no', and 'maybe' can transmit more information than a system with only 'yes' and 'no'. The basic terms apply to the capacity of an information source, but not to single instances. Information theory is concerned with averages. ~ I I 163 The source, channel, and receiver of information are distinguished, but the signal at any point in this sequence may be transformed (or transduced) into various material forms. Information is independent of mechanism. The code, or the syntax and semantics of the signal set, may be mdnipulated for ttw sake of efficiency or redundanc:y, rjer('ndinq on th(' demands on the systeln. The two primary meallS 0/ IIIdllif'llldl ion "r,' adjustments on the frequency and on the interdependence of signals. Shannon's first theorem says that codes can be made maximally efficient, if desired, so that channels can always be used at their maximum capacity. Shannon's second theorem is a generalization to cases involving a noisy channel, and says that any degree of noise can be offset by the use of redundancy. Having established these basic notions, Hockett begins a discussion of how they may apply to linguistics. His first point is that the continuous nature of speech and the discrete structures of langauge are not inconsistent and may be mathematically transformed into each other within information theory. Speakers (and field linguists) learn through experience to act as transducers to interpret continuous signals into discrete language units. Then Hockett addresses the extreme redundancy in language. Stress, for example, is redundant because it can be transmitted simultaneously with other signals. Writing is estimated to be about 50% redundant since we can read text in which half the letters have been removed. Problems of indeterminacy and transformation in morphology, and encoding through writing are muddled through in information theoretical terms, but as Hockett admits, 'Since there is currently no way in which all this can be disproved, it does not qualify as a scientific hypothesis; it is merely a terminology.' (p. 42) Hockett's final section outlines 'general implications' and touches on issues that would arouse discussion for years to come: the confusion of information and meaning, the tempting parallel between information and energy, and the idea that structuring of energy processes is accomplished by informational 'triggers , . The understanding of information theory in this review is unfortunately structural instead of regulative. The new ideas are seen as ways of describing the way language is, instead of the way it works or how it is used. For example, Hockett raises the idea that social information can be transmitted by variation in language, but immediately dismisses the possibility as uninteresting (Hockett 1953: 39) . 164 After the Hockett review, information theory was not absorbed by linguistics. Bar Hillel and Carnap (1952) made a valiant but uninterpretable effort at extending information theory into semantics. Greenberg (1956) made a well-conceived but superficial application in historical linguistics. Hockett continued to be interested and his 1977 volume contains early articles on several related topics. Zelig Harris tried early to examine syntactic structure in terms of statistical distribution of elements, and apparently is still at it (Harris ]qS?) Jakobson edited a volume on mathematical ]ingui~tic~ containing -a piece by him on information and communication (Jakob~J(Jn 1961) . The same volume also included a chapter by Rulon Wells referring to Church and Wittgenstein as the conceptual forebearers of irformation theory. Althou~h cluttered and unsystematic, On Human Communication by Colin Cherry (1957) was for years the best effort at applying information theory to language. It suffered, however, from too much engineering jargon and not enough appreciation of the human dynamics of communication. In 196 (...truncated)