Journal of Diplomatic Language

JOURNAL OF DIPLOMATIC   LANGUAGE
JDL I:4 (2004)

 Content Analysis of Short, Structured Texts: The Need for Multifaceted Strategies       

Dr. Harold W. Bashor, Ph.D.
American Graduate School of International Relations & Diplomacy
Paris, France

How can one analyze short, structured texts? Content analysis provides a systematic, replicable technique where validation normally takes the form of triangulation. Such mutual convergence lends credibility to the findings by incorporating multiple sources of data, methods, investigators, or theories. However, this paper examines the strengths and weaknesses of triangulation by comparing data collection methods for long, unstructured texts with short, structured texts. Considering the limitations of traditional triangulation, a multifaceted strategy has been tailored to meet the needs of specific, micro-linguistic analysis of a particular genre of text: international treaty preambles. In addition, a number of different perspectives to discern the following key elements and the dynamics of text have been examined: (1) purpose or overall agenda, (2) agent and his/her social-cultural standpoint, (3) the intended audience, (4) the socio-political context, and (5) most relevant to this paper, the language itself: syntax, semantics, style, rhetoric, and structure. The combination of these elements requires a multifaceted strategy that conventional triangulation often ignores in content analysis. This paper proposes a multi-layered approach with three different software programs. Although it has been argued in this paper that mutual validation of results is the ultimate goal, the capacity for methods to complement each other in the drive towards 'comprehensiveness' should not be assumed; nor should corroboration between methods be automatically considered unproblematic, and therefore precluding the need for further investigation.


INTRODUCTION

In an attempt to discover why so many international treaties have failed to achieve universal acceptance, this paper attempts to offer an innovative way of understanding international treaties by initially examining the micro-linguistic behavior of their preambles. In doing so, a unique explanation for a treaty's failure may be formulated--an argument that insufficient attention may have been paid to the treaty's rhetoric from the very inauguration of its discourse.

To begin, how can one systematically analyze short, structured texts? For the deconstructionist, language is everything. The world itself is "text" and the source of meaning can be found in the rhetoric of the text (Norris, 1982, pp. 18-22; Ducrot, 1984, pp. 95-98). By making these assumptions and deconstructing, analyzing, and comparing the style and rhetoric of preambular language, many complications and underlying inadequacies of preambular rhetoric may be exposed.

The Diplomatic Handbook defines a treaty's "preamble" as somewhat formal in style, setting out the names of the parties in the agreement, the names of the plenipotentiaries and the object of the agreement (Feltham, 1998, p. 87). However, Article 31 of the Vienna Convention on the Law of Treaties (1969), asserts that the preamble is an integral part of a treaty. The preamble is the first place to look for an explicit general statement of a treaty's objects and purposes that will govern the development of the treaty and promote its signature and ratification. The preambular language of any treaty, if found to be unclear and ambiguous, may thus be an initial reflection of the rhetorical inadequacies of the corpus of the text to follow.

The purpose for commencing with a study of the rhetorical structure of preambles is threefold. First, the genre of such short, structured texts allows the researcher to become familiar with computerized language analysis programs before analyzing the respective treaties in depth. Second, attention can be paid to the measurement abilities of the software chosen: did the programs produce the evidence that demonstrated the validity (or invalidity) of the inferences made? Finally, the analysis of these texts serves as a proving ground for the creation of new methodological strategies.

To exemplify rhetorical analysis of short, structured texts attention will be paid to the preamble of the Moon Treaty (1979). The "Golden Age" of the formation of international space law began in 1963 when the U.N. General Assembly adopted the Declaration of Legal Principles Governing the Activities in the Exploration and Use of Outer Space; however, it ended in 1979 when the Agreement Governing the Activities of States on the Moon and Other Celestial Bodies, the Moon Treaty, established the Moon and its natural resources as the "common heritage of mankind." The fifth in a series of United Nations space treaties from 1967 to 1979, the Moon Treaty was drafted by consensus and called for an international regime to govern the exploration and exploitation of lunar resources "as such exploitation is about to become feasible." The treaty also called for "equitable sharing by all countries "with special benefit for the developing countries." Although the United States and the former Soviet Union ratified the first four space treaties and were active negotiators in drafting the Moon Treaty, both states have since failed to sign and ratify the Moon Treaty. To this day, although fourteen states have signed the treaty, only ten states have succeeded in ratifying it.

In addition to deconstructing the Moon Treaty preamble, the preambles of the United Nations Charter (1945) and the Vienna Convention (1961) are comparatively analyzed considering their predominant role in establishing conditions under which respect for the obligations arising from treaties and other sources of international law should be maintained. Thus, it is assumed in this paper that these preambles should be exemplary models for all treaties.

The Limitation of Triangulation for Short, Structured Texts

This paper will confine itself to the third purpose for studying this genre of text: the importance of formulating new methodological strategies. Basically, content analysis is a systematic, replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding (Krippendorff, 1969, p. 103; and Weber, 1990, pp. 15-16). It also allows inferences to be made that can then be corroborated using other methods of data collection. In fact, validation of the inferences made on the basis of data from one analytic approach demands the use of multiple sources of information. In qualitative research, validation normally takes the form of triangulation, which lends credibility to the findings by incorporating multiple sources of data, methods, and researchers. However, triangulation in research that integrates quantitative and qualitative methods is a rarity in the International Relations (IR) discipline. Since classic IR models have been known to fail, the lack of this integration has spurred interest in pursuing means such as content analysis of international documents for capturing the gist of these human artifacts. Triangulation is, however, limited in the analysis of short, structured texts such as preambles. Table 1 compares the strengths and weaknesses of triangulation by examining multiple data collection methods, data sources, and researchers for large, unstructured texts in contrast to short, structured texts:

Table 1: Strengths and weaknesses of triangulation


Considering the limitations of traditional triangulation discussed above, multifaceted strategies should be tailored to meet the needs of specific, micro-linguistic analysis. When analyzing short, structured texts such as preambles, it is necessary to approach a text from a number of different perspectives to discern the following key elements and the dynamics of this particular genre: (1) purpose or overall treaty agenda, (2) agent and his/her social-cultural standpoint, (3) the intended audience, (4) the socio-political context, and (5) most relevant to this paper, the language itself: syntax, semantics, style, rhetoric, and structure. Table 2 reveals that the combination of these language elements requires a multifaceted strategy that common triangulation ignores in content analysis:

Table 2: Multifaceted Strategies for comprehensive micro-linguistic analysis


Comprehensive Multifaceted Strategy

To exemplify a multifaceted strategy, Diction 5.0, SphinxSurvey Lexica, and Nud*st 5 have all been employed to examine the sub-texture of the preambles of the Moon Treaty, the U.N. Charter, and the Vienna Convention. First, Diction 5.0 helped to determine the tone of the preambles by using programmed dictionaries that searched each preamble for five general features as well as thirty-five sub-features. SphinxSurvey Lexica was then used to generate factor maps from lexical cross-analyses for a graphical representation of the preambles. Finally, the preambles were imported into Nud*ist 5 and coded in order to concurrently monitor and manage the new ideas that had emerged, to further test hypotheses of inadequate rhetoric, and to confirm any trends qualitatively.

In sum, this paper was designed to report on the meticulous examination of the text of the Moon Treaty's preamble in light of other more successful and universally accepted treaties by using a multifaceted strategy. Creating a frequency table was the preliminary step taken which listed the lexicon of the preambles. Also, a comprehensive concordance created at the onset of the research was a valuable tool for qualitative analysis of the preambles. Initial Diction 5.0 analysis of the Moon Treaty preamble in comparison with those of the U.N. Charter and the Vienna Convention has confirmed what this research has suspected from the beginning: the Moon Treaty is far from characteristic of other multilateral treaties with respect to rhetoric.


CONTENT ANALYSIS: Diction 5.0

In order to deconstruct the Moon Treaty, the U.N. Charter, and the Vienna Convention preambles, the lexical choices of the multilateral negotiators have been analyzed in depth. The assumption made here was that the vocabularies of negotiators have revealed important clues about their philosophical worldview, psychic disposition, and social situation (Hart, 2000, p. 18). In general, diplomatic negotiators see themselves as experts in the ways of words; in fact, International Relations as a discipline can be considered a world of words. In the case of the Moon Treaty, however, this paper has alleged that these words may have been poorly contextualized, poorly theorized, and thus rhetorically inept.

The first tool used to investigate the lexical world of the Moon Treaty was Diction 5.0, a program that passes over a text systematically, objectively, and unobtrusively with the assistance of forty-seven dictionaries to examine texts for their verbal tone. This program has often been used to determine the authorship of anonymous works or to resolve conflict when authorship is questioned. Diction's dictionaries provide lexical classifications to produce standardized scores highlighting a text's features according to five master categories:


Results: The U.N. Charter, the Vienna Convention, and the Moon Treaty

By initially focusing on the lexical choices of the preambles of these three treaties, attention was immediately flagged to matters normally ignored by more conventional contextual analysis. At first sight in Table 3, the Moon Treaty (Moon) exhibits an unusually low score on Commonality, between twelve and sixteen standard deviations below the same scores for the U.N. Charter and Vienna Convention respectively.

Table 3: Master Category Scores (UN, Vienna, Moon)

The score on commonality reflects agreed-upon values of groups and rejects any idiosyncratic modes of engagement. The Moon Treaty's low score on Commonality is the result of low scores on Centrality and Cooperation. Centrality is measured by terms denoting institutional regularities and/or substantive agreement on core values: indigenous terms (native, basic), designations of legitimacy (constitutional, ratified), terms of congruence (conformity, mandate), and terms of universality (mankind, landmarks). The low score on Commonality is also due to a shortage of terms designating behavioral interaction that results in group processes.

Very low Commonality scores for the Moon Treaty may be understandable considering the ultimate goals of the U.N. Charter and the Vienna Convention as well as their historical context. However, it must be remembered that the Moon Treaty was committed to "equitable sharing" of the moon's resources by all states. With such language favorable to the developing countries at that time, it is understandable that support for the Moon Treaty by the two major space powers might have been problematic. Yet, it remains a paradox that the Moon treaty suffered such low levels of ratification by developing countries. The immediate absence of any terminology expressing "common ground" in the preamble may have been a potential limitation on its persuasiveness. This idiosyncrasy was monitored closely as the preamble was analyzed further.

The large inconsistency in Commonality can also be easily confirmed in Table 4 by examining the abundance of terms of universality, associations, and interaction in the U.N. Charter and Vienna Convention in comparison with the Moon Treaty. The Moon Treaty is comparable in length to the U.N. Charter (226 words) but is dwarfed by the Charter in Commonality terms. It should also be noted that the Moon Treaty preamble (219 words) is much greater in length than the Vienna Convention (143 words) despite the fewer number of terms denoting commonality:

Table 4: Terms of universality, associations and interaction

Conversely, it is also interesting to note that the Moon Treaty preamble has consistently scored the highest in all measurements of Realism. Such high scores in Figure 1 deserve attention. They reflect the Moon Treaty's mundaneness and avoidance of complex ideology. Again, while the other treaties examined have depended on more idealistic terminology such as all mankind, cooperation, justice, heritage, knowledge, and harmony for persuasion, the Moon Treaty was preoccupied with gregarious, avaricious gains: development, exploration, exploitation, derived benefits, use, uses, used. The lexical measure of Realism has thus shed light on the economic undertones found in the Moon Treaty's preamble. It could be argued such terminology previously outlined could be considered untimely considering the post-colonial period when the outer space treaties were proposed. A truly multifaceted strategy would require additional in-depth socio-political analysis of the historical context of the treaty.

Figure 1: Realism scores


Diction's built-in dictionaries have offered an innovative way to analyze the text of the Moon Treaty in comparison with more successful treaties. From the analysis it could be argued that the Moon Treaty preamble exhibits a high degree of ambiguity ascertained by the low scores on Certainty and by the low Type-Token Ratio. The Moon Treaty preamble is deficient in terms of group values, interaction, and cooperation-all indispensable in multilateral treaty-making.

However, only the preambles of the treaties were analyzed with Diction. Differences in the length of the preambles could have negatively impacted the variable scoring, as shorter texts contained fewer distinct topics while longer texts tended to emphasize the key principles of the treaty to follow. Because of the varying length of these condensed general statements and the risk that Dicton's five major categories (Activity, Optimism, Certainty, Realism and Commonality) may not have been sufficiently comprehensive to evaluate these texts, it was decided that the results should be confirmed with supplementary quantitative and qualitative analysis in line with the proposed multifaceted strategy.

For example, there were distinct structural differences in the characteristics of the preambles themselves. For example, the Moon Treaty's average word size was 5.06 characters per word, while the U.N. Charter and the Vienna Convention averaged 5.11 and 5.84 respectively. The fact that the average word size of the Moon Treaty was less than that of the other two preambles may not have had any effect on its persuasiveness, but it is another interesting characteristic that could be investigated further.

Keeping these problems and limitations in mind, the research proceeded with analysis of the same treaties using SphinxSurvey Lexica. This program's ability to generate factor maps from lexical cross analysis tables has introduced another innovative way to compare the textual data of the treaties' preambles. More precisely, the program generated new ideas with which to return to the preambles by identifying lexical clusters, testing the strength of their associations, and analyzing correlations between variables in order to perform multivariate analyses.

FACTOR MAPPING: SphinxSurvey Lexica
The first step of the analysis with SphinxSurvey Lexica was to generate a word lexicon of the words present in the preambles. After omitting tool words (those vital for constructing language but which convey little meaning) a frequency table by percentages was constructed. However, it was difficult to interpret any meaning from the tabulated data, the same data that was previously examined with Diction 5.0. Therefore, cross-tabulations were calculated in order to create a factor map, a graphical representation of the relationship between the values of the lexical table that might have otherwise been illusive alone with the analysis with Diction 5.0.

Results: The U.N. Charter, the Vienna Convention, and the Moon Treaty

The following factor map in Figure 2 has been constructed for the preambles of the United Nations Charter (UN), the Vienna Convention on Treaties (VIENNA), and the Moon Treaty (MOON) in order to analyze how the Moon Treaty preamble measures up to these models of classic treaty-making in the 20th century.

Figure 2: Factor Map-UN Charter, Vienna Convention, and Moon Treaty

The value of this factor map is the way in which it visually and intuitively displays findings which would otherwise have been time consuming and difficult to conclude with a frequency table. Proximity is determined by co-ordinates which indicate how two variables are associated and their degree of relationship (Kruskal and Wish, 1978, 7-11). Thus, for example, the proximity of co-ordinates reveals the extent to which the preambles are similar; the greater the distance between the proximity points, the less association between the vocabulary of the treaties and the larger their dissimilarity.

The size of the co-ordinates themselves indicates the number of corresponding occurrences or observations. The largest points (red boxes) represent the core preambles and the smaller points represent multiple correlation of the terminology among the treaties. For example, in the map above it was correctly concluded that the preambles of the Moon and the U.N. Charter are similar in size but smaller than the Vienna Convention preamble.

The two axes of the graph correspond to the two strongest factors and the value (in brackets) next to each axis label shows how much of the overall variance between occurrences is explained by this factor; thus in the example above Axis 1 accounts for 64.3% of the variance and Axis 2 for 35.6%. These are both relatively strong factors and the values are well distributed across the map. The great demarcation between the Moon Preamble and its counterparts confirms that it addresses very different issues.

The following areas deserve attention when evaluating the factor map: (1) the central area of the factor map where common features of the preambles all correlate, (2) the three main areas representing the preambles where the large boxes represent the lexical size of each preamble, and (3) the relationship, distance, and characteristics of the spaces between the main areas.

Figure 3: Factor Map-Central Area

Figure 3 illustrates the area most common to each of the preambles consisting of the most distinctive words shared by each preamble: international, promote, and progress. This configuration is reasonable considering the ultimate goals of international treaties to develop cooperation. However, in defense of a multifaceted strategy, the concordance reveals that promote is not used in the same context. Whereas promote refers to exploration and exploitation in the Moon Treaty, promote refers to social progress and friendly relations in the U.N. Charter and Vienna Convention.

This contextual difference only helps to reinforce the Diction's low scores on the Moon Treaty's Commonality discussed earlier-a low score of Commonality revealing that the Moon Treaty was not a "people's" treaty. According to its co-ordinates, the term progress is more closely related to the Moon Treaty and the U.N. Charter. This placement is due to the frequency of the word as a percentage of the total vocabulary of the texts. This may also be further delineated by the Moon Treaty's call for progress in exploration/exploitation in contrast to the Charter's call for social progress.

Figure 4: Factor Map-Moon Treaty
Figure 4 represents the main lexical area of the Moon Treaty which is bordered by a small area concentrated with terms completely cut off from the UN and Vienna areas. These terms are unique to the Moon Treaty and suggest the strong economic twist of the treaty (exploitation, derived, benefits, and achievement). Also, as will be discussed later, the absence of the term "peace" is compensated with a more negative equivalent (prevent conflict).

Figure 5: Factor Map-U.N. Charter
Figure 5 shows that the U.N. Charter has a monopoly on the human aspect of treaty-making (mankind, generations, men, women, life). Such rhetoric combined with "freedom" and "tolerance" offset the negative term "war" with its ultimate goal of saving "generations from the scourge of war." This is model rhetoric for treaty any preamble.

Similarly, Figure 6 reveals the Vienna Convention's monopoly on legal terminology which is understandable considering its goal of codifying customary law (rules, constitutional, codification, sovereign). However, it also shares the human aspect of treaty-making (friendly, peaceful, achievement, freedoms, independence). "Peaceful" occurs in the Vienna Convention as does "peace" in the U.N. Charter. However, this term or its derivative is not mentioned in the Moon Treaty preamble which only enhances its high scores for Realism.

Figure 6: Factor Map-Vienna Convention


The co-ordinate of "peace" in the Charter is not hidden from the Vienna lexical area so there is a correspondence between the two preambles. The fact that "peace/peaceful" coordinates do not lie closer to the common central area is surprising given the fact that the Moon Treaty was negotiated by the U.N. Committee on the Peaceful Uses of Outer Space (COPUOS). The Moon Treaty preamble does, however, refer to the prevention of conflict, but such vocabulary leaves the impression that conflict, not peace is inevitable. Figure 7's area of proximity between the Vienna Convention and the Moon Treaty is unremarkable except for the term equality. Reference to the concordance reveals that the Convention is more concerned with the "sovereign equality of States"; whereas the Moon Treaty aims to promote equality only in exploration/exploitation of lunar resources. Commonality is thus negated considering the technical inequality of space development.

Figure 7: Factor Map-VC > Moon


The strong correspondence between the Vienna Convention and U.N. Charter reveals a low degree of Concreteness which accounts for low Realism scores as indicated in Figure 8. Moon Treaty preamble nouns are extremely physical and tangible but few in number and often repeated. Such high scores in Concreteness point to little emphasis on ideology or abstract virtues. As seen here, there is an abundance of rich ideological terms (fundamental, justice, faith, equal rights, security, peace)-basic treaty-building blocks.

Figure 8: Factor Map-VC > Charter

Other Results and Discussion

As earlier mentioned, the Moon Treaty preamble is characterized by little variety of vocabulary, but the high frequency of concrete and material terms gives it the appearance of an inventory or a "to do" list. As if to compensate for the low Commonality and high Concreteness scores, the Moon Treaty preamble scores for Activity, Optimism, and Realism all tend to outpace those of the Charter and Convention. These stylistic features do not exist in isolation but form a distinctive tone-the tone of a treaty that needs to prove itself despite the lack of substance. It has also been argued that if a document uses many "understate" terms (little, some, other, apparent), it is also likely to use many "overstate" terms (natural, absolute, necessary). Namenwirth and Weber (1987) indicate that documents high in both overstate and understate terminology tend to discuss matters very defensively. It is also interesting to note that the Moon Treaty preamble attempts to defend its claims by recalling the four preceding outer space treaties for further validation even though it recognizes the "need to define and develop" the provisions of these earlier treaties to be applicable to the moon. The fact that the Moon Treaty preamble relies on this "band wagon" effect implicitly acknowledges the treaty's limited vision and need for acceptance.

The low variety of vocabulary in the Moon Treaty preamble conforms to Wendell Johnson's (1946) "Type-Token Ration" that divides the number of different words in a passage its total lexicon. A high score indicates a speaker's avoidance of overstatement and preference for precise, molecular statements. Conversely, a low score would point toward overstatement to the point of exaggeration as Table 5 illustrates:

Table 5: Type-Token Ratio
These calculations exhibit a low percentage of variety for the vocabulary of the Moon Treaty preamble which only strengthens this paper's proposition that the passage could be described as relatively imprecise or ambiguous. The relatively low Certainty scores have also revealed a degree of ambiguity by noting the low occurrence of Leveling Terms, words used to ignore individual differences and to build a sense of completeness and assurance. Ambiguity is also indicated by the general lack of resolute and totalizing terms (everyone, fully, completely, consistently). Instead the Moon Treaty preamble flounders with very general and non-specific adjectives (other, further, some).

Conclusion

This paper's multifaceted strategy of comprehensiveness demonstrates that the researcher must shift back and forth between the text and the output of the analyses progressively refining and validating the study's hypotheses. The use of multifaceted strategies implies a common ground, a central truth, a correct solution that could be approximated by different methods, neither complete nor perfect in themselves but jointly supportive of each other. Thus, the value of a multifaceted strategy is not that it guarantees conclusions about which we can be confident but rather it provokes a more critical, even skeptical, stance towards our data. It is hoped by presenting this strategy for short, structured texts such as treaty preambles that more discussion and elaboration will take place in the International Relations discipline, and be done so within the concerns that this model has attempted to address. It is further hoped that this strategy has contributed to the goals of treaty-making: the dynamic process of interaction, of attempts made and tested, and of learning and growing in the effort of persuasion for international cooperation.

References

Ducrot, Oswald (1984). Le Dire et le Dit. Paris: Les Éditions de Minuit.
Feltham, R.G. (1998). Diplomatic Handbook. New York: Longman Publishing Group.
Hart, Roderick P. (1997). Modern Rhetorical Criticism. Boston: Allyn and Bacon.
Hart, Roderick P. (2000). DICTION 5.0: The Text-Analysis Program. Thousand Oaks, CA: Scolari/Sage Publications.
Johnson, Wendel (1994). "Studies in language behavior: A program of research." Psychological Monographs, 56: 1-15.
Krippendorff, K. (1969). Models of messages: three prototypes. The analysis of communication content. G. Gerbner, O. R. Holsti, K. Krippendorff, G. J. Paisly and P. J. Stone (Eds.), New York: Wiley.
Krippendorff, K. (1980). Content Analysis: An Introduction to Its Methodology. Beverly Hills, CA: Sage Publications.
Kruskal, Joseph B. and Myron Wish (1978). Multidimensional Scaling. London: Sage Publications.
Namenwirth, J. Zvi and Robert Philip Weber (1987). Dynamics of Culture. London: Unwin Hyman.
Norris, Christopher (1982). Deconstruction: Theory and Practice. London: Methuen.
Weber, Robert P. (1990). Basic Content Analysis. Newbury Park, California: Sage Publications.

Commentary

Home
.