1 Network Creation

As a check on the validity of utilizing the netlit approach to creating literature reviews, we conducted an experiment in which five (junior) researchers independently coded the same set of articles and then compared the resulting networks. The researchers read fifteen articles, pulled from a section of James Vreeland’s International Organizations syllabus. No researcher had prior knowledge or specialization in international organizations. We expect this exercise to be representative of researchers in a higher variance group given low correlation of the limited prior knowledge on the topic (and therefore a harder test). Each researcher followed the standard netlit data entry approach and coded the articles’ relationships in a simple Excel spreadsheet containing three columns: from, to, and citations. Using the resulting spreadsheet, we then used the review() function from our netlit package to create the network diagrams for each researcher, which are shown below.













2 Network Similarity

After coding the relationships and creating the networks, cehck for network similarity using two approaches. We gave each coder a copy of every other coder’s spreadsheet and asked them to assign a similarity score for each edge in the other coder’s network. In essence, the coders asked themselves “does this conceptual relationship also exist in my network?” The similarity score could take three distinct values: 0 if the relationship does not exist in their own network, 1 if the relationship exists in their own network but with different labels, and 2 if the exact relationship exists in their own network. Using this data, we then created tables summarizing the network similarity, shown below. Diagonals are NA as coders did not evaluate their own network (all edges would appear in their own network). Off-diagonals represent the percentage of shared edges perceived by the row when examining the column’s network. For example, using the strict evaluation criteria, Researcher 2 claims that 67% of the edges in Researcher 1’s network also appear in their own network. Since naming conventions can easily differ across researchers in referencing the same concept, we consider overlap between two networks if the same concept occurs even if labeling might differ slightly (e.g. 1s and 2s are indicative of overlap, while zeroes are not). The main table presents these findings. (We also include a “strict” table that counts only 2s as relationships that exist in both coders’ networks). As the main table shows, accounting for labeling differences greatly increases the network similarity. Well over half of all edges are shared between all pairs of researchers.
Table 2.1: Perceived Similarity Across Researcher Findings (Main)
Researcher 1 Researcher 2 Researcher 3 Researcher 4 Researcher 5
Researcher 1 NA 0.50 0.55 0.68 0.62
Researcher 2 0.93 NA 0.75 0.68 0.74
Researcher 3 0.96 0.62 NA 0.71 0.82
Researcher 4 0.74 0.54 0.85 NA 0.64
Researcher 5 0.89 0.65 0.75 0.71 NA
Table 2.2: Perceived Similarity Across Researcher Findings (Strict)
Researcher 1 Researcher 2 Researcher 3 Researcher 4 Researcher 5
Researcher 1 NA 0.42 0.40 0.55 0.49
Researcher 2 0.67 NA 0.50 0.45 0.59
Researcher 3 0.44 0.29 NA 0.58 0.31
Researcher 4 0.26 0.27 0.55 NA 0.33
Researcher 5 0.63 0.46 0.65 0.66 NA

These quantitative results are supported by the qualitative feedback we elicited from the researchers. We asked the five researchers “if you were to separately use each of these networks to understand the current state of the literature – both what currently exists and what is missing – would you come away with similar conclusions?” and received positive responses from all five. For example, Researcher 3 stated that “after reading the 15 papers provided and then looking at everyone’s maps, the central causal relationships are present in all maps.” Research 5 reported that “except for some minor details, everyone captured similar causal relationships as the main ones.” This feedback increases our confidence that our method is replicable; starting from the same literature, five independent junior researchers with no prior knowledge on the topic arrived at similar conclusions regarding the state of the literature. In addition, these represented the modal type of researcher who might use our review method: junior researchers who are familiar with the discipline but are new to a specific topic.