Counting Articles in Nexis Uni

We wanted to count the number of articles on a number of keywords that were published in specific newspapers. This is a measure of salience. So we have a list of keywords (e.g. Boris Johnson, Jeremy Corbyn, Jo Swinson, Caroline Lucas), a date range (e.g. 1 January 2017 to 31 December 2017), and Nexis Uni to get the articles. In this case, we were not interested in the contents of the articles, so I downloaded the meta data (headline, publication, an empty “summary” column, and the date of the publication). Nexis Uni gives me a spreadsheet per keyword. With some 30 keywords, I did not want a manual approach.

What we wanted in the end was a spreadsheet, with all dates in the date range and the number of articles for each keyword as a column. Here’s how I did this in R:

sheets = list.files(pattern="*.XLSX")

This gives me a list of all the XLSX files in the folder. I use library(readxl) because it imports the dates properly, unlike some of the other options to open XLSX files in R.


for (i in 1:length(sheets)) {
assign(paste("N", i, sep=""), read_excel(sheets[i]))

Here I chose a loop to read in the data from the Nexis spreadsheets, each into a separate container. I guess I could have used a list or something, but for this project speed was no concern.

Next, we need a list of dates in the date range. The seq() already works, but to make this match the date format from the Excel documents, I need to wrap everything in the as.POSIXlt.

dat = as.POSIXlt(seq(from=as.Date("2017-01-01"), to=as.Date("2017-12-31"), by=1))

Now comes the actual work. First I create a vector with the dates of the articles in the Nexis spreadsheet for each keyword. Some dates in the date range have no articles, others have one, others still have more than one. I then use sapply() to match the dates, and colSums() to count the number of articles for a given day.

for (i in 1:length(sheets)) {
assign(paste("D", i, sep=""), eval(parse(text=paste("N", i, "$Date", sep=""))))
assign(paste("S", i, sep=""), eval(parse(text=paste("sapply(dat, function(x) D", i, "== x)", sep=""))))
assign(paste("H", i, sep=""), eval(parse(text=paste("colSums(S", i, ")", sep=""))))

At this stage I regret not using a list or data frame, but this code combines the different variables:

hits = data.frame(dat)
for (i in 1:length(sheets)) {
hits = cbind(hits, eval(parse(text=paste("H", i, sep=""))))

And then I add the names of the sheets and I have a data frame I can export with write.csv() or whatever.

colnames(hits) = c("date", sheets)

New Publication: The Gap between Public Preferences and Policies on Immigration

Just days after announcing the “SOM book” (Politicization of Immigration), I have the pleasure to announce another product from the SOM project: The Gap between Public Preferences and Policies on Immigration: A Comparative Examination of the Effect of Politicisation on Policy Congruence in JEMS. In this paper Laura Morales, Jean-Benoit Pilet and I examine the purported gap between (restrictive) public opinion on immigration and (expansive) policies by the elite.
Using data from the SOM project and a range of public opinion polls, we consider the situation across seven countries and 15 years (1995 to 2010). This provides a better insight in what is one of the most salient policy domains in contemporary Europe than was done previously. There is no evidence that strong anti-immigrant parties have anything to do with differences between public opinion and elite policies. Just like what I found in my monograph on political representation, it turns out that salience plays a key role. When negative attitudes in the population are combined with extensive media coverage, we observe high levels of policy congruence.

Women’s Representation in Multi-Ethnic Countries

A new paper by Leonardo Arriola and Martha Johnson examines women’s representation in multi-ethnic countries. They focus on ministerial appointments to executive cabinets in 34 African countries. They find that fewer women are appointed to cabinets in countries where ethnic groups are more politicized. Although my research focuses on representation in national legislatives, it shows that this mechanism seems to be at work more generally. My research argues that the salience of social divisions is relevant here: sometimes gender differences are relatively more important, sometimes ethnic differences are relatively more important.

In fact, when Arriola and Johnson note that there are more appointments for women in countries where there are more women in the legislature, they hint at the above, but never make it explicit. It is encouraging to see research by others corroborating findings, especially if they use different methods and a different focus.

Arriola, Leonardo R., and Martha C. Johnson. 2013. “Ethnic Politics and Women’s Empowerment in Africa: Ministerial Appointments to Executive Cabinets.” American Journal of Political Science. doi:10.1111/ajps.12075.

Ruedin, Didier. 2010. “The Relationship between Levels of Gender and Ethnic Group Representation.” Studies in Ethnicity and Nationalism 10 (2): 92–106. doi:10.1111/j.1754-9469.2010.01066.x.

Ruedin, Didier. 2013. Why Aren’t They There? The Political Representation of Women, Ethnic Groups and Issue Positions in Legislatures. Colchester: ECPR Press. ISBN: 9780955820397