Abstract: In this paper, we examine the social roles of software tools in the English-language Wikipedia, specifically focusing on autonomous editing programs and assisted editing tools. This qualitative research builds on recent research in which we quantitatively demonstrate the growing prevalence of such software in recent years. Using trace ethnography, we show how these often-unofficial technologies have fundamentally transformed the nature of editing and administration in Wikipedia. Specifically, we analyze „vandal fighting‟ as an epistemic process of distributed cognition, highlighting the role of non-human actors in enabling a decentralized activity of collective intelligence. In all, this case shows that software programs are used for more than enforcing policies and standards. These tools enable coordinated yet decentralized action, independent of the specific norms currently in force.

Download the full paper (PDF)

This week, I’m presenting a poster at WikiSym 2009 on “The Social Roles of Bots and Assisted Editing Tools.”  Most of the work is distilled from my thesis.

Abstract: This project investigates various software programs as non-human social actors in Wikipedia, arguing that their influence must not be overlooked in research of the on-line encyclopedia project. Using statistical and archival methods, the roles of assisted editing programs and bots are examined. First, the proportion of edits made by these non-human actors is significantly more than previously described in earlier research. Second, these actors have moved into new spaces, changing not just the practice of article writing and reviewing, but also administrative work.

Download the Poster (PDF)

Download the Extended Abstract (PDF)

And if you are interested in this topic, check out the full paper, The Work of Sustaining Order in Wikipedia: The Banning of a Vandal.

Jimmy Wales speaking at the conference keynote, by GreenReaper, CC BY-SA 3.0

A few months ago, I had the pleasure of presenting at the first (hopefully annual) WikiConference New York, sponsored by the Wikimedia New York City chapter with assistance from Free Culture @ NYU and the Information Law Institute at NYU’s law school. I know that I am atrociously late in writing this post, but I’m not really writing it for the Wikipedians out there. Rather, the WikiConference was an interesting experiment that seemed to apply Wikipedia’s philosophy towards editing to a conference, resulting in what the organizers called a “modified unconference.”

I had never heard of unconferences before, but they are apparently growing increasingly common in tech/programming circles, especially as precursors or followups to traditional conferences. The idea is that in order to keep administratve costs low, you don’t really organize the conference into pre-determined panels, roundtables, and keynotes. Instead, you have a general theme, a good number of open rooms, and a good number of eager participants, who set the topics of individual sessions for themselves and move from room to room on a fluid, ad-hoc basis.  The only rule is the “rule of two feet” – if you don’t like what is going on in the room you are in, leave and find another one.

The conference organizers apparently decided that this was too anarchistic, and instead opted to have a limited number of traditional sessions.  I was on one of the structured sessions, presenting my research on bots and assisted editing tools on the “Quality and Governance” panel.  It was also decided that the “open space”  time was to be segmented into blocks of concurrent sessions.  There was going to be a specific agenda for each of the open space sessions, but they were to be determined at the conference, not before; in addition, the process was to be open to anyone who wanted to propose a session.  While it seemed like an odd way to run a conference (and a bit scary seeing blank space dominate the schedule), it worked incredibly well.

Open space board at WikiConference NYC, by me, CC BY-SA 3.0

We had use of five rooms of various sizes, and one of them was dedicated for refreshments and mingling.  Outside of the largest room (which was used for each day’s opening keynote), there were sheets of paper taped to the wall, creating a table for rooms and timeslots.  After the first day’s opening keynote, sheets of paper, tape, and markers were passed around, and anybody could write something down, tape it to the wall under a timeslot/room combination, and that would be part of the initial schedule.

Given that most of us had never participated in this before, there was a good amount of milling around in front of the schedule wall – five minutes in, nobody had put up a single topic for any timeslot.  Feeling compelled to ake some initiative, I asked someone who was going to be on my panel that afternoon how he felt about a topic on macro-level decision making.  Specifically, I was interested in the approval of flagged revisions – the controversial software feature that would require some edits be approved before going live.  He suggested that I make it broader, and simply write “How do we make decisions?”

The open space wall, by Cary Bass, CC BY-SA 2.5

That seemed like a better and broader topic, so I grabbed some paper and one of the markers, wrote it down in my chicken-scratch handwriting, and taped it to the wall under the first timeslot for the second-biggest room.  Shortly after, three other sheets came up, on quite diverse topics: notability standards, libraries and librarians in Wikipedia, and translation/foreign languages.  Some had even put up sheets for other time slots, touching on nineteen issues that touched on just about every topic in and around Wikipedia.

According to the conventions of open space, the person who put the topic up was expected to start the session on time, say a few words to frame the issue, and then wrap things up at the end.  As the session began, I did just that, telling the room that I had originally thought of this as a discussion about the decision-making around large scale issues like flagged revisions.  However, it is probably good that I was not the moderator, because the room quickly got off the topic of macro-level decision-making and moved into the micro.  We ended up talking extensively about the wide variety of decisions that are made every day – whether to keep or delete a potentially unnotable article, to make an editor into an administrator, and more.

NewYorkBrad asking a question, by Sage Ross, CC BY 3.0

While this was not what I originally envisioned for the session, I was glad that the format had allowed such a swift change.  Had I been delegated to craft a speech, panel, discussion, or roundtable in a traditional conference, I probably would have taken it into a direction that most people did not want to go – of the twenty-something open sessions in the two days, nobody proposed a session on flagged revisions.  Unconferences are supposed to be directed by and for the benefit of the participants, and this was certainly the case.  In any case, the discussion on decision-making went rather well, although a moderator did end up emerging because our session ended up being one of the most popular open sessions, filling up the 75-person classroom.

Yet like in Wikipedia, the unconference didn’t simply devolve into a mass populist mob, reaching for the lowest common denominator.  The fact that we had multiple rooms, a couple of them small conference rooms, meant that less popular topics got their fair share of space.  One open session that I found interesting was on systemic bias – the fact that Wikipedia tends to implicitly favor certain topics, styles, or stances because of the demographic makeup of its contributors.  This tends to not be that popular of a topic, and only a handful of us showed up to discuss this (in my opinion) quite important issue.  However, this resulted in a very thought-provoking discussion among the five of us – that’s about three percent of the conference – who felt a need to identify, theorize, and fix Wikipedia’s systemic biases.

Open session: editing an article for the Wikipedia Signpost

Taken by GreenReaper, CC BY-SA 3.0

Another strength of the open unconference is its radical flexibility.  On the second day, the question/answer session in opening keynote speech turned into a strong debate among a few of the participants.  Because this stops others from asking questions, the typical move at conferences is to stop the debate and pledge to continue it later.  I’ve seen it happen at many conferences, but due to the rigid structure of most conferences, the continuing discussion rarely happens.  Yet in this case, the keynote speech was to be followed by open space sessions.  Realizing that there was an empty slot avaliable in one of the small rooms, the debate that emerged in the keynote Q/A was instantly given its own session.

We also had sets of lightning talks, which were presented in a keynote style.  For those of you who don’t know, lighting talks are short 3-7 minute presentations that anyone can give on the fly.  While lightning talks are held in many conferences I have been to, they tend to be pushed to the background.  Like poster sessions, lightning talks usually take place during established break periods (like lunch), or during other sessions.  This means that the only people who view them are other lightning talkers.  In our case, the lightning talks were after the lunch hour and when no other sessions were being held.  This way, I feel that the presenters got a much broader audience.

Andrew Gradman giving a lightning talk, by Sage Ross, CC BY-SA 3.0

In all, I think that the open unconference was a great success.  However, I don’t think that the “open space” model is adequate on its own – which is why I was glad that there were a limited number of keynotes and pre-arranged panels.  I was on one of the panels (discussing “Quality and Governance”), and got to give a standard 15 minute structured conference presentation, as did my fellow panelists.  I feel that that format is valuble, because I don’t think my research findings on bots and assisted editing tools (or any research findings, for that matter) could have been presented in an open space session or a lightning talk.  The two kinds of sessions are meant to facilitate two different kinds of activities: structured panels and keynotes frame discussions, while the open spaces let participants take it in any way they desire.  For example, I was very excited when the last open session of the conference turned into a user-driven showcase of assisted editing tools – completely unprovoked by myself, I promise.  Another session (one of my favorite) was a workshop in which all the participants worked collectively on writing a news article about the conference for Wikipedia’s community newspaper, the Wikipedia Signpost.

I’m not sure if these kinds of activities would have happened at a more traditional conference – and if they did, they would have probably required a lot more planning. One thing is certain though: the cost of the conference, which was the main reason for the unconference movement, was practically nil. It was completely run by volunteers, and only expenses were refreshments and food.

We throw around the words “collective intelligence” and “wisdom of the crowds” quite a bit to describe “Web 2.0″ sites like Wikipedia, but we hardly define what we mean when we use any of those terms, which is why they largely remain scare-quoted. Because of this, the door has been left wide open for scientists and journalistic defenders of science to critique Wikipedia and other social media sites as being relativist, collectivist mobs who can do no more than aggregate the baseline opinion of what the masses perceive to be Truth. While Wikipedia does have epistemic standards, the open question is how such an epistemology can be operationalized and enforced. To answer such a question, I examine Wikipedia in light of a distinction between an infrastructure of knowing (everything required to evaluate a statement as true/false) and an infrastructure of knowledge production (everything required to bring forth new statements with claims to truth/falsity). While the Wikipedian epistemology on the encyclopedic level is purely evaluative, refusing to publish original research and instead relying on reliable sources, this process is made possible by a non-encyclopedic form of knowledge production.

In short, in order for there to exist an infrastructure of knowing such that the evaluation of encyclopedia articles becomes possible, there must exist an infrastructure of knowledge production to generate and evaluate claims regarding the acts of editing. These include statements like “this edit is vandalism and needs to be reverted” or “this user is disruptive and needs to be blocked” – which require their own epistemic order for evaluation. Taking a cue from laboratory studies of scientific practice, I detail the way in which epistemic standards are “black boxed” into material technologies. In the same way that a mass spectrometer is the reification of dozens of now-unproblematic theories from physics, chemistry, and mathematics, so do various technological programs used by self-described “vandal fighters” reify Wikipedia’s epistemic standards. Similarly, in the same way that various technologies had to be developed to allow experimental science to trump philosophical reasoning (like laboratory reports, which made experimental findings circulatable), so have various technologies been developed that make Wikipedia’s mechanisms of epistemic verification and enforcement possible.

By detailing all the human and non-human actors at work in the banning of a vandal, I show how a group of seemingly-disconnected editors contributed to a process of knowledge production necessary for the enforcement of epistemic standards. In this way, collective intelligence was made possible in Wikipedia, but not because of a mystical or anarchistic wisdom of crowds. Instead, these encyclopedic epistemic standards were able to be enforced because various human and non-human actors were constantly working to hold together an infrastructure of non-encyclopedic knowledge production.

Link: Working Within Wikipedia: Infrastructures of Knowing and Knowledge Production (PDF, 901 KB)