The Samovar

Authority and Competence

The view that the people in charge know what they’re doing is implicitly prevalent. Although many people would say that the people in charge were incompetent, in fact there seems to be a widespread implicit assumption that they either do know what they’re doing or that someone else would know what they’re doing better. Underlying this is the assumption that it’s possible to know what you’re doing. In politics and economics, none of these are true – there is very little understanding of what is happening, what the effects of various actions will be, or what we should do. Understanding this is important, because at the moment there is a lack of critical thinking in politics. Although there is much criticism, of course, it usually fails to get at the root causes of problems and so the mass of critical feeling fails to achieve anything, and is wasted in irrelevancies.

As an example of this, the debate that was had a few years ago about extending the period that the police can hold people without charging them beyond 28 days in terrorism cases. Assumptions of competence pervaded this debate in many ways. For some, it was enough to note that the politicians thought that an extension was necessary to protect us. For others, that wasn’t enough, but the fact that the police thought that an extension was necessary for them to be able to protect us was enough. This was also the basis for a substantial amount of the debate in parliament. It was assumed that the fact that the police said they needed the extra time counted for something. Not everyone agreed that just because the police thought they needed it meant that they should get it, but it was universally agreed that their opinion counted purely on the basis that it was their opinion. Their perceived authority and presumed competence gave their opinion weight in and of itself.

However, very few people questioned why the police had come to their opinion, and what the evidence for it was. It turns out that one of the major claims was that in previous terrorist cases they had used more and more time, up to 26 days I think it was. The argument was that since they had used this much, it was probably the case that if the limit had been higher they could have profitably used more, and that future cases were likely to need even more. They were already at the limit, and this was holding them back.

This argument is terribly weak in many, many ways, but came under hardly any criticism at all (indeed, very few people even knew that this was the argument). But there’s no reason why it had to be like this – the weaknesses of this argument, and the counter-arguments against it, are not so complicated that most people wouldn’t be able to understand them. Rather, it was that the whole process of questioning the argument was made unnecessary by the fact that most people were willing to go along with the opinions of those in positions of authority based on their presumed competence. A more critically engaged society would be better able to protect itself against manipulation by those in power. We have all the necessary democratic mechanisms, but they count for nothing if we hand over our critical thinking to those in positions of authority (and that includes journalists).

In order to achieve a more critically engaged society, we need to understand competence better – where does it come from? What sort of things are we able to be competent about? How can we recognise it? We also need to dispel myths and misunderstandings about competence, which are widespread.

If we describe competence as being about having knowledge, then we can split it down into an explicit and tacit component. Explicit knowledge is something you can write down and tell others about, things like 2+2=4, the capital of France is Paris, etc. Tacit knowledge is everything that can’t be put into words, but that is still valuable. I can’t put into words what it is I’m doing when I solve a mathematical problem, but there’s definitely something I know that most other people don’t that makes me able to do them and them not. Tacit knowledge is built up from experience, thousands of particular cases, attempted and failed solutions of problems, etc.

The existence of tacit knowledge is very significant and certainly shouldn’t be underestimated. Indeed, this is a good argument that sometimes we do need to rely on the authoritative judgments of others (experts). But we shouldn’t make the mistake of just presuming that people do indeed have tacit knowledge about their area of work. First of all, you don’t get tacit knowledge about something just by doing it often, and secondly it is often easy to think that tacit knowledge is more general than it really is.

In the example of the debate above, the police may well have sincerely believed that they had the requisite experience and knowledge to make their judgment that they needed the extension to protect us from terrorism. But they didn’t. For a start, they haven’t dealt with nearly enough cases to get tacit knowledge that is worth much. Since the end of the IRA bombing campaigns, which were rather different to modern terrorism, there has only been one successful terrorist incident and a handful of failures. The police have had no great successes or failures in these matters – they have caught some people, but these have largely been fantasists who had no real ability to do the things they wanted to do. The fact that terrorism has been as little a problem as it has been is a consequence of the fact that there have been very few competent attempts at it, few individuals involved, and little will to carry it through. All the police know, then, is what they’ve done in the past, and what happened in consequence. Even on that basis alone, there’s very little to go on as chance and circumstance probably have as much to do with that as anything else. But further than that, there’s no basis for them to be able to know what would happen if they had got the extension they were looking for. They have no model, explicit or tacit, of the world that would allow them to make such a prediction, and no experience to go on. Almost certainly, what they actually had was the conviction that it couldn’t make it worse, and that it would make their lives easier.

It is vital, if we take someone’s judgment as an expert or authority, that we analyse what basis they might have for making the claims that they do, whether or not they could be in possession of explicit or tacit knowledge that would justify it, and where that knowledge could have come from. This is far from being a complete recipe for dealing with issues of authority, expertise and tacit knowledge, but even a better recognition of the importance of these issues would be a big step forwards.

One thing that needs to be addressed if we’re to achieve a more realistic understanding of these sorts of issues is cultural representations of expertise and competence – the myths that are portrayed in books, films, TV shows, newspapers and everyday discourse that support them. Films and TV shows typically portray hyper-competent individuals bursting with impeccable tacit knowledge. The reason for this on the one hand is obvious – a story about heroic, talented individuals is more interesting. Sometimes it’s obvious that people in these things act beyond human abilities, such as the hacker who can break into any computer system in the world in only 5 minutes. But other times the portrayal is more subtle and insidious, such as in The West Wing, a notionally realistic show about the US presidential staff in the White House. The technical competence of the heroes is constantly portrayed as a virtue in and of itself, and the show suggests implicitly that they have the right to rule by virtue alone of their greater ability to do so. Very few TV shows portray anything like reality in this regard, with the most notable exception being The Wire, one of the best things ever to be shown on TV, proving that it certainly is possible to do it (and be reasonably popular).

On a final note, these considerations apply very much to areas of life outside politics and economics. For example, some people might be shocked about the standards of evidence that count in science. But that’s another story.

Comments Off on Authority and Competence

Public knowledge
December 8, 2007, 2:56 am
Filed under: Academia, Anarchism, Internet, Manifesto | Tags: , , , , , ,

Wikipedia has a very bad reputation for accuracy, and recently it’s been getting a bit of a trashing for its internal politics. Despite this, millions of people continue to use it, and I think it’s easy to see why.

Despite its problems, Wikipedia is a better resource for the public dissemination of knowledge than almost anything else out there. It can be misused by blindly relying on what is included there, but this isn’t a reason to attack Wikipedia. You just have to approach it with the right attitude: a Wikipedia article is a starting point for further research, not an end point. It’s a means for discovering new information as much as a repository of information. We shouldn’t underestimate the importance of this. Discovering that certain knowledge exists is itself a very difficult and important thing to do.

Wikipedia articles are like a quick and dirty map of a knowledge space. They give you a rough idea of what something is about, even if the details may be wrong, and they suggest where you could go to find out more. As a sample, I picked the Wikipedia entry on dynamical systems more or less at random. As well as a decent length article, it has a bibliography of 17 books, including 13 serious academic books at varying levels and 4 popular mathematics books, and 22 internet links, including complete books available online, tutorials and the web pages of relevant research groups.

The nature of knowledge is that it is constantly expanding, and at the moment it is doing so at an incredible rate. Traditional repositories of knowledge like textbooks and encyclopaedias find it difficult to keep up, and are often years if not decades out of date. Wikipedia may be less authoritative than these, but it is often only days after a new discovery is made that a detailed write up is available on wikipedia with links to the original research paper for those who need more accurate information. Textbooks and printed encyclopaedias cannot compete with this.

It is interesting that much of the criticism of Wikipedia comes from those with a vested interest in doing so. The Encyclopaedia Britannica has criticised Wikipedia, and it’s obvious enough why they would do so because they’re in direct competition. But Wikipedia also gets a very bad treatment from the press, by people who are not directly in competition with it. The coverage from The Register (article linked to above) is a case in point. Their stories about Wikipedia are hostile almost to the point of absurdity.

So why is this? My feeling is that it’s because the model of public knowledge espoused by Wikipedia is a direct challenge to the elitist model of knowledge of journalists, and the reason they attack it so strongly is the same as the reason they attack blogs so strongly. Their whole reason for existence is based on the idea that they are providing something through their expertise and knowledge that cannot be obtained elsewhere (for free). If people could just directly access knowledge without going through them, why would be bother doing so? They feel their existence is threatened.

And they are right to feel that way. Wikipedia articles on new scientific discoveries are often much better researched than the write ups in newspapers, and Wikipedia authors often seem to have a better understanding of the discovery in question than the science writers in the newspaper. This shouldn’t be surprising: a newspaper typically only has one or two science writers (and they’re often failed scientists or those with only an undergraduate degree in science), whereas a Wikipedia article could be directly written by someone in that field or even by the original authors themselves. A newspaper article will never cite it’s sources because there isn’t enough space, but most Wikipedia articles do so (and those that don’t are conspicuously flagged).

Similarly, blogs often provide a much broader and more interesting range of political analysis than you find in a newspaper. One of the criticisms that traditional media such as newspapers level at blogs is that they don’t do investigative journalism, but in fact the heavy competition and diminishing revenues of traditional media mean that they are doing less investigative journalism than ever. When the US invaded Iraq, the traditional media were telling us how great everything was because their information was all coming through the filter of the military forces. On the other hand, Iraqi blogs gave a much broader picture.

Getting back to Wikipedia, the journalists and others would be right to criticise Wikipedia if the point of it was to provide an authoritative reference point for factual information. But this really shouldn’t be the point, and the criticism is fundamentally based on an inaccurate picture of the nature of knowledge. Truly authoritative knowledge is very rare. Anyone relying on a single source, however authoritative that source is, is making a serious error. Wikipedia shouldn’t be relied on in this way, but neither should an Encyclopaedia Brittanica entry or even a scientific textbook (and certainly not a newspaper article!). The critics cannot understand this point, or cannot concede it, because their view of themselves is that they are this sort of authority, and so they cannot comprehend the suggestion that this sort of authority is not needed.

So in defending Wikipedia from its critics, I am not – as they might imagine – denying the need for expertise, but attacking the false and elitist nature of expertise that they represent, and defending a view of knowledge that is inherently diverse.

As a postscript, a very interesting project is Scholarpedia. It is inspired by Wikipedia, but has a different balance of openness and expertise by essentially restricting editing rights to academics, with the level of control increasing with scientific status. As the front page of Scholarpedia states, “The approach of Scholarpedia does not compete with, but rather complements that of Wikipedia” (my emphasis). Scholarpedia is a recognition that both expertise and the dynamic, open approach of Wikipedia are important. At the moment, Scholarpedia is restricted to articles about theoretical and computational neuroscience, some mathematical fields, and astrophysics, but it will grow.