The recent sacking of David Nutt – formerly head of the Advisory Council on the Misuse of Drugs (ACMD) – for giving scientific advice that showed the stupidity of the government’s drugs policy, suggests taking a look at the role of expert advice in policy.

The problem is that the government can have an unstated policy of accepting expert advice when it suits them and rejecting it when it doesn’t. Such a policy is ideal for the government, because if the advice fits what they wanted to do anyway, they can claim that they are supported by evidence, and if it contradicts them they can in most cases easily shrug it off by claiming (correctly) that the point of expert advice is not that it should define policy, but that it should be taken into account as part of wider considerations, and that in this case, blah blah… The policy is equivalent in outcome to having no expert advice, but in some cases looks better. Alan Johnson’s statement in his letter to Nutt was extraordinary in tacitly recognising this:

I cannot have public confusion between scientific advice and policy and have therefore lost confidence in your ability to advise me as chair of the ACMD.

It is understandable then that other members of the ACMD are resigning, although it is not entirely clear how principled this stance is when everyone on that council must have known beforehand that they were helping to legitimate highly irrational policies. (I don’t want to be too critical though, maybe the strategy of working within a faulty system can do some good.)

So where does this leave the issue of expert advice? Can it play a useful role and if so, how? One possible way out of the problem above would be for the government to create advisory groups and commit itself to following their advice whatever it might be. There are various problems with this though. Firstly, it is subject to manipulation by selection of the members of the group. Secondly, it’s not clear that it would even work – Tony Blair stated the reason to go to war with Iraq “must be according to the United Nations mandate on Weapons of Mass Destruction”, but changed his mind when that mandate disappeared.

But a third and deeper problem with this and any other similar scheme is that it conceals the true nature of politics, and supports the false idea that government can be a purely technical exercise in doing whatever works. Politics is actually about conflicts of interests of different groups and classes. Portraying political issues as technical ones works to hide these truly political aspects. Governments and opposition parties are very happy to do this because they are both largely supporting the interests of the same classes/groups – typically the wealthier ones. This shouldn’t be surprising because the decision making part of the government and state largely consists of, is staffed by and supported by people in these classes.

I don’t want to suggest that there are not technical considerations in policy making, nor that expertise is irrelevant. In the case of drugs policy, for example, the evidence is overwhelming that tobacco and alcohol are more dangerous than cannabis and many other illegal drugs, making a mockery of government policy. However, I do doubt that an institutional arrangement can be devised which allows for a useful and non-political injection of expert advice into decision making. I would suggest instead that experts should be entirely independent of government. A well informed and scientifically literate press – something that is very far from what we have today – would be hugely preferable to any number of advisory councils selected by and working for the government. This would allow an injection of expertise into an explicitly political process, rather than supporting a fictional non-political one.

The view that the people in charge know what they’re doing is implicitly prevalent. Although many people would say that the people in charge were incompetent, in fact there seems to be a widespread implicit assumption that they either do know what they’re doing or that someone else would know what they’re doing better. Underlying this is the assumption that it’s possible to know what you’re doing. In politics and economics, none of these are true – there is very little understanding of what is happening, what the effects of various actions will be, or what we should do. Understanding this is important, because at the moment there is a lack of critical thinking in politics. Although there is much criticism, of course, it usually fails to get at the root causes of problems and so the mass of critical feeling fails to achieve anything, and is wasted in irrelevancies.

As an example of this, the debate that was had a few years ago about extending the period that the police can hold people without charging them beyond 28 days in terrorism cases. Assumptions of competence pervaded this debate in many ways. For some, it was enough to note that the politicians thought that an extension was necessary to protect us. For others, that wasn’t enough, but the fact that the police thought that an extension was necessary for them to be able to protect us was enough. This was also the basis for a substantial amount of the debate in parliament. It was assumed that the fact that the police said they needed the extra time counted for something. Not everyone agreed that just because the police thought they needed it meant that they should get it, but it was universally agreed that their opinion counted purely on the basis that it was their opinion. Their perceived authority and presumed competence gave their opinion weight in and of itself.

However, very few people questioned why the police had come to their opinion, and what the evidence for it was. It turns out that one of the major claims was that in previous terrorist cases they had used more and more time, up to 26 days I think it was. The argument was that since they had used this much, it was probably the case that if the limit had been higher they could have profitably used more, and that future cases were likely to need even more. They were already at the limit, and this was holding them back.

This argument is terribly weak in many, many ways, but came under hardly any criticism at all (indeed, very few people even knew that this was the argument). But there’s no reason why it had to be like this – the weaknesses of this argument, and the counter-arguments against it, are not so complicated that most people wouldn’t be able to understand them. Rather, it was that the whole process of questioning the argument was made unnecessary by the fact that most people were willing to go along with the opinions of those in positions of authority based on their presumed competence. A more critically engaged society would be better able to protect itself against manipulation by those in power. We have all the necessary democratic mechanisms, but they count for nothing if we hand over our critical thinking to those in positions of authority (and that includes journalists).

In order to achieve a more critically engaged society, we need to understand competence better – where does it come from? What sort of things are we able to be competent about? How can we recognise it? We also need to dispel myths and misunderstandings about competence, which are widespread.

If we describe competence as being about having knowledge, then we can split it down into an explicit and tacit component. Explicit knowledge is something you can write down and tell others about, things like 2+2=4, the capital of France is Paris, etc. Tacit knowledge is everything that can’t be put into words, but that is still valuable. I can’t put into words what it is I’m doing when I solve a mathematical problem, but there’s definitely something I know that most other people don’t that makes me able to do them and them not. Tacit knowledge is built up from experience, thousands of particular cases, attempted and failed solutions of problems, etc.

The existence of tacit knowledge is very significant and certainly shouldn’t be underestimated. Indeed, this is a good argument that sometimes we do need to rely on the authoritative judgments of others (experts). But we shouldn’t make the mistake of just presuming that people do indeed have tacit knowledge about their area of work. First of all, you don’t get tacit knowledge about something just by doing it often, and secondly it is often easy to think that tacit knowledge is more general than it really is.

In the example of the debate above, the police may well have sincerely believed that they had the requisite experience and knowledge to make their judgment that they needed the extension to protect us from terrorism. But they didn’t. For a start, they haven’t dealt with nearly enough cases to get tacit knowledge that is worth much. Since the end of the IRA bombing campaigns, which were rather different to modern terrorism, there has only been one successful terrorist incident and a handful of failures. The police have had no great successes or failures in these matters – they have caught some people, but these have largely been fantasists who had no real ability to do the things they wanted to do. The fact that terrorism has been as little a problem as it has been is a consequence of the fact that there have been very few competent attempts at it, few individuals involved, and little will to carry it through. All the police know, then, is what they’ve done in the past, and what happened in consequence. Even on that basis alone, there’s very little to go on as chance and circumstance probably have as much to do with that as anything else. But further than that, there’s no basis for them to be able to know what would happen if they had got the extension they were looking for. They have no model, explicit or tacit, of the world that would allow them to make such a prediction, and no experience to go on. Almost certainly, what they actually had was the conviction that it couldn’t make it worse, and that it would make their lives easier.

It is vital, if we take someone’s judgment as an expert or authority, that we analyse what basis they might have for making the claims that they do, whether or not they could be in possession of explicit or tacit knowledge that would justify it, and where that knowledge could have come from. This is far from being a complete recipe for dealing with issues of authority, expertise and tacit knowledge, but even a better recognition of the importance of these issues would be a big step forwards.

One thing that needs to be addressed if we’re to achieve a more realistic understanding of these sorts of issues is cultural representations of expertise and competence – the myths that are portrayed in books, films, TV shows, newspapers and everyday discourse that support them. Films and TV shows typically portray hyper-competent individuals bursting with impeccable tacit knowledge. The reason for this on the one hand is obvious – a story about heroic, talented individuals is more interesting. Sometimes it’s obvious that people in these things act beyond human abilities, such as the hacker who can break into any computer system in the world in only 5 minutes. But other times the portrayal is more subtle and insidious, such as in The West Wing, a notionally realistic show about the US presidential staff in the White House. The technical competence of the heroes is constantly portrayed as a virtue in and of itself, and the show suggests implicitly that they have the right to rule by virtue alone of their greater ability to do so. Very few TV shows portray anything like reality in this regard, with the most notable exception being The Wire, one of the best things ever to be shown on TV, proving that it certainly is possible to do it (and be reasonably popular).

On a final note, these considerations apply very much to areas of life outside politics and economics. For example, some people might be shocked about the standards of evidence that count in science. But that’s another story.

I’ve made a minor change to my comments policy for this blog in the light of the growing number of spam comments making it past the spam filters which I have to delete by hand.

There has been a lot of discussion of a new report by the Joseph Rowntree Foundation on attitudes towards tackling economic inequality (at Directionless Bones, Left Luggage, Sunder Katwala on CiF, Don Paskini at Liberal Conspiracy, and David Osler). Quoting Alderson at Directionless Bones, one of the key findings of the report is:

People didn’t seem to endorse the idea of ‘equality’ as a general principle as much as they endorsed ‘fairness’.

This is a point that several of the posts linked to above considered, and there has been a feeling that the left needs to find a new way to promote their view of the world to people (which traditionally is based on equality).

I find this interesting because for the last few years I’ve been coming to the view that the case for a left-wing politics should be based rather on the principles of fairness and freedom than on equality. Equality is important and essential, but I think it’s a consequence of fairness and freedom. I’ve argued this in much more detail in an earlier entry on capitalism. Essentially, a very unequal society will, in practice, also be necessarily an unfair one.

The report has caused quite a lot of distress because it showed that people are not against what they term ‘fair inequality’ – but I don’t think the left should be dispairing over this. The report also clearly showed that people think there are substantial levels of unfair inequality, and that this is a bad thing. This suggests to me that most people basically get and agree with the final point of the previous paragraph – high levels of inequality lead to an unfair society.

I suggest then that what the left needs to do is to push this analysis further and address the misconceptions that the JRF report showed that people have. Most people substantially underestimate the level of inequality that actually exists and overestimate the level of social mobility. Changing perceptions of these is difficult, but could make a significant difference.

As a final point, the report appears to be more of a blow to a state-centric form of socialism where equality is considered more important than fairness and freedom, and much less of a blow to an anarchist form of socialism which takes freedom and fairness to be fundamental. This is important and suggests the left should be considering a change of direction towards anarchist conceptions – and thankfully much of the left does, slowly, seem to be doing this (even if they don’t call it anarchism). In particular, the point about support for ‘fair inequality’ is very interesting with respect to the remuneration mechanism of parecon (which I written a few things about on this blog). Parecon allows for a certain amount of precisely ‘fair inequality’ – that is, inequality that comes from a choice to work harder or at more onerous labour. It is fair because anyone can make that choice (whereas not everyone can choose to be a doctor, banker, etc.). On the other hand, it absolutely rejects unfair inequality. As such, it seems that many people’s fundamental views of what society should be like resonate more with a pareconish or anarchist conception than a state-centric socialist one.

Shame about the title though. I think the words “Yes we can” should probably be banned. Anyway, here’s Robin Hahnel on “Change how the world works? Yes we can“:

Until capitalism is replaced, we want the tail to stop wagging the dog. Finance should serve the real economy instead of the other way around. If the financial sector improves the efficiency of the real economy, it is helpful. But if it misdirects investment resources to where they are less productive, it reduces production in the real economy by obstructing the flow of credit altogether. Then it is failing to accomplish its only social purpose. Jobs producing useful goods and services, and investments which help us to produce what we need with less human toil and less strain on the environment, are what count. Increases in the profit rates and stock prices of financial corporations count for nothing when they fail to correspond to real increases in productivity, as has too often been the case.

We have offered several positive alternatives to capital liberalization and to the governing structures and policies of the International Monetary Fund (IMF) and the World Bank, such as capital controls and a Tobin tax to protect smaller economies from volatile speculative flows. We have made suggestions on how national governments can restore competent regulation of their traditional financial sectors, and stressed the urgency of extending regulation to cover new financial institutions which were allowed to grow outside existing regulatory structures.

This will be of little interest to people who regularly read my blog, but might be of some interest to people who find their way here by the power of Google.

The standard way to compute fractals like the Mandelbrot set using Python and numpy is to use vectorisation and do the operations on a whole set of points. The problem is that this is slower than it needs to be because you keep doing computations on points that have already escaped. This can be avoided though, and the version below is about 3x faster than the standard way of doing it with numpy.

The trick is to create a new array at each iteration that stores only the points which haven’t yet escaped. The slight complication is that if you do this you need to keep track of the x, y coordinates of each of the points as well as the values of the iterate z. The same trick can be applied to many types of fractals and makes Python and numpy almost as good as C++ for mathematical exploration of fractals.

I’ve included the code below, both with and without explanatory comments. This 400×400 image below using 100 iterations took 1.1s to compute on my 1.8GHz laptop:

Uncommented version:

def mandel(n, m, itermax, xmin, xmax, ymin, ymax):
    ix, iy = mgrid[0:n, 0:m]
    x = linspace(xmin, xmax, n)[ix]
    y = linspace(ymin, ymax, m)[iy]
    c = x+complex(0,1)*y
    del x, y
    img = zeros(c.shape, dtype=int)
    ix.shape = n*m
    iy.shape = n*m
    c.shape = n*m
    z = copy(c)
    for i in xrange(itermax):
        if not len(z): break
        multiply(z, z, z)
        add(z, c, z)
        rem = abs(z)>2.0
        img[ix[rem], iy[rem]] = i+1
        rem = -rem
        z = z[rem]
        ix, iy = ix[rem], iy[rem]
        c = c[rem]
    return img

Commented version:

from numpy import *

def mandel(n, m, itermax, xmin, xmax, ymin, ymax):
    '''
    Fast mandelbrot computation using numpy.

    (n, m) are the output image dimensions
    itermax is the maximum number of iterations to do
    xmin, xmax, ymin, ymax specify the region of the
    set to compute.
    '''
    # The point of ix and iy is that they are 2D arrays
    # giving the x-coord and y-coord at each point in
    # the array. The reason for doing this will become
    # clear below...
    ix, iy = mgrid[0:n, 0:m]
    # Now x and y are the x-values and y-values at each
    # point in the array, linspace(start, end, n)
    # is an array of n linearly spaced points between
    # start and end, and we then index this array using
    # numpy fancy indexing. If A is an array and I is
    # an array of indices, then A[I] has the same shape
    # as I and at each place i in I has the value A[i].
    x = linspace(xmin, xmax, n)[ix]
    y = linspace(ymin, ymax, m)[iy]
    # c is the complex number with the given x, y coords
    c = x+complex(0,1)*y
    del x, y # save a bit of memory, we only need z
    # the output image coloured according to the number
    # of iterations it takes to get to the boundary
    # abs(z)>2
    img = zeros(c.shape, dtype=int)
    # Here is where the improvement over the standard
    # algorithm for drawing fractals in numpy comes in.
    # We flatten all the arrays ix, iy and c. This
    # flattening doesn't use any more memory because
    # we are just changing the shape of the array, the
    # data in memory stays the same. It also affects
    # each array in the same way, so that index i in
    # array c has x, y coords ix[i], iy[i]. The way the
    # algorithm works is that whenever abs(z)>2 we
    # remove the corresponding index from each of the
    # arrays ix, iy and c. Since we do the same thing
    # to each array, the correspondence between c and
    # the x, y coords stored in ix and iy is kept.
    ix.shape = n*m
    iy.shape = n*m
    c.shape = n*m
    # we iterate z->z^2+c with z starting at 0, but the
    # first iteration makes z=c so we just start there.
    # We need to copy c because otherwise the operation
    # z->z^2 will send c->c^2.
    z = copy(c)
    for i in xrange(itermax):
        if not len(z): break # all points have escaped
        # equivalent to z = z*z+c but quicker and uses
        # less memory
        multiply(z, z, z)
        add(z, c, z)
        # these are the points that have escaped
        rem = abs(z)>2.0
        # colour them with the iteration number, we
        # add one so that points which haven't
        # escaped have 0 as their iteration number,
        # this is why we keep the arrays ix and iy
        # because we need to know which point in img
        # to colour
        img[ix[rem], iy[rem]] = i+1
        # -rem is the array of points which haven't
        # escaped, in numpy -A for a boolean array A
        # is the NOT operation.
        rem = -rem
        # So we select out the points in
        # z, ix, iy and c which are still to be
        # iterated on in the next step
        z = z[rem]
        ix, iy = ix[rem], iy[rem]
        c = c[rem]
    return img

if __name__=='__main__':
    from pylab import *
    import time
    start = time.time()
    I = mandel(400, 400, 100, -2, .5, -1.25, 1.25)
    print 'Time taken:', time.time()-start
    I[I==0] = 101
    img = imshow(I.T, origin='lower left')
    img.write_png('mandel.png', noscale=True)
    show()

George Monbiot has an interesting article linking capitalism and privatisation with growing prison populations:

This revolting trade in human lives creates a permanent incentive to lock people up; not because prison works; not because it makes us safer, but because it makes money. Privatisation appears to have locked this country into mass imprisonment.

It’s not clear to me that this is enough to explain the whole problem, but it’s worth considering.

Alderson has an interesting piece on religion over at Directionless Bones:

[Alderson's view] also implies a certain set of priorities, that changing people’s lives is more important than changing their minds (though obviously not unrelated), and that often religion will persist regardless of rational arguments if the conditions that produce it persist.

The Mathematics Genealogy Project has a huge database of mathematicians, showing who was supervised by whom, and what students everyone had. If you’re a mathematician, you can use this to trace back who your mathematical ancestors were and it can be quite fun. Below is a chart I made of my own mathematical genealogy. It’s nice to see exciting names from the history of mathematics and science there, such as Poisson, Laplace, Lagrange, d’Alembert, Euler, the Bernoullis, Leibniz,  and Huygens (I stopped at that point). The dates are when they finished their doctorate, or if they didn’t do one, when they lived.

Reg article reporting on Nigel Inkster, former Assistant Chief of MI6:

There are limits to what we can sensibly aspire to…

Efforts to establish a global repository of counterterrorist information are unlikely ever to succeed. We need to be wary of rebuilding our world to deal with just one problem, one which might not be by any means the most serious we face.

…

We need to keep terrorism in some kind of context, for example, every year in the UK, more people die in road accidents than have been killed by terrorists in all of recorded history.

…

We should keep our nerve and our faith in our own values. Our own behaviour – especially with respect to the rule of law – is very important.