He was one of life’s larger personalities. ‘A big guy with a booming personality’ was one description. Born in 1927 to an army family, educated at a Wiltshire public school and Oxford University, he joined the Royal Marines at the end of the second world war and after a brief stint at a football pools company was recruited to the Metropolitan police. A fortnight later, Conrad Hepworth Dixon was called to an austere police section house in Beak Street in London’s Soho, stripped naked and ushered into a small room. ‘A man in white coat came in and stared at his lower half, examined his feet, and went wordlessly away. Shortly afterwards, Dixon was transferred to Special Branch,’ records his obituary. The branch only selected those it considered to be the finest specimens. In its view, it was the Met’s elite, revered and feared in equal measure, and Dixon was one of its most experimental leaders. It was Dixon, a chief inspector, who first had the ingenious idea of turning police constables into pretend political activists. The story starts with him.

It was 1968, a rather iconic year for protest. It is perhaps difficult nowadays to understand just how turbulent the late 1960s really were. Across the globe, governments were convulsing. Workers and students were taking to the streets in Paris, Prague, Berlin, Madrid, Mexico City and Chicago in spontaneous bursts of civil unrest. They had similarities and differences; they did not all have the same philosophy or aim, but they were loosely progressive or left-wing. They were united by their opposition to America’s war in Vietnam. And they shared what one historian called ‘the common spirit of youthful rebellion’.

In Britain, the establishment trembled as government ministers and police chiefs grew worried that protesters had the power to tear down Britain’s political and economic system. A revolutionary mood was in the air; many young people hoped there was more than a whiff of actual revolution too. They were enjoying the benefits of a good education and the prospect of a good job and they wanted to do things differently to their parents’ generation. They were more idealistic, free-spirited; they felt the world was theirs to shape. They rebelled against a society they saw as intransigent, drab and paternalistic. They distrusted authority – politicians, college administrations and police. It was a time of radical change in the arts, music, film and fashion. It felt like the old certainties were giving way to a new world. For the young, protest was the exciting engine of this change. For the authorities, it was a dangerous and very real risk to the established order.

A mood of panic gripped Whitehall as the protests over the Vietnam War intensified in 1967 and 1968. There were frequent marches in London, but one in particular shook the government: the Grosvenor Square demonstration of March 1968. Following a large rally in Trafalgar Square, thousands marched on the US embassy, which was surrounded by police. There was a standoff as the protesters refused to back down and mounted police rode into the crowd. Demonstrators broke through police ranks and streamed onto the lawn of the embassy, tearing up barriers and uprooting sections of the fence. More than 200 were arrested during a prolonged battle as stones, firecrackers and smoke bombs were hurled at the building.

Public disorder as ferocious as this had not been seen on the streets of London for many years. For some of the younger generation, it was a totally alien sight – and intoxicating. Black and white television pictures showing the police struggling to control the demonstrators, and punching and kicking them, shocked the country. One MP later complained about the ‘violent use of police horses who charged into the crowd even after they had cleared the street in front of the embassy’. In Scotland Yard, there was alarm – a sense that senior officers had been caught on the back foot.

That day was a watershed moment. When the protesters started to organise a second large demonstration scheduled for October that year, politicians and police were determined to clamp down. Two months before the scheduled march, Special Branch was sending weekly reports to the Home Office, chronicling how the protesters were preparing for the event. The dispatches reveal deep anxiety in police about what some feared was an impending insurrection. In one update, Dixon wrote that ‘a group of about 50 anarchists from Liverpool’ intended to attack an army recruiting office when the march snaked through Whitehall. Around 160 demonstrators from Glasgow, he reported, were ‘being advised to wear crash helmets and urged to carry ball-bearings, fireworks, hat pins and banner poles for use as weapons’.

One informant claimed that protesters would tie fishing line between parking meters to trip up police, while anarchists were reported to be devising a plan to loot shops, restaurants and travel agencies along Oxford Street. The list of buildings police believed were under threat was continually expanding and included the Bank of England, the Ministry of Defence, Home Office, the Hilton Hotel and even the Playboy Club. Bomb disposal experts were mobilised in case explosives were found. Ministers were so worried that they considered deploying soldiers – a suggestion rejected by senior officers at the Met who insisted they could control the streets.

Meanwhile the press inflamed the febrile atmosphere with lurid but dubious articles about protesters manufacturing and storing Molotov cocktails and attempting to buy small firearms. Privately, Dixon informed a senior officer the press reports were ‘a carefully constructed pastiche of information, gathered from a number of sources and spiced with inspired guesswork’. Brian Cashinella, one of the journalists who reported the claims, says the information had actually come from Special Branch. If the purpose was to discourage people from joining the protest, the strategy appeared to work, as more and more activists distanced themselves from the scheduled march.

Still the hysteria spread. Mary Whitehouse, the self-appointed guardian of British morals, breathlessly told police about a hot new piece of intelligence she had acquired. ‘I have received information from an American friend of mine that an organisation called “The Doors” who are a political extremist organisation are now in England. I do not think that their arrival here in the United Kingdom is a coincidence.’ A police constable dutifully recorded the tip-off, which, like much of the melodrama, was amusingly wide of the mark about the intentions of Jim Morrison and his fellow band-members.

When the big day of the second Grosvenor Square demonstration finally arrived, it was a huge anti-climax. Thousands of protesters came and marched, listened to speeches in Hyde Park and then quietly went home. One section of the demonstration again attempted to break the police cordon around the embassy, but without success. As for the arsenal of Molotov cocktails, one of the demonstrators found three milk bottles containing petrol and a piece of rag and handed them in to police.

However, hearsay can be as relevant as truth when it comes to influencing a national mood and the atmosphere remained volatile. Many still believed the country was on the cusp of social unrest. One damp squib of a march was not going to discourage Dixon, who was already determined to invent a new tactic to help quell the upheaval.

On September 10 1968, the Special Branch chief inspector had crafted a six-page memo. Stamped ‘Secret’, it was sent to a select group of the most senior officers in Scotland Yard. ‘The climate of opinion among extreme left-wing elements in this country in relation to public political protest has undergone a radical change over the last few years,’ Dixon warned. He spelled out how this change had escalated. ‘The emphasis has shifted, first from orderly, peaceful, co-operative meetings and processions, to passive resistance and “sit-downs” and now to active confrontation with the authorities to attempt to force social changes and alterations of government policy.’

He continued: ‘Indeed, the more vociferous spokesmen of the left are calling for the complete overthrow of parliamentary democracy and the substitution of various brands of “socialism” and “workers’ control”. They claim that this can only be achieved by “action on the streets”, and although few of them will admit publicly, or in the press, that they desire a state of anarchy, it is nevertheless tacitly accepted that such a condition is a necessary preamble to engineering a breakdown of our present system of government and achieving a revolutionary change in the society in which we live.’

Revolutionary times called for revolutionary solutions. When Dixon was finally asked by his superiors what could be done about the protests, he unveiled his grand plan. In Special Branch folklore, the ebullient commander is said to have replied: ‘Give me £1 million and 10 men, and I can deal with the problem for you.’ He got what he asked for. The framework for covert surveillance that Dixon constructed that year remained intact throughout his lifetime and beyond.

His radical plan was nothing less than a new concept in policing. The police had dabbled in undercover work before. By the late 1960s, the police had been deploying undercover officers to infiltrate and catch street criminal gangs, but had only done so sporadically. Officers impersonated drug dealers, robbers, contract killers, gamblers or antique fraudsters. But they only did so for short periods – usually just a few days or weeks. In between these covert assignments, officers would return to their station and resume their normal duties.

What Dixon had in mind was infiltration of a different order. He wanted Special Branch officers to transform themselves into protesters and live their fake identities for several years. For the whole time they were undercover they would never wear a uniform or set foot in a police station, unless, of course, they were dragged in, kicking, screaming and handcuffed, in character as a protester. They would be equipped with false ID documents, grow their hair long, and melt into the milieu of radical politics, emerging to feed back intelligence on any gathering conspiracy. These men, who would only penetrate political groups, would have a close-up, front-row view of what was really going on.

It is unclear how Dixon alighted on such a radical idea, but it was approved precisely because the authorities were so alarmed by the threat of the burgeoning protest movement. Labour prime minister Harold Wilson’s government was so supportive of the initiative to gather more reliable intelligence that it agreed to fund Dixon’s plan directly from the Treasury.

For his new recruits, Dixon had in mind a distinctive type of police officer. They needed to be skilled manipulators who were resilient, resourceful and sharp. They would find flats or bedsits, preferring those at the back of houses in case fellow activists went past at night and noticed the lights were off and no one was in. They would take up jobs with flexible working hours and travel, such as labourers or delivery van drivers, so they could disappear for, say, a day with their family, without arousing suspicion.

The elaborate pretence would count for nothing unless the physical appearance of the police officers was dramatically changed. The usual tidy short-back-and-sides haircut would have been an easy giveaway. These officers were required to grow their hair long, often down to their shoulders or even longer. Their hairstyles and resplendent beards fitted in with the alternative fashions of the day. One squad member recalls that he grew his hair out and put on John Lennon-style round glasses, but struggled to produce anything more than wispy facial hair. Another spy’s hair was so fine that he had to go to the hairdressers and have a perm. ‘I ended up looking like Marc Bolan – big hair!’ Years later, members of the unit had their hair shorn as they were sent to infiltrate Nazi skinheads.

The ragged appearance adopted by many of the police operatives gave rise to nicknames for their specialist unit within Special Branch. They were occasionally called the Scruffy Squad, but were more commonly known as the Hairies. More formally, the official name for Dixon’s new unit was the Special Demonstration Squad, usually shortened to SDS. Its official role was to provide ‘sufficient and accurate intelligence to enable the police to maintain public order’.

SDS officers evolved their own ritual over the years before crossing the threshold into their undercover lives. There were three parts to the ceremony. The first was to have a big leaving party to say goodbye to their Special Branch colleagues. For the restricted number within the branch who knew of the existence of the SDS, it was the unit into which police officers vanished. ‘It was a shadowy section where people disappeared into a black hole for several years,’ recalls one officer who infiltrated the revolutionary Socialist Workers Party in the 1980s.

Second, the would-be spies were taken into a room and interrogated by two senior members of the squad to test their cover stories. Another undercover officer remembers his test well: ‘What’s the name of your employer? What’s his phone number? Right, I am going to ring him now, and so on. They ask the kind of questions you might get tripped up by and make sure your cover is as solid as it can be.’ The last symbolic gesture was for SDS officers to hand in their warrant cards; it was too risky for them to be kept, even at home.

It was the final act in their transformation and it meant that police spies would never be able to arrest suspects, a factor that was key to the plan devised by Dixon. Normal undercover police who infiltrated criminal gangs gathered evidence in far shorter operations that could be presented in court as part of a prosecution. Like other branches of the constabulary, their job was to nail criminals. But the SDS was different: they were solely required to gain inside intelligence from political groups and slip it back to their handlers over a period of many years.

They would never be tasked with collecting evidence for use in a trial or called to be a witness in court. The reason was obvious: if any SDS officer disclosed their true identity to testify against activists from a witness box, the whole edifice of the spy operation would come tumbling down. For Dixon’s plan to work, it was essential for political activists to believe that the state would never go so far as to plant spies to live among them for years on end.

As with any closed group, the SDS quickly developed its own culture and language. The more accomplished members of the squad – those capable of totally submerging themselves into their undercover roles – became known as ‘deep swimmers’. Less committed spies were ‘shallow paddlers’ and derided for not managing to access the heart of their selected political movement. The SDS, which was 10-strong for many years, also had a nickname for the targets they were spying on. Political activists were called ‘wearies’, in recognition of what police saw as their ‘wearisome’ dedication to a political cause.

The SDS cherished their reputation as the Hairies, but insiders had another nickname for the unit. ‘The 27 Club’ was a name that recognised the date of the unit’s official foundation on October 27 1968, the day of the second large demonstration against the Vietnam War. The club even had its own unofficial motto that encapsulated the squad’s approach to undercover policing: ‘By Any Means Necessary’.

There is no doubt that many squad members relished the adventure of espionage, the thrill of constantly living on the edge at risk of being found out. Managing to make friends with activists and gaining their confidence was only half the battle. Protest groups were often on the lookout for infiltrators and the slightest slip-up by the police officer could stir suspicions. One spy says he was nervous every time he changed clothes and converted himself into his activist persona.

Life is full of weird, unpredictable coincidences, and the spies always ran the risk of unexpectedly bumping into someone they knew in their real life while undercover. SDS officers knew that at any moment they could be walking along the street with activists and randomly bump into someone who knew them as a police officer.

According to SDS folklore, that is precisely what happened to one officer who went undercover in the Socialist Workers Party in the 1970s. He was on a demonstration when a uniformed colleague recognised him through his dishevelled disguise. The SDS man’s reaction was instinctive and decisive: he attacked the police officer, grabbing his balls until he backed off. The spy then quickly slunk away.

These stories about what had gone wrong with past deployments were relayed to new recruits as a warning. Another cautionary tale handed down through the years concerned an undercover officer who was embedded in a tiny Trotskyist group, the International Marxist Group. A telephone tap had revealed to the SDS that his targets were growing suspicious about him. He was later invited to a pub by his activist friends, plied with alcohol and asked detailed questions about his past.

‘I remember drinking something in the region of nine or 10 pints of beer,’ he says. ‘I was very concerned that I was getting to the point where my guard would slip, that I would reveal something which would give something away or expose a colleague and I remember my mind seeming to stay ice cold. The rest of me felt like jelly but they had drunk along with me so they were showing considerable signs of wear as well, and I don’t know if I satisfied them or not but I was allowed to go and then I was met shortly afterwards by a colleague, then I just collapsed. I was absolutely drunk as a skunk, but I’d held it together until then.’

The episode told the SDS spy and his managers that it was time to quit. He had not enjoyed the undercover life and, in any case, it was affecting his family life. Every SDS officer knew that if the suspicions grew and they were compromised it would mean the end of their mission. They only ever had one chance; it was plainly too risky for a spy who had been rumbled to disappear and then reappear in a new group.

After the anti-Vietnam protests, one of the next movements to attract the attention of the SDS was the campaign against apartheid in South Africa. In the late 1960s, opposition to the regime was growing. A coalition of students, liberals and Christians sprang up to confront and disrupt white South African sportsmen competing in Britain. In late 1969 and early 1970, the activists laid siege to the South African rugby team with imaginative forms of civil disobedience.

Buying tickets like ordinary spectators, campaigners would gain access to grounds and jump over the barriers, past police and onto the pitch to interrupt the game. The battle of wits between the protesters and the police lasted months. Authorities deployed more police to form a barrier to guard the pitch and keep watch for activists leaping out of the crowd. At one game, barbed wire was laid down to deter incursions.

Campaigners adapted their own tactics. One slipped into the South African team’s hotel and glued the door-locks to their rooms. Another disrupted the tour by dressing in a smart suit and telling the driver of the team coach that he was needed inside the hotel. When the driver was gone, the activist sat in the driver’s seat and chained himself to the steering wheel.

Among those who were placed under surveillance by the nascent SDS was Peter Hain, an activist who would later become a Labour cabinet minister. One of the SDS spies who joined Hain’s campaign was called Mike Ferguson. The SDS operative’s handler, a fellow Special Branch officer named Wilf Knight, says the mole used cunning to climb the ladder of the anti-apartheid campaign. Knight recalls how, at one meeting, Hain told the group he believed they had been infiltrated by a spy. ‘There was one poor devil that Mike Ferguson looked down the room at and said, “I think it’s him,”’ says Knight. ‘He got thrown out, and Ferguson survived – bless him.’

Finger-pointing at bona fide activists, and wrongly accusing them of being infiltrators, become a well-used technique among the SDS officers who wanted to deflect attention. Ferguson survived any suspicion anti-apartheid campaigners may have had and completed his undercover tour. He later progressed to become the head of the SDS.

Another early target of the SDS was the International Marxist Group, a collection of left-wing revolutionaries at the vanguard of the campaign against the Vietnam War. ‘A small group numerically, but significant as an idea-producing faction,’ was how Dixon summed them up in one of his reports. Prominent in the group was a well-known radical of the time, Tariq Ali, who had, in Dixon’s view, ‘natural gifts as a mob orator’. Ali would later become a leading left-wing figure for many years.

The SDS spy who targeted Ali and his friends was sufficiently trusted to be allowed to guard the office of the anti-Vietnam war campaign. ‘I was aware that some of the keys that I was holding when I was babysitting those offices gave access to offices that we, or the Security Service, might be interested in,’ he recalls. When activists were out of sight, the SDS officer pressed keys into some Plasticine to make a mould from which copies could later be made. The filing cabinets of the campaign group were later apparently rifled through by MI5.

Breaking into the premises of campaign groups would become another common tactic for the SDS. The legality of such a tactic was dubious, but this was the SDS, the unit which believed ‘any means’ were justifiable and took its cue from the security services. This kind of stuff was routine for MI5, and they had long been assisted by reliable men from Special Branch. The former MI5 operative Peter Wright famously wrote in his bestseller Spycatcher: ‘For five years we bugged and burgled our way across London at the state’s behest, while pompous bowler-hatted civil servants in Whitehall pretended to look the other way.’

Dixon was given a free hand to develop the tradecraft of the undercover unit. The squad was buried so deep within Special Branch that it was immune to many of the grudging reforms undergone by the rest of the force. The branch operated on the traditional principle of ‘need-to-know’ – only those officers involved in an operation would be briefed on its details. Intelligence provided by SDS spies would be shared with other departments in the Met without anyone disclosing how it was obtained.

The SDS turned out to be a surprisingly easy secret to keep. Its spies were out of the office and out of sight, meeting only in safe houses across the capital. It is perhaps a cliché to call the SDS an elite squad, but to an extent it was. It prided itself on recruiting only the very best officers within Special Branch, which considered itself a cut above the rest of the Met. The covert squad did not operate like the rest of the police. There were very few rules and the spies did not feel bound by their rank when they discussed operations. The operatives were expected to approach problems in a creative way, eschewing the obedient, plodding mindset of a bobby.

The spying game was initially considered a man’s world. It appears that SDS operatives were all men until the early 1980s, when women police officers were needed to infiltrate the women-only peace camp set up outside the American nuclear weapon base at Greenham Common in Berkshire. Two female police officers were selected and duly dispatched to join the women protesters. Even after that, women were rarely recruited to the SDS. Insiders describe the unit as macho and competitive, a team in which it was frowned upon to discuss emotional issues despite the very obvious psychological strain being placed on operatives.

The unit was not just the brainchild of Dixon, but very much a product of Special Branch, which already had a controversial reputation for conducting a covert war against enemies of the state. The branch was founded in 1883 to capture Irish republicans who were planting bombs in London. Since then, its primary job had been to gather intelligence to protect the nation’s security. Some of its work was relatively uncontroversial. It included protecting public figures considered to be at risk, monitoring ports and airports for unwanted visitors, carrying out surveillance on embassies and helping to vet foreigners applying to become British citizens. But Special Branch, with its close links to MI5, was considered by many to be Britain’s political police. Its officers were characterised as foot soldiers for the security services, acting as their ‘eyes and ears’ on the ground.

Over the years, Special Branch had spied on suffragettes, pacifists, unemployed workers, striking trade unionists, antinuclear activists, anti-war campaigners, fascists, anarchists and communists. The job of Special Branch, as the then home secretary Merlyn Rees put it – perhaps too honestly – in 1978, was to ‘collect information on those who I think cause problems for the state’. In more formal terms, Special Branch spied on those deemed to be ‘subversives’. But the definition of subversion was a contentious one and there was always the concern police would target people they should not.

In 1963, five years before Dixon founded the SDS, subversives were officially defined as people who ‘would contemplate the overthrow of government by unlawful means’. By the late 1970s, the net had been cast wider and the definition amended to include all those who ‘threaten the safety or well-being of the state, and are intended to undermine or overthrow parliamentary democracy by political, industrial or violent means’. By 1985, concerned MPs on the home affairs select committee noted that the branch was acquiring ‘a sinister reputation of a force which persecutes harmless citizens for political reasons, acts in nefarious ways to assist the security services, is accountable to no one, and represents a threat to civil liberties’.

When Dixon was putting together his squad, Special Branch was expanding to cope with the perceived growth in the subversive threat. Indeed it seemed that the branch was constantly growing. The 225 officers attached to the Special Branch in the early 1960s had expanded to 300 by the end of the decade. By 1978, there were 1,600 Special Branch officers. Originally based only in London, branch officers spread out across the country into provincial forces. Much of the increase was a consequence of the urgent need to deal with the Irish terrorist threat following the outbreak of the Troubles in the late 1960s.

But the expansion was also attributed to the growing surveillance of political campaigners, thousands of whom ended up in confidential Special Branch files. Much of the intelligence in these files was generated from publicly available information. For example, desk officers scoured press clippings, noting down names of activists quoted in articles or recording the names of people who had signed petitions.

When the homes of activists were raided, branch officers would comb through the contents of address books, letters and cheque-stubs to work out whom they were in contact with. Moving up the scale, phones were tapped and mail was opened. This was the routine, systematic work of the branch.

SDS officers had the more glamorous role. They left office duties behind them to inveigle themselves into key positions in protest groups. Becoming the treasurer was an established method of quickly gaining access to financial records. So too was becoming secretary, or minute-taker. The spies were told to avoid rising so far up the hierarchy of a group that they became its leader. ‘As a rule of thumb, you could allow yourself to run with the organisation, but you had to stop short of organising or directing it,’ recalls an SDS spy who infiltrated the Socialist Workers Party at the time of the Falklands War. The best position was that of a trusted confidant, a deputy who lingered in the background.

Although Dixon brought the SDS into existence, he did not hang around for long. In 1973, just five years after constructing the apparatus for long-term infiltration of protest movements, the chief inspector quit policing altogether. He did not want to be desk-bound, grinding his way up the police hierarchy for the rest of his career. The former police officer went to Exeter University to study economic history and then completed a doctorate in the working conditions of merchant seamen. Bizarrely, in 1975 Dixon was a contestant on the popular television quiz show University Challenge. A black-and-white photograph shows the Exeter team seated in the familiar line-up of the show, waiting to answer questions, or as the programme’s famous catch-phrase would have it, their ‘starter for 10’.

Dixon’s young team-mates reflect the look of the era: long hair, moustaches, polo necks and big collars. The 48-year-old student stands out with his thinning black hair and bushy beard. His small eyes stare directly at the camera with the imposing air of someone who was not to be messed with. Dixon later lectured at maritime history conferences about the practices of foreign seamen and wrote a slew of books about yachting, one of which explained the intricacies of an electronic navigation system for boats. Each year, he spent a few months out on the open sea, sailing his boat off the coasts of Europe. He died in 1999, leaving behind a wife and four children. He was rewarded with a discreet obituary in The Times, the newspaper of record favoured by the establishment.

Prior to his death, Dixon did not disappear altogether from the SDS. He had chosen a new life but kept in touch with his old colleagues. Every five years, the spies, past and present, gathered to remember old times. In 1993, Dixon attended an SDS reunion to mark the 25th anniversary of the squad. Around 50 ex-spies and their managers came together at London’s Victory Club, near Marble Arch, and heard a speech from their founder. One attendee recalls Dixon looked like ‘a shanty old seaman’.

Over dinner, feeling secure that they were within a trusted circle, the spies traded tales of derring-do from their covert days. Revered among their number was of course Dixon, the founder of the squad. But there was another man who had gained the special respect of his colleagues – Bob Lambert. Suave and manipulative, Lambert’s own tour of duty as a spy a few years earlier was considered one of the most impressive deployments in the history of the SDS.