CHAPTER 14

The Lions in the Den: Japanese Counterintelligence

As the Japanese moved southward capturing more and more territory, no one was more surprised at the speed and extent of Japanese success than the Japanese themselves. The rapid military triumphs were both welcome and problematic. Now the Japanese were suddenly faced with the need to occupy and control much larger areas of South-East Asia than they had planned for. This entailed taking control on the ground and then establishing defensive systems to hold the newly captured territories and defend them from Allied counterattacks. Implicit in this was the need to establish effective internal security to meet the threats posed by civil unrest, insurgency and, importantly for our story, espionage by Allied intelligence organisations.

As Japan prepared itself for war with the West in the late 1930s, its intelligence system was little changed from the reforms introduced in 1907 following the end of the Russo-Japanese War. This war had shown Japanese commanders and government officials that the flow of intelligence had relied on opportunity and had been far too ad hoc to provide much benefit. The IJA therefore restructured its staff operations, combining responsibility for all foreign intelligence collection and assessment within the Second Department of the General Staff.¹

As Figure 14.1 shows, the Second Department of the General Staff was now organised into four sections, 5th covering the USSR, 6th Europe and the USA, 7th China and 8th being responsible for the conduct of conspiracy and propaganda operations. IJA SIGINT operations remained outside of Second Department as a stand-alone entity reporting directly to the General Staff. The outcome of this was that intelligence remained a tactical activity with little relevance for strategic planning or thinking. This was in line with the philosophy that the IJA had inherited, along with its staff system, from the Prussian model they had selected for the structure of their army.²

The biggest problem the modern historian has in identifying the effectiveness of the Japanese intelligence effort is that very little of the documentary evidence exists, as the Japanese burned all of their intelligence records before the occupation and, understandably, few ex-Japanese intelligence officers have spoken or written about their service between 1941 and 1945.³ This leaves the field empty, with the exceptions of Ken Kotani’s book Japanese Intelligence in World War II, and snippets of information in intelligence debriefings of IJA and IJN personnel conducted by war crimes investigators and intelligence agencies, particularly the US Strategic Services Unit, after 1945.

Figure 14.1: The IJA intelligence organisation

Like the IJA, the IJN had identified the same issues with its intelligence, and it too moved to centralise intelligence in the Third Department of the Naval Staff.4

The greater problem that was created by this approach was that intelligence remained strictly within its service silo and, even worse, within the silos that would be created within each service. Effectively, the IJN only conducted intelligence activity of importance to the IJN while the IJA did the same for itself, while individual generals only conducted intelligence of relevance to their own command. It would even transpire that individual commanders would withhold and use intelligence as a card during the interminable debates that led to decisions.

Alongside the reorganisation of the armed services’ intelligence collection and assessment systems, the Japanese also reorganised its HUMINT and SIGINT organisations: the Toku-Jo-Han (Special Intelligence Section), which was responsible for conducting SIGINT; and the Special Duty Agencies (SDA), charged with conducting overseas espionage. Those parts of the Japanese intelligence system that most concern our story—the internal security organisations, the IJA’s secret military police the Kempeitai, and the IJN’s equivalent Tokubetsu Keisatsutai (Tokkeitai)—remained unchanged.⁵

The Kempeitai was not strictly part of the IJA, as it was directly controlled by the War Department in Tokyo, just as the Tokkeitai was controlled by the Navy Department.⁶ The real difference between the Kempeitai and Tokkeitai was that the latter restricted itself to IJN bases, ships and areas under its direct control. The Kempeitai was left responsible for conducting counterespionage and internal security operations everywhere else.

In this role, the Kempeitai had the authority to conduct any investigations, apprehensions and examinations they deemed necessary. They were even able to extend their control over Japanese civilians, nominally the concern of the Japanese Consular Police, as this organisation was effectively subordinated to the local Kempeitai HQ.⁷ In China, away from the front line and areas where IJA formations were based, the Kempeitai was responsible for all counterespionage work, particularly in the major cities.⁸

The specific tasks of the Kempeitai were clearly laid out and were, in order of importance: first, the collection of information on the local community and for preserving peace and order; secondly, the investigation of foreign intelligence agents and the identification of their activities; and thirdly, the collection of other intelligence as directed.9

The Kempeitai was different from the IJA’s intelligence department because it was not subordinate to the IJA or even Imperial HQ, but to the War Ministry and the Provost Marshal General of the IJA.¹⁰ This gave the Kempeitai its secret police character and made it a fearsome organisation, even for members of the IJA.¹¹ That said, the control of the Kempeitai was delegated to the commanding general for a theatre, and Tokyo did not issue instructions directly to Kempeitai units within that theatre.¹² This was left to the commanding general.

The structure of the Kempeitai was highly flexible, and was changed within various theatres and commands to suit the needs of that command at the time. Its basic organisation is, however, set out in Figure 14.2.

The Kempeitai was not easy to join. Potential recruits had to be proven soldiers who had demonstrated consistent loyalty and dedication to the ideas of Bushido (the samurai code) and the superiority of the Japanese race. Oddly, there was a long waiting list of volunteers from other branches of the IJA wanting to transfer across. This suggests that service in the Kempeitai held significant attractions, because the training was even more brutal than that ordinary soldiers had to go through.

The training given to Kempeitai soldiers included judo, mob control, Shinto mental conditioning, indoctrination in the superiority of the Japanese race, clandestine tradecraft, espionage, running of agent networks and interrogation techniques. All of this was done within a very brutal, rough and tumble regime that tested the potential Kempeitai soldier to the maximum.¹³ The various schools—approximately four may have existed—produced around 300 graduates each per year, providing 1200 new soldiers annually.¹⁴

The Kempeitai—and on New Ireland and Bougainville, the Tokkeitai—were thus the main opponents the operatives of the Services Reconnaissance Department (SRD) and the coastwatch organisation faced when they operated behind Japanese lines. These elite counterespionage troops would demonstrate determination and aggression in hunting the Allied operatives on the ground, and their ferocious brutality severely limited the willingness of local populations to protect or support Allied operatives.

The effectiveness of these organisations can be gauged from the way the Japanese response to the activities of SRD and coastwatch parties changed from a more live-and-let-live approach, with Japanese patrols lacking aggression and determination, to the patrols chasing Allied operatives for long periods and using locals in tracking and attacking coastwatchers and SRD personnel.

Source: Strategic Services Unit, Japanese Intelligence Service, Section II, Japanese Military Intelligence in China, Section 6.a., CIA, www.cia.gov/library/readingroom/docs/JAPANESE%20INTELLIGENCE%20ORGANIZATIONS%20IN%20CHINA%20%20%20(WWII)_0001.pdf

Figure 14.2: Organisation of the Kempeitai (Japanese military police)

The enlistment of locals as guides and trackers was probably the single most effective strategy in making occupied areas unsustainable for Allied personnel.¹⁵ More than once, Eric Feldt, in his book The Coastwatchers, relates the concerns of experienced coastwatchers about the increasing use of locals to track them. These worries can also be found in the official record.¹⁶ As we will see below, once the Japanese were using local guides and carriers, whole areas became too dangerous for any Allied operatives to enter.

One of the other changes in the way the IJA operated was that they reined in the use of terror, which was proving counterproductive. Killing locals and stealing their food and animals, as well as raping and kidnapping their women, led to significant hostility among locals and a marked unwillingness to work for the Japanese. The fact that whole villages could simply lift up everything they owned and walk off into the jungle made this a significant problem in New Guinea, New Ireland and the Solomons, where the jungle afforded real protection to the locals. On Timor and the western end of the archipelago this was not the case.

In New Guinea, the way the Japanese operated made a marked impact on the Allied Intelligence Bureau (AIB), and particularly on SRD and ANGAU parties. At one time, in the Buna area, the Orokaiva tribe was being described in AIB intelligence reports as ‘extremely treacherous’ and having provided the Japanese with large parties of warriors.¹⁷ These warriors had attacked neutral villages that had refused to assist either the Australians or the Japanese in the Adjora, Serapi and Bofu areas. The reward must have been rapine, money and goods. At the same time, the Japanese were still executing anyone who offered resistance, beheading a villager from Lokanu, Kavieng, at Salamaua.¹⁸

The AIB was advised by the Far Eastern Liaison Office (FELO) that the local inhabitants would support whoever treated them better, but it was not that simple. The Japanese mixed bribery with harsh treatment of any dissent or failure to collaborate. One example of this is the bayonetting to death of eleven Biagi tribesmen at Deniki in September and the raping of women at Kokoda, Wairope and Buna, acts that alienated many locals in that area.¹⁹ The kidnapping of women to serve as comfort women in Japanese brothels further alienated local villagers.20 This was made worse by the wanton destruction of villagers’ gardens and the ill treatment of any locals conscripted by the Japanese for work. Unsupervised indigenous police working for the Japanese also indulged in rape, theft and mistreatment of locals. By 11 May 1943, however, the Japanese were trying to retrieve the situation by ordering that chiefs be treated with respect.²¹

The brutality of the Japanese meant that a local headman had little choice. If he did not supply manpower and material as demanded, he faced execution or other punishments for himself, his family and his village. It was therefore much safer to provide the support demanded by the Japanese.

On the other hand, the provision of support to transient AIB operatives presented the entire village with the certainty of severe punishment once the Japanese found out—and they would find out because, while their handling of intelligence processing and assessment was extremely poor at the higher level, their handling of tactical intelligence and, particularly their handling of security intelligence, was, as a result of a long history of surveillance operations in Japan, Korea and China, extremely good.

The system used by the Japanese was described in July 1945 by Corporal Koram bin Anduat of the British North Borneo Police. Corporal bin Anduat described how the Kempeitai operated in Borneo, and his testimony provides a rare insight into their counterintelligence efforts against Australian and Allied intelligence operations in occupied areas.

Anduat described how in Borneo the Japanese took over and completely disarmed the civil police, even taking their knives off them, and set them to work digging trenches, guarding facilities and even fishing.²² In time, some of the police were integrated into the Kempeitai’s network and carried out counterespionage duties and other support functions. At Lahad Datu, the head of the Kempeitai was an officer named Djumi, and, from Anduat’s description, this man was most likely a warrant officer.

Very quickly after their arrival, the Kempeitai established an extensive network of clandestine agents throughout the area, consisting of men, women and children, the latter both boys and girls aged between twelve and fifteen.²³ The clandestine male agents were distributed one to each village, regardless of whatever other agents were already in place. The Japanese would call a meeting of all the village headmen in an area and force them to provide lists of their male villagers. The list was then supplied by the Japanese military authorities to the Kempeitai, who would select a name and offer the individual paid work of an unspecified nature. If the selected villager agreed, then they became the secret Kempeitai agent in the village. If not, then another name was selected.24

Each clandestine agent was then tasked by his handler and told that under no circumstances were they to divulge to anyone the work they were to do, under threat of death or other punishment. A second clandestine network operated by a Japanese company, which, in the case described, was the Usira Company, provided the Kempeitai with a check on these agents and on how well they were protecting their status.²⁵

The description provided by Corporal bin Anduat shows that in the Lahad Datu area of Borneo, the Japanese were sticking to the systems and organisations they used in China and other places. In these areas, Japanese companies provided logistical and intelligence support to the IJA. The company had two roles: to obtain resources, including food and other supplies, and local labour for the IJA; and to maintain an extensive internal espionage organisation throughout the area. This espionage system was completely independent of the Kempeitai’s agent networks, although the company worked extremely closely with the Kempeitai.²⁶ If the company agents became aware of the identity of the Kempeitai’s agent in a village, that agent was brought in to Kempeitai HQ and imprisoned or executed.

At Lahad Datu, a Kempeitai warrant officer acted as the liaison officer between Kempeitai HQ and the village agents. Like most Kempeitai, he only wore his uniform in his office, going about the area he worked dressed in local clothing. This did not fool anyone, according to Anduat, as everyone in the entire district knew at one look that he was a Japanese dressed in Malay clothing. The poorly disguised warrant officer nevertheless conducted his rounds of the villages twice a week to collect intelligence from his agents. He hid the identity of the agents by visiting a number of houses, including the agents’. It was during these rounds that agents were paid, but on a results basis. If they had nothing, they got nothing. The pressure was therefore always on the agent to find something to report.

The unfortunate women recruited as agents were taken from their villages to work as comfort women in brothels, exclusively for the use of Japanese Kempeitai personnel. After a period serving as comfort women, some were selected and trained as clandestine agents. These women were used on surveillance operations and as field agents in their own right, with the job of recruiting networks of other women in an extended spying system reaching into all areas of village life.27

The Kempeitai also worked closely with the Kenchiji, the local Japanese district administrator, as well as the local Japanese companies. The result was a duplicated and overlapping surveillance and intelligence system that not only reported extensively on everything happening, but also double-checked itself. This system was highly effective in heavily populated areas, such as Timor and Java. It was particularly effective in Java because the Javanese had no love for their Dutch overlords and did not intend to support their return by assisting any white agents. The effectiveness of the Kempeitai’s system and the hostility of the Javanese and other populations towards the Dutch help explain the failure of the AIB’s operations in those areas.

In the east of the archipelago, the Kempeitai enjoyed similar success around the populated centres such as Lae and Rabaul, but not so much within the jungle-covered and sparsely populated hinterland. They did, however, make the local populations wary, and they did win converts who made it difficult for AIB and coastwatchers personnel to conduct operations.

The threat to AIB and coastwatcher operations did not just come from the Kempeitai and their local assistants, but also from the SIGINT operations conducted by the IJN and IJA right across the Pacific, and from the SIGINT listening posts established by the Kempeitai across the islands, including on Tabar, Lihir, Tanga and Feni islands.²⁸ In addition to these listening posts, each military post and larger IJN vessel had the capability to intercept and conduct DF on radio transmissions. The increased risk of compromise from SIGINT was well recognised within the coastwatch organisation as early as 1943.²⁹

The IJN’s Toku-Jo-Han had initiated its SIGINT operations as soon as it was formed, and it is claimed that by the 1920s they had broken the diplomatic codes of the United States, the United Kingdom and China. Given the access that the Toku-Jo-Han had to the telegraphic traffic carried on the mainline cables traversing China, it is completely unsurprising that they broke these systems.³⁰ Despite these successes, Toku-Jo-Han faced considerable problems in dealing with Allied communications and codes during the war. This does not mean they were not dangerous.

The organisation of Japanese intelligence within a given area of operations varied according to the perceived needs of the commanders responsible for it. SIGINT was an integral part of its activities.31 Indeed, we know from the Strategic Services Unit report that the Kempeitai in China had an integral SIGINT capability within its organisations and that this conducted detection activity similar to that of the Radio Security Service (RSS) operating in Britain, India and, later on, Australia.

Before the end of the war, the information the Allies had on Japanese SIGINT capability was limited, and they found it difficult to tell how useful SIGINT was to the Japanese conduct of the war. Postwar investigations and analysis clearly showed that while the Japanese intelligence effort completely failed to influence the strategic direction of the war, it did contribute significantly to Japan’s conduct of tactical operations. In SIGINT, this was as true as it was for HUMINT.

The reality was that both the IJA and IJN had significant successes in intercepting, if not breaking, Allied communications. In a postwar report by the Strategic Services Unit, it was found that the IJN had been intercepting US naval communications from call sign VHM located at Darwin. The traffic from this intercept was passed to Tokyo, and one source indicated that it was only in Tokyo that it could be broken.³²

Despite the dismissive tone of the source, the fact is that the IJN could have conducted traffic analysis and DF on the intercepted signals—and obviously did so, as the source knew the call sign of the station and its location of Darwin.³³ Later in his debriefing, this source disclosed that he and his Filipino colleagues believed they had identified the arrival of General MacArthur at Leyte when, on 18 October 1944, a station called ‘Tacloban Field’ requested a jeep be provided at the jetty for HALIFAX HIMSELF. They had already associated the codeword HALIFAX with MacArthur or his HQ and postulated that HALIFAX HIMSELF was MacArthur. The source then went on to describe in considerable detail the Leyte Gulf landings, including orders from HALIFAX for stricter radio discipline.

The source of this information appears to have been an English-speaking Filipino, either recruited or conscripted as one of a number of Filipino intercept operators working for the Japanese.³⁴ What is interesting about him is that he admits to the US interrogators that he was ‘greatly intrigued by the possibilities of long-skip VHF signals’ and he wanted ‘to know just how the sporadic E-Layer behaved in tropical regions’. This clearly indicates that he was much more than just a basic-level intercept operator, and that he was very conversant with the propagation of radio signals and the equipment he was using—a simple super-regenerative VHF receiver he had built and a Packard Bell disc recorder.35

The evidence provided to the Strategic Services Unit by this source indicates that he worked for the Japanese in both Manila and, after being evacuated on 29 December 1944, Shanghai.³⁶ This suggests that with the fall of Singapore and the occupation of the Philippines, the IJN and the IJA were able to access a very large and competent pool of English-speaking personnel. There can be little doubt that the individuals recruited functioned as intercept operators, traffic analysts and interpreters, thus providing the Japanese with a considerable boost in their effectiveness against Allied communications.

The importance of this lies in the fact that as Allied units and personnel were overrun, their communications became highly vulnerable to the Japanese SIGINT effort. It also brings into question the effectiveness of the very low-level codes and ciphers used by the SRD and coastwatch organisations, and it helps explain the ease with which Japanese forces were able to interdict SRD and coastwatch parties later in the war. The Japanese were competent in identifying radio traffic from such parties and were quite capable of identifying the radio reporting coming from islands far away from their immediate operating areas. Using a radio to communicate from Japanese-occupied areas was a dangerous undertaking, and the only advantage was that the terrain made it difficult for the Japanese to get to locations quickly, so coastwatchers and other AIB operators could get clear before they arrived.

Of course, this advantage could be countered, and it was. The answer was in fact simple. The IJA quickly realised that sending a single patrol, or even two, on a direct trek towards the area from which a signal came allowed sufficient time for the Allied party to make good its escape. The change in tactics was to insert a large number of smaller patrols, led by local guides, along the coastline of the island from which a signal was detected. These patrols would then sweep inland, cutting off the retreating Allied party or forcing it deeper into the jungles and mountains to make good an escape. The issue now facing the party was that the long line of sweeping Japanese patrols meant it had to cover long distances very quickly and this led to it spending most of its time on the run. The outcome of this was that even if they weren’t caught, they were prevented from observing activity along the coast and, even if they did observe activity, they could not afford to stop or set up the radio to send the intelligence.

The Japanese had yet another extremely important source of intelligence they could exploit in their battle against the AIB and its parties in Japanese-controlled territory. This source was the captured AIB operatives, whose rudimentary training did not prepare them to resist the sophisticated interrogation techniques of experienced Kempeitai officers. Interrogation would prove to be the most bountiful source of detailed intelligence the Japanese developed on the AIB, its various organisations, and their functions, personalities and resources. In fact, by 1945, the Kempeitai had completely mapped the entire AIB organisation, its operations, plans, personalities and techniques.37

All of the intelligence obtained on Allied espionage operations in the Southern Army area was passed back to its HQ at Saigon, where a dedicated section collated and studied it to build the picture of Allied intelligence in South-East Asia. The intelligence derived also appears to have been communicated to Imperial HQ in Tokyo.³⁸

All this meant that by the end of 1942, following the capture of the LIZARD II party, it is likely that the Kempeitai on Timor began putting together a very good picture of the way AIB parties operated. They would also have obtained good information on how the AIB parties were being prepared and despatched. All of this intelligence would have been collated and analysed in Singapore and Saigon, and appropriate countermeasures put in place. The history of the AIB’s operations in Timor and Java clearly shows, as we will see later, that the AIB operatives being inserted into these areas were doomed from the very beginning. The AIB failed to develop a proper appreciation of the enemy they faced. Had it done so, it might have realised the impossibility of its ambitions for setting South-East Asia ablaze and it would have saved the lives of many good men and countless locals. That, however, is a later part of this story. Now we need to look at the AIB, its founding, and its hopes and ambitions.