CHAPTER FIVE
THE SECRET WAR
Battles were won because we had advance knowledge of enemy plans, could influence those plans, and could anticipate enemy actions by methods heretofore concealed.
A Man Called Intrepid,
Sir William Stephenson.
The Australian Government was unaware that MacArthur’s predictions of enemy movements were derived from a source that he was unable to confide to the government or even to Curtin.
Prior to the war in Europe, the British Secret Service had obtained a German cipher machine, which they later developed to enable Churchill and the key Chiefs of Staff to monitor both German and Japanese naval, army and air movements. The stolen machine was originally known as an “Enigma” machine – a name taken from the Greek, meaning “puzzle” – and was probably the most jealously guarded secret in military history.
It is now evident that deciphering the enemy’s codes played a major role during World War II. For 35 years after the war, military historians tended to believe in such concepts as “great men”, like Churchill and Roosevelt, or gifted military leaders, such as Montgomery and MacArthur. The defeat of the Axis powers was brought about by these leaders and the men they led, coupled with the eventual superiority of western firepower. But an incalculable advantage was in knowing beforehand what the Axis powers intended to do. The Allies possessed the Enigma machine as early as 1939; but it wasn’t until 1974 that historians learnt of its existence and necessitated a revision of the history of World War II. We now know that the Allies had privileged access to most of the secret enciphered and coded messages that emanated from both German and Japanese High Commands.
This single advantage, which neither the Germans nor Japanese seriously suspected, was the decisive factor in winning both the European and Pacific wars. The British called the information gained from it “Ultra”, and the Americans “Magic”. If there were some shadowy areas where Ultra’s effectiveness was diminished by the need to keep it secret, they were outnumbered by the decisive battles, which were won as a result of its information.
In the Pacific theatre, the Battle of Midway Island in June 1942 was the turning point in the Pacific War. Ultra information played a crucial role in this when precise dates were intercepted from Japanese Imperial Headquarters of an impending attack by Japanese forces on Midway Island. This information was vital to the successful deployment of the American Pacific Fleet.
Again, almost a year later, Ultra information played a key role with the assassination of Admiral Isoroku Yamamoto. A Japanese signal was intercepted and decoded, which gave the precise date, time and itinerary of Yamamoto’s planned inspection of the Bougainville and Truk area. The message was rushed to Washington where, after a midnight meeting with the Secretary of the Navy, President Roosevelt made the decision to ambush Yamamoto’s aircraft over Bougainville. The death of this revered hero shattered the morale of Japanese forces.
The part played by Ultra in the Pacific is still only vaguely known. Ultra’s existence was first authoritatively revealed in 1974, when F. W. Winterbotham published The Ultra Secret. Soon after there appeared the international bestseller – A Man Called Intrepid by William Stevenson. This book detailed the part played by Sir William Stephenson (not the author) in the secret war and who, on Churchill’s orders, established in New York and Bletchley Park in London, a high level, secretly liaisoned organisation called British Security Co-ordination (BSC). This organisation dealt with Ultra information and passed on the deciphered interceptions to the appropriate authorities.
A primitive version of the Enigma machine had been obtained by the British Secret Service as early as 1938. Its function was explained by a Polish engineer who had been working in a German factory, which was building the secret cipher machine. The German machine was theoretically incapable of having its cipher broken because of its incredibly complex operation. However, the Pole took careful note of the various components and, with the help of other Polish workers in the factory, was able to piece together its function. The Pole was then smuggled to Paris where he reconstructed the machine from memory for the British Secret Service. Although it had common parts, he was unable to provide cipher experts with any of the specifics they required and a decision was made to steal a production model. After learning that one such machine was to be delivered to a German frontier unit, the truck carrying the machine was ambushed and the cipher machine stolen. The truck was then set on fire and fake coils, springs and rotors were planted in the charred remains to fool the German investigators.
The Japanese had purchased an early uncomplicated version of the Enigma machine from the Germans in 1930, and adapted it for their own use for diplomatic traffic. Its use was also shared with the Japanese military. Although diplomatic and military traffic was concealed by complicated codes, it became increasingly apparent that Japan was also moving towards naked aggression. The US Army’s Signal Intelligence Service duplicated the Japanese machine and called it “Purple”. By September 1940, Japanese codes were being intercepted and gradually deciphered. From this point on the Americans gave it the code name “Magic”.
After September 1939 there was always the possibility that Britain would be conquered by the Nazis. Churchill appointed the Canadian industrialist William Stephenson as the Chief of British Secret Intelligence to set up a combined American and British secret intelligence organisation in neutral United States. Stephenson established BSC in New York with the covert approval of President Roosevelt who realised that America’s own intelligence organisation had many shortfalls and was incomparable to that of Britain. In these troubled times, he saw the need to collaborate with Churchill. American William “Wild Bill” Donovan was appointed by Roosevelt to be groomed as Stephenson’s successor and to set up BSC’s American equivalent, the Office of Strategic Service (OSS). Until Donovan was ready to take over, Stephenson co-ordinated both American and British intelligence.
Through Stephenson, Britain ran the most intricate, integrated intelligence organisation in history. His mandate assumed that Britain might be conquered, so New York was chosen as the junction box through which all Allied secrets could be pooled as it was the commercial and communications centre of the world. If Britain fell, as seemed possible, the secret war could still be directed from BSC.
It was from this shadowy world that the “beat Hitler first” policy was formulated and the task of keeping that knowledge secret rested with Stephenson. Had this clandestine alliance between the United States and Britain become known to the American press or Congress, there could have been a call for the President’s impeachment through violation of American neutrality.
When the stolen Enigma machine arrived in London, Stephenson established his main secret intelligence organisation at Bletchley Park, 60 miles from London. It was here that the German ciphers were broken and the information used by Churchill, without compromising his source, for Britain’s profit. As a result of Stephenson’s diplomatic persuasion, the US Army Signal Intelligence Service handed over a model of its Japanese version to the British, which was also installed at Bletchley Park. Churchill now had access to both German and Japanese traffic.
By 1942, Stephenson’s secret intelligence network had grown throughout the world with intercepted Japanese wireless traffic coming from Australia, India, Ceylon, New Guinea and Singapore direct to Bletchley with copies to New York. Over 30,000 experts were now linked to the British Secret Service but only a handful of people knew or had access to its information.
Churchill was reluctant to impart Ultra’s existence to Curtin. He distrusted the Australian Government, which he felt was complacent. Since the invasion of Australia seemed possible, Churchill considered that the Australian Government would compromise Ultra by using the information in its own defence against Japan. Indeed, Churchill only told Roosevelt as much as he felt he needed to know.
Through Ultra it was learned that there was an impending bombing raid on Coventry in Britain. Churchill decided not to evacuate the city or to warn the population, in the event of risking Britain’s pre-knowledge of the bombing to the Germans. Many people were killed in that raid. Although the decision was considered right, Churchill later wrote that Coventry was the most painful decision he had to make during the war.
To reduce the risk of an Ultra leak, Churchill confided its existence to the fewest British leaders possible. He felt it was necessary to confide in Roosevelt to demonstrate that Britain could repel an invasion from across the English Channel. Churchill urgently needed the American destroyers and other defence hardware, which the American President had promised, but first he had to prove that this hardware would not be captured by the Germans and eventually used against the United States. Churchill stated: “We must make an exception with Mr Roosevelt. To him, and to him alone, the truth should be confided…”
Meanwhile, at Bletchley Park, Special Liaison Units (SLU) were being trained to decipher and rephrase Ultra information, so that if it was intercepted by the enemy during redirection, it would not resemble the original signal. These units were then sent to various locations throughout the world. Such a unit, SLU9, was sent to Brisbane, Australia, at the beginning of 1942 under the control of Australian Colonel Alastair W. Sandford. Sandford had received his training at Bletchley and was the only man empowered to convey Ultra information to American, British, and Dutch or Australian military Chiefs of Staff.
It is interesting to note that the only Australian war records uncovered of Sandford’s activities appeared in the Army list under Special Wireless Section, Australian Intelligence Corps. His army career was recorded up until the beginning of 1941, after which he seems to have vanished. He re-emerged in the latter half of 1943 where the Army List records him as being attached to the Central Bureau, Intelligence Corps – the same organisation set up by General MacArthur which evaluated Ultra information.
Colonel Sandford also channelled all intercepted Japanese wireless traffic to Arlington House, Nebraska Avenue, Washington, where it was analysed by Stephenson’s second in command. This man was Commander Laurence F. Safford – code-name “Crusader” – who headed the intelligence unit at Washington Naval Headquarters (NEGAT). According to Safford, every wireless transmission emanating from the Japanese in the Pacific, including Japanese submarine traffic off the coast of Australia, had been intercepted since the beginning of 1942.
With the installation of the Australian SLU, information containing Japanese naval, army and air movements was now being received in London, Washington and Brisbane.
Situated on the sixth floor of the Australian Mutual Provident (AMP) building in Queen Street, information gained from Ultra suffered from severe restrictions imposed by General MacArthur regarding who should have what information. Although rigid security was imposed, only some of the information obtained was siphoned to Australian military leaders in Melbourne without compromising its source.
The person in charge of setting up SLUs around the world, Fred Winterbotham, a senior Air Staff representative in the British Secret Intelligence Service for 10 years, records the distribution of information in his book, The Ultra Secret. On a visit to Brisbane in early 1942, he wrote:
I wanted to make my number with the Australian Chief of Air Staff, Air Commodore Jones. I found him unhappy about the amount of information that was being passed to him from Brisbane. He certainly wasn’t getting all that he should have, so I decided to look into the Brisbane distribution as soon as possible … I found the Ultra set-up in Brisbane was most efficient, but it suffered a little from restrictions imposed by General MacArthur, on “who should have what”. Nevertheless, I was able, with the co-operation of Colonel Sandford, the young Australian officer in charge who I had known in London when we were teaching him the job, to sort out some of the distribution problems …
Winterbotham recounts that no history of naval warfare in World War II can now be regarded as complete without mention of the information, which was available to the Allied admirals. It was this information, which created the sea air battles in the Pacific. The Japanese plan aimed to outflank New Guinea by going far to the east to the Coral Sea before closing on Port Moresby. The plan was received on 17 April 1942 and passed to Admiral Nimitz. It gave him time to move his ships to meet the threat and, in fact, fight the Battle of the Coral Sea in early May. The Battle of the Coral Sea was not the decisive victory but it did stop the Japanese moving southwards, and the threat to Port Moresby was averted.
After the outbreak of war, the overall Australian intelligence organisation had come under the control of General Sir Thomas Blamey. Its operations were limited but after the fall of Singapore, the Special Operations Executive (SOE), quartered in New York and a secret cell of BSC, sent British businessman G. S. Mott to whip it into the most effective intelligence organisation in the Pacific. Mott had received his formal training in intelligence at Singapore prior to the war and was given the wartime rank of Colonel.
Mott developed a reputation as a cold and calculating man who had wide business interests in both Australia and the Far East. After the fall of Singapore, he harboured a deep resentment against the Japanese who had caused the collapse of his Asia-wide business of Maclaine, Watson & Co Ltd. The American Allison W. Ind, attached to MacArthur’s Allied Intelligence Bureau (AIB) in Australia, in his book Spy Ring Pacific, describes Mott as a moody, dark, saturnine man with a short temper, who had the ability to act quickly and get things done. It is now known that Mott was guided by Stephenson and reported directly to London.
Surrounded by a small staff of specialist British, Dutch and Australian officers, Mott established a security tight base behind the walls of a fashionable Toorak house called “Airlie”. It was also known as the “department of dirty tricks”.
After meeting with some opposition from the Australian Government, Mott formed the Inter-Allied Services Department (IASD), which was merely an innocent cover name for his activities. His organisation intercepted highly revealing cryptograms from the Japanese, which Mott channelled directly to 10 Downing Street and BSC in New York. He also kept Churchill informed of Curtin’s attitudes and wartime policies. Even General MacArthur, when he established his own Allied Intelligence Bureau, was unaware of the full extent of Mott’s activities.
With the entry of America into the war, MacArthur became one of the few Americans to have access to Ultra information. He established his own intelligence organisation in Melbourne under the fiery leadership of Colonel C. Willoughby. Washington offered the services of clandestine agents attached to the American department of BSC, titled Office of Strategic Services (OSS, the forerunner of the CIA) and led by William Donovan. But MacArthur refused, saying that he disliked having personnel under his command whom he could not control. Willoughby also resented anyone interfering in his intelligence domain. Nevertheless, “Wild Bill” Donovan channelled the appropriate Japanese intercepted traffic to MacArthur, which became the governing factor of his strategies.
MacArthur was extremely grateful for Donovan’s information and later described him as the most determined, resourceful and gallant soldier that he had ever known. Donovan replied that he knew too much about war to glory in it. He added that wars were made by politicians who neglected to prepare for it.
By June 1942 MacArthur had amalgamated Mott’s secret intelligence organisation into his own AIB, labelling it the Central Bureau. This section was controlled entirely by Willoughby and Mott who distributed information only on a need-to-know basis, which included the Supreme Commander of the South-West Pacific.
Allison W. Ind, in his book, A History of Modern Espionage, described the Central Bureau as having accomplished the incredibly difficult task of establishing wireless reporting stations in the East Indies. Within the Central Bureau, MacArthur also formed an Allied Translator and Interpreter Section (ATIS), which translated Japanese radio interceptions and captured documents. This section was led by American Colonel Sydney F. Mashbir, a master intelligence agent with a long and distinguished career as a translator, and who had a fine insight into the Japanese mind. Prior to Mashbir’s arrival in Australia, the ATIS was headed by Queensland-born Oxford University graduate George Caiger who had spent 10 years as an English lecturer in Tokyo before the war.
By mid-April, intercepted Japanese communications, together with northern coast watcher reports, revealed to Mott and Caiger’s organisations details of Admiral Yamamoto’s plans for the imminent sea invasion of Port Moresby, New Guinea. From there the Japanese intended to carry out further attacks on Australia.
On 17 April Colonel Willoughby in Melbourne passed the plans to MacArthur and Admiral Chester Nimitz, Commander in Chief of the United States Pacific Fleet based in Hawaii. Despite this ample forewarning, MacArthur and Nimitz were still unable to muster the forces necessary to inflict a decisive defeat on the Japanese naval forces in what was to become known as the “Battle of the Coral Sea”.
Yamamoto’s plans announced the forces to be used would be three aircraft carriers, five heavy cruisers, four light cruisers, 12 destroyers, and Captain Hankyu Sasaki’s submarine force. Willoughby also advised Nimitz that the invasion would occur during the first week of May. MacArthur informed Curtin and the Australian Government of the impending invasion and added that this information was derived from coast watcher reports.
Admiral Yamamoto’s objective was to launch a seaborne invasion of Tulagi in the Solomon Islands and then Port Moresby, thus securing Japanese lines of communications to the north of Australia. Once this objective was achieved, the Japanese would then be able to launch air attacks against an isolated Australia, and also against Nauru and the Ocean Islands, culminating in the capture of their phosphate deposits.
Once Yamamoto had issued his invasion plans, however, he found it difficult to determine the strength of the opposing forces. His limited intelligence organisation revealed the presence of only a small Allied naval force in the Coral Sea. Due to this uncertainty, Yamamoto added another two submarines to Captain Sasaki’s Third Submarine Company, with orders to locate and “destroy the enemy fleet”.
Unknown to Yamamoto, Rear-Admiral Fletcher and his USS Yorktown force were already near the Coral Sea area by 25 April. Nimitz ordered the USS Lexington force, which was already a week out of Pearl Harbor, to proceed to rendezvous in the Solomon Sea on 1 May with the heavy cruiser Chicago and the destroyer Perkins, which were escorting the fleet oiler Tippecanoe from Noumea. In addition, the Australian Squadron, under the command of Rear-Admiral Crace of the Royal Navy, was ordered to reinforce the Yorktown and Lexington groups, and sailed from Sydney accordingly.
The main action of the naval battle did not occur until 8 May when planes from the American aircraft carriers sank the Japanese carrier, the Shoho, and seriously damaged another, the Shokaku. The Americans lost the Lexington and two others ships. The Yorktown was also badly damaged. Although the Americans suffered greatly, it was considered a victory because the Japanese fleet turned back to Rabaul due to a lack of adequate air defences. After learning of their withdrawal, Yamamoto ordered them south again to “annihilate remaining enemy forces”, but because of the distances between the opposing forces, no further contact was made.
The Battle of the Coral Sea was the first in naval history of enemy surface vessels being beyond sight of one another, and the battle being decided by their opposing aircraft. The result checked the Japanese in their southward drive, their objective to invade Port Moresby was denied them, and the capture of Nauru and Ocean Islands was postponed. Also, because of the outcome of the forthcoming Battle of Midway, a massive 300-aircraft attack on Townsville and the planned invasions of New Caledonia, Fiji and Samoa had to be postponed and eventually abandoned in July.
The War Cabinet met on 13 May 1942 and, largely based on MacArthur’s appreciation of the previous day, severely criticised the conduct of the Coral Sea operation. MacArthur had observed that the backbone of the striking power was the aircraft carriers, which did not belong to his command. The results of this meeting expressed the government’s views to be:
… rather disappointing, the more so as we had ample warning of the enemy’s intention, the prospective date of the attack and the strength of his forces. With the advantage of this information we should have been able to concentrate the superior strength necessary to have ensured a complete victory. As it was, an opportunity to inflict losses on the enemy was lost.
Curtin cabled both Churchill and Roosevelt of the War Cabinet’s viewpoints. He also emphasised in the cable that in view of the Coral Sea engagement, it was vital to build up and maintain Australia’s strength sufficiently to repel any further Japanese attacks.
This communication to Churchill and Roosevelt was not sent through secret channels. The cable was an open one, in which Curtin stated that, “We knew the strength of the enemy concentration, we knew his intentions, and we knew the prospective date of his attack…” This knowledge he would not have divulged so freely had he been aware of Ultra and the security surrounding the information obtained in it.
From the outset Churchill had insisted on the utmost secrecy for Ultra, evidenced by the example of his inaction over the German bombing of Coventry. To inform Curtin of its existence was an added and unnecessary risk, as Churchill doubted that Curtin had sufficient strength to sacrifice Australians under similar circumstances.
Indeed, the closing sentence of Curtin’s cable, “Fortune will not continue to favour us with these opportunities if we do not grasp them”, suggests that he would have used these opportunities, as Churchill feared, to protect Australia against attack – a step he could not have taken without arousing enemy suspicion of Ultra’s existence when it would be realised by the Japanese that their codes had been deciphered.
The Coral Sea engagement, though not a military success did, however, grant Australians a reprieve from the imminent threat of invasion. A tide of relief swept through the nation. Although the Japanese failed to capture Port Moresby, parts of Sasaki’s Third Submarine Company were already in Australian waters, watching and waiting to strike a blow at the heart of urban Australia, thousands of miles from the New Guinea war front.