CHAPTER EIGHT

Florida

NED HALLOWELL WAS RESTLESS. LIKE SOME OF THE ENLISTED men, the colonel worried that the regiment was doomed to months of tedium and fatigue work while the navy lobbed shells into Charleston. Hoping to prod his superiors into action, he steamed down to Hilton Head on February 3 to remind them “that his 10 companies are full & that he has 52 unassigned recruits from Vermont.” Doubting that the Fifty-fourth was as battle-ready as Hallowell claimed, General Gillmore accompanied Ned back up the coast, where he ordered a grand review of all troops. Although rarely inclined to give black soldiers their due, Gillmore was impressed by what he saw, and after the drill he told Hallowell to prepare to break camp. On February 5, orders arrived just as the men were eating dinner. Gillmore directed General Truman Seymour to sail for Jacksonville, Florida—which had already changed hands several times—“effect a landing there,” and push forward to Baldwin, a rail junction twenty miles to its west. The Fifty-fourth was to be part of a force of 7,000 men. The soldiers shipped out that night under the cover of darkness, although a movement that large was visible even in Charleston, where General P. G. T. Beauregard guessed that they were headed for Savannah. Twenty-eight ships sailed south, with Hallowell aboard the General Hunter, a transport named for the relieved commander.¹

By comparison to the regiment’s earlier voyages, the short trip down the coast was smooth. The first two steamers entered the mouth of the St. Johns River and began the twenty-five-mile trek upriver to Jacksonville. Just as the vessels rounded the point above the town late in the afternoon on February 7, about fifty Confederate cavalrymen rode into sight and fired a volley at the General Hunter, wounding its mate. “Every man began to load without orders and rushed for the gangway to get on shore,” Gooding reported. Soon “the men were rushing pell-mell up through the streets” in pursuit. Most of the Confederate riders escaped, but the Fifty-fourth captured thirteen prisoners. Hallowell posted seven companies around the town as advanced pickets, with the black soldiers supported by mounted riflemen from the Fortieth Massachusetts Infantry.²

None of the men was impressed with Jacksonville. After being abandoned by Southern forces, occupied by U.S. soldiers, and then evacuated once more, much of the town was in ruins. “Poverty, ignorance, filth, fleas, alligators and rebellion encompass the State from the bar on the St. Johns to Apalachee Bay,” groused a journalist traveling with Seymour’s troops. Nor were the locals welcoming. “The faces of the ladies in Jacksonville indicated a sort of Parisian disgust as the well-appointed Union army, composed in part of Lincoln’s ‘niggers,’ filed through the streets,” Gooding observed. But the men maintained a “respectful silence” and refused to respond in kind. If they acted the part of professional soldiers, Gooding hoped, the Floridians might respond with “respect and courtesy.” Whatever the behavior of the women who watched the Fifty-fourth march by, their reaction to seeing black men in blue was no worse, Gooding mused, than what they faced while on leave “in some parts of the free North.”³

Having secured Jacksonville, the Fortieth Massachusetts and five companies from the Fifty-fourth followed the railway west toward Baldwin. Hallowell ordered a dozen men to secure the telegraph lines, while others found themselves back in the familiar role of fatigue duty, chopping and hauling the wood “kept constantly on hand” for arriving locomotives. On February 8, about ten miles into their march, the vanguard stumbled into an equally surprised Confederate encampment. The “sable sons of Africa charged with a yell that sent utter terror and confusion into the hearts of the enemy,” one soldier later boasted. The two regiments captured a rebel battery of five guns, more than 100 horses, and a large number of small arms. The Fortieth pursued the fleeing Confederates, and roughly 100 threw down their arms and promised to take an oath of allegiance to the United States.⁴

Finally, on February 13, the soldiers, all “well tired out,” arrived at Baldwin, an unimpressive hamlet but a junction of four railroads and so, one captain remarked, “quite an important place to hold.” While the companies awaited further orders, the men constructed tents from rail timbers and rubber blankets. Then the skies opened, and the makeshift shelters proved inadequate. “Here’s two inches of water in my boots,” a soldier complained. “No tea or coffee, no meat, no bread.” Worse yet, Gooding’s hope that their polite efficiency might bring out the latent Unionism in the local population proved misplaced. While patrolling the streets, one white officer happened upon a hostile woman who warned him that he was “in a terrible position.” When he paused to ask why, she snapped that “if you are taken prisoner, you will surely be hung because you command nigger troops.”⁵

WITH THE SIEGE OF CHARLESTON CONTINUING TO REQUIRE NO INFANTRY support, Gillmore decided to reinforce the troops already in Florida by ordering the Fifty-fifth and Colonel James C. Beecher’s contraband regiment, the First North Carolina, to sail for Jacksonville. Orders arrived to break camp on February 10; in a lament common to soldiers everywhere, George Garrison was surprised by both the abrupt departure and Gillmore’s disinclination to inform them whether they were “to remain in that place or go on some expedition after getting there.”⁶

The steamers docked at Jacksonville on the afternoon of February 15. Colonel Alfred Hartwell marched the regiment through town and reported to Seymour, who, unsurprisingly, thought the best use of the men was in fatigue duty. The regiment bivouacked seven miles outside Jacksonville at Fort Finnegan, an abandoned Confederate camp the soldiers promptly renamed Camp Shaw. “Our officers and men place but little confidence” in Seymour, one soldier admitted, as the general was known to be a close associate of conservative former general George McClellan, who was running for the Democratic presidential nomination. But the newer recruits of the Fifty-fifth were pleased to be near the seasoned men of the Fifty-fourth. The former “are soldiers, and only care to be treated as such,” Fox confessed, “while our men are fearful all the time,” having not yet tasted battle. Fox suspected his company yet required “a little stricter discipline.” Private John M. Smith and three others were about to show just how tragically prescient he was.⁷

After deserting from Camp Meigs on July 20, 1863, Private Smith had been recommended for mercy upon his arrest in Boston. The hard-luck soldier, who had either lost his father or been abandoned by him as a child before being raised in a predominantly white poorhouse in Brunswick, rejoined his company on Folly Island in early October. But he spent most of the winter suffering from syphilis in the regimental hospital, where he was dosed with potassium iodide and small amounts of mercury. The twenty-one-year-old was finally ready for duty on January 15, but the other men in Company B thought him one of the worst soldiers in the unit. Charles Fox regarded Smith and his only two friends, Privates Spencer Lloyd and John Wesley Cork, as “known rascals.”⁸

Cork and Lloyd, ages twenty-three and twenty-one, respectively, were evidently old friends. Both hailed from Wilmington, Delaware, and had been mustered in on the same day in May. Lloyd listed his profession as a porter, while a company clerk identified Cork as simply a laborer. As the company marched down the road to Camp Shaw, they passed an isolated house and noticed that the only inhabitants appeared to be a young wife, her infant, and an old man. Cork and Lloyd had been in Florida only two days before they approached Smith and a fourth, unnamed, soldier with their plan.⁹

Aware of the hardships facing Floridians, and especially single women whose husbands were away with the Confederate army, the four approached the house on February 17. They assured the woman, Sarah Hammond—a widow, as it turned out—that the army had “lots of provisions” to spare, which they were disbursing to the local population. With her baby on her hip, Hammond set out for the camp, but she had gone only 200 yards when the four caught up to her. Lloyd asked her “for a hug.” Hammond tried to walk around them and headed back toward her house, yelling as she went, but Lloyd threatened to shoot her if she “hallooed so as to alarm the people at the house.” Lloyd threw Hammond to the ground, and as she told it later, he “then violated my person first—then Private Cork violated me while [Smith] held [her] down.” Hammond’s child was crying, and Smith promised to kill her “and the child too” if both were not quiet. “I cried and screamed and begged them to let me loose,” she later insisted, but Cork and Lloyd held her “down by the arms” while Smith raped her. The fourth assailant, never named, “then violated my person—He then got up and broke and ran ways towards Jacksonville.” The other three shouldered their packs and strolled off toward the camp, leaving Hammond to stumble back toward her home.¹⁰

Just as Hammond was heading for a neighbor’s farm in search of help, Captain J. D. Hodges, a white officer with the First North Carolina, happened by. Sarah rushed to the fence, shouting that “those four men with packs on their backs that has just gone down the road did what they pleased to me [and] threatened me with their guns.” Hodges galloped after the men and followed the three soldiers—the fourth assailant having vanished—back to the camp. As they entered the gate Hodges instructed Lieutenant George Mussey of the Fifty-fifth to watch them while he reported the incident and rounded up a guard of eight soldiers. The posse arrested the men, and with the permission of the camp’s commander, Hodges escorted them back to Jacksonville and delivered them to the provost marshal. Military justice being swift, trial was set for the following evening.¹¹

Around sunset, General Seymour ordered a panel convened “for the trial of four Soldiers of the 55th Mass. Vols., charged with committing Rape upon the person of a white woman.” The four-person tribunal included Charles Fox, who just months before had advocated lenience for Smith as a deserter. Captain William Anthorp of the Thirty-fourth USCT was assigned the unenviable task of defending the accused. Unaware that the army considered abusing local “secesh” women a serious offense, the three regarded the brief proceedings as a mere formality to appease local sensibilities. After Hammond and Hodges provided testimony, Cork rose to ask Sarah if he “threaten[ed] to take your life, if you did not let [him].” Lloyd also implied that Hammond had initiated the encounter, asking: “Did you not pick your own place to lie down?” Hammond responded that Smith had in fact “threatened to shoot me while [Cork] was struggling with me,” and in reply to Lloyd, she insisted that when she refused to lie down, “he then took hold of me and pushed me backwards.” The hearing lasted seven hours, ending at four in the morning of Thursday, February 18. “The evidence here closed and the prisoners having no defense to make,” the clerk scribbled, “the Court was then closed.” The three prisoners were instructed to await the verdict outside.¹²

The court sent a chaplain to find the three, who were sitting under guard by a campfire, “engaged in light and easy conversation.” The soldiers glanced up, reading their fate in the chaplain’s somber expression. “Are we convicted?” one asked. When the minister remained silent, they again argued that Hammond “was particeps criminis”—a willing accomplice—and that the incident hardly “amounted to rape.” As their courtroom demeanor had indicated, “they were astonished that so much importance should be attached to so unimportant an affair.” When the chaplain informed them that they would be executed that evening, Smith was stunned. “Why were we not taught that such consequences would result from such an act?” Smith wondered. Fox then called the regiment together to announce the men’s sentences and to emphasize the importance of keeping “their good name.” Aware that many white Americans regarded black men as barbarous savages, Fox “could not help showing [his emotions] a little in [his] voice.” The ranks were silent, but Fox guessed the regiment “to be on the right side” of the matter.¹³

As the sun began to set, the provost marshal marched the condemned men to an awaiting wagon. “Calmly, and with a firm step, they took their seats in the cart,” the chaplain observed, before he stepped forward to deliver a prayer. General Seymour lived down to his reputation by barking: “Served them right, now let any other man try it if he dares,” as if the three were somehow representative of the nearly 1,000 soldiers in the regiment. A driver carried Lloyd and Cork to a hastily constructed gallows at Camp Shaw, where their deaths were witnessed by the Second and Third South Carolina. Private John Smith, age twenty-one, was hanged before the Fifty-fifth in the town square of Jacksonville, illuminated, Fox recorded, only by “the dim moon-light.” The regiment “could but see the justice of the sentence,” he added. The army left the bodies hanging for twenty-four hours.¹⁴

The executions were widely reported in the Northern press, including in Smith’s Maine, and reaction broke along predictable lines of race and political affiliation. One conservative Manhattan paper ran the headline “Wholesome Hanging at Jacksonville,” maintaining that the three soldiers “shared a righteous fate.” Most Republican papers covered the story without editorial flourish, other than to note that Smith had previously “been pardoned for desertion” and that the three “were among the hardest men in the regiment.” Writing to the Weekly Anglo-African, Private George Stephens conceded that “the base crime of rape [was] second in its heinousness to willful murder,” yet he thought it wrong that “a black man should be hung for a crime if a white man is not treated with the same punishment.” That sentiment was shared by Dr. Esther Hill Hawks, who took offense at General Seymour’s words. “If the same measure had been meted out to white officers and men who had been guilty of the same offense toward black women,” she confided to her diary, Seymour “might have grown hoarse repeating his remarks.”¹⁵

Hawks was more right than she knew. Twenty-one percent of all U.S. soldiers executed by the military were black, although by the war’s end they constituted only 8 percent of the army. In every case where a black soldier was hanged for rape, the accuser was white. Southern whites, of course, had long employed rape as a weapon of terror against black women, and during the war years reactionary whites used the threat of assault to intimidate black women and their families. But the men of the Fifty-fourth and Fifty-fifth understood that John Smith’s behavior endangered their hopes of exchanging military service for citizenship. As one white New York feminist argued, were Congress to give the vote to black men but not to women, the result would be “fearful outrages on womanhood, especially in the southern states.”¹⁶

AS THE MEN SORTED OUT THEIR EMOTIONS THE MORNING AFTER Smith’s execution, they all assumed that they were in for a quiet, lengthy stay in northern Florida. Literally without loss of life, the regiments had successfully carried out Gillmore’s plan, capturing Jacksonville and the rail junction of Baldwin and driving Confederate cavalrymen into the backcountry. Despite being instructed to entrench and await further orders, Seymour wrote to Gillmore on February 17, informing his commander that he intended to take portions of six regiments and destroy the railroad near the Suwannee River, roughly 100 miles from Jacksonville. The thirty-nine-year-old Seymour, a veteran of the Seminole War, thought himself familiar with Florida’s terrain, and he also suspected that he faced only tattered remnants of cavalry and militia units. “Greatly surprised” to hear of the incursion, Gillmore dispatched his chief of staff to Jacksonville “to stop the movement.” Choppy seas prevented the order from arriving at Camp Shaw until midday on February 20, by which time Seymour and the troops would be in the thick of the largest battle waged in Florida.¹⁷

General Beauregard had initially guessed that Gillmore’s intended target was Savannah, but as it became clear that the objective was the Jacksonville area, he moved to bolster Florida’s defenses. General Joseph Finnegan was already posted at Olustee Station with several regiments, and Beauregard ordered General Alfred Colquitt to hurry south and guard the rail lines west of Baldwin. By February 18, Finnegan commanded 4,600 veteran infantrymen, 600 cavalrymen, and three batteries of twelve guns. Serving with the Confederates was Colonel Abner McCormick, who in a reference to the Smith hanging assured his men that the invading force was “made up largely of negroes from Georgia and South Carolina, who have come to steal, pillage, run over the state, and murder, kill, and rape our wives, daughters and sweethearts.” McCormick exhorted his troops to “teach them a lesson,” vowing that he would “not take any negro prisoners in this fight.”¹⁸

The order to move reached Hallowell’s hand on the morning of February 19, and by 8:30 a.m. five companies of the Fifty-fourth were marching behind their colonel as part of the 5,500-man-strong force, a larger number of Union soldiers than were engaged at Wagner. Three New York regiments took the lead, advancing along the bed of the Atlantic and Gulf Central Railroad. Companies from two New England regiments trailed, with Pennsylvania recruits in the Eighth USCT next. Inexplicably, almost all of the regiments in the front were untested raw recruits. As part of Colonel James Montgomery’s brigade, the First North Carolina and the Fifty-fourth brought up the rear in what quickly became a very long column. After marching twenty miles, some members of the Fifty-fourth wondered if “there was any danger of the rebels getting into our rear.” Seymour, unworried, pushed forward and declined to send scouts or skirmishers in advance of his main force. That night the soldiers unrolled their blankets along the track, the full moon, one soldier remembered, “brightening the white, sandy earth” and the “moss-laden limbs of the huge pines which stood sentry-like on the roadside.”¹⁹

Hard marching began again the next morning at eight-thirty. At the top of each hour, the soldiers paused to catch their breath, “but we cannot say the troops fairly rested,” an accompanying journalist reported. The road they followed was “of loose sand, or boggy turf,” and in low areas the soldiers were “covered knee-deep with muddy water.” Noon passed, but Seymour refused to break long enough for the men to gnaw at their hardtack. Having covered sixteen miles, the procession ground to a halt at two o’clock on Saturday, and the Fifty-fourth, “weary, exhausted, faint, [and] hungry,” yanked off their knapsacks and fell to the ground in the shade of some fallen trees. Suddenly, from far to their front, they heard an explosive volley, followed by the roar of cannon. “That’s home-made thunder,” one soldier said aloud. Ned Hallowell was already on his feet, yelling at the men to grab their rifles, when an orderly galloped up, asking to see their colonel.²⁰

The Fifty-fourth was two miles behind the fighting, and Hallowell ordered his men to abandon all gear but their firearms and cartridge boxes and “move forward at the double-quick.” As they closed in on the mayhem, they encountered “hundreds of wounded and stragglers,” many of them from the untested Eighth USCT, who begged them to retreat, shouting, “We’re badly whipped!” and “You’ll all get killed.” Seymour’s advance forces, Hallowell quickly grasped, had blundered into entrenched Confederates concealed behind a thick wood. Although the Confederates, it was later discovered, had five fewer companies than did Seymour, their 5,000 soldiers—3,000 more than had defended Wagner—had begun to dig defensive lines, while other companies started shifting to the side to flank the panicked American troops. While Hallowell brought his men into formation, Seymour galloped up. “The day is lost,” the dumbfounded general shouted, “you must go in and save the corps.” Hallowell pulled out his saber, the familiar metallic hiss lost in the din, and led the Fifty-fourth to the left, while the First North Carolina charged to the right of the Seventh Connecticut. The Fifty-fourth dashed into the melee “with a cheer,” remembered a white soldier, with Company D bawling out their battle cry: “Three cheers for Massachusetts and seven dollars a month.”²¹

Having led his invasion force into a trap, Seymour compounded his errors by maneuvering his advance regiments onto a small patch of high ground between two swamps, one to the front, which hindered their advance, and the other to their rear. Finnegan’s men, who knew the area well, continued to flank Seymour, so that the Confederates fired from both an entrenched front and from the side, with sharpshooters hidden in the trees. Hallowell and his men splashed through 200 yards of swamp, he later reported, “driving the enemy from some guns, and checking the advance of a column of the enemy’s infantry.” As the shattered companies of the Eighth USCT and Seventh Connecticut—most of whom had enlisted only two months before—fell back in disarray, the Fifty-fourth and the black soldiers in the First North Carolina continued to advance, protecting both flanks. Colonel Charles Fibley, commander of the Eighth, took a bullet to his heart, and five Confederate balls killed as many color-bearers. Each time, soldiers from the Fifty-fourth recaptured the Eighth’s flag. To the rear, the Fifty-fourth’s band struck up “The Star Spangled Banner,” and one veteran swore that even above the unceasing musket fire, the “soul-stirring strains” could be heard, inspiring “new life and renewed energy into the panting, routed troops.”²²

To better make sense of the chaos swirling about him, Hallowell mounted a stump about fifty feet behind the front. Despite the danger posed by Confederate sharpshooters, he could not resist being proud of his regiment. One soldier glanced up and “observed Col. E. N. Hallowell standing with a smile upon his countenance, as though the boys were playing a small game of ball.” The determination of the Fifty-fourth encouraged the battered New York regiment; although nearly out of ammunition, they fixed bayonets and raced forward to join the black troops. Some of the newer black recruits, who had joined after the fight at Wagner, were so nervous that they forgot to pull the ramrods out of their muskets and sprayed the Confederates with both balls and short spears. A sergeant bawled for those who had lost their ramrods to drop the balls down the barrels and then hammer the stocks hard against the ground. Not only was the tactic successful, but some of the veterans realized that they could load and fire faster using that method than with a ramrod.²³

The firefight continued for more than two hours, the Fifty-fourth standing their ground and pouring withering fire into the gray line. “The musketry firing had now increased to one loud, continuous peal, amid which was heard the rapid crackling of guns,” one officer recalled. But after firing nearly 20,000 cartridges, they began to run short. Hoping to drive back the Confederate left flank, Lieutenant Colonel William Reed of the First North Carolina led his regiment in a charge. The outnumbered black soldiers got within twenty yards of the Confederate line before a murderous volley took Reed’s life and forced them to retreat. From his position on the left, Private Joseph Wilson of the Fifty-fourth squinted across the field toward the North Carolinians. “Men fell like snowflakes,” he later wrote. Grasping their banner in his left hand—his right hand all but shot off—the First’s color sergeant led the survivors back toward American lines. Colonel James Montgomery turned to the officers nearest him, urging them to save themselves. “Now men, you have done well,” he sighed. “I love you all. Each man must take care of himself.” The Fifty-fourth had never cared much for Montgomery or his instructions, and nobody, one soldier sneered, planned “to retire in his bushwhacking way.” Sergeant James Wilkins, a twenty-one-year-old housepainter from New Haven, instead shouted “Rally!” and his company shifted behind him, forming a neat line along the dirt road. We would “not retreat when ordered,” a member of the Fifty-fourth later bragged.²⁴

In this 1894 lithograph of Olustee produced by the Chicago firm of Alexander Allison and Louis Kurtz, black troops march across a cleared field toward entrenched Confederates, while smoke rises into gently rolling hills. In reality, the battle was waged in a pine forest, neither side had the opportunity to fortify the flat, swampy ground, and the rail line was not as close to the fighting as depicted. Courtesy of National Park Service; photograph by author.

But they would listen to Ned Hallowell, however reluctantly. By dusk, he understood that if they remained on the field, his men would be either killed or taken prisoner. The First North Carolina continued to fire, but their methodical retreat exposed his right to Confederate horsemen. “Both flanks were being folded up,” a soldier observed, “and slaughter or capture would have been the inevitable result.” Seymour decided to pull out and limp back toward Baldwin. The conservative general who had never had any use for black soldiers admitted that the Fifty-fourth “was the only colored regiment that was worth a d[am]n,” and he ordered Hallowell’s men, together with what remained of the First North Carolina and the Seventh Connecticut, to form a rearguard to protect his retreating forces. Even then, one soldier insisted, their blood was up, and “the men had no idea of obeying the firm order.” Hallowell had to tell them three times to leave the field. “It was a sorrowing spectacle to see our little army, so hopeful and so gallant,” wrote Private George Stephens, “in such a precipitate retreat.”²⁵

Their retreat back down the tracks “was steady and cool,” one soldier maintained, but the danger was far from over. The Fifty-fourth paused to fire every 200 yards, the black smoke briefly enveloping the kneeling men. The Confederate infantrymen chose not to follow, but cavalrymen on their right continued to shoot, and sporadic cannon shot fell into their ranks. A stray ball creased Sergeant Swails’s right temple, gouging a furrow “two inches long and fracturing the skull.” More concerned for his company than his own well-being, Swails herded his men toward Baldwin until he grew insensible and “fell exhausted beside the road.” Captain Lewis Reed, age twenty-one, bundled the bleeding sergeant into a nearby cart. Swails “deserves special praise for his coolness, bravery, and efficiency during the action,” Hallowell later reported.²⁶

So ended the fight at Olustee, the bloodiest battle fought in Florida during the Civil War, as well as Lincoln’s hopes of having a restored government for the state in place in time for the November elections. It was another defeat for the Fifty-fourth, but another grim test of their mettle, and so another victory for their growing reputation. Captain Luis Emilio recorded total American losses at 193 killed, 1,175 wounded, and 460 missing, for a total casualty list of 1,828, or 34 percent. Ten more later died of their wounds, making total U.S. losses at Olustee approximately the same as at Wagner. The Fifty-fourth suffered three officers wounded, 13 soldiers killed, 63 wounded, and eight missing. The First North Carolina paid a higher price, with 230 casualties, as did the undertrained recruits in the Eighth USCT. Confederate losses were far lighter. Ninety-three men died, 847 were injured, and six went missing, a casualty rate of 19 percent. “The losses of the 54th Mass. were not severe,” Fox assured his wife. Yet the clash proved devastating for one family in New Bedford. Our “hearts are saddened by mournful events and our eyes moistened with tears of grief,” a soldier wrote to the Mercury and the Weekly Anglo-African. “Corp. James Henry Gooding lost his life in that battle.” Gooding was in the thick of the melee, “carrying the flag on to victory,” when he was shot, the soldier reported. “He enlisted from the highest motives, believing the destiny of the black man of his country was in this rebellion,” as he believed “that now was the day of salvation.”²⁷

The Fifty-fourth reached the encampment near Baldwin around one o’clock on Sunday morning, just as companies from the Fifty-fifth, led by Sergeant Major James Monroe Trotter and including Private Nicholas Said, began to arrive from the east after marching more than twenty-four hours. “Here a scene that beggars description was presented,” one soldier wrote. “Wounded men lined the railroad station, and the roads were filled with artillery, caissons, ammunition and baggage wagons, infantry, cavalry, and ambulances.” Still leery of a Confederate counterassault, the Fifty-fourth secured the disordered withdrawal by securing the road that ran beside the rails. “Exhausted as was our black regiment,” Captain Wilkie James bragged, “it was still necessary to cover the retreat of our men as best they might.” After marching 110 miles and taking part in a bloody battle, the soldiers reached Camp Shaw near Jacksonville on Monday afternoon, but not before having to lash their horses to a disabled train filled with the wounded, which they dragged for the last ten miles of the retreat.²⁸

Although all present praised the Fifty-fourth for the unhurried, orderly fashion in which they protected the rear of the retreating forces, there had been no time to collect the bodies of the slain or assist those who were badly wounded and assumed to be dying. The “greater part of the wounded and all the killed were left on the field,” one soldier lamented, and a reporter for the Savannah Republican agreed that both the “dead and wounded” of the retreating army “lay thick on the field.” Before the battle, Colonel McCormick had threatened to give no quarter, and his men hastened to make good on that warning. One young Georgia cavalryman reported hearing shots coming “from every direction” after the battle was over. When he inquired of a Confederate officer what the cause of the firing was, the man coolly replied: “Shooting niggers, Sir.” The officer explained that he had tried “to make the boys desist but I can’t control them.” Yet another officer conceded that after the Fifty-fourth were gone, “our men slayed the Negrows & if it had not been for the officers their would not one of them be spaired.” After several junior officers sought to put an end to the executions, Confederate infantrymen simply switched to quieter methods. The “boys went over the battlefield and knoct the most of the wounded negros in the head with lightwood knots,” Private Joab Roach explained. The brutal retaliation was both revenge against black soldiers who fought them as equals and a lesson for those slaves yet at home. One soldier urged his wife to tell their slaves, “if they could have seen how the negroes are treated” at Olustee, “I think that would cure them of all desire to go.”²⁹

In the aftermath of the battle, Seymour was skewered by the Northern press for leading his troops into a trap, and Gillmore, having no desire to take the fall for the disaster, publicly announced that Seymour had defied his orders by leaving Baldwin. Gillmore also leaked the rumor that he intended to have Seymour arrested, and while he did not follow through, he did make sure that Seymour understood that his career was over. Gillmore ordered General Israel Vogdes, who was then taking part in the Charleston siege, south to Jacksonville to take over Seymour’s division. Not surprisingly, conservative editors contended that the fault lay not with the invasion’s commander but with his black troops. A lieutenant from Rhode Island reported that it “was our misfortune to have for support a negro regiment, who, by running, caused us to lose our [five abandoned artillery] pieces.” The comment was picked up by the Democratic press, which railed against “the cowardice of negro regiments” and used the debacle as “conclusive proof that the blacks are unfit for soldiers.”³⁰

The Republican press fired back, with the Hartford Connecticut Courant writing, for instance, that “[o]n the battle-field [the black soldier] is fast vindicating his claim to the respect of the world.” But now, for the first time, a large number of whites—many of them self-identified as previous skeptics—were willing to say the same. “Had it not been for the glorious Fifty-forth Massachusetts,” one soldier ventured, “the whole brigade would have been captured or annihilated.” While other companies panicked, he added, the Fifty-fourth “was the only regiment that rallied, broke the rebel ranks, and saved us.” Even Seymour admitted that the “colored troops behaved creditably—the Fifty-fourth Massachusetts and First North Carolina like veterans.” (Anxious to parcel out blame, the general instead pointed to the Seventh New Hampshire, “a white regiment from which there was every reason to expect noble service.”) Yet another white soldier marveled at the Fifty-fourth’s refusal to “retreat when ordered.” If the Fifty-fourth “has not won glory enough to have shoulder straps,” he wondered, “where is there one that ever did?” The Amherst Farmer’s Cabinet agreed: “They fought gallantly and lost heavily, sustained the reputation they had gained at Wagner.”³¹

Perhaps the strangest praise came from three Confederate deserters who wandered into the Fifty-fourth’s camp a month after the battle. Deciding their war was over, they arrived at the garrison in search of food, raising their hands in surrender. The “three rebs” hardly cared for their black captors, but neither did they wish to fight on against them. “You black soldiers fight like the devil,” one admitted. “It is twice we met you. Once at James Island, and the other day at Olustee. We know all the Massachusetts flags. You peppered us like hell.” Coming from battle-hardened Carolinians, such compliments meant a good deal to men far from their homes and families. The exchange, which the soldiers were anxious to share with the Weekly Anglo-African, might help erode “prejudice, the curse of the North as slavery is the curse of the South.” If so, one wrote, “then we will suffer more, work faster, fight harder, and stand firmer than before.”³²

ONCE THEY COULD BE SHIPPED UP THE COAST, THE WOUNDED FROM the Battle of Olustee kept the doctors busy at No. 6. Dr. Esther Hill Hawks was stitching a wound in a soldier’s arm when he looked up. “Is you Mrs. Hawks,” he asked. “There is only one woman in dis world who could do that.” The soldier identified himself as Henry Krunkleton, the eldest of the five Pennsylvania brothers to enlist. Hawks had cared for his three brothers the previous July, when two were wounded on James Island and one at Wagner. “My brudder told what you did for the 54th soldiers in the hospital.” Upon inquiring, Hawks discovered that Henry had remained in Mercersburg to look after the family farm. But after his brother Cyrus was killed on James Island, Henry “had come out with the 55th.” He lived to return home and marry his fiancée.³³

Also resting in No. 6 was Acting Sergeant Major Stephen Swails, although not for long. The bullet had cracked his skull along his left temple, and the only remedy then available was time and patience. The doctors judged his wound to be “severe but not mortal,” and Gillmore granted Swails a two-month furlough to convalesce. Mid-April found him boarding the Elmira train for Manhattan and then transferring on to Boston for shipment back down the coast. (The army deducted $9 for transportation costs from the salary he still refused to accept.) While in Elmira, Swails presumably visited Sarah Thompson and their children in nearby Corning, but as always, he evidently had no interest in legalizing their relationship.³⁴

On the battlefield, however, he displayed no ambivalence. Now thirty-two years of age, the former wastrel had discovered that he was a skillful leader of men, a gifted practitioner of the dark trade—as did his colonel. Shortly after Olustee, Hallowell wrote to Governor Andrew, insisting that Swails had “fought so splendidly” that he deserved promotion to second lieutenant “whenever he recovers from his wounds and rejoins the regiment.” Weary of battling with Washington over pay and doubting that the Lincoln administration would soon consent to promote a black man, the governor decided his title gave him power enough over state regiments, and on March 11 he agreed. Swails was informed of his promotion when he reached Jacksonville on May 12. His men were overjoyed. “Swails, colored, has received his commission as a 2d lieutenant,” one informed the Weekly Anglo-African. “That is a starter.” The men in the Fifty-fourth saluted him as a commissioned officer, but as Andrew had feared, General John Foster, the latest commander of the Department of the South, refused Swails’s “muster into service in that grade” unless explicitly granted authority to do so by Washington, “he being partially of African blood.” The army continued to pay Swails as a sergeant major, a moot point so long as the entire regiment declined wages.³⁵

Foster was undoubtedly being honest about awaiting orders from Lincoln or Stanton, as his post had evolved into an unstable, rotating position; over its three-year existence, six men had held command of the Department of the South, with David Hunter and Gillmore each serving twice. Yet Foster’s response also masked his genteel racism. “Sergt Swails is so nearly white that it would be difficult to discover any trace of African blood,” he remarked. Maybe, Foster reasoned, that explained why Swails was “so Intelligent.” Charles Duren, himself a second lieutenant in the Fifty-fourth, agreed, finding it unfortunate that Swails had decided not to pass for white at the war’s start. “About Sergeant Swails there is some doubt about his being a black man,” Duren wrote to his father in Maine. “I know he is not black, but I mean a negro,” he continued, trying to puzzle his way through his confusion. “White or black he is brave” and “qualified for the position of an officer.” Still, Duren could not bring himself to support the “mixing [of] colored officers with white.” Perhaps the answer was the complete segregation of America’s armed forces, at which time Duren was “ready to step out” of the Fifty-fourth and turn the regiment over to black officers.³⁶

Just as Swails was in transport down the coast from Boston, a curious note appeared in the New Bedford Republican Standard. “‘James Goodrich,’ Co. C., 54th regiment, reported a prisoner in the hands of Florida rebels, is supposed to be James H. Gooding, of this city, who was said to have been killed at Olustee.” For Ellen Gooding and her parents, this glimmer of hope, coming two months after the battle, was almost too unlikely to credit. But corroboration came within the week, as the Weekly Anglo-African included Gooding in their list of wounded, placing his injury in the hip. Normally the last to know about battlefield losses, the War Department listed Gooding as being “wounded in thigh,” although they mistakenly bucked the young corporal back down to the rank of private.³⁷

The reports were not wrong. Gooding was alive, if wounded, and was among the eight men missing from the Fifty-fourth, as were George Brown, a twenty-year-old farmer from West Chester, Private Isaac Hawkins of Medina, New York, and one of the regiment’s three George Washingtons, this one a Philadelphia-based mariner. The prisoners also included soldiers from the Eighth USCT, the First North Carolina, and the Seventh Connecticut. For several weeks, Confederates held the Olustee wounded in Tallahassee, but as they improved, survivors were packed into boxcars for shipment to Camp Sumter, a prisoner-of-war camp adjacent to Andersonville, Georgia. Eleven black soldiers arrived on Sunday, March 27, Gooding apparently among them. Given Davis’s edict and the Confederate Congress’s resolution, a colonel responsible for their induction questioned General John Winder, the officer in charge of prison camps, as to whether the captured blacks should be treated as soldiers or fugitive slaves. Winder replied, as had so many before him, that they should be treated as ordinary prisoners of war until he heard from his superiors. In the meantime, the black soldiers were segregated into a small encampment near the fort’s south gate.³⁸

Built only several months before, in February 1864, the prison camp originally covered roughly sixteen acres, although during Gooding’s time there its fifteen-foot-high walls would grow to encompass another ten. The first contingent of 860 U.S. soldiers had arrived on February 15, but by the summer’s end the total number reached 31,693. Winder appointed Henry Wirz, a Swiss-born captain, as the camp’s commander. The Confederates initially planned to construct wooden barracks, but wartime shortages forced the prisoners to instead put up ramshackle huts made out of sticks, clay, and tattered tents. The wounded were treated in thirty-five large tents judged too worn for army use. Medicine was nonexistent. By early March, of the 13,218 soldiers who had arrived at the Andersonville depot, 1,026 had died. The conditions appalled even Confederate surgeons, who complained that the beef rations were crawling with maggots and that soldiers suffering from gangrene lacked clean water to wash their sores. Rations were invariably in short supply, and when prisoners reached the final stages of emaciation from chronic diarrhea, the camp’s officers simply cut off their food and allowed them to die. Some corpses lay in the sun for days before burial.³⁹

Much of the manual labor within the camp was delegated to what was known as “the Negro Squad,” comprising the approximately seventy black prisoners. The group was put to work burying the dead in ditches seven feet wide by three feet deep, but only after stripping the deceased of their uniforms and washing their clothing. On one occasion, Wirz fired a warning shot at George Brown, whom he had sent to collect shovels but who did “not run as fast” as the captain wanted. Wirz originally ordered George Washington to be whipped, and for the crime of “carrying [stolen] onions to the sick men of the hospital,” Isaac Hawkins received 250 lashes. Wirz had sentenced Hawkins to 500, but after it became clear that the New York mariner would die before the whipping was finished, the captain ended the torture.⁴⁰

Over the course of the war, Andersonville Prison housed 45,000 U.S. prisoners, nearly 13,000 of whom died, most from scurvy and dysentery. This August 1864 image shows tents and shanties built up to the “dead line,” a light fence twenty feet inside the stockade’s main wall. Any prisoner who even touched the “dead line” was shot without warning by guards. Courtesy Library of Congress.

The prison authorities reserved their greatest animosity for black men who wore stripes on their shoulders and believed themselves to be the equal of Southern officers. Wirz especially disliked those he dubbed “white negroes—mulattoes—in the prison.” That meant that light-skinned Corporal Gooding faced hardships unique even by the violent, heartless standards of Andersonville. Either because of his wounded thigh or owing to his pride, or both, the young sea cook—who had braved arctic winds and drafted letters of complaint to presidents and governors—refused to labor with the “Negro Squad.” For this offense, he was beaten so badly that his body was broken and limp when his tormentors finally dragged him away. James Henry Gooding died on July 19, 1864—a year and a day after Wagner—five weeks shy of his twenty-sixth birthday. This time the army would not hear of his fate until May 22, 1865. Wirz, for his part, would be executed for war crimes the following November.⁴¹

“WITH A RETURN TO THE MONOTONY OF CAMP THE QUESTION OF PAY again became a source of discontent,” Luis Emilio observed. Friendly Northern editors reported that just before Olustee, Hallowell had promised that he would refuse to allow his regiment “to fight as they had received no pay.” But when the orders to march arrived, William Lloyd Garrison wrote, “there stood the old 54th with levelled bayonets.” Even so, the fighting at Olustee aggravated their grievances. They had fought bravely, more so than some white regiments, yet those white units continued to receive higher pay. Emilio was right in thinking that growing discontent over the seemingly endless pay issue was eroding morale in both regiments. The Fifty-fifth had now been in the field for six months, and Charles Fox guessed that “much suffering must exist among” their families back home. George Garrison also noticed that men in his company were “beginning to show signs of insubordination,” warning his mother that “if they are not soon paid you need not be at all surprised if you hear of some serious outbreak.” Just prior to the battle, Stephen Swails had sparked talk among the regiments by demanding to either be paid as a white soldier or be released from duty, as Garrison agreed that “we ought at once to be mustered out of the service by the Government.” Unless Congress resolved the matter quickly, “our efficiency as a Reg’t will be at an end.”⁴²

The inevitable explosion came in late May. A detachment from Company A refused to turn out for guard duty. “Great turmoil” shook the Fifty-fourth, one officer wrote. Hallowell ordered his captains to load their pistols, and then he shouted for the regiment to fall in. Instead, the men continued to loiter about, and the commotion drew spectators from other units. A frustrated Hallowell adopted a balanced policy of coercion and conciliation. Drawing his revolver, he marched down the line of resisting soldiers, pointing the barrel at each of their heads while barking: “Do you refuse to go on guard?” One by one, the cowed men shouted: “No, sir!” The crisis momentarily averted, Hallowell then promised the men that if they would continue to perform their duty, he would leave for Washington to plead their case.⁴³

Aware that the army had little interest in fending off angry colonels, Hallowell bypassed General Foster and sought out one of his subordinates for permission to travel, hinting that he had already received “authority from the War Department” to travel to Washington “on important military business.” Even with a pass in hand, Ned was risking severe reprimand, but he was desperate, and in any case, fighting Quakers like the Hallowell brothers had long ignored secular authorities in pursuit of a moral good. Worried junior officers wished him luck. Hallowell “left here yesterday for the North to hurry up the payment of our two Reg’ts,” George Garrison told his mother.⁴⁴

Compounding Hallowell’s anger was the fact that he and the white officers in the Fifty-fourth had not been paid since November 1, 1863, the paymaster apparently thinking that nobody in the regiment would accept their wages. Hallowell demanded that Washington “send some one with funds to pay them,” adding that it might “save useless trouble to state that the enlisted men will refuse to take pay unless you are prepared to pay them according to the terms of their enlistment.” The soldiers knew of their colonel’s efforts on their behalf, but Hallowell’s labors did little to dampen their anger, especially as the government’s decision to charge black soldiers for uniforms resulted in a growing debt. One private in the Fifty-fifth calculated that the “average amount of clothing charged to each man is $53,” which the War Department continued to deduct from their accounts. Even worse, when the Fifty-fourth was ordered to Olustee, they took only “what they had on their backs.” Upon returning to Jacksonville, they discovered that much of their clothing and equipment had been accidently destroyed by a fire—yet the government planned to charge them for new uniforms. “In many cases the men’s clothing account amounts to so much, that when pay comes but little will be due them,” Ned complained. “It seems that the 54th is bound to fight its way through everything.” Most of the soldiers, as one groused, came to doubt they would live to see “the day when the government will do justice to the 54th and 55th Regiments, and pay us what is justly our due.”⁴⁵

Each mailbag brought plaintive letters from home, reminding the troops that the problem extended far beyond their perceived rights as soldiers of the republic. “Is it not perfectly delightful,” one private derisively asked, “to hear your wife and darling babies crying to you for the necessities of life.” A desperate wife wrote to beg her husband to accept any pay, however modest, as she “had been driven by want and the cries of her children for bread, to yield to the tempter,” that is, to sell her body. How “our families are to live and pay house-rent, I know not,” lamented another soldier. James Monroe Trotter was not yet married, but the sergeant major knew what was being said around the Fifty-fifth’s campfires. “That the American Congress should regard us so indifferently makes us sometimes sad,” Trotter reported to a friend of the governor’s. But that was nothing by comparison to the sorrow of “thinking of the necessities of a dear wife and little ones and other beloved ones at home.” If the men in his company remained patient, he decided, it was only thanks to “Jehovah, who will not suffer this war to end until every trace of Slavery is gone.”⁴⁶

As the temperatures rose, so too did the men’s anger. It helped little that both regiments were ordered to return to the Carolina coast. Lincoln had promoted Ulysses S. Grant to lieutenant general, handing him command of all Union armies. After Grant assigned General William T. Sherman the Division of the Mississippi, Washington believed that their combined efforts would place pressure on the Confederate interior and weaken the defense of Charleston. The War Department ordered Gillmore to take 16,000 men and leave the coast to participate in the march against Richmond, while the two Massachusetts regiments, joined by the newly formed Eighth USCT, would return to Folly and Morris Islands. Of this larger, grand strategy, the men knew nothing, only that their unpaid sojourn in the heat and sand was to continue.⁴⁷

In early June, Lieutenant Thomas Ellsworth of the Fifty-fifth ordered his company turned out for inspection. Among those caught unprepared was Private Wallace Baker, a Kentucky-born farmer. Although he had seen action at Gettysburg with the Second Massachusetts, the youthful Ellsworth was regarded by his men as “one of the new batch of Lieutenants promoted from some white regiment.” Ellsworth ordered Baker back to his quarters, but the private refused. “I shan’t do it,” he snapped, “I’ll go to the guard house first.” Had wiser heads been present, the spat might have been defused. But Ellsworth, all of twenty-three, feared his authority was in danger, while Baker, equally young and aggressive at nineteen, was weary of taking orders from sons of privilege. Ellsworth was drawing his sword when Baker landed “two violent blows” to his face, shouting: “You damned white officer, do you think that you can strike me, and I not strike you back again?” Baker lunged for Ellsworth’s sword, unsuccessfully, and as two black sergeants ignored the lieutenant’s pleas for assistance, Ellsworth finally subdued the smaller man and dragged him off to the guardhouse.⁴⁸

Unlike John Smith, who was surprised by his sentence of death, Baker knew exactly what was coming. By the war’s end, black men constituted one in thirteen U.S. soldiers, but nearly 80 percent of those executed for mutiny. Even Boston newspapers denounced Baker as “a hard character.” When Chaplain John Bowles went to tell Baker of his fate, he found him “sitting on the side of his bunk in his shirt sleeves with quiet indifference.” Bowles tried to convince Baker to kneel and pray together, but the condemned man instead suggested that the chaplain might simply pray for him. “I came out here to fight the rebels and I would not mind being killed in battle,” he told Bowles. But he had “done nothing worthy of death, and they might shoot and be d[amne]d! God would make it all right.” Baker did ask that somebody write his mother in Kentucky “and let her know what had become of him.” Should they ever get paid, he added, his salary should be sent to her.⁴⁹

The morning of June 18 found Baker calm and prepared, one witness observed, to meet “his death with stoical indifference.” A detail of white soldiers from the Fifty-fourth New York marched him down the beach on Folly Island “into an open space of ground, where the grave was already dug.” The entire regiment had been ordered to observe the execution, and to several of his friends, Baker shouted, “They have me in a pretty tight place today, boys.” Baker took a seat on his coffin, with his hands tied behind his back and a blindfold covering his eyes. The Reverend Bowles dutifully reported that Baker “fell pierced with five balls,” including two to the head. “The Government which found no law to pay him except as a nondescript and a contraband,” Governor Andrew noted mordantly, “nevertheless found law enough to shoot him as a soldier.” A private in the Fifty-fifth agreed. “As the Government has refused to pay us,” he insisted, one might “argue from these premises that the court had no jurisdiction in the case.”⁵⁰

At long last, embarrassed by the press attention devoted to Baker’s execution, Washington began to act. Though the Massachusetts soldiers did not know it, Governor Andrew had kept up a steady flow of complaints to Lincoln, Stanton, and Attorney General Edward Bates. “I will never give up the rights of these men while I live, whether in this world, or the next,” Andrew assured Senators Charles Sumner and Henry Wilson. The sly politician also used a recent act of Congress against the administration. As a new law allowed Northern states to meet their military quotas by recruiting runaways from Confederate states, Andrew informed Lincoln that Massachusetts was prepared to entice black Virginians north with a generous state bonus, which would dash the president’s hopes of enlisting Southern blacks in the USCT. The governor wrote Lincoln again on May 27, immediately making a copy of his missive available to the Liberator. Black soldiers “have fallen in battle on James Island, in the assault upon Fort Wagner, or in the affair of Olustee,” he reminded the president, so these men, “bearing honorable wounds,” were hardly mere laborers. At the same time, both Ned Hallowell and Alfred Hartwell—despite being instructed not to by General Foster—posted letters to Andrew and Stanton, demanding that their regiments either be paid in full or “be sent back to Massachusetts and mustered out of the service.”⁵¹

As Andrew suspected, Bates desired a quick end to the debate. Despite his Virginia birth and reputation as a conservative legalist, the attorney general submitted a short, almost brusque opinion to the president on April 23. Ignoring the Militia Act, Bates instead pointed to the Second Confiscation Act of 1862, which authorized the president to employ African Americans in any way he saw fit. Bates chose to focus on the case of Reverend Samuel Harrison, the Fifty-fourth’s unpaid chaplain. Harrison was neither a soldier nor a laborer, he reasoned, and so under the terms of the July 1861 Volunteer Service Act the chaplain was entitled to the same monthly salary of $100 paid to white ministers. Although Bates avoided any mention of black combatants, the logic of his opinion pointed to equal pay for all servicemen. As William Lloyd Garrison crowed, the attorney general essentially ruled not only that black men deserved the same pay as whites, but that if “persons of African descent could be lawfully accepted as private soldiers, so also might they be lawfully accepted as commissioned officers.” Bates even encouraged the president to consider it his “duty to direct the Secretary of War to inform the officers of the Pay Department of the Army that such is your view of the law.” If Lincoln desired legal cover, he now had it.⁵²

Yet Congress continued to muddy the waters. After Senator Wilson introduced a resolution to eliminate racially based salaries the previous February, Sumner was able to advance legislation to equalize pay for all soldiers as of January 1, 1864, with the law retroactive to the final months of 1862. Sumner was unable to defeat an amendment, however, favored by moderate Republicans who continued to fret about the costs involved, a complaint they never once advanced regarding white soldiers. The amendment sought to underpay the first contraband regiments. Under the law, finally passed in mid-June, only African Americans who were free as of April 19, 1861—just days after Lincoln called for troops following the attack on Fort Sumter—would be eligible for back pay. “The argument which prevails in the Senate,” William Lloyd Garrison fumed, “is simply that it costs too much to be honest.” And even then, 49 of 184 House members cast a no vote, all of them Democrats and most from the Midwest and the new state of West Virginia.⁵³

Lincoln signed the bill into law on July 4, even though he professed to find the Congress’s distinction regarding back pay and enlistment dates confusing. So too did the attorney general, who confessed to Edwin Stanton that he thought the law “very peculiar in its phraseology,” and he urged the War Department to pay all soldiers according to the promises made at the moment of their enlistments, “and not under the act of June 15, 1864.” Although the vast majority of original enlistments in the Fifty-fourth had been born free—or, like James Henry Gooding and William Carney, pretended that they had been—that was not the case with newer recruits in the Fifty-fifth. When the paymaster issued a circular directing commanders to question each soldier, “under oath,” as to their status, Hallowell supplied the obvious solution. In what came to be known as the “Quaker Oath,” he suggested that his men swear only that they “owed no man unrequited labor on or before the 19th day of April, 1861.” Since neither former bondmen nor white abolitionists believed that slaves owed anything to their previous owners—just the reverse, in fact—the vast majority of black soldiers raised their right hands and swore the oath. By summer’s end, most white officers of black regiments had adopted the strategy. A handful of soldiers, still offended by Washington’s cheapness, refused to take the pledge. “But we have no harsh words for the many who were equal to the occasion by swearing to their freedom,” remarked an approving Pen Hallowell. Those bondmen, such as Henry Jarvis, who had risked their lives to race for Union lines, Pen reasoned, “had overcome too many difficulties in their escape from the South, to be seriously annoyed by this grotesque proposition to swear away their back pay by denying their freedom.”⁵⁴

WITH THAT, NED HALLOWELL CALLED BOTH REGIMENTS TOGETHER to “announce that the question is justly settled,” and that the paymaster would arrive soon. “I pray that soon all will be all right,” James Monroe Trotter sighed. While the troops awaited his appearance, the soldiers idled away the hot afternoons, reading and rereading shared copies of the Weekly Anglo-African. Trotter and Nicholas Said had been teachers during their civilian days, and the Fifty-fifth pitched two large tents, salvaged some “suitable desks and benches” for furniture, and turned the shelters into evening classrooms. Other soldiers poked into holes in the sand to extract the egg sacs of spiders, which one private found surprisingly good. “Wash them down with brackish black coffee.” The Fifty-fifth, another soldier wrote the Liberator, “in all probability, will see no more active service for the remainder of the season, stationed as it is, in its old position on Folly Island.” As ever, soldiers made that assumption at their own peril.⁵⁵

Although the Confederates had abandoned the southern coast of James Island during the siege, the northwestern edges remained vital to Charleston’s defense. Fort Pemberton, a hastily constructed battery of some twenty guns, sat far up the Stono River, guarding the waters west of the harbor. Confederate skirmishers used Pemberton, as well as the smaller Fort Lamar, as bases from which they occasionally harassed U.S. gunners on the lower portions of the island. Most of the time, soldiers from the two Massachusetts regiments exchanged only a few random shots with Confederate pickets. George Garrison and three companies from the Fifty-fifth clashed with a small number of pickets on May 21, driving them three miles north toward Pemberton. The companies suffered no serious casualties, although the “Rebels, so far as could be made out, lost quite heavily in killed and wounded,” Garrison reported. “Three of their officers were shot by our men, and even seen to fall from their horses.” Just days later, Garrison’s company ran into another patrol, but running short of ammunition, the “Rebels occasionally shake their guns at our men, but don’t venture to fire them,” Garrison observed. Reporting back to Colonel Hartwell, Garrison noted that “the rebel pickets have their horses with them, which is a pretty good sign that they have but a small force to oppose us.” Hartwell, in turn, sent word along to General Foster that south of the fort, James Island was lightly defended, at best.⁵⁶

With Fort Pemberton removed, shore guns could be advanced closer still to Charleston, and so, in late June, Foster began to plan for what became the largest military maneuvers of the summer in coastal Carolina. Deciding upon a three-point assault, Foster ordered two divisions to shift west before striking north up the Edisto River and across John’s Island to disrupt the Charleston and Savannah Railroad. At the same time, Hartwell was to march the Fifty-fifth into the central portion of James Island, quite near to where the Fifty-fourth saw their first action one year before, and take Fort Lamar before advancing on Pemberton. Because so many of the officers in the Fifty-fourth were ill or temporarily assigned to other units, Foster instructed the 103rd New York and the Thirty-third USCT (formerly the First South Carolina Infantry) to accompany them, while the navy outside the harbor kept up its fire on the city. For the Fifty-fifth, who had missed most of the fighting at Olustee, the incursion was an opportunity to at last prove their mettle.⁵⁷

On the night of June 30, four companies from the Fifty-fifth boarded boats on the north shore of Folly Island that would ferry them across the inlet to Long Island. Initially, Sergeant Trotter wrote, their landing “was disputed only by the mud and some of them sunk nearly up to their necks in the mire.” The trek toward Fort Lamar involved “plunging, wading and part of the time almost swimming” across a series of swamps, but just as dawn broke they reached dry land. Suddenly, they could hear rifle fire to their front. As they had been at Olustee, the Confederates were fewer in number but well entrenched, and soldiers from both the New York and USCT regiments panicked and began to race by the Fifty-fifth in retreat. Hartwell tried to slow their flight, but when that failed, he shouted to “bring forward the 55th!” Trotter’s company had already fixed bayonets and were charging ahead, cheering and shouting, “Remember Fort Pillow,” invoking the name of the Tennessee fort where victorious Confederates had massacred defeated black soldiers the previous April. The ground to their front was littered with trees felled by the defenders, who added grapeshot to their musket volleys, raking the field before them. “But onward we plunged,” Trotter remembered, “getting nearer and nearer the battery and very soon the enemy seemed to get confused.” When the Fifty-fifth got within 200 yards of the Confederates, they made a final, desperate rush, “yelling unearthly” cries. Having first believed they had won the day, the Confederates realized they were about to be overrun. The skirmishers broke, racing for their horses and abandoning two cannon. Company F reached the cannon first; they spun it around and fired a farewell blast at the retreating Confederates. “O how they did fly!” Trotter would recall. But the Fifty-fifth was weary following two days and nights of “wading through the mud and water.” And with the 103rd and Thirty-third in full flight, Trotter sourly observed, they “had no support, the other Reg’ts having failed us.”⁵⁸

Wounded soldiers were carried back to a battlefield hospital on Long Island, where harried surgeons did their best, one remembered, “in the open air, without tents.” The colonel of the New York regiment reported that the opening Confederate volley “killed seven of my men and wounded many others”; the Thirty-third USCT suffered six killed and thirteen wounded. Hartwell recorded twenty-nine casualties in the Fifty-fifth, including eleven dead and another eighteen wounded. Among the dead was Sergeant Alonzo Boon, an illiterate laborer who had been befriended by Trotter. At Readville, Trotter had impressed upon Boon “the importance of learning, telling him that I would gladly help him, and that when he could read I would recommend him for sergeant.” Boon had taken grapeshot to his leg, and he died while surgeons attempted an amputation. His company buried him near where he fell, hammering a crude board into the ground that read: AS HE DIED TO MAKE MEN HOLY, LET US DIE TO MAKE MEN FREE. Despite Hallowell’s failure to successfully promote Swails in rank, Alfred Hartwell forwarded Trotter’s name to the War Department for promotion to second lieutenant. That did little to ease Trotter’s pain. “No one’s death has made me feel so sad as his,” he admitted. “In truth I loved him.”⁵⁹

The two westward thrusts were also slowed by boggy terrain, and while the three-pronged assault succeeded in driving the Confederates back into the northern edge of James Island, the effort to coordinate the incursions failed. Unable to march north, the weakened Fifty-fifth dug in where they were, and several companies of the Fifty-fourth were moved up from Morris Island to support their lines. Although the skirmish of July 2 was thus of less consequence than the fighting on James Island the previous year, and although it received far less press attention than did the battles at Wagner or Olustee, the hitherto untested troops were justifiably proud. “I say, could you have seen the old 55th rush in,” one soldier wrote the Liberator, “you would have thought nothing human could have withstood their impetuosity. We know no defeat.” Garrison agreed, reporting that “this was the first time this regiment was ever under a hot fire,” and that their task was especially difficult, as “they had to charge through retreating white and black troops in the face of murderous fire.” This time no editor confused the Massachusetts regiment with the faltering troops of the USCT. That criticism, in fact, was made by a soldier in the Fifty-fifth, who complained that “conscript regiments here will not fight.” Had there been more companies from the Fifty-fifth and Fifty-fourth in the skirmish, he believed, “Charleston to-day would either be ours or else we would be at her gates instead of occupying an old post.”⁶⁰

James Monroe Trotter was born to an enslaved mother and a white father who evidently cared enough for James and his mother to liberate them. When Trotter enlisted, he was teaching in Chillicothe, Ohio, a haven for freed slaves where he met Virginia Isaacs, his future wife. He would be the second man of color in the U.S. Army promoted to second lieutenant. Courtesy Cornell University.

The Fifty-fifth was still basking in the glow of their well-deserved laurels when late in the day on September 28 word spread that the paymaster had arrived and would begin disbursements the next morning. Some of the earliest volunteers in the Fifty-fourth had enlisted eighteen months before. Soldiers reached for their fiddles, one officer chuckled, and “songs burst out everywhere.” Veterans of Wagner and Olustee danced about the camp, and “boisterous shouts [were] heard.” It took $170,000 just to pay the men of the Fifty-fourth, each soldier receiving on average $200. Not having been paid for months, Hallowell stepped forward to collect $562.64. Although a few soldiers indulged in what a disapproving Luis Emilio described as “lavish and foolish expenditures,” the vast majority made plans to ship their earnings home, either through Adams Express Company, a freight and cargo business, or one of the chaplains. The Fifty-fourth shipped $100,000 home, while the newer recruits of the Fifty-fifth hurried an equally impressive $60,000 to their families. Some were in debt to sutlers, the civilian merchants who sold provisions to soldiers in the field, and they paid those men off too. Among those on Morris Island working as a sutler was disabled veteran Lewis Douglass. Still awaiting a response to his proposal from Amelia Loguen, he had returned to the regiment he missed and treasured. “I feel more self reliant, more independent than I should had I forever hung around home,” he confided to his fiancée. After collecting the money owed him, both by the army and by his employer, Lewis shipped home $50.⁶¹

Enlisted men and officers alike decided that an official day of celebration was in order. As overjoyed as the soldiers were, their victory was about far more than money in their pockets. “No, it was because a great principle of equal rights as men and soldiers had been decided in their favor,” Trotter reflected, “that all this glorious excitement was made.” The Fifty-fifth’s band led the regiments “toward a piece of rising ground” on Folly Island, where the men adopted a series of resolutions. The soldiers vowed to “stand now, as ever, ready to do our duty” in “crushing this wicked rebellion, and preserving the national unity.” They promised also to forgive those who failed to “appreciate our motives, in connection with the pay question,” being convinced that their critics would one day “see the error of their way.” Most of all, they swore to prove themselves “worthy of the responsible position assigned us by Providence in this, the grandest struggle of the world’s history between Freedom and Slavery.” White Northerners, in the main, had enlisted to stitch their torn nation back together. But from the day they stepped down from the train in Readville, the black recruits had sought to bring on a revolutionary conflict and to “prove our fitness for liberty and citizenship in the new order of things now arising in our native land.”⁶²

AS OCTOBER BEGAN, THE FIFTY-FOURTH WAS GIVEN THE TASK OF guarding Confederate deserters and prisoners of war, not so much as an intentional insult to Southern whites but rather because General Rufus Saxton could imagine “no better officer than Colonel Hallowell” to perform the task. Problems at home in Elmira, on the other hand, continued to bedevil Stephen Swails, who was given yet another thirty-day furlough on November 6. As the War Department was not yet prepared to recognize his commission, Hallowell reluctantly ordered Swails not to wear his lieutenant straps home, but rather “in future wear the uniform of a 1st Sergeant.” Two days after Swails’s departure came Election Day. Most of the men cheered the news of Lincoln’s victory, some because they regarded the Great Emancipator as God’s chosen instrument of freedom and others, still irritated with his administration over the protracted pay fight, because they loathed George McClellan and the Democrats’ conciliatory platform. Hallowell was thrilled. He thanked “God that ‘long Abraham was [president] four years longer’ and that he was an honest man.” Ned confided to his father that he was “very tired of the war” and longed for home, yet he was “determined to see it through.” In northern Virginia, “Grant has the rebel power by the throat,” and to the Fifty-fourth’s west, “Sherman is tearing their vitals out.”⁶³

Hallowell was more right than he knew. Three days after Lincoln’s reelection, Sherman, then in Kingston, Georgia, telegraphed Washington, requesting that General Foster sever the Charleston and Savannah Railroad. That crucial lifeline had been the target of the unsuccessful July incursion by the Fifty-fifth, and Sherman wanted the task accomplished by December 1. Foster handed the assignment to General John Hatch, who began to select the 5,000 men believed necessary. Hatch intended to steam upriver past Beaufort, land at Boyd’s Neck, and then march overland to capture the junction at Grahamville. As part of his Second Brigade, Hatch decided to take companies from both the Fifty-fourth and Fifty-fifth, with Lieutenant Colonels Charles Fox and Henry Hooper—who had been promoted from major after Olustee—under the joint command of Colonel Alfred Hartwell. Although this would be the first time the two units might see combat together, Foster thought it unnecessary for Hallowell, who knew the regiments better than any other officer, to participate, opting to leave him behind with two companies to guard the islands.⁶⁴

Foster hoped that Charleston would not notice the departure of so many men, so he ordered the tents left standing on Morris and Folly Islands. The troops slipped away to Hilton Head late on the night of Sunday, November 27, leaving behind a disconsolate Hallowell. He rode down the beach, “the empty tents looking like ghosts.” With eight companies of his “beloved regiment” gone, he found the camp too quiet; “everything is so lonely,” he complained to his mother. After Hatch and the men vacated the island, Hallowell contacted Foster, begging him “to send some one to relieve me in time for me to catch the regiment before it went into action.” The young colonel had stood with his men at Wagner and Olustee, and the thought that they might see combat without him left him nearly frantic with worry.⁶⁵

His brother Pen, reading about the affair in Philadelphia, later concluded that the two regiments were doomed to be ordered into indescribable folly by senior officers who had been promoted far above their level of competence. And nature was of no help. As the troops sailed up the winding tributaries that poured into Broad River, heavy fog and hidden undergrowth slowed their advance and snagged their boats. Some of the companies were forced to disembark about half a mile below their designated target of Boyd’s Neck; others reached the dilapidated wharf, but “not an army transport was to be seen,” according to one officer, the vessels having either gotten lost in the fog or sailed up the wrong estuary and become grounded in the shallow waters. Yet all was not lost. The railroad was guarded by Confederates stretched thin along its line, and a picket force of roughly seventy-five Southern soldiers was all that stood between the U.S. forces and the depot at Grahamville. Hatch instructed Hartwell’s regiments to bivouac at the Neck, while he scouted ahead with several regiments of USCT and a small contingent of sailors. But his maps were worthless, and twice they got lost, losing valuable time.⁶⁶

In an effort to sever the Charleston and Savannah Railroad line in support of General Sherman’s arrival from Atlanta, Colonel Alfred Hartwell and sixteen companies from the Massachusetts regiments ran into entrenched Confederates at a rise called Honey Hill two miles east of Grahamville (here spelled Grahamsville). For advancing the colors into heavy fire, Corporal Andrew J. Smith became a posthumous recipient of the Medal of Honor in 2001. Courtesy Library of Congress.

Not until daybreak on November 30 did the troops finally begin moving down the main plantation road toward Grahamville. Portions of two New York regiments led the way, followed by the Twenty-fifth Ohio and the Third Rhode Island Artillery. The second brigade followed, led by Hartwell and sixteen companies from the Massachusetts units. But downed and damaged bridges slowed their progress, and by 11:00 a.m., they had advanced only three miles. By then, Confederates in the interior had grasped the objective of the invasion and shifted troops toward the junction. Led by Colonel Charles Colcock, between 1,400 and 2,000 troops dug in along the crest of Honey Hill, two miles east of Grahamville. Although the hill was little more than a rise in the road and the semicircular line of earthworks was crudely constructed, Colcock placed seven cannon atop his lines. The road turned just before reaching Honey Hill, so attacking troops would not spy his earthworks until the last moment. To one side of the road sat an empty field of tall grasses, and beyond that a cotton field. On the other side stood a small, whitewashed church backed by a swampy forest that followed the highway for nearly forty yards, transforming the path almost into a narrow bridge, and one easy to defend.⁶⁷

The advance portions of Hatch’s force walked directly into the teeth of the Confederate defenders. “The enemy came by the former road, and turned the angle apparently before they were aware of an opposing force,” reported a journalist traveling with Colcock’s men. Infantry inside the church opened fire, while others set fire to the high grass, forcing the New York regiments to keep to the road or seek cover in the swamp. Colonel James Beecher, now leading the Thirty-fifth USCT, led his men forward, but “our batteries,” the Southern journalist recorded, “opened upon them down the road with a terrible volley of spherical case.” The regiment fell back in disarray. As at Olustee, the Massachusetts men were far in the rear when they heard the first sounds of gunfire. Hartwell was immediately in the saddle, his sword in one hand, his hat waving like a flag in the other. “Follow your colors,” he bellowed as he galloped to the front, directing companies of the Fifty-fifth up the side of the road toward the guns, while shouting for those in the rear to divide up, with some wading into the swamps and others racing through the still-burning fields. Sergeant Trotter led one of the companies toward the guns. At one point, the path grew so narrow that the men raced only four abreast, making the Confederate use of grapeshot murderously effective. “It was like rushing into the very mouth of death going up this road facing 7 pieces of death dealing cannon,” Trotter remembered. “But when commanded to charge ’twas not [ours] to refuse,” and “in we rushed cheering and yelling.”⁶⁸

As Hartwell galloped toward the hill’s crest, a charge of canister caught him, blasting his sword from his hand and killing his horse. The dying animal rolled, pinning the young colonel beneath it in the mud. A lieutenant ran forward, cut the saddle from the beast, and helped Hartwell to stagger toward the safety of the woods. Two more balls caught him, and Hartwell shouted to be left behind, but two other men turned to help. One was himself then wounded, but the four eventually made it to the trees. The men bound Hartwell’s wounds before returning to the fray. Sergeant Robert King, the eighteen-year-old color-bearer of the Fifty-fifth, Charles Fox later wrote, “was blown to pieces by the explosion of a shell,” but Corporal Andrew J. Smith picked up the banner and advanced toward the hill. As they approached the earthworks, a strange moment of silence was followed by a renewed explosion of “grape, canister, and bullets at short range,” and the survivors fell back in retreat, tripping over the bodies of the fallen.⁶⁹

At the same time, Hooper arrived with two companies of the Fifty-fourth. He had posted the remainder of the regiment along the road all the way back to Boyd’s Neck to secure the column’s rear. Just then, General Hatch’s chief of staff appeared on the scene, looking, Hooper thought, much “excited.” The general demanded that the Massachusetts men again “charge.” Hooper was stunned. “Where?” he asked. “Charge! Charge!” was all the aide could say. Wisely, Hooper avoided the main road, instead leading his men into the trees, warning them to seek cover as they advanced and to use their ammunition as sparingly as possible. Beecher and what was left of the Thirty-fifth followed. “It was a perfect jungle all laced with grape vines,” he remembered. One of those in the second assault later told Trotter that the forest was so “dense and marshy [that] it was almost impossible on this account to maneuver more than half our troops.” As that side of the Confederate earthworks was “very much exposed,” the journalist from the Savannah Republican worried, the black soldiers might have flanked their lines had enough made it “through the swamp and up the hill,” but the bog made for a good barrier.⁷⁰

A few made it across the marsh. Two of those who did were Virginia runaways Peter Drummond and Henry Jarvis, whose company briefly captured one of the Confederate guns. “O God, I’ve got you,” Jarvis shouted in triumph. To his side, Confederate infantrymen spun about and fired. Jarvis took a ball in the arm. “I kep’ on fightin’ till a ball struck my leg an’ I fell,” he said later. He was struck “once more in de same leg,” the second ball shattering the bone. Drummond and Jarvis had been friends since age six; they had run away together, sailed to Liberia together, and enlisted together, and Peter was not about to leave Henry behind. Together with Augustus Brooks, Drummond dragged Jarvis back into the trees. “I should have bled to death ef all our men hadn’t been drilled in usin’ a tourniquet, an’ supplied wid bandages.” The two barely had time to tie up Henry’s leg, stick a knife into the knot, “an’ twist it tight ‘fore I fainted.” As a second retreat began in earnest, Sergeant Trotter was also hit, although the wound was not severe enough to keep him from leading his company back down the road toward Boyd’s Neck.⁷¹

After what had transpired at Olustee, the black soldiers refused to leave any wounded behind. Even so, the next morning the Confederates were stunned by the carnage that lay in front of their lines. The bodies “lay five deep as dead as a mackerel,” sighed a Confederate gunner. The reporter for the Savannah Republican concurred, writing that he “counted some sixty or seventy bodies in a space of about an acre, many of which were horribly mutilated by shells, some with half their heads blown off.” At No. 6 General Hospital in Beaufort, Jarvis “stood it all for to keep my leg,” and the surgeons tried to save it, but “pieces of de bone [kept] a comin’ out,” and at length they sawed it off four inches below his pelvis. American casualties were severe, with 89 killed, 629 wounded, and, despite the best efforts of the troops to leave no man behind, 28 captured. Confederates reported only 47 total casualties, with 8 killed and 39 wounded. Trotter guessed that the Confederate casualties were not that much less than U.S. casualties, “although they try and make it so.” But while that was invariably true, they had, as Trotter conceded, “such natural positions as more than made up for such deficiencies” of numbers.⁷²

The Northern press largely blamed the tragedy on fog, shallow inlets, and poor maps rather than Hatch’s inept leadership. In truth, all played a role in the defeat. But in later years, one soldier concluded “that the Rebels had a trap set for us and we were marched right into it.” Reading of the battle, Pen Hallowell agreed that if Hatch could not be blamed for the miscalculations that took place on November 29, the next day’s order to storm the hill was “not free from the charge of down-right recklessness.” The narrow roadway and the swamps to one side made the assault, Pen decided, sadly reminiscent of Wagner. A veteran of the Fifty-fourth who survived both battles drew a similar comparison. “Wagner always seemed to me the most terrible of our battles,” thought Sergeant Charles Lenox, “but the musketry at Honey Hill was something fearful.” As the sad news filtered back to Morris Island, Ned Hallowell thought the only positive aspect to his missing the action was that he was “alive to comfort those who lived through it.” He understood that had he been there, it might have been him, rather than Hartwell, who lay in a bed in Beaufort, or even in the ground. “It’s very hard Mother to make up your mind that you are willing [to] die,” he wrote home, “but it’s ever so much harder to content yourself at [camp] while others are suffering. Oh Mother I wish I was a boy, so that I might cry out.”⁷³

Hallowell sent that letter in December, by which time the Fifty-fourth and the Fifty-fifth were preparing to celebrate their second Christmas on the islands. Despite their losses, the men rightly took heart in the Northern praise for the way they fought in the face of impossible odds. William Lloyd Garrison anointed the Fifty-fifth as “heroes of all the hard fights that have occurred in the department since their arrival” on the coast. (Privately, Garrison was happier yet to hear from George, who miraculously “escaped without a scratch.”) The festering issue of pay had at long last been resolved, if not before the tragic execution of Private Wallace Baker, a man, unlike John Smith, mourned by his fellows. Though the matter of black promotion and advancement remained a source of discontent, the bombardment of Charleston continued, and from the number of Confederate deserters who wandered into the camps, it was clear that the city would soon fall, and if so, that was in no small part thanks to their sacrifices.⁷⁴

On December 18, however, word arrived that General Sherman was displeased that the rail lines were not yet severed. Both regiments understood that their battles, both political and military, were far from over.