Colored Orphan Asylum burns during New York draft

In her day, Elizabeth Oakes Smith was a national figure. Born in Maine in 1806, she and her family moved to New York City in the late 1830s, where she joined literary circles and emerged as a prominent feminist essayist, lecturer, and poet. By the 1850s, she was living with her family on Long Island. The forces of the Civil War, however, led to her downfall. In 1861, her favorite son, Appleton Oaksmith, was arrested for outfitting old whaling ships for the slave trade. After spending several weeks imprisoned at Fort Lafayette in New York and Fort Warren in Boston, Oaksmith was transferred over to civil authorities and convicted in the federal court in Boston in the summer of 1862. Before he could be sentenced, though, he escaped from jail and exiled himself in Havana, Cuba, where he became a Confederate blockade-runner.

Elizabeth Oakes Smith spent the war years seeking a presidential pardon for her son. She wrote to Abraham Lincoln and met with Secretary of State William H. Seward, but nothing ever came of her efforts. As the war continued, she became increasingly embittered toward the Union and believed that Appleton was the victim of a malevolent administration in Washington, D.C. Although she had shown sympathy for African Americans and abolitionism before the war, her Democratic politics became increasingly evident in her diary by the midpoint of the war. She grew to especially hate Seward.

Elizabeth Oakes SmithElizabeth Oakes Smith spent much of the prewar years championing women’s rights, and was sympathetic to abolitionism. But when her son was arrested for slave trading and she could not obtain his release, she grew increasingly bitter toward the Lincoln administration.

In the summer of 1863, Oakes Smith experienced one of the greatest terrors of her life—the New York City Draft Riots. For several days in mid-July, working- class men, women, and children—mostly Irish Democrats—lashed out against Lincoln’s conscription and emancipation policies. At least 105 people died during the five days of rioting, many of whom were African Americans who were viciously targeted by the mob. During this ordeal, Oakes Smith saw a dense crowd standing around a lamppost, upon which hung the body of Colonel Henry O’Brien, the Irish commander of the 11th New York Volunteers, who had ordered his troops to fire above the crowd the previous day in order to disperse them.

Sadly, O’Brien’s men had killed a mother and her 2-year-old child in the process. Now the rioters would exact their revenge. She also encountered Jeremiah G. Hamilton, a 56-year-old African American millionaire on Wall Street. Her diary, which is held with her papers at the Albert and Shirley Small Special Collections Library at the University of Virginia, is one of the most remarkable accounts of this period. The following excerpts trace her reaction to the Gettysburg Campaign and then the horrific events that transpired in New York City shortly afterward.

June 17
The whole country is in a ferment because of the movement of General Lee north—Harrisburg is threatened—the Govenor [ sic ] of New York has been called upon for troops by the Govenor of Pennsylvania. The people have responded generously to the State call, but desired guarantees of the Govenor that they shall go only on short service—to defend the north—not fight the South—they will not trust the authorities at Washington.

June 18th
The excitement still continues. The people call for Gen. McLellon [McClellan] to head the army of the Potomac. We are now reaping the bitter fruits of the imbecility, and treachery of this Administration. I sometimes wonder how the historian will deal with this period. The Abolitionists have anticipated time by publishing their own version of events, called a History of the Rebellion by [John S.C.] Abbott, which suppresses facts and misrepresents them.

I am startled and distressed at events, and find how little I have understood the world—how far off the ideal seems now—that once was so near: but to one thing I still cling, the intrinsic worthiness of our humanity—the wholesomeness, and upwardness of its attributes—and then comes the question whence comes all this distortion?….

June 19
….Noon. The papers have come—and the panic is subsiding—it seems that Lee is on the retreat. I do not think all this demonstration is for nothing—I believe he designs to attack Washington, and dictate a peace from the Capital….

June 24th
A day of heavy work—found myself worn in body and spirit—cross and miserable.

The country is in a state of ferment, and yet it is suppressed—no faith in our rulers—I see the Govenor of New Jersey has directed the Malitia which was collected for the defense of Pennsylvania, to be disbanded, judging that there will be no occasion for their service. They refused to serve under the Administration….

Lee’s army within 18 miles of Pittsburg. 10 o’clock P.M. Going to bed—dissatisfied with myself—oppressed with a terrible sense of weariness and despondency. I am indeed cast from my moorings, drifting—drifting—whither?

June 25
The Confederates in large force in Pennsylvania—the papers contain some terrible records of savage cruelty and atrocious crime perpetrated by the Black soldiers. The negro in our cities and villages also has started upon a career of crime unparrallelled in our history. We are threatened with a war of races, in which the poor Negro must and will be the great sufferer….

June 27
[Discussion of Lee’s invasion and Confederate privateers omitted.]

I believe this treacherous Administration designs to allow the Confederates to take Washington—having no capacity to manage our national affairs—finding themselves hopelessly involved, they design to let matters take very much their own way, and then under plea of necessity, acknowledge the Southern Confederacy:—then will come the contest for northern supremacy, when the wily Seward hopes to ride into the Presidency of the North….

June 30th
….The Confederates are within four miles of Harrisburg. Gen. Meade has been called to the command of the army of the Potomac—Gen. Hooker relieved probably because of his interference. This is bad, people say, in the face of the enemy—but the Administration will not care to give battle—they wish for a general disruption in public affairs. They have inaugurated the reign of falsehood.

July 1st
….The Confederates are within ten miles of Washington—they do not seem inclined to battle—with a grim wit they say they have come North to hold Peace meetings. So disgusted have the people become with the present Administration that they seem rather disposed to receive, than repel the invaders. Many say openly that the prospect of restoring the Constitution would be better under Jefferson Davis, than under Abraham Lincoln. Immense Peace Meetings prevail through the country, and resistance to the enrollment is the common sentiment….

July 3d
Great anxiety about the expected battle—I am by no means sure that our arms are successful against Vicksburg. The repulse of our troops at Port Hudson was a shocking, and bloody disaster….

July 6th
A terrible battle has been fought at Gettysburg. The Union army is said to be triumphant, with a loss of twenty thousand men—Good God! what horrible carnage—I am sick at the record—my whole soul revolts at this sanguinary conflict. Such a victory is as ruinous as defeat. There must be a compromise—for neither party will yield, and each seems an equal to the other in point of courage and persistency….

July 20th
The past week has been one replete with anxiety and not without incident. On Monday 13th inst. I took the early train for the City, where the most appalling scenes awaited me. Our fool-hardy, despotic rulers in spite of the warning of observant men, and the indication of the masses, that revolt would come, have persisted in enforcing the Conscription Act. On Saturday the action of Draft created no disturbance, although much jeering and derision followed. The City was greatly excited through Sunday—On Monday I went to the City, and what I now describe, I saw myself. I had occasion to visit the offices of several lawyers upon business—as I passed along the streets large bodies of people thronged the corners—a great crowd was about the Tribune Office, and the white faces of the workmen within, with their little paper caps upon their heads now and then appeared at the windows—but retreated at a sort of growl from the crowd which beset them.

I was obliged to call upon places in this vicinity, and was not unwilling to learn by my own observation the exact spirit of the crowd. I saw a respectable working-woman threading her way through the living mass, as I was doing myself, and found it convenient to use her and her basket as an entering wedge. Sometimes it would cross my mind, that if paving stones should take wing my position would be a dangerous one—but I had little anxiety for myself—indeed! let me confess it: own to the truth.

I was intoxicated—drunk with excitement. I said of myself—“Oh thou drunk, but not with wine.” I had seen the people submit to so many arbitrary measures—seen them go like sheep to the slaughter in this stupidly managed war—seen them die without a word for measures repugnant to them—seen the encroachments upon our liberties made daily by this corrupt Administration, and yet the people were silent—bitter—cursing deep, not loud, and I began to lose all hope—I wished I could do…something to rouse them—but nothing seemed able to do this—the people tamely cowered under oppression—the radical Editors lied and deceived them as did the rulers at Washington, and I despaired for American freedom. I knew the draft was repugnant to the genius of the country—I knew that the…burden of the war fell upon our working men, and the clause of exempting those who were able to pay the $300 threw all the burden upon the poor men, still it seemed as if the people would submit—

More on the Draft

But now there was a recoil—five thousand men were up in arms—there was a perfect howl of rage and indignation from the masses. I said to my pioneer of the basket—“what is the matter? what are the people about?”

She gave me a fierce look—“They wont be carried off to the war—that’s what is the matter.”

“Well, would you have your husband carried off in this way?”

“If they do, they ve got to fight me first,” was the prompt reply. She went on with a hard sneering laugh—“Eh! you ladies can pay the $300 and keep your men to home.”

I said no more—passing into Broadway the shops were nearly all closed—the stages had been stopped or converted into conveyances for the insurgents—I walked up to the University building where I found Dr. Elliott, and Edward, on my way I passed several police men, haggard, dusty—exhausted—I said to myself—the mischief works—these insolent ruffians, who have lately fairly trod upon the people—knocking them down, firing upon them in mere wantonness, will now find a check—I grew ruthless in my indignation—for their insolence and cruelty had become a public cry.

About three o clock P.M. Capt Ellott invited us to a dinner at Delmonico in 5th Avenue: scarcely were we seated when the waiters rushed in barring the doors and closing the windows, and there came that great sound as of the sea—the tumult of the people. The rioters had burned down the colored orphan asylum and several other buildings—they paused and for a brief space it was doubtful whether they would not force the building. Soon there was a cry—“there goes a nigger,” and the cruel, remorseless multitude, three thousand strong were in pursuit of the unhappy fugitive. He was without doubt torn in pieces. As we made our way up to 36th street, all was dire confusion—mad uproar—police men, Military, citizens and rioters, in one vast conclave. I was shocked and ashamed to hear these well to do and luxurious people—the denizens of that vicinity urge the fire of the military—there was no expression of pity—no sympathy for the poor laborer, who in his mad vengeance sought a sort of justice—a wild revenge one most true.

Scene of a lynching during New York draft
riotsRioters, many depicted as stereotypical Irish, jeer at a lynched African American on Clarkson Street, by the Hudson River docks. Violence against Blacks was a hallmark of the Draft Riots, and many African American families left Manhattan in the wake of the upheaval.

Early on tuesday morning I was obliged to go to Wall Street. Before noon the outbreak had assumed such proportions that all business was suspended—stages and cars could not run, and the frequent discharges of the military told that hot work was in progress. I tried to make my way up town—I could not get across the City to take the train for home—nearly exhausted I was struggling onward when a carriage stopped in front of me, and J.G. Hamilton Esqr asked me to ride home with him—I did so and remained long enough to rest, and obtained information that struck me to the heart, and which I inwardly resolved to impart to the people.

While resting in the Hamilton parlors, which opened upon 29th st. down which masses of people were constantly passing with the debris of the insurgents, I observed a demonstration which was quite touching. I ought to say that Mr Hamilton is I think an eastern Indian—his complexion is darker than that of a mulatto, but his features are Caucassian [ sic ]—and he is a highly cultivated man—his children also are dark, but with fine black eyes, and long hair in ringlets—not at all [illegible word]. Two of these stepped out upon the balcony as they probably have always been in a habit of doing to watch the passers by in the street. Mr H. sprang from his chair—took them in and closed the window— He understood the hazard growing out of their dark complexions. I shall never forget the expression of anguish upon his face.

Soon after I took my leave, and my Host advised me to get at once into Lexington Avenue as a safer retreat from the crowd. I followed his advice—but made a circuit which brought me into the midst of the insurgents. Oh! my soul—what a sight presented itself! Masses of infuriated women, tossing their arms wildly—weeping women and children, and pale desperate men—pools of blood—broken furniture burning ruins. In a low calm voice I began to talk with the people—I told them what I had just heard that five thousand men would reach the City in the five o’clock train, and they had orders to march at once to this Avenue [2nd] and rake the whole length of it with grape shot—no warning given—the first round would be this iron hail. I went from group to group and told this and urged them to go to their houses. A poor, lank boy of thirteen kept close to me—at length I said to him, “My poor boy—go home—keep out of these dreadful scenes.”

The child burst into a perfect paroxysm of tears and sobbed out—“I have’nt got any home”—a woman explained that his Mother had been for some time dead—the father had been killed in the army—and the child lived upon the kindness of others. Observing a pale desperate looking man leaning against a wall I assayed a word with him—“Madam, (he said in good accent) it may be easy to tell us what to do—but I will not obey this draft. I may as well be shot here as anywhere. Look here—(pointing to blood upon the walk)—the soldiers fired upon us—not a word—no warning—and I took a child up—shot through the head—covered with blood—I looked at him—it was my own child—I will have revenge.”

Death of Colonel O’Brien in front of drug
store Colonel Henry F. O’Brien commanded the 11th New York Volunteers, which skirmished with the rioters. Discovered by a mob in a drugstore, he was dragged out, beaten, hanged, and tortured to death. Oakes Smith’s remonstrances on his behalf were rebuffed. Colonel Henry F.
O’BrienColonel Henry F. O’Brien

At length a great burly fellow eyed me with a savage frown—and muttered between his teeth—“I know what you are—you are one of them aristocrats from the fifth Avenue—you’re a Spy”—looking round upon his followers—“she’s a Spy—sent here from the Black Republicans!” I saw I was in some danger—but I did not flinch—I do not think I turned pale—I repelled the charge in a calm, firm voice, and went on—he following and muttering—but I did not fear—a superhuman strength seemed mine. I knew better than to leave them—so I kept on talking in [a] low calm manner, advising them as seemed best. I had now reached 33d street, the disorder rather increasing. A short distance from me was a dense mass surrounding a lamp post, upon which the infuriated multitude had suspended the body of Col O Brine, an Irish officer, who had commanded his troops to fire into the crowd. It was a sickening sight—for four hours they tortured the unhappy man—prolonging his sufferings with a fiendish fury. I expressed compassion for him—but they justified their conduct on the ground that he was a traitor to his countrymen—

The deep shadows began to overspre[a]d the neighborhood—and waving my hand I turned up fourth street, quite a group following me and thanking me for what I had done.

The people everywhere repelled the imputation that their object was plunder. It was only opposition to the Draft, they said, and disgust at that clause in the act by which the rich man could exempt himself by paying $300 while the poor man was compelled to go.

It is not generally known what gave the first impetus to the Riot against the draft. I was told several times by the people with whom I talked, the Rioters—and they all told the same story.

It seems that somewhere in the vicinity was a widow woman—Irish, who had six sons. Of course the six were enrolled for the Draft, and by a singular fatality—the whole six were drafted—the young men were aghast—a crowd followed them to the Mother’s door; when the announcement was made to the Mother—she uttered a wild cry—“a yell,” the people called it, and rushed shrieking into the street, tearing her hair and tossing her arms above her head. The effect was electrical—and the fierce passions of the people broke out at once, sweeping all before them.

Like a moth to a flame, Oakes Smith ventured into Manhattan during the riots. Some of the locations she noted are numbered here: 1. New York Tribune offices; 2. Delmonico’s; 3. The Colored Orphan Asylum; 4. 36th Street, which she recalled was full of “police men, Military, citizens, and rioters”; 5. After briefly visiting with Jeremiah G. Hamilton on Wall Street, she headed to his home at 68 E. 29th Street; 6. Near this corner, a mob first attacked Colonel Henry O’Brien of the 11th New York Volunteers.

I returned home on wednesday morning. In the cars I found Mr Hamilton—looking haggard and internally excited, but outwardly calm, and determined. I said at once go home with me. He remained nearly a week….Oh how anxiously we waited returns from the great City.

The Radicals have done their utmost to exasperate all classes—in order to have martial law proclaimed in the City. God save us from such calamity—the streets of New York would run blood—and the prisons be filled with men and women suffered to be obnoxious to the powers that be.

this article first appeared in civil war times magazine

See more stories

SubscriBE NOW!

Civil War Times magazine on Facebook Civil War
Times magazine on Twitter

Jonathan W. White is professor of American Studies at Christopher Newport University. In August 2023 he published a biography of Appleton Oaksmith titledShipwrecked: A True Civil War Story of Mutinies, Jailbreaks, Blockade- Running, and the Slave Trade with Rowman and Littlefield.