First thing, I moved out of the South—for good, I thought. I wasn’t bitter or anything like that, I just wanted to be closer to the center of the entertainment business. It’s funny about the South. It did a lot to me, but it also made me what I am. My roots, my religion, and my music all come out of the South. Generations of my family, as far back as I can trace them, lived around one little area in South Carolina. I never even traveled outside the South until that first recording session at King.
I moved my father into the house I had on Ell Street in Macon and bought a big twelve-room Victorian place in St. Albans, Queens, in New York City. I bought it from Mr. Bart and Cootie Williams, the great trumpet player with Duke Ellington. The Ell Street house was available because Dessie and I had broken up.
She was a good woman—we were together for a lot of years—but she did a lot of wrong things. I guess I did, too. We fought a lot. She went through a lot of money, but she didn’t know any better. She often played pinball, the kind that pays off—if you win; otherwise, you lose money. She lost a lot—a lot. She also bought presents for men with my money. Once she bought a present for a fella in the hospital and didn’t even buy one for me. She was a hardworking girl, but she didn’t know any better.
The house in St. Albans needed a lot of work—the basement was full of water, things like that—so I hired a bunch of people to work on it while I went on tour. The move was just one part of several big changes I was getting ready to make. First, though, I wanted to do another live album.
Even before the Apollo record got so big, I was talking to Pop about doing another. I wanted to do it in Washington at the Howard because I figured that was a good audience for me. Fop said, “It doesn’t matter where you cut it, Jimmy, you’re going to have a wild audience now.” So in November, after we left the Apollo with all the long lines and everything, we went into the Royal and cut Pure Dynamite: Live at the Royal. At just about the same time, Atlantic Records was recording all their people live at the Apollo: Otis Redding, King Curtis, Ben E. King, Doris Troy, Rufus Thomas, and the Coasters.
I think Pop was wrong, though. It did matter where we cut the album because we had some acoustic problems with the Royal. The album didn’t sound bad, but it didn’t sound as good as the Apollo recording. Live at the Royal was kind of lost in the shuffle anyway because when it was released in February 1964 the other record was still climbing the charts, hanging in there much longer than anybody thought it would.
A big hit album like that changes things. One thing it was changing was my audience. It was bigger, naturally, and the racial makeup was changing. Before that, the audience was almost exclusively black. A few hip whites came, but not many, not like they did for a rock ’n’ roll singer like Richard, say. I remember one young man, a white kid, who slipped backstage at a gig in Florida in 1959 when I was still scuffling. He knew everything on the Please album. I couldn’t believe it.
In 1963 there must’ve been a whole lot of white kids like him because the crowds at places like the Maryland beaches were about 35 percent white, and I know the Apollo album wasn’t being played on white radio stations. For a performer, getting popular like that brings pressures and opportunities at the same time. Lots of times, it’s hard to tell ’em apart.
There was talk about changing the show. Some people thought we should put in more stuff for the white audience. Pop was against it from the getup. “Listen,” he said, “the whites are coming to see exactly what you’ve been doing for blacks, the gutbucket stuff. You’ve discussed this only with black people. You don’t know how white folks think about it. You’re seeing them and they’re out there and they seem to be having a good time, so what do you want to change for? If they wanted Bobby Rydell, they’d go to see his show.” At first we didn’t listen to him. We threw in some tunes like “The Wanderer” by Dion and some pop tunes that were hot then. It didn’t take us but a minute to realize they were out of place. We jumped right back into our own thing.
I was anxious, though, to break out in some kind of way. Didn’t look like I was going to do it on King. And Mr. Nathan wouldn’t pay me the money I thought I had coming, and every time I asked for any kind of big advance against royalties, he wouldn’t give it to me unless I let him extend the contract. But it looked to me like if anybody owed anybody, King owed me. And I was tired of fighting Mr. Nathan all the time. I loved him like a father, but I was tired of fighting him. It was a time of change at King, too. Mr. Nathan had been sick for a while, but when he got his health back he would be running the company full time again. His ideas about artist development and promotion were old-fashioned, not like Mr. Neely’s. Around this time, there was a chance Mr. Neely would buy him out, but when it didn’t come through I was ready to go.
Around the end of 1963, after the Royal album, Pop and I formed a company called Fair Deal Productions, to be run by Marty Machat, the lawyer who handled the Rolling Stones and others. I signed with Fair Deal, and Fair Deal signed me to a production deal with Mercury/Smash Records. Mr. Machat took the position that my contract with King had run out. He also got me signed to an American Federation of Musicians contract. See, my contract with King was a personal services contract for vocals, which was standard in those days to keep the company out of trouble with the union. My name never showed up on the session reports as a musician even when I did play, but King Records, like most of the other companies, was subject to the AFM union contract. Later on, when it all wound up in court, Mercury based their case on signing me as an AFM artist.
The first thing I did for Smash was cut Byrd on a tune called “I’m Just a Nobody,” and it became a nice little regional hit for him. Then I cut him and Anna King on a duet—“Baby, Baby, Baby”—that came out in March 1964. The first tune by me on Smash came out in April—“Caldonia.” It seemed right to start out on the new label with the song that started me out as a performer way back when I had the Cremona Trio. I was also cutting things for the Showtime album that came out on Smash a little later and for the Grits and Soul LP for them that was all instrumental.
Meantime, King released “Oh Baby, Don’t You Weep” in January, reissued “Please” in February, and put out “Again” at the same time “Caldonia” came out on Smash. Then Mr. Nathan filed a lawsuit to keep me from recording for Mercury. While the thing was tied up in the courts, I kept cutting on Smash.
I kept on touring, too, and in late May I was back in the Apollo. Anna, Bobby, and Sugar Pie were featured, and we had Patti LaBelle and the Bluebelles, too. At that time they had a big hit with “Down the Aisle.” Patti’s real name was Patti Holt, from Philadelphia. The rest of the group was Cindy Birdsong (who later joined the Supremes when Florence Ballard left), Sara Dash, and Nona Hendryx. Sara and Nona later became solo acts on their own. They were with the revue for quite a while. Patti was as fantastic then as she is now.
It was the same thing at the Apollo as the last time—lines in all directions as far as you could see. It was nice because I was able to sleep in my own house and drive in to work in the morning. New York was home. I guess that’s why it seemed like the right time to do a homecoming concert in my other hometown, Augusta.
Now that I was off parole and was really making it, I was looking forward to going back there—until I found out the audience in Bell Auditorium would be segregated. I wasn’t heavy into the human rights thing yet, but I knew I didn’t want to play a segregated show in my own hometown, especially after the crowd had integrated itself when I played Jennings Stadium, the baseball park there. When I had played it a few years before, the audience had been segregated, as usual, but during the show the white kids started coming down toward the stage. Before long there were white kids and black kids crowding around the stage, dancing and hollering and having a good time. They had integrated themselves. I didn’t want to turn right around and let ’em be resegregated. I had grown up with the signs that said “White Drinking Water” and “Colored Drinking Water,” and it always seemed to me that water didn’t have any color.
There was a long discussion with the promoter. He said it was the city. I said I wouldn’t do the show—contract or no contract. That was the way we left it. The way Bell Auditorium was set up in those days there was a little section of about one thousand three-hundred seats behind the stage that was used for small affairs. The rest of the seats, several thousand of them, were on the exact opposite side of the stage. If a big black show was in there, the main seats would be for blacks and the other one thousand three hundred would be for whites. If it was a white show, it would be the other way around. You had to play to two audiences at once; when you were facing one, you had your back to the other. The stage had two curtains, one on each side, but I didn’t know that. When I had been there to see a show or whatever I had never seen another side.
When the show started, we ran out and faced the audience—the major side that we thought was the only side—but we heard people yelling and screaming from both sides. I turned around, and there were all the white folks on that side. And a lot of ’em knew me, were friends of mine, yelling “James, hey James!” That got to me. They were friends of mine and the black folks were friends of mine, but they couldn’t sit together. I finished the show, but I said to myself, “Never again!”
We had a few days off after the show, and I got all the fellas to hang around so I could show ’em where I was born and raised, and I took ’em all around, showing ’em the canal, the house on Twiggs, and the Richmond County Jail.
Next we went to Macon for a show at the Auditorium there. Over the years we had played in all the clubs in the area, but we hadn’t played the Auditorium but one or two times back when I had “Please” and “Try Me.” This was kind of a homecoming for me, too, because I had moved away. I had played the Macon Auditorium about two years before, with the audience segregated. The way it worked, if a white act was appearing, then the whites got to sit on the main floor and the blacks in the balcony; if it was a black act, then it was blacks on the main floor and whites up top.
We got into Macon a few days before the show and found out that it was going to be segregated like before. I had just been through all that in Augusta and I wasn’t going to go through it again, not in my second hometown. I told Pop and Mr. Brantly, “I’m not going to do it. Not no way.”
“The advertising’s all out,” Mr. Brantly said. “The tickets have all been sold.”
“I don’t care,” I said. “I’m not going on. And if they say the people aren’t going to be separated and I go on and see they are, I’m walking off.”
Pop said, “Jimmy, just close your eyes to it. Just take the money and get out of town. Why come in here and try to change these people’s policy?”
“Because it’s not right, Pop.”
The funny thing about it was that integrating for a black show would benefit the white people in the audience; they would have the same chance at the good seats as the black folks. I was determined to do my show that way or no way. But I didn’t want to just sit there and refuse, I wanted to get it done. So I went to see a friend of mine, a very influential white fella who ran a car lot in Macon, and told him the situation.
“Let’s just try it this one time,” I said. “All people are going to do is walk in together and sit down. Most folks are escorted, so they’re with their man or their woman. After the show, everybody goes home. Meantime, all they’re doing is standing around in a place together. Nobody’s going to bother nobody.”
He said he’d see what he could do. I think he went around and talked to some of the city officials and got them prepared for what was going to happen. Didn’t make a lot of noise about it, just explained things. The audience didn’t know anything about it until they got there for the show. What we did was close the balcony until the main floor was filled. Then we opened the balcony back up. The show went off without a hitch. I think a lot of people were surprised that the building was still standing afterwards, but I wasn’t.
The next time I went back to Augusta for a show—about five or six months later—I went to the leaders of the white community and the black community and talked to them about what had happened in Macon, saying that we could do it in Augusta, too. So we closed off the small rear seating section and the balcony in Bell Auditorium until the main part was filled, like we’d done in Macon. It came off fine. The next day the papers wrote it up, saying it was the first time there’d been an integrated event and complimenting everybody. After that, the audiences were integrated for all the shows. A little later, President Johnson signed the 1964 Civil Rights bill that outlawed segregation in public places, but it took a long time to take full effect in show business; it was more than a year later before a lot of venues in the South quit segregating the audiences.
In June I went into the Mercury studios and cut “Out of Sight” and “Maybe the Last Time.” Like everything else for me that year, they were an ending and a beginning. “Maybe the Last Time” was a heavy gospel-based number, all about appreciating friends and everything while you can because each time you see somebody may be the last time, you don’t know. Turned out that session was the last time I ever used the Famous Flames on record. They were a good stage act, but they couldn’t really sing all that good. With recording becoming more sophisticated every day, this showed up more on records. Sometimes in the studio some of them had been on dummy mikes anyway; they didn’t know that at the time because I didn’t want to hurt anybody’s feelings. After the “Last Time” session, I used only Byrd and those I told him to hire, so musically and as far as personnel went, the song was another kind of ending for me.
“Out of Sight” was another beginning, musically and professionally. My music—and most music—changed with “Papa’s Got a Brand New Bag,” but it really started on “Out of Sight,” just like the change from R & B to soul started on “Think” and “I’ll Go Crazy.” You can hear the band and me start to move in a whole other direction rhythmically. The horns, the guitar, the vocals, everything was starting to be used to establish all kinds of rhythms at once. On that record you can hear my voice alternate with the horns to create various rhythmic accents. I was trying to get every aspect of the production to contribute to the rhythmic patterns. What most people don’t realize is that I had been doing the multiple rhythm patterns for years on stage, but Mr. Neely and I had agreed to make the rhythms on the records a lot simpler.
“Out of Sight” went out of sight on the charts when it came out. It didn’t take time to build, it didn’t make it first on the R & B chart and then cross over, it just took off across the board. But even though “Out of Sight” was the biggest hit I’d ever had up to that point, I didn’t put out a single new vocal release for almost a year afterward because of the court fight with King. It was almost like the period between “Begging Begging” and “Try Me,” except this time the dry spell was after a big hit. There were still plenty of James Brown releases because with all the masters they had stockpiled King was able to put out a lot of my stuff. And they reissued a lot of my old stuff. They put out “So Long” and “Dancin Little Thing” right before “Out of Sight” came out, and later on they released “Tell Me What You’re Gonna Do” and “I Don’t Care.” They reissued “Think” and “Try Me” and put out a record with a medley of three songs on one side and “Fine Old Foxy Self” on the other. It went on and on like that for the next year.
In October, just about the time that I should have put out a new song, King got an injunction that prohibited me from releasing any more vocals on Mercury/Smash. Mercury even withdrew the Out of Sight album that had just come out. The case eventually went up through the courts; on appeal, the appellate division said I could only sing on King and play on Mercury. They divided me in half. If I wanted to sing, I was going to have to go back to Mr. Nathan. Just when I should have been reaping the rewards of ten years of hard work, I was stuck.
Nothing could stop me from performing, though. Right after the injunction I went back into the Apollo, but it was strange because Little Willie John was supposed to be there. But he had gotten arrested for killing a man at a party in Seattle. It was sad because it happened at Little Willie John’s engagement party. The man he killed refused to give up a seat to a lady; Willie got on him about it, they got into an argument, and Willie wound up stabbing the man. I went back a long way with Little Willie John—my first gig at the Apollo, recording on King together, and playing a lot of shows together in the early days. We always had a friendly rivalry, and I loved him. I knew he was looking at some hard time, and I was worried because he was a little cat and I didn’t think he could survive too long in prison.
That week at the Apollo the people were lined up like always, even though it was cold and snowing. The Apollo’s business had been down to about a fourth of normal since August when a policeman had shot and killed a fifteen-year-old boy and a riot had broken out. Even after things settled down again, people were staying in, but they had come out for me in spite of the weather and everything. Looking out at those lines, thinking about Little Willie John and everything he was going through, and thinking about what I was going through, I suddenly realized how grateful I should be. See, back then a ticket to the Apollo was good for the whole day and night; you could stay for as many shows as you wanted. So a lot of the people out there were going to have to wait indefinitely to get in, but they were willing to do it.
I rounded up everybody on the show and told them to put on their coats and hats—we had some work to do. We had urns of coffee made and got some plastic cups, and then we went outside and went up and down the line, serving coffee to the people who were waiting to get in. I told them I appreciated their waiting in the snow and that I would try to do a good show for them. We did that for the rest of the week there, and when any act wasn’t on stage, it was out there serving more people during the shows. I think they were grateful to us; I know I was grateful to them.
Looking back on it, I think 1964 was a strange year. A lot of tragedy came to a lot of people I went way back with in the business. Rudy Lewis, who’d been in the Drifters for five years, died of a heart attack at age twenty-seven. Not too long after Little Willie John got arrested, Ray Charles was busted at Logan Airport in Boston for heroin, and not too long after that Sam Cooke was shot to death in Los Angeles under peculiar circumstances.
I was bouncing around between two labels and wasn’t sure I was going to sing on record again, but I had to feel blessed. It seemed like a lot of things from the past were slipping away, a lot of changes were coming around, to me and to everybody, but I was surviving. And I believed in the future. It was like standing at a crossroads. There I was playing the Apollo with the Five Royales, a group I had imitated when I first started out Little Willie John was probably gone for good, and it seemed like a lot of cats were dying. Yesterday was gone. But I had faith in the direction of my music. And at the end of the year I knew I’d seen the future when I encountered two new acts: the Rolling Stones and the Jacksons.