Born "Gilberto Miguel Calderón" in New York City, Cuba's parents moved from Puerto Rico to New York City in the late 1920s and settled in Spanish Harlem, a Latino community located in Manhattan. Cuba was raised in an apartment building where his father had become the owner of a candy store located on the ground floor (street level floor). His father had organized a stickball club called the Devils. Stickball was the main sport activity of the neighborhood. After Cuba broke a leg he took up playing the conga and continued to practice with the conga between school and his free time. Eventually he graduated from high school and joined a band.
In 1950, when he was 19 years old, he played for J. Panama and also for a group called La Alfarona X. The group soon disbanded and Cuba enrolled in college to study law. While at college he attended a concert in which Tito Puente performed "Abaniquito". He went up to Tito and introduced himself as a student and fan and soon they developed what was to become a lifetime friendship. This event motivated Cuba to organize his own band. In 1954, his agent recommended that he change the band's name from the Jose Calderon Sextet to the Joe Cuba Sextet and the newly named Joe Cuba Sextet made their debut at the Stardust Ballroom.
In 1962, Cuba recorded "To Be With You" with the vocals of Cheo Feliciano and Jimmy Sabater. The band became popular in the New York Latin community. The lyrics to Cuba's music used a mixture of Spanish and English, becoming an important part of the Nuyorican Movement.
In 1965, the Sextet got their first crossover hit with the Latin and soul fusion of "El Pito (I Never Go Back To Georgia)". The "Never Go Back To Georgia" chant was taken from Dizzy Gillespie's intro to the seminal Afro-Cuban tune, "Manteca". Jimmy Sabater later revealed that "none of us had ever been to Georgia". 
Along with fellow Nuyorican artists such as Ray Barretto and Ricardo "Richie" Ray, Cuba was at the forefront of the developing Latin soul sound in New York, merging American R&B styles with Afro-Cuban instrumentation. Cuba was one of the key architects behind the emerging Latin Boogaloo sound, which became a popular and influential Latin style in the latter half of the 1960s. In 1966, his band which included timbales, congas, sometimes bongos, bass, vibraphones, and the piano among its musical instruments, scored a "hit" in the United States National Hit Parade List with the song "Bang Bang" - which helped kick off the popularity of the boogaloo. He also had a #1 hit, that year in the Billboards with the song "Sock It To Me Baby".
On April 1999, Joe Cuba was inducted into the International Latin Music Hall of Fame. In 2004, he was named Grand Marshall of the Puerto Rican Day Parade celebrated in Yonkers, New York. He was also the director of the Museum of La Salsa, located in East Harlem, Manhattan, New York.
Cuba died on February 15, 2009 in New York City, after being removed from life support. He had been hospitalized for a persistent bacterial infection.