From Detroit, at the urging of Louis Armstrong, Jeffries moved to Chicago where he performed in various clubs. One of his first gigs was in a club allegedly owned by Al Capone. Jeffries began his career working with Erskine Tate and his Vendome Orchestra. Tate signed the 19-year-old Jeffries to a contract with his Orchestra at the Savoy Ballroom in Chicago. His break came during the 1933 Chicago World's Fair A Century of Progress International Exposition singing with the Earl Hines Orchestra on Hines' national broadcasts live from the Grand Terrace Cafe. His first recordings were with Hines in 1934, including "Just to be in Carolina". By 1940, he was singing with the Duke Ellington Orchestra and then recorded with him from 1940 to 1942. His 1940 recording of "Flamingo" with Ellington, released in 1941, sold more than 14 million copies in its day. His name had been Herbert Jeffrey, but the credits on the record mistakenly called him Jeffries, so he renamed himself to match the typo. During his time with the Duke Ellington Orchestra as a lead vocalist, Jeffries proved his talent as a mature singer, demonstrating his wide vocal range in such songs as "I Don't Know What Kind of Blues I've Got," "The Brownskin Gal," and "Jump for Joy" (all 1941). The 1944 single "My Little Brown Book" by Ellington and his Famous Orchestra, on which Jeffries provided vocals, reached No. 4 on Billboard R&B chart.

In his teens Jeffries developed a fine voice; initially he sang in higher registers. He started out his singing career as a lyrical tenor, but, on the advice of Duke Ellington's longtime music arranger, Billy Strayhorn, he lowered his range to mimic the vocal stylings of crooner Bing Crosby. Jeffries became a "silken, lusty baritone," according to music critic Jonny Whiteside.

In 1945, Jeffries had a hit on the Billboard R&B chart with "Left A Good Deal In Mobile" (No. 2), on which he was accompanied by pianist Joe Liggins and his band Honeydrippers. Then, he moved to Europe and performed there for many years, including at nightclubs he owned. He was back in America by the 1950s, recording jazz records again, including 1957 collection of ballads, Say It Isn't So.

In 1995, at age 81, he recorded The Bronze Buckaroo (Rides Again), a Nashville album of songs on the Warner Western label.