Costa began playing tennis at the age of five. He first came to the tennis world's attention as an outstanding junior player. In 1993, he reached the French Open junior final and won the Orange Bowl. He turned professional later that year and quickly established a reputation as a strong clay court player. Spanish former player and commentator for Spanish television Andrés Gimeno used to call him "the man with two forehands", because he could hit with the same accuracy and strength both forehand and backhand. In 1994, he won two challenger series events and was named the ATP's Newcomer of the Year.
Costa won his first top-level singles title in 1995 at Kitzbühel, beating the "King of Clay", Thomas Muster, in a five set final, and it was Muster's first of his only 2 losses on clay in 1995. Costa ended Muster's streak of 40 consecutive clay match wins and his 11 consecutive final wins . Costa won three further titles in 1996. In 1997, he won another two singles titles and was part of the Spanish team that won the World Team Cup. He won another two singles titles in 1998, including the Tennis Masters Series event in Hamburg and played at magnificent level at the French Open, only a brilliant Marcelo Rios could stop him at fourth round, but he established the bases for his future winning in 2002. Three further titles followed in 1999.
In 2000, Costa helped Spain win its first Davis Cup. Despite being knocked out in the first round in men's singles, he also captured a bronze medal in the men's doubles at the 2000 Olympic Games in Sydney, partnering Álex Corretja.
Going into the French Open in 2002, Costa had not won a tour title of any kind since 1999 and was not considered to be among the favorites. Seeding 20th, he defeated Richard Gasquet, Nikolay Davydenko and Andrea Gaudenzi to reach the fourth round, where he defeated two time defending-champion and former world No. 1 Gustavo Kuerten in straight sets. He followed up with a five-set victory over Argentina's Guillermo Cañas in a quarterfinal. He then defeated fellow-Spaniard, former world No. 2 and long-time friend Àlex Corretja in a four-set semifinal. In the final, Costa came up against another Spaniard, future world No. 1, Juan Carlos Ferrero. Ferrero had been in fine form in the run-up to the event and most observers considered him to be the heavy favourite going into the final. But Costa won in four sets, 6–1, 6–0, 4–6, 6–3, to claim his first Grand Slam title. Costa destroyed Ferrero, who could only win nine points during Costa's service games in the first 2 sets. The win propelled him to his career-high singles ranking of World No. 6 in July 2002.
Coming into the French Open as defending-champion in 2003, Costa spent a total of 21 hours and 15 minutes on court, winning four five-setters before eventually being knocked-out in a semifinal by Ferrero (who went on to win the title).
During his career, Costa won 12 top-level singles titles. In 2005, he captured his first tour doubles title (in Doha, partnering Rafael Nadal).
Citing recurring injuries and lack of desire, he officially announced his retirement from competitive professional tennis on 21 April 2006, at the completion of the Open Seat 2006 in his hometown in Barcelona. In his last tournament, Costa defeated American Vincent Spadea and Slovakian Dominik Hrbatý before losing 6–1, 5–7, 7–5 to Ferrero in the third round.
In December 2008, Costa was named Spain's Davis Cup captain, replacing Emilio Sánchez Vicario. Costa became the most successful Spanish Davis Cup captain to date as he led Spain to two Davis Cup titles in 2009 and 2011, before handing the captaincy over to Àlex Corretja.
Costa was coaching ATP pro Feliciano López.