The agency did not give details, but reports suggest the accusations
involve prostitution in Cartagena, which is hosting the Summit of the
Americas.
Agency spokesman Edwin Donovan told the BBC the agents had been replaced and Mr Obama's security would not be affected.
Mr Obama arrived in Colombia on Friday to attend the summit.
In a statement, Mr Donovan said the agency took allegations
of misconduct "very seriously" and said an investigation had been
launched.
"There have been allegations of misconduct made against
Secret Service personnel in Cartagena, Colombia prior to the President's
trip," he said.
"Because of this, those personnel are being relieved of their
assignments, returned to their place of duty, and are being replaced by
other Secret Service personnel."
According to The Washington Post, 12 agents were recalled, and at least one was suspected of involvement with prosititutes.
More than 30 leaders are in the Colombian port city for the summit, which will see talks on economic policy and trade.
The BBC's Arturo Wallace in Cartagena says that, even before
the incident with the agents, the summit seemed likely to be an awkward
one for the US government, with debates on the legalisation of drugs and
sanctions against Cuba promised.
Ecuadorean President Rafael Correa is boycotting the summit
because of Cuba's exclusion, and the attendance of Venezuelan President
Hugo Chavez is in doubt.
Earlier, Colombian police said two small explosions took place in Cartagena, but caused no casualties or damage.
Those explosions followed at least one other blast in the capital, Bogota, near the US embassy, officials said.