Andre Esteves was arrested at his home in Rio de Janeiro. Documents were seized both from there and from the bank he heads in Sao Paulo.
At the same time, the leader of the governing Workers Party in the Senate, Delcidio Amaral, was also arrested.
Both men are accused of obstructing the course of the investigation.
Mr Esteves is Brazil's 13th richest man, worth an estimated $2.5bn (£1.7bn), while Mr Amaral is the first sitting legislator to be detained in scandal.
After his arrest early on Wednesday, police released a recording in which Mr Amaral appears to suggest that the imprisoned former head of Petrobras' international division should refrain from disclosing any evidence about Mr Amaral and Mr Esteves.
In exchange, the Petrobras executive, Nestor Cervero, would be helped to escape from Brazil to Europe, reports said
Image captionFederal police released a tape that seems to implicate Senator Amaral in the scandal
The tape was secretly recorded by Cervero's son. He told Brazilian media that his father and he had disagreed with the escape plan.
Later on Wednesday, the Senate held an emergency session and backed up the Supreme Court's decision to arrest Mr Amaral.
More than 100 people have now been arrested in the two-year investigation, including politicians and former top executives at Petrobras, and some of the country's biggest construction firms.
Although Brazil's President and former Petrobras Chair Dilma Rousseff has not been directly implicated in the scandal she is under severe political pressure over its implications, says the BBC's Wyre Davis in Rio de Janeiro.
The president of the Workers Party, Rui Falcao, has published a statement distancing himself from the disgraced senator.
He said was "perplexed by the facts" that led to the arrest.
"None of the acts attributed to the senator is connected to his activities for the party.
"For that reason, the Workers Party does not feel obliged to lend him any solidarity," added Mr Falcao.
>>> Read more: Vietnam’s Textile Industry: Fail To Reach Target
