JOHANNESBURG, May 31— South Africa said today that it was ''no longer sure'' that United States-sponsored negotiations could be held to secure a withdrawal of Cuban soldiers from Angola in return for a settlement to bring independence to neighboring South-West Africa.
The statement, by Foreign Minister Roelof F. Botha, seemed to represent a new setback for the Reagan Administration's policy of ''constructive engagement'' in the region. The policy includes the idea that a Cuban pullout from Angola would persuade South Africa to relinquish its disputed hold on South-West Africa, a former German colony widely known as Namibia.
South Africa's position was made known a week after Pretoria acknowledged for the first time that it had sent troops into northern Angola. The South Africans say their commandos were on an espionage mission, but Angola says they had been sent to sabotage American-operated oil installations in the northern enclave of Cabinda.
Two South Africans were killed and one was captured in the clash with Angolan troops. The captured officer, Capt. Wynand du Toit, said at a news conference in Luanda, the Angolan capital, this week that his mission was to sabotage oil installations owned in part by a subsidiary of the Gulf Oil Corporation, which earn some 90 percent of Angola's foreign exchange. The raid has been widely interpreted here as a bungled effort to bring economic pressure on the Angolan Government to accept South Africa's demands in the region.
'Seeking a Pretext'
''The Luanda Government has informed South Africa through United States channels of its intention to break off negotiations with the republic,'' Foreign Minister Botha said. ''If indeed this is the position of the Luanda Government, it would appear that they were seeking a pretext on which to break off negotiations on the question of a Cuban withdrawal from Angola, and it would reveal that they have never been serious about this problem.''
The statement said South Africa was committed to accepting a formula proposed by five Western nations - the United States, Britain, Canada, France and West Germany - for the independence of South-West Africa, ''provided a firm agreement is reached on Cuban withdrawal from Angola.''
''In the light of the position adopted by the Luanda Government, the South African Government is no longer sure to what extent this goals remains attainable,'' Mr. Botha said.
A withdrawal of the 25,000 to 30,000 Cuban troops supporting Angola's Marxist Government is a principal goal of American policy in the region.
Since ''constructive engagement'' was initiated over four years ago by Chester A. Crocker, Assistant Secretary of State for African Affairs, he and his assistant, Frank G. Wisner, have held many negotiating sessions with Angola and South Africa. The effort sought a trade-off by which the Cubans would withdraw from Angola and the South Africans would pull out of South West Africa, where Angola-based insurgents are fighting a low-key guerrilla war. Mr. Crocker recently put forward a set of compromise proposals.
South African Rebels Cited
But the issue has now been clouded by South Africa's assertions that Angola is playing host to insurgent groups opposed to both South Africa's control of South-West Africa and to its policies of racial separation.
South Africa's statement today seemed designed to shift responsibility for the breakdown in talks to Angola and to respond to the debacle caused by its soldiers in northern Angola.
''Sooner or later,'' it said, ''the Luanda Government will have to realize that they cannot continue playing host to tens of thousands of foreign forces and to terrorist groups such as the A.N.C. and Swapo, all of whom are hostile to peace and stability in southern Africa.'' The A.N.C. is the African National Congress, the most prominent of exiled groups seeking the violent overthrow of the South African Government. Swapo is the South-West Africa People's Organization.
The South African statement did not address the assertion by Pretoria's neighbors that it is the source of instability in the region. It said, for instance that ''there can be no peace in southern Africa unless all foreign forces are removed and unless all the countries of the region agree not to accommodate on their territory those whose declared policy it is to commit acts of violence or subversion against other states in the region and to use violence to achieve political objectives.''
The implication seemed to be that South Africa's incursion into northern Angola did not fit this category.
Pretoria's Plans for Namibia
The statement also seemed to reinforce speculation here that South Africa wants to pursue its own settlement in Namibia in a manner that will ignore both United States policy and the Angola-based insurgents.
Next month, South Africa plans to install an administration in South-West Africa drawn from anti-insurgent political groups, and it has called a multiparty conference. In its statement today, South Africa said it would consult with groups, including the United States and the multiparty conference, to determine ''whether withdrawal negotiations can still be conducted.''
The incident in northern Angola has been interpreted by South African commentators as certain to strengthen calls in the United States for withdrawal of business investments here.
Moreover, South African political analysts said, the premises of ''constructive engagement'' now seem undermined. The policy is based on the notion that confrontation with South Africa's white rulers will not bring change, but persuasion will.
