(CBS News) KABUL - In some of the strongest language yet from a U.S. official, Defense Secretary Leon Panetta said Thursday that the U.S. was "reaching the limits of our patience" with Pakistan allowing militant groups to enjoy safe haven inside parts of the country.

Panetta would not rule out stepping up drone attacks inside Pakistan if those safe havens are not eliminated, or sending in ground troops - an option which would infuriate Pakistan's leaders and public.

"It is extremely important that Pakistan take action to prevent this kind of safe haven from taking place, and from allowing terrorists to use their country as a safety net in order to conduct their attacks on our forces," said Panetta, who made a brief, unannounced visit to Kabul on Thursday.

"We have made very clear that it is difficult to achieve a secure Afghanistan as long as there is a safe haven for terrorists in Pakistan from which they can conduct attacks on our forces," Panetta said.

Panetta in Afghanistan as Karzai says NATO strike killed 18 civilians

While stressing that the U.S. was not at war with Pakistan, Panetta indicated that the country's leaders were not doing enough to secure their territory.

"We're at war with those who would attack our country and attack our troops," the defense chief told Fox News. "Unfortunately, they reside in the [tribal areas] in Pakistan. The United States will do whatever we have to do to protect our forces."

When asked if that included possibly sending U.S. troops into Pakistan, Panetta wouldn't rule it out.

"I'm not going to go into particulars," he said. "It's about protecting our forces and trying to urge the Pakistanis to take the steps they have to take to control the situation."

The relationship between the U.S. and Pakistan has deteriorated considerably since November, when a coalition air strike killed 24 Pakistani troops at a remote post on the border with Afghanistan.

Pakistan responded by closing their borders to coalition supply trucks and demanding an official apology from the U.S., which they have not received.

Pakistan has also asked the U.S., publically at least, to end the drone strikes which target Taliban and al Qaeda-linked militants in the border region.

Panetta made it clear on Wednesday that the strikes, which have killed a large number of al Qaeda's senior leadership during the past year, would not stop.

Over the past two weeks, there have been eight drone attacks in Pakistan's northwest tribal areas