The decision to travel to amber list countries is a "personal responsibility", skills minister Gillian Keegan has told Sky News.
But she echoed the PM’s warning yesterday that they are "not somewhere where you should be going on holiday".
Many people have reportedly travelled to destinations on the amber list since the ban on foreign holidays was lifted on Monday.
More than 150 flights are reported to have left on Monday to places such as France, Greece, Spain and the US.
Amid confusion over the government’s position on holidays, Boris Johnson said yesterday that travel to amber list countries had to be for "some pressing family or urgent business reason".
Under pressure from Labour leader Sir Keir Starmer at Prime Minister’s Questions Johnson said the government is trying to move away from legislating for everything and more towards guidance.
But he reiterated: "You should not be going to an amber list country on holiday."
Asked about travel to amber list countries this morning, Ms Keegan told Sky News: "This is the time for cautious personal responsibility… The British public have been amazing both in getting the vaccine and… adhering to all the lockdown rules… But… this first step is the first step in the next stage of the journey. And we think it’s the right step."
On the amber list, she added: "This is guidance from the government. We haven’t legislated to ban people from going on holiday.
"And, as with many of these things that we’ve had throughout the pandemic, this has been about really relying on the great British public to be sensible and looking at the guidance that we put in place and taking their own decisions."
But she cautioned: "We wouldn’t advise going on holiday to the amber list countries."
EU ambassadors, who met on Wednesday, are understood to have backed plans to allow fully vaccinated UK holidaymakers to visit the bloc for non-essential travel this summer.
Confusion over the UK government’s position left many in the travel industry expressing frustration, especially as Foreign Office (FCDO) guidance appeared to indicate that people were safe to visit many of the countries on the amber list.
George Morgan-Grenville, founder and chief executive of travel company Red Savannah, said the industry was "facing a disaster" and she couldn’t understand why it "has to bear the brunt of so much confused and mixed messaging from ministers".
Travel firm TUI said it will continue to operate to destinations "where borders are open and FCDO advice allows travel", while Hays Travel said it would also take bookings for "amber destinations that are allowed".
Shadow health secretary Jonathan Ashworth says the government was wrong to ease the rules on foreign travel.
He told Sky News: "People just want clear guidance about what is the right thing to do. I probably wouldn’t have introduced… the international travel restrictions easing, this week. I’d have done it in a different way.
"Our shadow home secretary, Nick Thomas-Symonds, has long been calling on your programme and other programmes that we need a comprehensive border policy.
"Our borders have been as secure as a sieve throughout this crisis – it’s why we are seeing these variants bounce across the world and come and hit us and there’s probably going to be more of this as well."
It comes as the next stage of England’s roadmap out of restrictions on 21 June is at risk due to a rise in cases of the Indian variant.
Cases of the COVID strain, also called B.1.617.2, have doubled in the UK in the last week after it was first thought to have entered the UK via travellers from the sub-continent.
Imperial College London epidemiologist Professor Neil Ferguson, a member of the government’s Scientific Pandemic Influenza Modelling Group, told BBC Radio 4 that rules around travel into the UK were not strong enough.
He said: "I think one thing the import of this Indian variant has shown us – everybody coming back from India had to home quarantine, had to test – is an imperfect way, it basically didn’t work."