A third of travellers from Germany, Italy and Spain, and a quarter from France, said they would be less likely to visit the UK following a Leave vote.
Four in 10 EU residents are worried that Brexit could make holidaying in the UK more expensive, according to research from Travelzoo.
The company estimates that the fall in visitor numbers could cost Britain’s tourism industry up to £4.1bn a year, based on data from VisitBritain and the Office for National Statistics (ONS).
Joel Brandon-Bravo, UK managing director of Travelzoo, said: “Our neighbours in Europe clearly don’t want the UK to leave the EU, and the impact of this sentiment could translate into a significant drop in bookings to the UK from the largest European countries.
"When combined with a potential loss of more than 10 per cent of visitors from North America, as indicated in our research, it’s clear that Brexit could be very bad news for the UK’s tourism industry."
However, some analysts believe a Leave vote could provide a boost for domestic tourism, with 28 per cent of Brits concerned it could cause European trips to get more expensive.
More UK nationals than ever are already opting to holiday at home this year amid uncertainty about the result of the vote.
What does Brexit really mean for the industry?
The issue has proved divisive in the hospitality sector, with a William Reed survey finding that a third of foodservice operators think Brexit would be ‘good for business’.
JD Wetherspoon launched a range of Brexit-themed beer mats this week comparing the morals of the International Monetary Fund (IMF) – which backs the Remain campaign - to FIFA.
However, hospitality recruiter The Change Group has warned that restaurants, hotels and pubs will be left short-staffed if migrants are unable to take hospitality jobs following a vote for Brexit.
The firm said only a third of applicants for top chef roles in London were from Britain, with the figure falling to a fifth for chef de partie jobs.
Travelodge CEO Peter Gowers has argued that the hospitality industry risks ‘sleepwalking in to danger’ over the issue.
“If the result leads to it being more difficult for people to visit this country then we have a problem as an industry,” he said.
“As business people we need to take a lead on this issue. If people can’t get in to the country then the infrastructure and level of service are all irrelevant.”
Travelzoo’s survey was taken online by 4,950 people across France, Spain, Germany, the US and Canada. Atomik Research surveyed 2,004 consumers in the UK and 1,003 consumers in Italy.