Twenty fire engines and 120 firefighters and officers have been called to the scene in William Street.
Pictures posted on social media show large quantities of black smoke pouring from what appears to be the roof of the hotel.
The Mandarin Oriental is home to the two-Michelin-starred Dinner by Heston Blumenthal restaurant, as well as Daniel Boulud’s Bar Boulud.
The hotel completed a multi-million pound renovation, dubbed the “most extensive in its 115-year history”, last month – which included a revamp of its 181 rooms and suites and a “soft renovation” of Dinner.
London Fire Brigade said in a statement: “Twenty fire engines and 120 firefighters and officers have been called to a visible fire at a hotel in William Street in Knightsbridge.
“The Brigade's 999 control officers have taken around 40 calls to the fire, which is producing a lot of smoke.
“The Brigade was called at 1555. Fire crews from Chelsea, Kensington, Hammersmith, Battersea and other surrounding fire stations are at the scene.
“The cause of the fire is not known at this stage.”
The hotel said in a statement: "We are aware that there is a fire at Mandarin Oriental Hyde Park, London, and that the fire brigade is at the hotel.
"As the situation is still unfolding, we have no further details to share at this time, but will provide an update soon."