Elvedgård near Veflinge was first mentioned in 1436. Back then the owner was the esquire Hartvig Hesten.
In 1475 another owner, Frederik Barsebek, had a papal letter allowing him to transform the courtyard into a monastery for the Carthusian Order. However this never happened.
Over time, many claimed the right to the manor house, but in 1555, after much dispute, the farm surpassed to the brothers Jørgen og Otte Brahe, who were both rich and important men. Otte Brahe was the father of the world-famous Danish astronomer Tycho Brahe. However, it was his second son, Axel Brahe, who later on became an influential person in the Danish government, who inherited the manor house in 1571.
After the death of Axel Brahe his daughter Anne Brahe took over, but she had to pawn Elvedgård which would then have several owners through the years.
1721 the new owner Ulrik Frederik Brüggemann built a new one floored three-winged main building with tile roof. The main building, placed on an old mound, probably built by the previous owner Axel Brahe in 1590, also had an attic and foundation wall, while the side wings were half-timbered.
Claus Hansen Wedel inherited the manor house in 1746, and for the next 200 years until 1969 it was owned by the Vedel family
1800 the then owner, Claus Vedel Simonsen, took down the top floor of the towers and connected the rest of the towers to the side wings.
In 1832 the famous historian, Lauritz Schebye Vedel Simonsen, inherited Elvedgård and lived here for the rest of his life. Because of his attachment to the local area he donated money for a new school and a parish library in Veflinge.
In the 1860's the then owner, Julius Christian Frederik Wedel-Heinen, tore down the old wings, built new ones and rebuilt parts of the castle.
The farm buildings of Elvedgård burned down in 1915, and a new three-winged building with white plastered foundation wall and red tiled roof was errected northeast of the main building.
The main building of Elvedgård is still surrounded by three moats. On the other side there is an old Romantic garden, which is criss-crossed by canals.