A mountain spur lying high above Bad Driburg has been used as a refuge for a very long time. The saxons are believed to have erected a retreat, protected by a stone-earth rampart. Due to the sharp drop on the south-eastern side no further fortifications were necessary. During Carolingian times the ramparts were replaced by stone walls. The castle was destroyed by Charlemagne in the war with the saxons. He erected a church and gave the castle ruins to the diocese of Paderborn. It is unclear whether the Iburg was the site of a Saxon sanctuary.

Between 1134 and 1142 benedictine nuns settled in the Iburg, but quickly left the area because “of the inconvenient location, the barrenness of the soil and the inhospitable forests”. In 1189 a new castle was erected by bishop Bernhard II. of Paderborn, which was destroyed in 1444 by Otto von Braunschweig.
The name Iburg originates in the word Eibe (= yew tree) and therefore means Eibenburg (“yew tree castle”). Even today you can see a yew tree growing in the middle of the castle ruin. In fact, both yew and juniper are the only coniferous trees native to the forests between Egge and Weser.