The castle of Ormož is situated in the south-eastern part of the Bronze Age settlement which was protected with an earthen rampart and a defensive ditch. The ditch winds along the castle complex to the east and, at the end of the palace yard, turns towards the former Hungarian gate.
The Lords of Ptuj started building the castle complex in the last third of the 13th century. It remained their property until 1438 when the family of the Lords of Ptuj died out.
The first building phase most probably comprised the south wing due to its exposure; it housed the residence of the castellans of Ptuj, Lords of Ormož. Only later the huge yard, which was surrounded by a wall, got a mighty tower at the north-east corner in order to reinforce its protection. The events at the end of the 15th century sped up the development of the castle. After it was burnt down in 1487, it passed in the hands of the family Szekely in 1490 that had it restored with the purpose to settle there. They also had the north and west wings built, and probably had started the east wing as well. In 1532 the castle was damaged by the Turks; in 1540 it was restored as part of the town fortifications. After the attack of the Kruci on 12th February 1704, the castle was derelict. Between 1710 and 1740 it was restored in the Baroque style by the family Pethe. At the beginning of the 19th century, the castle was bought by Jože Pauer, an industrialist, who had it restored in the Classical style and built a wall around the south-eastern castle garden. The last owners of the castle were counts Wurmbrandt whose coat of arms is built-in above the main castle entrance.