Burg Scôzes was the oldest building in the city, and the natural heart of it. Through the years it was known by many names: "Burg Scôzes", "Burg Scôzky", "Scôz Burg" or simply "Scôz" (a word which actually can be translated as "fortress" or "stronghold"). In the later years the castle was simply referred to as the "Burg".
Situated on the west bank of the river, it's powerful guns could control the vital Market Island Grosz Inszel (later renamed Panardo Inszel).
When the Grand Dukes of Scôz began to use the Burg as their official residence, it also meant that the city became more important.
In 1348 the Cathedral was completed on the south end of Panardo, and in 1410 a bridge was built between these two Power Structures of the city. An alliance between earthly and heavenly power.

This painting of the city is probably the oldest known picture of Scôz am Rhein. It shows the city as seen from East, and was commissioned by Grand Duke Ottokar in the early 1500's. Both the Burg and the Cathedral are clearly visible. The Arch Bishop of Scôz had his own fortress not far from the Burg, but that building was demolished by Grand Duke Wolraf in 1412. Two big towers can be distinguished on the Burg. The high tower to the left is "Riezgne Krgd" ("The Tall Power"), and was erected in the early 1400's. The big central tower of the Burg, the "Dickzke Wrat" ("The Fat Anger") can be seen to the right of the building. This tower was the oldest part of the castle, and later served as Royal Treasury for the Grand Dukes of Scôz.