With the death of John de Somery in 1322 the extensive estates of the Barony of Dudley entered into a period of confusion and disputed ownership.

Only in 1327 did John de Sutton, after imprisonment in the Tower of London and the extraction of a forced disclaimer to the property, eventually inherit the family estates of his wife Margaret de Somery.

The Castle had a relatively calm succession of the de Sutton lords, all name John.