By the 1830s Norwich Castle was so tumbledown that prisoners in it were using its ramshackle exterior to escape down. But one person was determined to save it. Trevor Heaton tells the story of a man named Stone.

Norwich Castle seems so solid, so massive and so permanent that it's hard to believe that its future has ever been in serious doubt over the centuries.

But for every person over that time who has been happy to see it there, and happy to spend public money to preserve it, there have been plenty of others who would be just as happy to have pulled it down in the name of progress (and saving on the rates).

Norwich isn't unique in this, of course, and in towns and cities up and down the land similar debates have been played out over the years. So many have resulted in those castles being levelled in the name of 'moving with the times' and protecting the civic purse.

Sometimes the citizens haven't had a say in matters. Oliver Cromwell cut a swathe through many a medieval castle by 'slighting' them – with the help of gunpowder – to deny them to the Royalists. If the Parliamentarians hadn't held sway in Norwich then that too, surely, would have been the fate of its own castle.

In later centuries the railway age was to have its effect. Berwick Castle, for so long the bastion of Englishness against the Scots, was helpless in the path of the new permanent way.

Not Norwich though. The castle is still here, and loved more than ever – a fact which is keeping visitors flocking to its latest exhibition, The Square Box on the Hill, which is celebrating the building's long, long story until Sunday June 3.

The castle's fate was not just down to the skill of its Norman builders. It must have had a succession of 'champions' over the last 900 years, many of whom we will never know the names of.

But we know about Francis Stone. He was the leader of efforts in the early 1830s to repair the crumbling keep, then part of the County Gaol complex, for posterity.

And it needed repair – really needed repair. The ravages of time had played havoc on ancient structure. 'Stones were falling and the prisoners were actually escaping,' explained exhibition curator Paris Agar.

Its poor state of repair was thrown into even more sharp focus by the buildings inside and alongside it. William Wilkins' prison, replacing the Sir John Soane complex, had been built from 1822-27. Its pristine Aberdeen granite was in stark contrast to the crumbling Caen stone of the old keep.

But what to do? The debate swung backwards and forwards, often through the pages of the Norwich Mercury. There was a school of thought (and a School of Painting) which wanted it left untouched, its crumbling facades adding a picturesque touch to Norwich's skyline.

It didn't help that the idea of what actually constituted the 'perfect' medieval castle was unclear to many. Views had been coloured heavily by Sir Walter Scott's immensely popular medieval adventure Ivanhoe, published in 1820, but also by the love of the 'Gothick' which had come to the fore a generation or two earlier.

The aptly-named Francis Stone was the driving force behind the campaign to protect the building by facing it in a new layer of stone. So you thought stone cladding was an Eighties thing? Norwich Castle got there 150 years earlier.

Norwich architect Stone, like the Wilkins father and son architects before him, was as passionate about the past as he was for the future. Francis had a bit of a thing for Norfolk's bridges, recording the county's ancient survivals in detail.

The County Surveyor for almost 30 years, he is a real unsung hero of the castle's long story, working tirelessly for the castle to be repaired, but also meticulously surveying the inside and outside of the crumbling building - stone by stone, crack by crack - in three volumes of annotated watercolours in 1826, 1833 and 1834.

On July 3 1833, Stone warned the Quarter Sessions – the visiting judges – about the parlous state of the building. He told them that the exterior surface of the castle was 'so mutilated by time that stones have fallen to persons passing by'. The justices has ordered loose stones from the North and West sides of the castle, but this had left the west side with 'so unseemly an appearance it was deemed advisable to go no further'.

It was resolved that Stone should come up an estimate for repairing and restoring the West side. It actually took the best part of a year before the justices were able to consider those estimates, which by now also included the South and North sides.

The County Surveyor said the walls above 26 feet were 'perforated throughout' and a partial repair would not be effective. His estimates were £1565 each for the West and South sides, and £1000 for the North (the East side abutted the new castle gaol buildings) – the equivalent of just over £510,000 in today's terms.

In the event, the project overspend was to be a further £1600, largely down to some of the stonework having to be thicker to ensure strength and stability, and also urgent repair work to two large cracks that appeared during the work.

Eventually a tender was accepted from a Norwich stonemason, a Mr Watson, and the minute books record how the work proceeded, with the battlements and buttresses removed for the safety of the workmen.

Stone was still involved with the work, although it is clear that there must have been some disquiet at the cost or quality of the worksmanship, as a second opinion from 'Mr Blore' was ordered by the justices in November 1834, the County Surveyor being requested to give 'every information to Mr Blore which might be useful'.

Logistics and cost sadly meant that the castle was not refaced in the Caen stone from Normandy as the original had been. Instead, Bath stone was chosen.

Sadly, Stone was not to see the fruits of his labours. He died in August 1835, aged 65, only a year into the five-year refacing project.

Even allowing for the occasionally flowery prose typical of the age, it is clear that Stone was held in high regard by all who knew him. The Norwich Mercury reported: 'His professional acquirements, as well as his practical skill were of a high order; his integrity was unimpeachable, his friendship, active and firm, his benevolence void of affectation, and his devotion to the fine arts exemplary…

'Few man indeed are taken away who awaken more sincere or deeper regret, whether of a private or public nature, or of whom it may be more truly observed, had it pleased Heaven, 'He should have died hereafter'.'

Stone was replaced as project supervisor by Anthony Salvin, who had his own ideas of what a medieval castle should look like. He replaced the 13 crenellations-a-side battlements with nine – and no, there's no evidence that he was a triskaidekaphobic; he just preferred it that way. Happily, one of his other ideas – to put turrets on each corner – was quietly dropped.

The final result looks so perfect that more than one visitor has assumed the medieval castle is a later confection, as Paris explains. 'So many people who come to the city don't realise that it's been refaced – they think it's some sort of Victorian construction rather than Norman.'

The castle was to face one more long dark night of the soul, in the 1880s following the decamping of the County Gaol to its current position on Mousehold Heath in 1883. Once again, happily, the knock-it-down camp was overruled, and from 1887 to 1894 it was converted into the museum we see and love today.

And although Stone had died before seeing the refacing project completed, his legacy lives on in another way. Not only are his three volumes of watercolours works of art in themselves, but the information that Stone so carefully collected is now helping 21st-century contractors preparing the multi-million-pound Norwich Castle: Gateway to Medieval England project to know exactly what lies beneath the Bath stone, something which would surely have pleased such a great champion of this much-loved building.

The Square Box on the Hill continues at Norwich Castle Museum and Art Gallery until Sunday June 3, combining beautiful artwork, hidden stories of the building's history and recalling the nostalgia of school visits to this iconic site. It includes prints, models, paintings, architectural plans and memorabilia, many of which have never been on display before. Find out more on www.museums.norfolk.gov.uk