Due to the French who had burned the town in 1339 and again in 1377, All Saints was in a ruinous state by the 15th century when it was rebuilt in the perpendicular style. One Richard Mechynge desired in his will, 1436, to be buried in the 'new church of All Saints', so the rebuilding must have taken place around then.*

Although it has been often stated that the church has been on its present site since Norman days, there is a certain body of evidence tending to show that it may have been very much nearer the sea, from whence it had to be removed like its neighbour (St. Clement).**

Its registers date from 1559.

It has  a large churchyard with many tombstones.