The site of a former colonial port that thrived throughout the mid-18th century, most of the town was reportedly burned down by the British in 1776