The Vidalia Academy was founded in 1877 and housed in a one-room building originally located on Mr. H.C. Allen's farm near the site of Latta. In 1879, the structure was moved a short distance north of Latta, nearer to a road crossing Buck Swamp. In 1883 a music room was added and the Academy continued to operate until 1898 when it was merged into a large new school in Latta.

The original Vidalia building, now located on Main Street in Latta, was at one time under the ownership of A. LaFon LeGette, Jr., who foresaw its historical significance and ensured its preservation.