Born in 1858 in Rutherford County, James Vester Miller was the son of a slave, Louisa, and her white owner. After the Civil War, his mother took her three children and made her way to Asheville. There Miller's interest in building developed and he was soon considered one of the city's master masons. He formed a company, Miller & Sons Construction, which specialized in churches and commercial buildings during the late 1880s and early 20th century. Among his noted works are the Post Office & Federal Building, later torn down, St. Matthias Episcopal Church, St. James A.M.E. Church, and Hopkins Chapel. He was also instrumental in the establishment of Violet Hill Cemetery for African Americans in West Asheville.