James McConnell Smith built the house with his wife, Mary Patton Smith, around 1840. Smith was one of the region's wealthiest men, owning as much as one-third of Asheville and over 30,000 acres on the French Broad River. The home is the oldest surviving building in Asheville and the oldest brick structure in Buncombe County. It served as a second home for Smith, as the family's main dwelling was in Asheville. The house was later purchased at public auction in 1858, by Smith's daughter, Sarah Lucinda, and her husband William Wallace McDowell. The house was saved from demolition in 1975 by the Western North Carolina Historical Association and the Junior League. The Smith-McDowell house is home to the historical association and serves as a museum.