Building 4 located on Officer's Row and also known as The Surgeon's Quarters was constructed of sandstone in 1863-1864 by English Mason Charles May. It is one of the oldest buildings in Idaho. A brick addition was built in 1880 and partially destroyed by fire in 1950.
The building was part of a cavalry fort built by the US Army in 1863 during the middle of the Civil War. The Fort was intended to protect both miners and the interests of the US Government in the gold of the Boise Basin and silver of Silver City as well as the Pioneers of the Oregon Trail. By 1919, the campus was loaned to the US Public Health Services to be used as a hospital. And by 1938 it was transferred to the Veterans Administration to take on its present role as the Boise VA Medical Center.
The entire campus is listed on the National Register of Historic Places.