The buildings that comprise the Swift House Inn today are the product of over a century of adaptation to the needs of the Swift and Stewart families that lived here from 1815 until 1981. The original portion of the Main House was built by Samuel Swift (1782-1875) in 1814 when he was at the beginning of a long career of public service. Shortly after being admitted to the bar in 1808, he started a bookstore, printing business, and eventually a newspaper, which was established in 1812. The focus of the paper, in part, was to communicate war news, since Middlebury was so close to the enemy's country and to the expected seat of war.

After the war, Samuel Swift settled down to practice law first serving as Judge of Probate, then of the County Court, and eventually serving in the state legislature. He returned to literary pursuits later in life, when at the request of the Middlebury Historical Society, he wrote histories of Addison County and the Town of Middlebury - still major sources for the study of area history.

Shortly after Judge Swift's death in 1875, the house was bought by Governor John W. Stewart (1825-1915). M. Stewart's political career included eight years in Congress and one term as Governor of Vermont. Stewart was proudest of his role in the 1860 Republican National Convention when he was influential in turning the Vermont delegation from William H. Seward to Abraham Lincoln.

The house served as home base for the Stewart family, which consisted of Governor Stewart, his wife Emma Battell Stewart and their children: Philip Battell, Elga and twins Jessica and Robert Forsyth. They lived in the house for only part of the year, using it as a base from which they traveled, attended school, or sat in Congress. Summers were spent in Middlebury, with side trips to Lake Champlain and to Ripton where Mrs. Stewart's brother, Joseph Battell ran the Bread Loaf Inn. This is now the summer campus of the Middlebury College School of English and home of the Bread Loaf Writer's Conference.

Mrs. Stewart, who spent winters in Boston or New York, had one of the first telephones in Middlebury installed to connect her home with the Congregational Church so that she could hear the services she was unable to attend. She also kept the church supplied with fresh flowers from her gardens and greenhouse. In 1885 the Stewart's hired Clinton Smith, Middlebury's most prolific Victorian architect, to design and build the horse barn and a carriage house located on the northeast corner of the property. Smith's other work is evident throughout Middlebury, in the County Courthouse, the Methodist Church, the downtown Beckwith Block, and numerous other buildings, commercial, public and private. In 1906 Governor Stewart removed the original summer kitchen wing and replaced it with a two-story addition. Stewart's architect, Harding & Seaver of Pittsfield, MA, designed the new wing so that it would appear to have been built at the time the house was originally built. At the time of this remodeling, doors and porches were added to both the east and west facades of the original house, as well as a large arched window on the east side.

In 1915, on the death of Governor Stewart, the house passed to his daughter, Jessica (1871- 1981). After a romantic and brief first marriage to J. Walter Sylvester, a minister who died of tuberculosis within eight months of the wedding, she married an older man, Charles M. Swift, grandson of the original owner of the house. Still another lawyer, he had made his legal reputation in his younger years in a complicated property case that he spent 40 hours summing up for the jury. He later gave up practicing law and went into the business world where his major accomplishment was building a streetcar system for the Philippines. Mr. and Mrs. Swift traveled around the world, cruised on his yacht in Florida, maintained a New York apartment and built a summer home complete with a golf course on Lake Champlain in Ferrisburgh. They used the Middlebury house occasionally and made it available to friends and relatives.

In 1939 ten years after the death of her husband, Mrs. Swift gave up her New York apartment and made her Middlebury house her primary residence. She continued her family's tradition of service with her donations to local institutions, which included building a library wing for the Sheldon Museum that contains her family papers. She also donated her early childhood home, built by her great-grandfather and now known as the Community House, to the people of Middlebury. In addition she donated her home in 1943 to be used as a restaurant for charitable purposes by Sir Wilfred and Lady Anne Grenfell. After Mrs. Swift's passing in 1981, her home in Middlebury and its contents were eventually auctioned off and the property was turned into an Inn. Today, the Swift House Inn is owned and managed by Dan and Michele Brown.