The Webb House is an elegant example of Federal architecture from the last quarter of the 18th century. Originally, the building is believed to have served as an inn situated near the junction of Stirling Creek and Main Street in the village of Stirling (also known as Winter Harbor, now Greenport). The oldest part of the building may have been erected by Captain William Booth around 1720. When William Booth died in 1723, the business was inherited by his son, Lt. Constant Booth. In 1786, Orange Webb took out a loan to purchase the property.

Some time in the late 1700s or early 1800s, the house was moved to the corner of Main Street and Route 25 in Greenport, where it was used as a farmhouse. Orange Webb passed the house and property on to his son David, who in turn passed it on to his wife Elizabeth Webb.

After Elizabeth Webb sold the house and grounds at auction in 1820, it was owned by a number of different local families including Kings, Youngs, and Jeromes. In 1941, the Sinuta family bought the house and surrounding farmland. Virtually no modernizations had been made to the house up to that point and the Sinuta family chose to build a modern house on the property. The Webb House fell into disrepair.

In 1954, rumors that the house was going to be demolished led a number of concerned citizens to attempt, unsuccessfully, to raise money to save the structure and turn the house into a museum. In the summer of 1955, George Latham, one of the founders of OHS, purchased the house and moved it to Orient to house his collection of antiques. The house was moved from its foundations and traveled a half-mile down the road and across fields to a barge in Stirling Creek. It was then floated to the foot of Harbor River Road in Orient and transported to its current location along Poquatuck Park. In 1981, Latham bequeathed the house and most of its furnishings to OHS. The Society offers tours of the house during the summer months.