The residence was designed by Irish-born architect James Hoban in the Neoclassical style. Hoban modeled the building on Leinster House in Dublin, a building which today houses the Oireachtas, the Irish legislature. Construction took place between 1792 and 1800, with an exterior of Aquia Creek sandstone painted white. When Thomas Jefferson moved into the house in 1801, he and architect Benjamin Henry Latrobe added low colonnades on each wing to conceal what then were stables and storage. In 1814, during the War of 1812, the mansion was set ablaze by British forces in the burning of Washington, destroying the interior and charring much of the exterior. Reconstruction began almost immediately, and President James Monroe moved into the partially reconstructed Executive Residence in October 1817. Exterior construction continued with the addition of the semicircular South Portico in 1824 and the North Portico in 1829.

Because of crowding within the executive mansion itself, President Theodore Roosevelt had all work offices relocated to the newly constructed West Wing in 1901. Eight years later, in 1909, President William Howard Taft expanded the West Wing and created the first Oval Office, which was eventually moved and expanded. In the Executive Residence, the third floor attic was converted to living quarters in 1927 by augmenting the existing hip roof with long shed dormers. A newly constructed East Wing was used as a reception area for social events; Jefferson's colonnades connected the new wings. The East Wing alterations were completed in 1946, creating additional office space. By 1948, the residence's load-bearing walls and wood beams were found to be close to failure. Under Harry S. Truman, the interior rooms were completely dismantled and a new internal load-bearing steel frame was constructed inside the walls. On the exterior, the Truman Balcony was added. Once the structural work was completed, the interior rooms were rebuilt.

The present-day White House complex includes the Executive Residence, the West Wing, the East Wing, the Eisenhower Executive Office Building, which previously served the State Department and other departments (it now houses additional offices for the president's staff and the vice president), and Blair House, a guest residence. The Executive Residence is made up of six stories: the Ground Floor, State Floor, Second Floor, and Third Floor, and a two-story basement. The property is a National Heritage Site owned by the National Park Service and is part of the President's Park. In 2007, it was ranked second on the American Institute of Architects list of America's Favorite Architecture.