The palace is named after Mount Potalaka, the mythical abode of the bodhisattva Avalokiteśvara. The 5th Dalai Lama started its construction in 1645 after one of his spiritual advisers, Konchog Chophel (died 1646), pointed out that the site was ideal as a seat of government, situated as it is between Drepung and Sera monasteries and the old city of Lhasa. It may overlie the remains of an earlier fortress called the White or Red Palace on the site, built by Songtsen Gampo in 637.

The building measures 400 metres (1,300 ft) east-west and 350 metres (1,150 ft) north-south, with sloping stone walls averaging 3 metres (9.8 ft) thick, and 5 metres (16 ft) thick at the base, and with copper poured into the foundations to help proof it against earthquakes. Thirteen storeys of buildings, containing over 1,000 rooms, 10,000 shrines and about 200,000 statues, soar 117 metres (384 ft) on top of Marpo Ri, the "Red Hill", rising more than 300 metres (980 ft) in total above the valley floor.

Tradition has it that the three main hills of Lhasa represent the "Three Protectors of Tibet". Chokpori, just to the south of the Potala, is the soul-mountain (Wylie: bla ri) of Vajrapani, Pongwari that of Manjusri, and Marpori, the hill on which the Potala stands, represents Avalokiteśvara.