In 1783, Nguyen Anh was defeated by Tay Son and ran away to Con Dao Island. Together with fishermen, Lord Nguyen Anh founded three villages: An Hai, An Hoi, and Co Ong. Nguyen Anh sent Prince Canh followed by Ba Da Loc to France to ask for reinforcements to fight against the Tay Son troops. Lady Phi Yen, the second wife of Nguyen Anh, advised him not to do so. Nguyen Anh got angry with her and thought that his wife might have been dealing with Tay Son troops.

He intended to kill her. Just before Tay Son troops attacked the island, Nguyen Anh put his wife in prison and went to sea. Prince Cai, who was then four years old, wanted his mother to come with them. Nguyen Anh got angry and threw his son into the sea. The body of Prince Cai drifted onto Co Ong Beach where it was buried by villagers.

According to the legend, Lady Phi Yen was rescued from the stone prison by a tiger and an ape. She lived with the villagers, taking care of Prince Cai's grave. After an unfortunate encounter with a man, she killed herself to keep her pride.

The villagers mourned her and built a temple in her honour. In 1861, the French colonialists occupied the island and destroyed the temple. In 1958, the temple was rebuilt on its present location.