According to the geographical book of "Dai Nam Nhat Thong Chi" (Dai Nam Comprehensive Encyclopaedia), Yen Binh district formerly belonged to Thu Vat area, Yen Binh district of Tuyen Quang province, spanning along the banks of the Chay River. Yen Binh town - the centre of Yen Binh district today, was formerly Ky Ma commune of Tong Dao Ngan ("Tong” was a feudal administrative unit that was smaller than a district and bigger than a commune today), Thu Vat area, Yen Binh district.

Senior residents of the area said that Ky Ma was once the locality taking the charge of taking care of horses and building the horses' hooves for the army barracks of Gia Quoc Cong Vu Van Uyen and Vu Van Mat, two loyalists of the Le Dynasty (16th century). The brothers helped the Le Dynasty defeat rebelling forces and revitalise the country.

Yen Binh township today has two streets named after the two brothers. Their temples are located on a hill in Group 9 of the township.

On Vu Van Uyen Street, there is a newly-constructed cultural work – Hoang Lien Pagoda, which covers over 4,000sq.m. The pagoda was built with contributions of Buddhist followers.

In 2018, the construction of the pagoda started, with a design comprising a gate, Dai Hung Bao Dien (main building), Nha tho To (house for worshipping heads of the pagoda), and an office of the Vietnam Buddhist Sangha in Yen Binh district, and auxiliary works.

The main building of the Hoang Lien Pagoda covers 360 sq.m, with a height of 9.44m, comprises "Dinh”-plan houses with one main house with a width of 3.8m, and four side houses.

It has typical architectural style of a Vietnamese pagoda with decoration of lotus, daisy, bamboo arranged in dragon shape, and four sacred animals (dragon, unicorn, turtle and the phoenix).

"Hau cung” – the place to worship "Three Treasures” of Buddhism – is decorated in traditional style. The statues of the "Three Treasures” are made with a high level of sophistication and high aesthetics, creating a magical, cozy, pure and peaceful space.

The pagoda creates a highlight for the natural, historical and cultural tourism of Yen Binh – the northern gate of Yen Bai province.