Lippi was born in Florence to Tommaso, a butcher. Both his parents died when he was still a child. Mona Lapaccia, his aunt, took charge of the boy. In 1420 he was registered in the community of the carmelite friars of the Carmel, where he remained until 1432, taking the Carmelite vows in 1421 when he was sixteen. In his Lives of the Artists Vasari says: "Instead of studying, he spent all his time scrawling pictures on his own books and those of others." The prior decided to give him the opportunity to learn painting.
Eventually Fra Filippo quit the monastery, but it appears he was not released from his vows; in a letter dated 1439 he describes himself as the poorest friar of Florence, charged with the maintenance of six marriageable nieces. In 1452 he was appointed chaplain to the convent of S. Giovannino in Florence, and in 1457 rector (Rettore Commendatario) of S. Quirico in Legania, and made occasional, considerable profits; but his poverty seems chronic, his money being spent, according to one account, in frequent amours. Vasari relates Lippi's visits to Ancona and Naples, his capture by Barbary pirates and enslavement in Barbary, where his skill in portrait-sketching helped to release him. From 1431 to 1437 his career is not accounted for.
In June 1456 Fra Filippo is recorded as living near Florence to paint frescoes in the choir of the cathedral. In 1458, while engaged in this work, he set about painting a picture for the convent chapel of S. Margherita of Prato where he met Lucrezia Buti, the beautiful daughter of a Florentine named Francesco Buti, she was either a novice or a young lady placed under the nuns' guardianship. Lippi asked that she might be permitted to sit for the figure of the Madonna (or perhaps S. Margherita). Under that pretext, Lippi engaged in sexual relations with her, abducted her to his own house, and kept her there despite...