When you first start reading the poem, you can see that the Duke thinks the Duchess is very beautiful- "paint could not hope to reproduce". She was also kind-hearted and easy to please, but as you read on you realise the Duke thinks of these qualities as 'faults' because she doesn't reserve her attention for him; he believes he is better than everyone else due to his rank and power. He misinterprets the fact that everyone is fond of her and accuses the Duchess for having multiple affairs when actually she was just responding with her natural kindness. This shows that the Duke was very selfish, and you could say he only liked the Duchess because of her beauty.
The Duke Alfonso and Lucrezia de' Medici were only married for two years before Lucrezia died under suspicious circumstances. Many say the Duke poisoned her, which shows how the Duke was only in love with her looks and how selfish he was. In the poem the Duchess' death is caused by the Duke, as shown in the line "[He} gave commands; Then all smiles stopped together". Her death was caused when the Duke realised he couldn't control her; she still had her faults and the Duke hated them. Even after death he tries to control her by trapping her in a painting, where the "spot of joy" on her cheek is frozen in time, leaving the Duchess constantly smiling at those who look at her, if the Duke allows.
Even though it is the Duke narrating the poem and he is the one complaining about the way the Duchess acted, it is not him that you pity- it's the Duchess who you empathise with for having such a horrible husband.