Sure, there may be some lingering jealousy on Petunia's part. She wanted to be "special" like Lily, but wasn't. Their parents were so proud of Lily, and Petunia got left out. Yadda yadda yadda.
I think this is the bigger issue: Petunia hates magic because it killed her sister, and the scar on Harry's head is evidence of that. There may have been a chance for her to make up with Lily, and for them to salvage a relationship. But now that the Potters are dead, she feels incredibly guilty that she will never be able to do that. Perhaps she even thinks that Lily died without knowing that Petunia still loved her.
Too, the letter from Dumbledore explains about Lily's sacrifice for Harry, and how that love and protection live on through Petunia. I don't think it's a stretch to say that Petunia wishes she could have died for Lily and saved her in the same way that Lily died to save Harry.
Basically, all this is to say that Petunia feels guilty and resentful about what happened to Lily, and that's why she treats Harry the way she does. He is a living reminder of who his mother was; he had Lily's eyes, and Petunia has to look into those eyes every day, even though Lily is gone. I know that Lily and Petunia were not on good terms at the time of the Potters' deaths, but they were still sisters and they still loved each other.
Of course, none of this is an excuse for abusing Harry as the Dursleys do, but I think it explains a lot about Petunia. Vernon on the other hand, is a completely different story. Personally, I just think he's ignorant, selfish, a bigot, and very cruel.