I would be honest early so that she has enough time to find a babysitter. If you don’t feel comfortable saying that the wedding isn’t really appropriate for children, due to the menu, time and other factors, you can phrase it in a way that sounds like you’re thinking of her and her spouse.
For example: “I don’t think you and Bill will have fun if you’re worried about Sara. Why don’t you hire a sitter to watch her so you can celebrate with us.”
You may also want to add that you want her in the family photos and then the sitter can take over.
If your wedding is at a hotel you can have the sitter stay in a hotel room so the parents can run up if they want to. And you should splurge on a few kid-firendly DVDs, snacks and toys -- a kid-friendly guest room gift of sorts.
If your sister insists that her daughter attends the wedding don’t fight her on it, just order a kid's meal and make sure there’s room at her parents’ table for her.