The bridal party and your immediate families should receive an invite and so should guests who have travelled from out of town. When you know there are guests staying at a local hotel the night of your wedding I recommend inviting them to the post-wedding brunch. It’s the polite thing to do.
If somehow they find out there’s a brunch and they weren’t invited they may be insulted and now you’ve turned the wonderful memory of your wedding into an awkward/uncomforable situation.