If you are planning to have a destination wedding because you want to keep the wedding small and intimate then don't invite more people than those few you want to include. Never assume that because you're having a destination people won't travel to it. They may have always wanted to go to that location and your wedding is the perfect excuse to go.
The general rule is that you are safe to assume that 10% of your invited guests will send their regrets, but when it comes to a destination wedding I recommend you only send invites to those must-haves and not take a chance on inviting others who you don't really want there but feel that you have to invite them.
The good thing about a destination wedding and guest lists is that your family and friends will probably assume you are only inviting a limited number and won't be insulted when they don't receive an invite in the mail.