If a week or two passes and he's still awkward, try bringing it up. Be up front and say that you feel things may be weird but don't know why - that you're unsure if you did something to hurt him but you want to make things better. Make sure to re-iterate that you really wanted to go with him, but the timing was off, and that you aren't just saying that to be nice but really mean it.
If things DO change, then STILL bring it up because it's important to crush misconceptions and miscommunications before they get built upon. If you chicken out on this step, you may actually lose him if he decides to give up on you and move on. Communication is always important, people can't read minds, and what may be obvious from your perspective may not be from his. Keep this bit of information in mind even if you two get together, too many couples make the mistake of assuming their partners can "know" what's wrong and why it's wrong.
The best way to bring this up even when things are normalized is to just ask him about his trip, and subtly lead the discussion into the point where he stopped replying. The script kidna follows the same format - you want to make it clear that you WANTED to go and you weren't just politely rejecting. Only in this case, you should muster up the courage to ask HIM somewhere to make up for it. As a "I felt bad rejecting such a kind offer, I still wanna hang with you" kind of deal.
BUT honestly, at some point just take the leap to ask for an actual date. Let him know you're interested in him - he already threw a pretty hard and solid hint he's into you (who invites someone they just met recently on an all expense paid vacation? if he was uggo you'd be posting this story in the creeper threads.) So just put some effort in on your part as well. I'm usually against the idea of office romance, but you seem like you're headset on going through with it anyway, so just FUCKING DO IT.
Good luck, Anon. Get him a tatter tot topped chilli cheese.