Let them in on the plans, but they cannot be the key to the plans, then say if you are leaving at 10:00 leave. Then tell them you didn't want to be late, since they were, you thought something had come up. If everyone is bringing something for a potluck, give them something that it does not matter if it gets there or not. Then when they do show up you are happy to enjoy there company, if they don't show it does not put a damper on the plans for every one else.