With few people left to fight for an Britain’s place outside the EU, it may only be a matter of time before we rejoin the bloc