There are probably legitimate concerns about Gibraltar actually. 1. It is practically a tax haven, and has privileges because of London financial markets; 2. it uses outside labour, implying an open border; 3. it has an international airport, again, involves issues to do with freedom of movement.
It you combine that with the, shall we say, Ruritanian tendency amongst UK politicians, you can see why EU is raising Gibraltar.