As long you are actively passing, you can stay in the passing lane. The problem is when people start "cruising" in that lane. No matter how fast you think you are going, there will always be someone faster and you will impede traffic sooner or later.
If you are passing a whole group of cars while someone comes up behind you, keep going of course...you may of course need to adjust your speed, but keep going till you can safely move over. But if there is a car right behind you, chances are, you're in the wrong lane.
It's pretty simple.
