Abbott should start to worry. A return of Rudd as prime minister will see Labor's fortune turns for the better and Abbott's turns for the worse.