A challenge of the patent won't take all that long, perhaps as much as a year.
They either can prove prior art or they can't.
If they could then they should have done so before the patent was granted. The fact that they didn't clearly suggests to me that their prior art claim is total BS.