If market opens at significantly different price, you may be forced to liquidate and loose more than expected.
Also worth mentioning you might be on the hook to buy it back at any time; after all, the person you borrowed it from may themselves wish to sell it. If widespread, this is the basis of "short squeezes" (e.g. of GameStop fame/infamy), if a lot of short sellers are trying to buy it back at the same time