I think the point of a Tom Swifty is that the adverb does double-duty. On the one hand it depicts a state of mind or emotion appropriate for the phrase described, and at the same time it somehow also touches on the subject itself. Thus in
"I really don't care who stole the flower from my bouquet," said Tom lackadaisically
"lackadaisically" implies not caring, and at the same time, the stolen flower means Tom is now lacking a daisy. Or in
I lost my lawsuit, said Tom plaintively
losing a lawsuit makes one sad and therefore merits a plaintive comment, after filing it has made one a plaintiff.

Just any old appropriate adverb won't do; there should be some wordplay, too.