Ever since Britney Spears joined the newly-revamped panel on 'X Factor,' reports have run rampant of her walking off the set during auditions. But when it happened on Monday, she wasn't alone -- in fact, it may have been her fellow judge, Demi Lovato, who was the first to hit the door.

The trouble apparently started during the auditions in Greensboro, N.C. when a drag-queen -- dressed in cat glasses, a bedazzled jean jacket and boots -- reportedly butchered Spears' hit '(You Drive Me) Crazy.' PEOPLE said Brit was so upset by the mess that she stormed off from the judges' table, but it seems there's more to the story.

"Oh, my God, it was just horrible," said L.A. Reid, adding that Spears wasn't the only one who left -- he and Lovato did, too. "We just couldn't take it anymore."

"I felt really uncomfortable, and I had to walk away," Demi explained. "He [thought] that [he did] such a good job, and I felt so bad. I laughed at first, but I felt really bad."

And Britney? Well, maybe she wasn't upset, just hungry: "I was following [Demi] ... She was like, 'Let's go,' and I was like, 'Okay.' It was our [dinner] break anyway."

As for Simon Cowell, he actually stayed put. "Simon has a little more tolerance [and] patience than we do," Reid said. Which begs the question: when did that happen?

Watch Demi Lovato, L.A. Reid and Britney Spears Talk About the Greensboro 'X Factor' Auditions