Houses aspire to Glory, but by no means to they get there.
Bad deaths in battle, disease, curses, maltreatment of their yeofolk, gambling away their monies… possibly even a terrible love affair that goes horribly wrong? Too many knights errant fallen, few nobles of note to hold up the name… ooo, there’s a thought: an Earl who gives out easy Tests for their family? Lowering the standard for nobility for their House and by implication, the nation?
It rather depends what sort of reaction you want:
“Oh, are they still around?” for a House fallen into penury and having not done much of note.
“Those scum?” for a house convicted of Sorcery, Heresy, or similar. There have been a couple of these in play, probably a bad idea to jump in as a scion of one of those…
“Indeed. I have heard of them…” for a House that failed to keep promises, whipped it’s yeofolk (who likely packed up and left), and generally acted like empty braggarts
“Ah yes. A tragedy worthy of song…” for a House struck down by curses, disease, or bad luck.
Here’s an interesting example of a prosperous House falling: “Lord Tytos was known as the Laughing Lion. He wanted to be loved, so he was generous and forgiving, and was taken advantage of ruthlessly. Those unable to pay their debts, he laughed and forgave them. People laughed at him, and he laughed with them, while the prestige and power of his house melted like a sandcastle in the rain.”
(Okay, that one’s from A Song of Ice and Fire, but it’s a good example of how someone who’s a bad choice for leading a House can screw it up.)