i think this would probably explain the phrase "silly goose"
I imagine the whole point is to say that someone is "sillier than a goose" famously known for, not being silly at all. Arguing that you are so silly, that you look like a goose being silly.