I doubt that Hitler would have cared if "we" had called him on anything. If you're referring to Germans, actually, lots of people called Hitler on it. The Social Democrats and Communists in Germany quite vocally. That's why he was never able to really win a free election in Germany. He took power as the result of a shoddy political deal, not because he ever won a majority in the Reichstag. In other words, he was called on it by many people, but he still took power.
So, when you say Hitler should have been "called on it" - how?