When Fudge says “The downfall of the self-styled Lord Voldemort”, ze is referring to when Voldemort tried to kill Harry as an infant.
Harry Potter fans will remember Fudge being afraid to say “Voldemort” out loud. In Voldemort's Children, I'm not including the issue of being afraid to say the name. Why not? Because it doesn't add to the themes I'm trying to develop. (By comparison, in Harry Potter, that convention does add to a major theme – namely, the theme of overcoming one's fear. Voldemort's Children sneers at that theme.)