Yes you lose those points. But they are not replaced by zero points.
If he made the semis, the points earned for winning drop off and would be replaced by the semis points. If Nadal or Djokovic were close enough in the rankings, or one of them one, they possibly could pass Federer.
If for some reason like injury Federer doesn't play Wimbledon. Or if something crazy happens like he loses early, the Wimbledon points falling off his ranking would be replaced by points from another event. The rankings are based on your best "X" results over a year. I don't know what X is now, it used to be 12. If it is still 12, then whatever Federer's 13th best result over the last 12 months is, that would be added back to his ranking points.
To summarize, his total ranking points would go down, but probably not by the full amount of whatever a Wimbledon win is worth.
Compare this to say whatever happens to Serena Williams, Venus, or Sharapova. They don't play in a lot of tournaments, so sometimes when they don't defend points, their points from 12 months ago drop of and are replaced by zero, which is why their rankings go up and down so much.