Put the ice cream out to soften. In a large punch bowl, whisk egg yolks with a few tablespoons of sugar until lighter in color. While whisking, add a few ounces of brandy to "cook" the eggs. Add the cream, then the half and half, then enough milk (around 2/3 of it), then taste. You want the mixture to be a bit richer than half and half. Add a bit more milk if you need to. Add vanilla, then taste, and correct for sugar. Add most of the remaining brandy, remembering to taste - this should have a detectable amount of alcohol, but should not be overpowering or raw tasting. With a scoop, add about half of the ice cream to the punch bowl, and whisk it in. Float the remaining ice cream in the bowl for cooling. Whip the egg whites to medium peaks with a bit of sugar. Add the whites to the bowl, and mix in gently. Grate some nutmeg on top of punch, then pass the grater for guests to garnish as they will.
Some people have issues with raw eggs. If you do, this is not for you. As kids, we used to get it without the alcohol, and we're alright. Recently, I have come to like about 20 oz of Cognac (Ferrand 10, etc - decent, but nice) and the remainder Jamaican rum, like Mt. Gay XO or Appleton 12. We also use Blue Bell Homemade Vanilla, because it's available and fairly rich.