Aflatoxin M1 (AFM1), a human carcinogen, is found in milk products and may have potentially severe health impacts on milk consumers. We assessed the risk of cancer and stunting as a result of AFM1 consumption in Nairobi, Kenya, using worst case assumptions of toxicity and data from previous studies. Almost all (99.5%) milk was contaminated with AFM1. Cancer risk caused by AFM1 was lower among consumers purchasing from formal markets (0.003 cases per 100,000) than for low-income consumers (0.006 cases per 100,000) purchasing from informal markets. Overall cancer risk (0.004 cases per 100,000) from AFM1 alone was low. Stunting is multifactorial, but assuming only AFM1 consumption was the determinant, consumption of milk contaminated with AFM1 levels found in this study could contribute to 2.1% of children below three years in middle-income families, and 2.4% in low-income families, being stunted. Overall, 2.7% of children could hypothetically be stunted due to AFM1 exposure from milk. Based on our results AFM1 levels found in milk could contribute to an average of −0.340 height for age z-score reduction in growth. The exposure to AFM1 from milk is 46 ng/day on average, but children bear higher exposure of 3.5 ng/kg bodyweight (bw)/day compared to adults, at 0.8 ng/kg bw/day. Our paper shows that concern over aflatoxins in milk in Nairobi is disproportionate if only risk of cancer is considered, but that the effect on stunting children might be much more significant from a public health perspective; however, there is still insufficient data on the health effects of AFM1.