No, dairy is not bad for endurance athletes unless someone has a dairy protein allergy or suffers from lactose intolerance. Dairy products are a good source of calcium, vitamin D, and potassium and provide protein, and carbohydrates, which promote recovery, muscle growth, and bone strength. Dairy products are also rich in leucine, which has been shown to stimulate muscle protein synthesis and help build muscle post-workout 
That said, dairy isn’t absolutely essential for athletes to perform at their best. Emerging research indicates that plant-based diets may improve V02 max, boost fat loss, increase glycogen stores, and reduce inflammation 
For endurance athletes who are vegan, lactose-intolerant, or allergic to dairy protein, there are several plant-based milk alternatives on the market, fortified with calcium, vitamin D, and other nutrients, however, many lack protein. Soy milk offers a comparable amount of protein to cow's milk and can be used in place of dairy for post-workout recovery.
 Jäger, R., Kerksick, C. M., Campbell, B. I., Cribb, P. J., Wells, S. D., Skwiat, T. M., Purpura, M., Ziegenfuss, T. N., Ferrando, A. A., Arent, S. M., Smith-Ryan, A. E., Stout, J. R., Arciero, P. J., Ormsbee, M. J., Taylor, L. W., Wilborn, C. D., Kalman, D. S., Kreider, R. B., Willoughby, D. S., Hoffman, J. R., … Antonio, J. (2017). International Society of Sports Nutrition Position Stand: protein and exercise. Journal of the International Society of Sports Nutrition, 14, 20. https://doi.org/10.1186/s12970-017-0177-8
 Barnard, N. D., Goldman, D. M., Loomis, J. F., Kahleova, H., Levin, S. M., Neabore, S., & Batts, T. C. (2019). Plant-Based Diets for Cardiovascular Safety and Performance in Endurance Sports. Nutrients, 11(1), 130. https://doi.org/10.3390/nu11010130