Anthony was born in Portland, Oregon on May 15, 2000. He's the son of Crystal McCrary and Greg Anthony. Greg won an NCAA title with UNLV in 1990 and was an 11-year NBA veteran. As a child, Cole moved to New York City. Anthony originally played baseball but decided to focus on basketball in fifth grade. He lived in New York City through his junior year of high school before transferring to Oak Hill. During Anthony's freshman and sophomore years at Archbishop Molloy, he excelled and led his team to the CHSAA Class AA city championship finals. As a junior, Anthony averaged a league-high 23.4 points, 7.8 rebounds and 4.1 assists per game and was named to the USA Today All-USA third team. At Oak Hill as a senior, Anthony averaged 18.5 points, 10.2 rebounds and 10.2 assists per game and became the first player in school history to average a triple-double. Anthony was voted onto the 2019 USA Today All-USA first team. Anthony also won MVP at the 2019 McDonald's All-American Game, the Nike Hoop Summit and the Jordan Brand Classic. After the 2018-19 high school basketball season, Anthony was named the Virginia Gatorade Player of the Year for success in both basketball and academics. As a freshman at North Carolina, he was named to third-team All-ACC despite missing 11 games and averaged 18.5 points, 5.7 rebounds and 4.0 assists per college contest.