Harry Potter and the Prisoner of Azkaban is widely considered to be one of the best outings of the entire franchise, and a lot of that is down to the vision that Alfonso Cuarón brought to the franchise. The auteur, who is also behind films like Y tu mamá también, Gravity, and Children of Men, brought a unique take on one of the darkest Potter books, and it really paid off.

Fans of the franchise have long wondered why he didn’t return to direct any more of the movies. Now, producer David Heyman has elaborated on this, explaining the answer is pretty simple.

“We asked Alfonso if he wanted to do the fourth, and he said, ‘Cabrón,'” Heyman said on Harry Potter: The Official Film Podcast, referencing a Spanish slang term. “I think he was exhausted. I look at pictures of Alfonso from then – Alfonso is one of my best friends, and he’s my son’s godfather, and I love him. But I look at him from then, and I just look [at him]… Because he was exhausted.”

Harry Potter and the Goblet of Fire went on to be directed by Four Weddings and a Funeral helmer Mike Newell. He only signed on for one movie, also, before passing the baton to David Yates, who directed the final four films.

Heyman was speaking as part of the Harry Potter: Official Film Podcast, which is set to turn into the official series podcast when the HBO Max series begins airing this Christmas. The new show will adapt Harry Potter and the Philosopher’s Stone in its first season, before tackling a new book with each subsequent season.

You can watch the Harry Potter movies on HBO Max. For more, check out our guides to the best shows on HBO Max to stream now.