'I was super frustrated to leave the show,' he admits.