Esprit chairman leaves one day after CEO announces departure