gary

It could be erect-crested Penguin although I can not rule out Fiordland.
I remember the Dunedin Penguin rescue centre keeping some injured  Erect-crested Penguin, which indicate an occasional occurrence of the species in southeastern South Island. Also the habitat fits this rocky cliff nesting species.
In contrast, Fiordland Penguin nests in forest, and I remember they are rarely recorded in southeastern South Island.
All by my guess, no point of evident-based ID.