VMS UI_OpenSSL: generate OpenSSL errors when things go wrong.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2063)
(cherry picked from commit c922ebe2)