VMS UI_OpenSSL: if the TT device isn't a tty, flag instead of error

On all platforms, if the controlling tty isn't an actual tty, this is
flagged by setting is_a_tty to zero...  except on VMS, where this was
treated as an error.  Change this to behave like the other platforms.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2064)