Fix tests of TEST tests, as it were

Fix warning/bug in rc5test
Remove useless/warning-only test from dsatest.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3387)