Add some soft asserts where applicable

This is an internal API. Some of the tests were for programmer erorr and
"should not happen" situations, so a soft assert is reasonable.

Reviewed-by: Rich Salz <rsalz@openssl.org>