Add missing return value checks

Also correct the return value from the the "prime" application

Reviewed-by: Richard Levitte <levitte@openssl.org>