Handle BN_mod_word failures.

As of 37258dad

 and the corresponding upstream
change, BN_mod_word may fail, like BN_div_word. Handle this properly. Thanks to
Brian Smith for pointing this out. See BoringSSL's
44bedc348d9491e63c7ed1438db100a4b8a830be.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>

GH: #1251