Fix calling convention bug in ecp_nistz256_ord_sqr_mont

The rep parameter takes an int in C, but the assembly implementation
looks at the upper bits. While it's unlikely to happen here, where all
calls pass a constant, in other scenarios x86_64 compilers will leave
arbitrary values in the upper half.

Fix this by making the C prototype match the assembly. (This aspect of
the calling convention implies smaller-than-word arguments in assembly
functions should be avoided. There are far fewer things to test if
everything consistently takes word-sized arguments.)

This was found as part of ABI testing work in BoringSSL.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/8108)