Fix XDR decoding of large values in xdr_u_int

Our ancient RPC value internally decodes 32-bit wire values into a
signed long, which is then casted to the appropriate type.
xdr_u_int() contains a check intended to catch wire values that don't
fit into a u_int on platforms with 16-ints, but on platforms with
64-bit longs it was failing on values of 2^31 or larger because the
sign-extended value appeared larger than UINT_MAX.  Fix the check by
casting the value to uint32_t before comparing.

This bug, in combination with a poor choice of types in
kadm_rpc_xdr.c's xdr_krb5_enctype(), prevented negative enctype values
from being transported properly in kadmin's change_password command
result.

ticket: 6753

git-svn-id: svn://anonsvn.mit.edu/krb5/trunk@24210 dc483132-0cff-0310-8789-dd5450dbe970

diff --git a/src/lib/rpc/xdr.c b/src/lib/rpc/xdr.c
index ff67e90f6d021e7811bb62b617d2f1cd185c9e48..8689ee31661b3cdda5496ba9cb82acb3eeb0cf4f 100644 (file)
--- a/src/lib/rpc/xdr.c
+++ b/src/lib/rpc/xdr.c
@@ -145,7 +145,7 @@ xdr_u_int(XDR *xdrs, u_int *up)
                if (!XDR_GETLONG(xdrs, (long *) &l))
                        return (FALSE);
 
-               if (l > UINT_MAX)
+               if ((uint32_t)l > UINT_MAX)
                        return (FALSE);
 
                *up = (u_int) l;