use and set the fd_set pointers instead of the actual values, as then we

work properly with the multi interface when the user has provided the
fd_sets!