Assert that the source/destination address is in kernel space when appropriate. This is mostly to give ourselves an error if we're doing something we were not expecting to be doing.