Did you tried to check how many active WiFi connections [2.4ghz N for direct connection, G or N if you are using router] are in your surrounding [inSSIDer for Android]? Remote Play can crap out if there is too much congestion.
http://crowd-logic.com/blog/wp-content/uploads/2014/02/iNS.png
You cannot force PS4 to work on specific channel, but you can do so with Vita [in WiFi options menus] and with router. Changing active channel has helped several people to get rock-solid connection in their apartments where they can catch a lot of WiFi networks of their neighbors.
This one looks like it has "USB pass through" built-in.
http://www.dx.com/p/plastic-gaming-plastic-hand-grip-for-ps-vita-black-yellow-144071#.VBFlnqO0NNo
edit - nope, it's just locking switch
Thanks for the suggestion but I'm pretty sure that's not the problem. I'm literally in a cabin in the woods and the closest house is at least half mile away.