I have play alot of games on my PS3 with the DS4, while my PS4 was on standby. Not one time have pressing the PS button turned on the system and started controlling both systems.
I have had my PS4 turned on while check something on my PS3 and it have never controlled both. I don't why or how this is happening but if it is true, I am sorry.
As for refunds, I don't know. It sucks.
If a DS4 is hooked up to a PS3 (or PC) and not a PS4, it becomes a standard generic USB controller, which is why it won't turn on a PS4.
However, if you sync one up to a PS4 and then hook it up to a PS3 (or PC), the wireless connection to the PS4 doesn't disconnect since...
A) The PS4 is the only device the DS4 connects to over a wireless connection.
B) The DS4/PS4 connection is the primary connection.
The reason you can control both is because the PS3 accepts standard USB protocols. When the DS4 is plugged into it, the PS3 detects a standard USB controller and acts accordingly. The PS3 has no idea there is already a wireless connection established because it...
A) Has no idea what a PS4 is and can't connect to it's Bluetooth protocol
B) Doesn't see a Dual Shock, it sees a generic wired controller.
As for the return, Sony should go ahead and do it, couldn't hurt.