Unless you have some understanding with the sender about always being available, or if you knew ahead of time that you'd be receiving a text (e.g., you're making plans for a night out), it's not rude at all to put off responding. That's what asynchronous communication is for. Sometimes, you have to take ownership of your own time and not beholden to the schedules of others.
Putting off a response for days instead of hours, well, that may be a bit much, because it could be a passive aggressive way of punishing the person who texted you. Okay to delay that long for thoughtful, verbose email responses. Not as easy to justify for phone calls and texts.