-
Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
Nintendo files multiple Nintendo Switch patents (controller, dock, architecture*)
TheMagician
Member
I wonder if you can use the camera and mic from your phone.
Graphics Horse
Member
Honestly how much stuff this little device can do is so exciting to me. Now we just need some patents about the Pro Controller to confirm gyro
I'd be surprised if it didn't this time around, no excuse for it to not replicate what the grip can do, except for the palm facing IR and maybe NFC.
Skittzo0413
Member
Yeah, that's what I expected. It's interesting, though, as that patent was one of the first things that came to mind when I watched the reveal video, with the Switch's form-factor being seemingly perfect to accommodate the tech. The joycon grip would potentially even allow the use of the passive controllers while docked. But choosing not to include it in this patent would be an odd decision if they were actually going ahead with using the functionality in Switch.
It does introduce a couple of questions, though, as there is more going on on the sides of the Switch than the patent covers. It's difficult to see in most photos, but the lighting in this photo allows you to see a few regularly spaced marks down each side of the Switch:
Here's a zoomed version which I've brightened a bit to make it easier to see:
There's clearly a lot more going on here than you can see in the patent diagram:
Aside from the communication interface at the bottom, the only feature the patent shows along the side of the Switch is a single hole to allow the controller to lock into place. It's possible that what we're looking at is just additional grooves/holes/whatever to help guide the joycon into place and lock it there, but I don't really see the need for it.
Anyway, it'll be interesting to finally find all these things out come January.
That's a good find, I'm not sure what that could be when looking at the patent. You could very well be right that those are the IR communication windows talked about in the other patent... It's also possible that there have been some more recent patent applications filed which aren't published yet, showing some of the things that are absent from this patent application (dock shape, kickstand location, SD card slot behind kickstand, etc.) which may include those IR windows. I guess we'll get a better look at that come January.
Looking at those images again, they look like they could be little roller-type protrusions designed to help smoothly slide on the joycons. Or maybe they make the *Switch* noise heard in the trailer.
But the hardware was still capable of supporting 8 separate wireless controllers. That's the point you were making right? Why would the Wii U be capable of that but not the Switch?
No, not necessarily. There are some features described which are separate embodiments that cannot coexist together. Like the IR camera on the button of the joycon R can be a visible light camera instead of IR- you can't have it be both. Also consider the last few images showing the more standard slider on the sides of the tablet- it would be hard to have both that and the setup shown in the earlier figures at the same time.
The final product will be designed to do a lot of the things described in most of the main embodiments, but not everything there.
One theory from the past month is that the IR camera on the joycon R could be used to scan QR codes.
The Switch has two IR LEDs on the upper part of the screen, which can be seen, while it is docked. That's probably why the screen isn't completely covered (also a little space for some notifications). This way a IR sensor could work while docked and on the go.
The big question is, how are supposed to use the right Joycon, since it would be needed to be hold upside down? You can't reach the ZR and R shoulder buttons, which would be perfect as triggers (like the B-Button on the Wii-Remote).
Edit: Wait I am wrong. There actually are no IR LEDs on the Switch ... So I have no idea.
-shadow-
Member
I honestly wouldn't be surprised if their reasoning for excluding it is "You can grab the system from the dock and have a headphone jack." We know that the Joy-cons won't have them so yeah...
Oh, I see, thanks.
It was a genuine question on my part (I know little about inner IR camera workings) and I've totally missed the emitter references in the patent, by the way.
I really really hope IR gets a second chance, it was a truly brilliant and revolutionary addition to traditional control schemes IMHO.
Jorok Goldblade
Member
The IR is probably just used for initial calibration, the some other tech (magnetometer / compass sensor probably) is used to prevent drift.
You wouldn't be using that screen for the VR anyway. Patent says headset.
Skittzo0413
Member
I refuse to use the hyphen when "joycon" just looks so much better and is much easier to type.
I like it. There will probably be name variations for other types of joycons, such as those from the patent pictures, or others that could come.
I repost this picture as I love the idea of interchangeable controllers:
CaviarMeths
Member
^ Man, I forgot about those multi-directional clickable scroll wheels. The possibilities for that were vast. Now I'm going to miss it.
Yeah, it wouldn't surprise me if they're rollers or something like that to guide the joycon into place, but it would be a little unusual to need so many of them. (I'm also guessing that the switch noise is the locking mechanism clicking into place)
It does seem like there are a couple of different things going on there. There are two smaller, bright circles equidistant from the top and bottom and then a number of evenly spaced elongated marks which are a bit duller (or simply not as reflective at this particular angle). The two brighter marks would actually seem pretty well placed for little IR sensors for the passive controller tech, as between the two they'd cover the entire controller, and the fact that they're round and reflective would be a match for small lenses.
Of course, I could just be reading too much into a handful of pixels, as they're also well placed for a structural element, for example locking pins, or even just screws holding the main Switch body together.
Starchasing
Member
The IR camera is for hand recognition. Thats what the patent says:
[0211] A window portion 68 is provided in the lower side surface of the housing 51. The right controller 4 includes an infrared image-capturing section (an infrared image-capturing section 123 shown in FIG. 31), the details of which will be described later, to detect a hand movement and/or the gesture, etc., of the user by means of the infrared image-capturing section 123.
[0451] The right controller 4 includes the infrared image-capturing section 123. The infrared image-capturing section 123 includes an infrared camera for capturing an image in a region around the right controller 4. In the present embodiment, the infrared image-capturing section 123 is used to capture an image of a hand of a user. The information processing device 1 identifies an input made with the hand (e.g., a gesture input, etc.) based on information (e.g., the position, the size, the shape, etc.) of the hand of which the image has been captured. The infrared image-capturing section 123 can rely on ambient infrared illumination, or in one example includes an illuminating section for outputting infrared light
CrunchyFrog
Member
Reads like the same stuff the MS Surface Pro keyboard add-on does, yeah.
Graphics Horse
Member
Edit: oh shit I should have read that description: :lol
It could still double as both, maybe? It won't be interfering with another purpose when the cover is used.
Skittzo0413
Member
Yeah it's a very interesting picture and you're right that it's very hard to gather much from it, but it's certainly something that's not shown in the patent application. Another thought is that the two shapes outside of the central shape could be little indents where the inner shoulder buttons of the joycons can be recessed. And it's a bit odd that it looks like there is an upper terminal in addition to the lower terminal shown in the patent application. Or rather than terminals those could be locking engagement members. Very interested to see some detailed shots of this thing!
And it's definitely an indication that they've added/removed features from the patent application since the June filing date.
Right, the user I was responding to was asking about how you'd scan QR codes without a camera though, and I believe an IR camera could be used for that in addition to the biometric/gesturing uses.
Isn't an "angular velocity sensor" the same thing as a gyroscope? Or am I mistaken there?
Correct. It's like my Pixel tablet (which I am using to post this): it detects magnets in specific parts of the case for shutting the screen off. Pixel tablet has a handful of magnets for its keyboard attachment.
It's cool stuff and very handy, but nothing earth shattering.
This, please. I want to be able to play Splatoon Switch on the Pro Controller the way it was meant to be played, with gyro controls.Honestly how much stuff this little device can do is so exciting to me. Now we just need some patents about the Pro Controller to confirm gyro
Graphics Horse
Member
Even the DS's lid does it with a magnet sensor
It's probably unlikely...but I don't suppose a magnetic force sensor built in to the Switch could be used for the positional tracking of the joy-cons?
Just wondered if they might have a better tracking system this time so they track more like a PlayStation Move but without the need for a big glowing bulb.
It would certainly make sense if they do intend to do something VR wise.
Boss Doggie
all my loli wolf companions are so moe
I like it. There will probably be name variations for other types of joycons, such as those from the patent pictures, or others that could come.
I repost this picture as I love the idea of interchangeable controllers:
whoa, this looks like some kamen rider add-on lol
Starchasing
Member
I like it. There will probably be name variations for other types of joycons, such as those from the patent pictures, or others that could come.
I repost this picture as I love the idea of interchangeable controllers:
The patent says that a controller without electronics can be attached to the R Joy Con. the movements would be detected with the IR camera... any ideas of how that would look like?
joshbob1985
Member
I am now quivering with excitement,
Skittzo0413
Member
Does it say that specifically? If so that sounds exactly like the patent from earlier this year that Thraktor has been talking about in this thread.
Graphics Horse
Member
Presumably attachments would have a pair of rails like the one on the tablet for the right controller to slot into, with some extention on the bottom end, then you'd hold the whole thing backwards and have a fishing rod crank or whatever at the top. The sensor would see specially marked internals rotating through a special window in the attachment.
Starchasing
Member
Yes, it says that a controller device can be atached and that the ir camera would see the controller device movements
The concept of the patent is; the portable unit communicates wirelessly with tethered unit to access the wired connection. Semantics of whether the Wii U Gamepad has a typical processing guts or not, the method of wireless communication and/or range has absolutely no baring on the matter, especially since the patent doesn't go into great details about the requirements or limitations of this feature. To reiterate; The Wii U Gamepad can and do wirelessly communicate with the stationary device to access the wired connection. I won't engage in any further argument with anyone who insists it doesn't.
On another note...
The most surprising thing for me is just how many features they've managed to cram into those little joy-cons. I'm really curious about the battery life. I kinda do hope there's enough space to fit a AAA battery, and that Nintendo or a 3-party offers the option, cause I'm a big fan of rechargeable AAA/AA.
Also, with all the comments of how weird it is to have the IR pointer at the botton of the joy-con, it may actually work well with "side grip" method of holding the joy-con(the 'new' method shown in the filed images, but face down instead). That way, you can point, and your fingers have easy access to the top shoulder buttons(which now are comfortably at the when holding the controller face down in "pointer mode.")
My guess is that the locking mechanism is at the top (which makes sense as the release button is at the top of the joycons) and the communication terminal is still at the bottom where it's best protected.
I suppose it shouldn't be too surprising that a few things would change from a patent filed the best part of a year before launch, but it is interesting to consider what those things might be. For example, if you look at the photo I posted before, there appears to be a difference between the two sides of the Switch:
On the left side of the Switch we can see the three central elongated marks (although less visibly elongated), along with a mark down at the bottom alongside the terminal, but we can't see the two bright circular marks I was talking about before. Now, this could simply be down to the different angles of the two Switches relative to the camera and light source, and that they're still there but simply not visible. However, if they are only on one side it makes their purpose particularly interesting.
As an example, let's say they are IR cameras. Nintendo actually have a couple of slightly older patent applications (which Disorientator posted to GAF here) which deal with using IR cameras for hand-tracking. The second one in that post (20160073017) is quite interesting for a few reasons:
Firstly, the patent deals with an IR camera situated on the right hand side (but not the left) of a tablet to use for hand-tracking. Secondly, the two named inventors of this patent, Yoshiyasu Ogasawara and Shinichi Kinuwaki, are also named inventors on the later patent dealing with passive controllers (along with two others, although none of the four are named on the patent in this thread).
What we're looking at in the photo is a pair of small dots on the right hand side (but not the left) of a tablet which happen to match the size and reflectivity one would expect for small IR cameras. This patent uses a single IR camera combined with a simple linear depth sensor, but it's possible that Ogasawara and Kinuwaki realised that they could implement both of their inventions by using two IR cameras instead of one, allowing the Switch to do the following:
- Full 3D hand tracking (using basically the same technique as Leap Motion)
- Support for cheap passive controllers with buttons anywhere on the controller (a single camera probably wouldn't be able to cover the entire thing)
- Support for 3D sticks/slide pads on the passive controllers via dual-camera depth measurement (this isn't covered by the controller patent, which only mentions buttons and rotating dials)
Furthermore, if they combined it with the first patent in Disorientator's thread (also by Ogasawara and Kinuwaki) they could use RGBIR sensors instead of IR and do all of the above, plus in theory it would be able to make full colour 3D captures of real-world objects to bring into games (not covered by any of the patents to my knowledge, but should be technically possible once you combine everything together).
Nintendo have a lot of experience working with cheap IR cameras as input methods from the Wii, and there's no reason they couldn't do something like this quite cheaply (although the colour 3D captures would require slightly more expensive higher-res cameras). We've also had hints at unannounced features, and comments like Todd Howard's discussion of an NDA'd demonstration at E3 as "probably the best demo I've ever seen" might make sense in the context of some crazy unannounced tech like this.
I just had a look through, and it says it here:
It seems strange to locate the IR camera at the bottom of the right joycon grip for this purpose, though. The controls would have to be located below the right joycon, which would make them near-impossible to use while attached and awkward when not attached. Especially when you could include the camera on the side of the Switch body instead (even just one camera on one side) and the attachment rails are already there and can accommodate an ergonomic control layout. It's possible that they've just included it here speculatively (as it's described as an example of alternative use rather than a core feature), or that they have changed their mind since the patent was submitted and moved it onto the Switch body itself.
I suppose it's also possible that they just included it here in a limited fashion to cover their bases, and were also working on another more detailed patent of the "extension controller" mechanism itself (as described above) and we'll see it in a few months.
It does warrant the question, though, that if they have added IR cameras the to side of the Switch since this patent was issued, then why do photos show us that there's still one on the bottom of the right joycon? Wiimote-style pointing tech (which I think most of us were assume it was for) doesn't seem to be covered in this patent, and all the other functionality which is mentioned could probably be as well or better handled by an IR camera in the Switch body.
Starchasing
Member
My guess is that the locking mechanism is at the top (which makes sense as the release button is at the top of the joycons) and the communication terminal is still at the bottom where it's best protected.
I suppose it shouldn't be too surprising that a few things would change from a patent filed the best part of a year before launch, but it is interesting to consider what those things might be. For example, if you look at the photo I posted before, there appears to be a difference between the two sides of the Switch:
On the left side of the Switch we can see the three central elongated marks (although less visibly elongated), along with a mark down at the bottom alongside the terminal, but we can't see the two bright circular marks I was talking about before. Now, this could simply be down to the different angles of the two Switches relative to the camera and light source, and that they're still there but simply not visible. However, if they are only on one side it makes their purpose particularly interesting.
As an example, let's say they are IR cameras. Nintendo actually have a couple of slightly older patent applications (which Disorientator posted to GAF here) which deal with using IR cameras for hand-tracking. The second one in that post (20160073017) is quite interesting for a few reasons:
Firstly, the patent deals with an IR camera situated on the right hand side (but not the left) of a tablet to use for hand-tracking. Secondly, the two named inventors of this patent, Yoshiyasu Ogasawara and Shinichi Kinuwaki, are also named inventors on the later patent dealing with passive controllers (along with two others, although none of the four are named on the patent in this thread).
What we're looking at in the photo is a pair of small dots on the right hand side (but not the left) of a tablet which happen to match the size and reflectivity one would expect for small IR cameras. This patent uses a single IR camera combined with a simple linear depth sensor, but it's possible that Ogasawara and Kinuwaki realised that they could implement both of their inventions by using two IR cameras instead of one, allowing the Switch to do the following:
- Full 3D hand tracking (using basically the same technique as Leap Motion)
- Support for cheap passive controllers with buttons anywhere on the controller (a single camera probably wouldn't be able to cover the entire thing)
- Support for 3D sticks/slide pads on the passive controllers via dual-camera depth measurement (this isn't covered by the controller patent, which only mentions buttons and rotating dials)
Furthermore, if they combined it with the first patent in Disorientator's thread (also by Ogasawara and Kinuwaki) they could use RGBIR sensors instead of IR and do all of the above, plus in theory it would be able to make full colour 3D captures of real-world objects to bring into games (not covered by any of the patents to my knowledge, but should be technically possible once you combine everything together).
Nintendo have a lot of experience working with cheap IR cameras as input methods from the Wii, and there's no reason they couldn't do something like this quite cheaply (although the colour 3D captures would require slightly more expensive higher-res cameras). We've also had hints at unannounced features, and comments like Todd Howard's discussion of an NDA'd demonstration at E3 as "probably the best demo I've ever seen" might make sense in the context of some crazy unannounced tech like this.
I just had a look through, and it says it here:
It seems strange to locate the IR camera at the bottom of the right joycon grip for this purpose, though. The controls would have to be located below the right joycon, which would make them near-impossible to use while attached and awkward when not attached. Especially when you could include the camera on the side of the Switch body instead (even just one camera on one side) and the attachment rails are already there and can accommodate an ergonomic control layout. It's possible that they've just included it here speculatively (as it's described as an example of alternative use rather than a core feature), or that they have changed their mind since the patent was submitted and moved it onto the Switch body itself.
I suppose it's also possible that they just included it here in a limited fashion to cover their bases, and were also working on another more detailed patent of the "extension controller" mechanism itself (as described above) and we'll see it in a few months.
It does warrant the question, though, that if they have added IR cameras the to side of the Switch since this patent was issued, then why do photos show us that there's still one on the bottom of the right joycon? Wiimote-style pointing tech (which I think most of us were assume it was for) doesn't seem to be covered in this patent, and all the other functionality which is mentioned could probably be as well or better handled by an IR camera in the Switch body.
I don't think the Ir is for wii mote style pointing tech. Many patents and other signs point towards hand recognition and or cheap controller extensions.
If you put the IR in the side of the tablet its great for additional controllers, but would not work while docked. The grip would need IR cameras too.
If you put IR on the R Joy Con, its great for hand recognition but it would make additional controllers a bit weir due to the position of the IR camera.
My bet is that they will use it for hand recognition. They have been trying similar game mechanics in wii party u.
I like it. There will probably be name variations for other types of joycons, such as those from the patent pictures, or others that could come.
I repost this picture as I love the idea of interchangeable controllers:
lol this will never happen. Or it will and be totally unsupported like 95% of Nintendo peripherals.
Cuningas de Häme
Member
That one guy who posted his blog (Nintendotimes it was, I think) pointed out that one picture where the joycon was held like a joystick...
Imagine a flying game with real joysticks! That is something that's missing from other consoles...
Ooh, ooh! Starfox with real joysticks! Or Rogue Squadron...
Imagine a flying game with real joysticks! That is something that's missing from other consoles...
Ooh, ooh! Starfox with real joysticks! Or Rogue Squadron...
Skittzo0413
Member
Very intriguing theory. I do like the idea of two IR cameras detecting gestures leap-motion style, but my big question about this theory is- why would they have the IR controller sensing functionality relegated to the right side of the Switch when the right joycon has been shown in the patent to have more functionality typically than the left joycon? The idea behind the IR connection patent is to make very cheap controllers that likely have no electronic components themselves, yet the right joycon has been shown to have IR functions and NFC functions. If Nintendo wanted to allow you to replace only a single joycon with a cheaper one, completely devoid of electronics, then why would they choose that right one with more features rather than the left one with few features (especially when the left one is shown to be replaceable even in this patent application). This image would fit your theory, but it's still strange to me that the right joycon- the one being replaced- is the one with more functionality, not less:
The only reason I can think of to do that is if the IR windows on the right side of the Switch (assuming that's what they are) is intended mainly for gesture/object recognition with the controller communication being either a minor feature or an afterthought.
And to answer your question at the end, the patent explicitly discusses using that lower IR window on the right joycon for biometric identification (ID user by their vein pattern) fairly extensively, so I'd be willing to assume that's the major use of that feature. It could still be useful for simulating touch input on the TV as an afterthought, even without such a mention in the patent application.
Also, I just thought of this- could the IR cameras on the side of the Switch tablet be usable as positional trackers when the Switch is placed into a HMD? Assuming the HMD has windows allowing the light through.
forknknife
Banned
I don't get it. Isn't the IR sensor on the bottom of the right joycon? Then it will be pointing downwards in every scenario, right? Except if you would turn it around that is, but then you would be griping the controller from the L/R buttons side which would be awkward as fuck, right? Or am i not getting this? Or, in the case it is for hand gesture recognition, how would it work with the IR facing downward?
My head is hurting, Nintendo!
My head is hurting, Nintendo!
Skittzo0413
Member
(Edit: One of) The functions of that IR sensor mentioned
forknknife
Banned
Wow, really? For what purpose? Anti theft? To log in? seems like too much trouble for just a simplified way to login. Maybe this could mean they're going serious about their online and it's a way to simplify payments and so on.
You're right, 5% represents a niche, Yet it's more than enough to make the medium better than it currently is. You don't seem to have enjoyed the dreamcast days, with so many arcade games and peripherals. I'm the kind of gamer that wants this back.
Yes, it does talk wirelessly to a different unit that has a tethered connection. No it wouldn't work the same way. The Wii U Gamepad uses a unique implementation of Wi-Fi Direct to transmit audio, video, and input back and forth between the console. There are no network packets passed to the Gamepad; they strictly go to the console.
It's the difference between a terminal hooked up to a mainframe (Wii U + Gamepad) versus your Wi-Fi router disseminating your Comcast (Switch).
So again, they appear the same if by same you mean "I have a screen that is showing a thing and I can send inputs and it responds".
iamaustrian
Member
It seems this Thing is Packed with All Sorts of stuff. I Hope cooling won't be a problem
GameXplain time, 50 min video: https://www.youtube.com/watch?v=wlWBK8sq3eQ
GameplayWhore
Member
I recently found out that my Virtual Boy seems to no longer function, so I'm up for some VB VC. They could even "Game Boy Color" it and allow you to choose alternate hues to use for the view. Or alternately try to figure out some sort of context- or location- sensitive colour alteration, so the monochrome game appears to be multicolored, not unlike how the Vectrex used physical coloured transparent overlay sheets to spice up the grayscale vector graphics screen it came with. Not an easy proposition for most games, but fun to think about.
Refreshment.01
Member
i talked about this in the list... The Joycon doesn't seem to have a battery inside, just a plug and power controller to recharge the Joycons.
And hopefully it doesn't contain a battery, this would mean the JoyGrip would be cheap enough to include in one of the SKUs or in all of them (assuming there is more than one).
Talked about this back in the October video when discussing the function of the LEDs in the grip.
To clarify, The LED's in these Switch controllers work like the ones in the Wii Remote to id player number and battery status. if you watch the Switch Pro in the trailer, you will see that there's 4 LEDs in them, so it clearly means there can be 4 Pro controllers connected to a single Switch.
The patent also talks about the mix and matching of different controllers and Joycons set ups (NES style, Dual handed, within shells, etc). So it seems likely that, for example, you could have 3 Joycons and one Pro connected to a single device.
Has anyone seen Google's Project Soli demo and PointGrab's hand recognition tech on youtube? It seems PointGrab functions is what Nintendo is aiming to do as their touch screen functions on tv solution. Project Soli functions maybe for the IR and depth sensors on the side of the tablet? Maybe its intended for you to place the Joycons on the table in front of you and it will recognize your hand for basic navigation purposes during non gaming apps which is why they chose for the IR sensors to be on the bottom of the Joycons?
You have to throw all that out and look at one thing. Can Nintendo make a fun VR game or incorporate fun VR experiences into their games with this tech?
And the answer points to yes.
Do the math. Virtual Boy came out 20 years. Switch is a traditional console and handheld first and foremost. VR capability is included for next to nothing except for the cost of the headset. Nintendo makes great games and experiences using low cost tech.