Nintendo Patent Application - Accessory and Information Processing System

#1
So 3 Nintendo patent applications were published today.

They seem to be a continuation on the concept also discussed here, which basically talks about a portable device with an infrared camera, distance sensor, etc.

2 of them (United States Patent Application 20160231807 and United States Patent Application 20160232675) seem to explore the hand/gesture tracking functionality (as in the older patent) in more detail, but the 3rd one is a bit more interesting, especially in regards to the latest NX rumors.

It talks about attaching an accessory (e.g physical buttons) to the side of the portable device, and using the infrared camera to read the button presses instead of actually using any form of electrical connection between the two.

It took me a while to realize, but this basically means that the accessory (physical buttons in this case) doesn't have to contain any electrical components thus significantly lowering its' cost, as it's basically a plastic shell with the camera/software doing all the work.

Questions raised:
- Same infrared/camera setup must be installed on the other side of the device if two attachable accessories are to be supported. (higher cost)
- Will latency be an issue? How fast can the infrared camera/software detect the button presses without errors?
- What about analog sticks? The patent shows the detection of rotational movement (e.g. turning a knob). How would something similar work for the analog stick and how precise would be?


United States Patent Application 20160231773

Publication Date: 08/11/2016
Filing Date: 01/29/2016

Abstractt:
An example accessory can be attached to an information processing device. The accessory includes a housing, a movable operation section, and a movable portion. An inside of the housing is visible from an outside thereof through at least a portion thereof. At least a portion of the operation section is exposed on the outside of the housing. The movable portion is placed at a position inside the housing that is visible from the outside of the housing through the portion of the housing. A position and/or an attitude of the movable portion changes in response to the operation section being operated.
BACKGROUND AND SUMMARY
There are conventional accessories which, when connected to a portable device, add functions to the portable device. For example, there are techniques where an external controller as an accessory is connected to a connector provided on a controller of a game device. This enables a wider variety of game operations using two controllers.

Conventional accessories have a circuit configuration for communicating with a portable device or a circuit configuration for performing an information process for generating information to be transmitted to a portable device, and there has been room for improvement in simplifying the configuration of the accessories.

Thus, the present specification discloses an accessory, an information processing device, an information processing system, a storage medium storing an information processing program, an operation determination method and a process performing method, with which it is possible to simplify the configuration of the accessory.


FIG. 8 shows an example operation device before and after it is attached to a portable device;


FIG. 9 shows an example configuration where an operation device is attached to a portable device;


FIG. 6 shows the external appearance of a non-limiting example operation device;
As shown in FIG. 6, a hole (opening portion) 21a is formed in the housing 21. Thus, the operation device 20 is configured so that the inside (the inner side) of the housing 21 can be seen through the hole 21a. Note that the housing 21 may have any configuration such that the inside of the housing 21 is visible from the outside through a predetermined portion of the housing 21. For example, in other embodiments, the predetermined portion may be formed by a transparent member instead of the provision of the hole 21a in the predetermined portion.

Here, in the present embodiment, a reference marker 21d is provided on a wall surface on the inside of the housing 21 (the inner wall surface of a surface 21c in FIG. 6). Movable portions 23A to 23D to be described later are provided on the inside of the housing 21 (see FIG. 7). The hole 21a is formed at such a position that the movable portions 23A to 23D and the reference marker 21d are visible when one sees the inside of the housing 21 from the outside of the housing 21 through the hole 21a.


FIG. 7 shows an example internal configuration of the operation device shown in FIG. 6;
FIG. 7 shows an example internal configuration of the operation device shown in FIG. 6. Note that FIG. 7 shows only a part of the configuration inside the housing 21. As shown in FIG. 7, the operation device 20 includes the movable portions 23A to 23D provided inside the housing 21. Note that where the movable portions 23A to 23D are not distinguished from each other, the designation “movable portion 23” may be used to refer to any of the movable portions 23A to 23D.

As shown in FIG. 7, the movable portions 23A to 23D are connected to the operation buttons 22A to 22D, respectively. That is, the movable portion 23A is connected to the operation button 22A, the movable portion 23B to the operation button 22B, the movable portion 23C to the operation button 22C, and the movable portion 23D to the operation button 22D. The movable portion 23 moves (downward in the present embodiment) in response to an operation on the operation button 22 connected thereto. In the present embodiment, each operation button and a movable portion connected thereto are configured as an integral part. Note however that the movable portion may be configured so as to be moved in accordance with the movement of the operation button, and an operation button and a movable portion connected thereto may be separate parts in other embodiments.

In the present embodiment, at least a portion of the movable portion 23 (the hatched area shown in FIG. 7: hereinafter referred to as a “recognition portion”) is of a material (color) such that the infrared camera 4 can distinguish this portion from other portions (other than the portion of the reference marker 21d) of the wall surface on the inside of the housing 21. The recognition portion may be of a retroreflective material, for example. Then, the infrared light from the illuminating section 7 is reflected more toward the infrared camera 4, making it easier for the infrared camera 4 to recognize the recognition portion. This also makes it easier for the infrared camera 4 to recognize a recognition portion that is located away from the infrared camera 4. The movable portion 23 is connected to the lower portion of the operation button 22. Here, for each of the movable portions 23A to 23C, the recognition portion is placed under (directly below) the corresponding operation button 22. For the movable portion 23D, on the other hand, the recognition portion is displaced from a position that is directly below the corresponding operation button 22D (see FIG. 7). That is, the recognition portions of the movable portions 23A to 23D are arranged so that the recognition portions are visible through the hole 21a.


FIG. 11 shows an example captured image obtained by an infrared camera;
FIG. 11 shows an example captured image obtained by the infrared camera 4. The captured image shown in FIG. 11 is an image to be obtained when the operation buttons 22A to 22D are not operated (in the unoperated state). As shown in FIG. 11, the captured image includes an image (reference marker image) 30 of the reference marker 21d and images (movable portion images) 31A to 31D of movable portions 23A to 23D. Thus, the reference marker 21d and the movable portions 23A to 23D are placed at such positions as to be included in the image-capturing range of the infrared camera 4, with the operation device 20 attached to the portable device 1. Thus, with the operation device 20 connected to the portable device 1, the infrared camera 4 can capture an image of the reference marker 21d and the movable portions 23A to 23D inside the housing 21 through the hole 21a of the housing 21. That is, the following members (a) to (d) are placed at such positions that the infrared camera 4 can capture an image of the reference marker 21d and the movable portions 23A to 23D in such a state as described above:

(a) infrared camera 4 of portable device 1;

(b) hole 21a of housing 21 of controller device 20;

(c) reference marker 21d of controller device 20; and

(d) movable portions 23A to 23D of controller device 20.


FIG. 12 shows another example captured image obtained by the infrared camera 4. The captured image shown in FIG. 12 is an image obtained while the operation button 22A is operated (depressed). When the operation button 22A is operated, the movable portion 23A moves downward. Therefore, in the captured image, the movable portion image 31A is located below its position in the unoperated state, as shown in FIG. 12. Thus, when an operation button 22 is operated, the position of the corresponding movable portion image 31 changes from its position in the unoperated state. Therefore, based on the position of the movable portion image 31 in the captured image, it is possible to determine whether the operation button 22 corresponding to the movable portion image 31 is operated.
 
#2
Hmm. That does look sort of like a controller looking thing sticking to the side of the screen.

And yes, I know this doesn't indicate how the final version will look or even function.
 
#6
I understand they are going for broad coverage with this patent, hence the extremely simplistic rendering... But my gosh that looks funny xD
 
#13
I guess this vindicates the Eurogamer rumor of detachable controller sides for the NX.
No it doesn't, primarily because the eurogamer rumor specifically implied that they would have function once detached (2 player mode).

As described in the patent, these controllers are electronicless husks of plastic.
 
#16
They should make the ports based on USB 3.0/PCI-E.

That way, you take the controllers off, plug it into the docking station, and the docking station can provide an extra GPU to the device (and you've got your controllers in your hands).

Just a thought.
 
#25
Cool beans. This is how the circle pad pro communicated with the OG DS, no?

That required the IR blaster on the 3DS, this is a direct connection to the NX itself.
This patent is also infrared no? IR = infrared.

It talks about attaching an accessory (e.g physical buttons) to the side of the portable device, and using the infrared camera to read the button presses instead of actually using any form of electrical connection between the two.
What advantages does IR have over a normal, physical connection? Future proofing (aka firmware updates could accept more accessories than initially planned?) Cheaper? Less parts that can break?
 
#27
Using infrared lights, sensors and reflectors to detect the physical movement of the buttons seems like the most insane, over-engineered way of going about this. Is there some advantage of doing it like this is isn't apparent?
 

Neoxon

Junior Member
#29
No it doesn't, primarily because the eurogamer rumor specifically implied that they would have function once detached (2 player mode).

As described in the patent, these controllers are electronicless husks of plastic.
Wasn't there a mock-up that had the parts connect to a separate controller brain while the NX is in Dock Mode? 10k mentioned that it was in line with what Nate told him (who made 10k realize that he was played like a fiddle up until recently).
 
#41
Wasn't this idea already done with the 3DS' Circle Pad Pro?

EDIT: Someone already beat me to it. I did see how it has "some" electrical components, if this has absolutely none, I'd like to see it working eventually.
 
#42
That's actually hilariously smart.

No controller ports/less expensive.

No controller parts/lighter and less expensive.

No exposed electronics when the controllers are off/less chance of breaking/reduced costs for repairs.

As an electrical engineer I see this as a big boon. The issue ultimately is latency, but with the screens in phones nowadays the tradeoff isn't as great. This is very very clever and fits right in with Nintendo's philosophy of using old tech in creative ways.
 
#46
Know what else this does? It makes it REALLY cheap for third party accessories and customization of the hardware... Have big hands and need big spaced out buttons? Mad Catz got you.