Support NeoGAF

[jeff_rigby]

SCEA’s Research and Development Organization is looking for a senior level engineer to support its PSVR SDK (Playstation Voice Recognition) development for all PlayStation games for current and future platforms.

To help handle/compile standard web based grammar support for multi-languages for voice recognition grammar and dictionary.

To provide the PSVR SDK support, coordination, and improvement for all games and to PS4 system for multi languages, particularly for Japanese. Including AEC support, VSH integration, and performance debugging for applications.

Notice it's web based and in the PS4 the Webbrowser and native support libraries are always loaded (WebGL desktop). For Games a SDK for game developers and of course for the PS4 UI just expand on what we have now.

Note that I expect the browser to have Voice Recognition and since DLNA CVP2 = Vidipath is a HTML5 browser app then it will have Voice Recognition also.

As to timing, Voice recognition is via the Tensilica Xtensa HiFi DSP which is in the Southbridge and it appears Sony is now working with routines in the Southbridge. I expect Sony to target the casual user making the PS4 one of the easier CE platforms to use with Media.

Cadence-Tensilica delivers the software support you need, with support for more than 140 audio, voice, speech recognition, and audio and voice enhancement software packages.

Always On audio Technology is why the PS4 and XB1 camera (which has the mic) has it's own USB port that can be powered separately from the main USB port. HiFi Mini is not needed in a powered by the mains Game console. It just needs to support AOAC network standby power levels of less than 500mw. The HiFi audio DSP can also do basic face recognition while the Xtensa IVP can do a more accurate Face detection.

Tensilica Audio DSP Group at Cadence, discusses always-on audio functionality. Gerard details features like voice trigger, sensor fusion, and low-power audio playback, and explains how Cadence’s HiFi DSP solution can help you successfully implement always-on audio technology

Demo at CES 2015 of an AOA Listening tablet using Tensilica DSP.

"Playstation" = key phrase even when the PS4 is off and functionality similar to the above video where the Key Phrase is "OK Google". For this to work similar to how the tablet works requires CEC for the PS4 to turn on the TV and a TV that turns on as fast as possible.

 

[jeff_rigby]

Bringing Virtual Reality to the Web: VR, WebGL and CSS – Together At Last!.

WebGl, WebVR and the Metaverse

The PS4 Open Source list includes Lua which was used by the PS3's Home as the scripting language. Could there be a webGL 3D "Home" in the PS4's future? 3D on 3D TVs and VR head mounted displays...

use something cheap like in this video to head track and you can support a form of VR on a 2D TV and more on 3D TVs and HMDs.

 

[Caleb187]

lolwut: The Thread

 

[ShemhazaiX]

Bringing Virtual Reality to the Web: VR, WebGL and CSS – Together At Last!.

WebGl, WebVR and the Metaverse

The PS4 Open Source list includes Lua which was used by the PS3's Home as the scripting language. Could there be a webGL 3D "Home" in the PS4's future? 3D on 3D TVs and VR head mounted displays...

use something cheap like in this video to head track and you can support a form of VR on a 2D TV and more on 3D TVs and HMDs.

Was so disappointed when I actually read enough of that to understand what the image was saying. I thought it was going to be people turning the internet into virtual reality. As in, the pre-existing internet. Like maybe turn webpages into walls or something.

Now I'm sad.

 

[jeff_rigby]

lolwut: The Thread

It sometimes seems like I'm far afield in what I bring to a thread but there is logic behind it. The Xtensa DPU processors are in: all AMD APUs, the XB1 and PS4, many Smart TVs, many phones and tablets. Cadence-Tensilica made a point of mentioning a Khronos OpenVX (Virtual reality) partnership.

The Xtensa processors aren't just for media (Codec hardware acceleration and DRM) or voice and gesture recognition. Typically a GPU provides 3x to 6x better performance/watt than a CPU for typical graphics chores while the Xtensa IVP provides a x10 to x20 improvement. The article points out that the IVP provides near FPGA performance improvements (Over CPU and GPU) but is easier to program.

January 6, 2015 - DeNA, a global leader in developing and publishing mobile games, today launched Protocol Zero, an innovative stealth-shooter for Samsung Gear™ VR Innovation Edition powered by Oculus technology. The first immersive first person action game released for Gear VR, Protocol Zero engages players in an intense, heart-pumping 360° experience. A demo version is available now for free on the Oculus Store exclusively for the Samsung Galaxy Note 4.

The Samsung Note 4 has Tensilica Xtensa processors as does the XB1 and PS4 and I expect the Nintendo NX will also. A Xtensa processor of sufficient power can do Voice and gesture, Codecs, DRM and can post process video frames for a Oculus display.

 

[Shin Johnpv]

lolwut: The Thread

aka every jeff_rigby thread.

 

[jeff_rigby]

aka every jeff_rigby thread.

SCEA’s Research and Development Organization is looking for a senior level engineer to support its PSVR SDK (Playstation Voice Recognition) development for all PlayStation games for current and future platforms.

To help handle/compile standard web based grammar support for multi-languages for voice recognition grammar and dictionary.

To provide the PS VR SDK support, coordination, and improvement for all games and to PS4 system for multi languages, particularly for Japanese. Including AEC support, VSH integration, and performance debugging for applications.

Point is the Guy hired to support Web based Voice recognition and Grammar which we know uses the Xtensa HiFi DSP.

 

[tensuke]

jeff_rigby is like misterxmedia with sources

 

[herod]

lolwut: The Thread

Futile scattershot speculation that provides no useful information to anyone, the thread.

For the low percentages of estimates that are in any way correct, it's still useless information anyway even after it comes to pass. The low hit-rate just compounds the problem; there is no worthwhile actions one can take from researching how a closed system like the Playstation line is constructed, for anyone who is not already privy to the facts and under NDA.

 

[jeff_rigby]

jeff_rigby is like misterxmedia with sources

When the PS3 and PS4 are updated (by sometime around June) they will get both HTML5 and DLNA updates with the PS3 also getting Playready EMBEDDED updates. Since the above job posting is March 23 2015, expanded voice recognition will come later this year as HTML5 updates with a Voice recognition SDK for developers.

I'd guess a voice recognition feature for the Browser is coming.. which will also be used for Vidipath since it's a HTML5 app. (A Sony employee mentioned there will be more updates at shorter intervals this year and so far the last two were 6 months apart.)

Sony plans to support one RUI menu for all DLNA servers in the home with their Store and Playstation Vue as part of the menu. How well the DLNA Client is done determines to a large extent whose platform gets used thus who owns the living room....this includes Voice Recognition control.

Misterxmedia is a Fanboy with little knowledge of what's not possible. I know that both Sony and Microsoft have a similar media roadmap and are using nearly the same hardware. The differerence is the PS4 using GDDR5 had to move the ARM buss and Xtensa processors out of the APU into Southbridge with it's own DDR3 memory while the XB1 which uses DDR3 memory had to include expensive 32 MB SRAM and couldn't support a larger GPU because of memory bandwidth. Speculating a larger hidden second GPU shows ignorance, speculating a much smaller second GPU as I have done may be wrong but it's possible and likely if you understand that XTV will be a part of IPTV streaming modes and needs a small low power GPU which is why the Yukon slide included two GPUs and both Sony and Microsoft have two GPU patents (low power and high performance).

 

[tensuke]

What would you use VR for in the browser though? Why wouldn't voice commands be system-wide? I don't know of any website that makes use of VR on the site itself.

I do wish they'd update the browser to support html5 audio, I can't seem to get it working in any format (at least since the last time I tried) and it makes me sad because I want to get a thing working on it.

 

[jeff_rigby]

Futile scattershot speculation that provides no useful information to anyone, the thread.

For the low percentages of estimates that are in any way correct, it's still useless information anyway even after it comes to pass. The low hit-rate just compounds the problem; there is no worthwhile actions one can take from researching how a closed system like the Playstation line is constructed, for anyone who is not already privy to the facts and under NDA.

1) It's not scattershot since it's based on what I know to be the media roadmap and EPA/Energy Star and EU powerboards dictate to a large extent design criteria for the Consoles and future PCs.

2) This time the PS4 and XB1 are based on 2016+ to 2020 (Roy Read 25X more power efficient) mid performance AMD SoCs with each implementing a work-around for the lack of HBM, HSAIL, Vulcan, SPIR and VISC CPU Architecture With 3X Better IPC . AMD Kaveri and Carizo as well as the XB1 and PS4 use the same Xtensa processors from Cadence IP and 2016 AMD SoCs will use more

3) Understanding what's coming (world wide three network convergance IPTV, VOIP and IP (including China) - Vidipath - HTML5 - WebVR) and a cheap AMD SoC with PS4 performance allows one to understand that the services not the hardware are the future of Sony and Microsoft and to properly invest in the Stock market.

That you think it's scattershot with a low hit rate means you don't fully understand what's coming. It may seem like Jeff is crazy in posting all this but the implications are HUGE and few are understanding what it means.

 

[jeff_rigby]

What would you use VR for in the browser though? Why wouldn't voice commands be system-wide? I don't know of any website that makes use of VR on the site itself.

I do wish they'd update the browser to support html5 audio, I can't seem to get it working in any format (at least since the last time I tried) and it makes me sad because I want to get a thing working on it.

Voice commands will be system wide. A decision was made to use Khronos, the W3C and Browser standards (APIs included) for just about everything. This way all platforms with a browser can support the same standards...this is the thrust of Vidipath = DLNA CVP2 as a HTML5 app.

http://www.neogaf.com/forum/showpost.php?p=158918182&postcount=1572
http://www.neogaf.com/forum/showpost.php?p=159044770&postcount=1573

The Roku 3 ($100) now supports Voice Recognition for it's system menu....you will hear more about voice recognition control in the near future.....

 

[Oppo]

Jeff actually does know what he's talking about. he just uses a bazillion acronyms.

 

[jeff_rigby]

Jeff actually does know what he's talking about. he just uses a bazillion acronyms.

For the big picture yes, for the details, timetable and minutia I am as ignorant as just about everyone else.

In early 2011 I emailed Geoff Levand who was in charge of the Webkit Javascript engine port to the PS3. He laughed at some of my Webkit speculation and it took me more than a year to understand why....I was vastly underestimating the importance of HTML5, my asking if a HTML5 browser was coming to the PS3 amused him. Can we now all see why?

The browser in the PS4 will eventually be updated to nearly 100% of the HTML 5.0 standard then HTML 5.1 next. The browser in the PS3 too but of that I'm not as sure.

 

[tensuke]

Misterxmedia is a Fanboy with little knowledge of what's not possible. I know that both Sony and Microsoft have a similar media roadmap and are using nearly the same hardware. The differerence is the PS4 using GDDR5 had to move the ARM buss and Xtensa processors out of the APU into Southbridge with it's own DDR3 memory while the XB1 which uses DDR3 memory had to include expensive 32 MB SRAM and couldn't support a larger GPU because of memory bandwidth. Speculating a larger hidden second GPU shows ignorance, speculating a much smaller second GPU as I have done may be wrong but it's possible and likely if you understand that XTV will be a part of IPTV streaming modes and needs a small low power GPU which is why the Yukon slide included two GPUs and both Sony and Microsoft have two GPU patents (low power and high performance).

It was just a friendly jab at how seemingly nonsensical your posts can be to those of us who have a hard time digesting all this info. I have no doubt you know what you're talking about as opposed to mxm who just makes stuff up. I realize it's hard to look at technical data and accurately predict the timetable and implementation rate for certain things. Still love your posts/threads.

 

[Fisty]

A low-powered "media mode" is definitely what the consoles need, no reason to be running Netflix and playing media files on the apu

 

[jeff_rigby]

A low-powered "media mode" is definitely what the consoles need, no reason to be running Netflix and playing media files on the apu

Exactly...but as a Game console the EU and Energy star give Microsoft and Sony some slack only requiring voluntary compliance except for Standby modes. The ARM blocks in both the XB1 and PS4 as well as DDR3 memory in the XB1 and a separate 256MB DDR3 memory for the PS4 southbridge are to my mind reasons to believe IPTV modes are coming.

If HBM and 2.5D interposers had been ready in 2013 then both would have used them and the XB1 Yukon design wouldn't have changed to the Durango.

The Jaguar as a CPU with programmable microcode might be firmware update-able to use VISC routines with 2-3X the IPC in this console generation or the next. This addresses complaints about the Choice of Jaguar over more powerful CPUs in the game consoles which we all thought was a TDP (power) issue. A CPU with a higher IPC (Instruction per cycle) is needed for AI which requires single thread performance and my understanding is it's a significant issue with the XB1 and PS4.

 

[Blanquito]

Exactly...but as a Game console the EU and Energy star give Microsoft and Sony some slack only requiring voluntary compliance except for Standby modes. The ARM blocks in both the XB1 and PS4 as well as DDR3 memory in the XB1 and a separate 256MB DDR3 memory for the PS4 southbridge are to my mind reasons to believe IPTV modes are coming.

If HBM and 2.5D interposers had been ready in 2013 then both would have used them and the XB1 Yukon design wouldn't have changed to the Durango.

The Jaguar as a CPU with programmable microcode might be firmware update-able to use VISC routines with 2-3X the IPC in this console generation or the next. This addresses complaints about the Choice of Jaguar over more powerful CPUs in the game consoles which we all thought was a TDP (power) issue. A CPU with a higher IPC (Instruction per cycle) is needed for AI which requires single thread performance and my understanding is it's a significant issue with the XB1 and PS4.

The link doesn't work for me. Have another source?

 

[le-seb]

http://www.softmachines.com/wp-content/uploads/2015/01/MPR-11303.pdf (I think)

 

[jeff_rigby]

This is not a Xbox is better than PS4 post so read carefully.

New Xbox features now in preview builds: Some of these features (TV tuner control) are part of soon coming Cable Labs and W3C extensions to HTML5 and others are features a Media Hub will support to the coming Vidipath ecosystem. They are supported now by Microsoft inside it's ecosystem (They can control the DRM and insure content providers that their media is secure). I.E. the same thing will eventually be supported by the PS4 and XB1 to the entire Vidipath ecosystem. Microsoft as a software company is getting W3C TV features in the XB1 before the PS4.

Miracast (Handheld screen to TV screen) is also an up-coming feature. It requires encryption and encoding at the handheld and decryption and unencoding at the Xbox1 using Xtensa Stream processors. Since the PS4 has the same in Southbridge, the PS4 should be able to do the same with slightly more efficiency.

If you have looked at a list of WiFi SSIDs in your home and have a PS4 you will notice it's listed even if you are using a wired Eithernet connection. So WiFi direct is possible, the rest is software and certification. When, after DLNA like with Microsoft and part of the second screen and media server update.

Wi-Fi CERTIFIED Miracast™ is a groundbreaking solution for seamlessly displaying multimedia between devices, without cables or a network connection. Users can do things like view pictures from a smartphone on a big screen television, share a laptop screen with the conference room projector in real-time, and watch live programs from a home cable box on a tablet. Miracast connections are formed using Wi-Fi CERTIFIED Wi-Fi Direct®, so access to a Wi-Fi® network is not needed – the ability to connect is inside Miracast-certified devices.

Miracast is an industry-wide solution, so the technology works well across devices, regardless of brand. Connections are easy to set up and use since the devices choose the appropriate settings automatically. Miracast supports premium content—like Blu-ray feature films, live television shows and sports, or any other copy-protected premium content—allowing you to watch what you want, where you want.

 

[le-seb]

If you have looked at a list of WiFi SSIDs in your home and have a PS4 you will notice it's listed even if you are using a wired Eithernet connection. So WiFi direct is possible, the rest is software and certification. When, after DLNA like with Microsoft and part of the second screen and media server update.

It's been known for a while, Jeff.
WiFi direct can already be used on PS4 to connect a PS Vita when using Remote Play.

 

[jeff_rigby]

It's been known for a while, Jeff.
WiFi direct can already be used on PS4 to connect a PS Vita when using Remote Play.

So you think Miracast is coming to the PS4? I was concerned since Microsoft mentioned WiFi direct and Sony hasn't as well as the PS4 WiFi radio only having one channel at a time. I guess the difference might be related to the XB1 needing a WiFi channel for their controller while the PS4 uses bluetooth for it's controller.

Wi-Fi Direct uses the same silicon

Manufacturers don't need to add extra radios to their kit: the idea is to have Wi-Fi Direct as part of the standard Wi-Fi radio. It's backwards compatible too, so you don't need to throw out your old Wi-Fi-enabled kit.

The Wi-Fi alliance currently claims that more than 1,100 devices have been certified since October 2010, including televisions, smartphones, printers, PCs and tablets.

Wi-Fi Direct is in DLNA, iOS, Android and BB OS and even your new Xbox

In November 2011, the Digital Living Network Alliance (DLNA) announced that it was including Wi-Fi Direct in its interoperability guidelines.

 

[jeff_rigby]

4K media DRM requires Playready 3.0 and Sony will be using Playready in the PS4 which I believe will be a 4K blu-ray player with a firmware update.

Playready 3.0 requires hardware to support DRM.

Again, the PS4 and XB1 use the same Xtensa stream processors on a ARM bus as AMD's Kaveri and likely include a ARM CPU that supports Trustzone as a security/DRM processor just like all AMD APUs. We now know why AMD includes the ARM Trustzone processor... Windows 10 requires hardware support for Playready 3.0 (TEE level DRM) which will be required for 4K playback from internal 4K blu-ray player or on-line streaming. The 4K blu-ray digital bridge likely the same TEE level DRM.

The PS4 follows all ARM TEE level recommendations while the XB1 and most PCs will not even though both have a ARM trustzone processor. Everything DRM as well as on-line commercial transactions occur inside the PS4 southbridge as a SoC while the XB1 and most PCs have a separate Southbridge and DRM has to be split up between APU and Southbridge or dGPU and southbridge.

If you read the Movie industry requirements for DRM, it requires a firmware update-able, revocable DRM with watermarking. Further, encryption from Source to Sink and for streaming DRM, that's Playready into the PS4 southbridge and HDCP 2.X out of the southbridge (see below). Having unencrypted video from southbridge to a HDMI chip would violate best practice and Content provider guidelines. Further TEE level DRM for on-line purchases also requires the GPU (for customer assurance ICONs) to be inside the same SoC with all IO and everything managed by a Trustzone like processor (protected virtual processes). The Xtensa DPU can also be used as a low power GPU.

The PS4 has a custom Panasonic HDMI chip. Why custom if not to support HDMI 2 and HDCP 2.X. HDCP can take place in the PS4 southbridge where the Trustzone processor and Cadence-Tensilica Xtensa DPU stream processors reside. The HEVC codec in both the XB1, PS4, Kaveri and Carrizo, in fact all codecs, compression and streaming DRM are software based on Xtensa processors. (not officially confirmed)


Leaked Sony documents on 4K blu-ray and the digital bridge (requires Tee level DRM = trustzone processor just like Playready 3.0)

 

[GAMEPROFF]

So 4K capability can be patched?

 

[jeff_rigby]

So 4K capability can be patched?

No, the hardware has to support the necessary features then it can be firmware updated.

It looks like all AMD APUs can support Playready 3.0 because they have ARM trustzone processors and since the UVD (Universal Video Decoder) in AMD APUs is a ARM Trustzone managed Xtensa processor with earlier versions not as powerful but able to use the GPU with Kaveri and later using a more powerful version not needing to use the GPU they may be all able to support HEVC. Kaveri's Xtensa processor does support HEVC Codecs. The XB1 has the ARM buss inside the AMD APU while the PS4 has the ARM IP in Southbridge with it's own 256 MB of DDR3 because the PS4 APU is using GDDR5 and even at the slowest clock GDDR5 would not allow for mandated by the EU and Energy Star power levels/modes.

PS4 is Feature-proved" means they have a list of coming features with the hardware designed/proved to be able to support those features.

The issues with 4k blu-ray are:

1) The Disk size and that is going to be 3 layer with the Panasonic tweak that gives 66 GB/layer. All drives after about 2012 can support this.
2) HEVC (h.265) 4K codec and the OP in this thread points out Kaveri, XB1 and PS4 use the same Xtensa processors that can support HEVC.
3) HDMI 2 with HDCP 2.2 my post above this one explains that or this post.
4) DRM which includes the above, the digital bridge and again the reference to Windows 10 using Playready 3.0 for streaming. With the PS4 having a Southbridge that can support ARM Trustzone TEE level DRM it's just again a firmware upgrade needed to support anything so far envisioned likely including the recommendations of DSTAC for Downloadable security for the Cable card replacement.

 

[Oppo]

any update on guesses for when they roll out the DLNA support? it must be nigh, or I would hope.

 

[Jonnax]

any update on guesses for when they roll out the DLNA support? it must be nigh, or I would hope.

They seem to be updating every six months and the last big one came out in March. They did say they want to speed this up but so for there hasn't been much news. Maybe E3, though I don't know if they'd want to announce: "Here's what you been waiting for! VIDEO PLAYBACKKKKKK"

 

[jeff_rigby]

any update on guesses for when they roll out the DLNA support? it must be nigh, or I would hope.

I have no idea....

I was blown away by Comcast: X1: XFINITY for VidiPath Overview

Comcast has complied with the FCC mandate and now offers Vidipath service.

To use XFINITY for VidiPath, you will need:
An XG1 DVR set-top box connected to the home network via a MoCA network adapter
MoCA network adapter must be purchased separately
Pace XG1v1s do not support XFINITY for VidiPath
XFINITY Internet service
XFINITY for VidiPath-compatible Smart TV connected to the home network

Note: There are currently no consumer-ready devices that are compatible with VidiPath; however, several manufacturers may have these devices ready by the end of 2015.

So when will Sony update the PS4, when Playready 3.0 releases or before? Will they wait till October when they will be selling 4K Blu-ray players that are also Vidipath STBs and digital bridge Streamers to other Vidipath platforms?

The Comcast Vidipath overview is wrong on several points. 1) My X1 DVR has both RJ45 and Moca network support as does my Comcast supplied Cable modem. 2) It's not just Smart TVs as Vidipath clients, they don't mention the XB1, Windows 10 PCs, PS4 and other ARM STBs that mention 4K and 4K blu-ray streaming support.

Edit: That no-one has released a vidipath client has to be due to DTLA not activating the DTCP-IP Key server or industry agreement and DTLA compliance. Vidipath only requires Playready 1 for 1080P content and in Videos the Vidipath reference clients demoed were ready a year ago. So I expect all Vidipath clients including the 15 stated to be certified and waiting since late 2014 will launch at the same time to share advertising and consumer education. There is a chicken and egg adoption/startup issue and consumers have to see value before they will buy vidipath. DTCP-IP was proposed in 1995, almost made it in 2008 and with the FCC mandate and 4K blu-ray digital bridge should breach the chicken and egg barrier.

My fear is that the industry, due to Microsoft and Sony's influence, will wait till October for the release of the 4K blu-ray players with the digital bridge which both the XB1 and PS4 will support. If rumors are correct then Microsoft will have a Xbox 360 mini as a Vidipath client and Game server to XB1 and Windows 10 PCs and I would guess Vidipath clients too. All these devices sharing media and games over a home network should make a splash big enough to justify educating the consumer and generating sales to get their platforms in front of as many consumers as possible to sell media, games and services.

The 1TB PS4 and XB1 are I think part of this...industry consensus is that 1TB is the minimum size Hard Disk for a 4K blu-ray with a digital bridge. External network drives can store encrypted copies, the player would need a DRM mechanism with on-line authorization = Playready 3.0? And if this is true it explains Sony and Microsoft wanting to wait and the initial requirement by Microsoft that the XB1 needs to be always on-line for DRM.

1080P Vidipath clients can be extremely cheap ($40-$90) and the higher end clients ($200 and above) need some way to differentiate themselves = Games, Digital bridge streaming, UI and control schemes like Voice and Gesture. XTV/browser is a, likely late 2016 feature that the cheaper Vidipath clients won't or won't support well.

 

[jeff_rigby]

Proof the XB1 has HEVC codecs (Xtensa IVP processors see OP). Encoding HEVC is much harder than decoding which makes the images on the right amazing. HEVC for Skype and game streaming means the Xtensa IVP in Kaveri can encode as well as decode and older AMD APUs and all Intel CPU/GPU combinations using a combination of older Xtensa processor and GPU shaders can do the same. It does mean that older AMD APUs can't game stream to the XB1 as the shaders are being used for the game. This explains the XB1 to PC but not PC to XB1 being mentioned.

(from Arnold Beckenbauer and Pada on Beyond 3d) . onQ123 posted on the Xtensa processors there but no-one read the links and Grail jumped on him without reading the links or thinking about what has to be supported in a Game console for Media.

AMD in older APUs uses a ARM trustzone managed Xtensa stream processor and GPU shaders.
In Kaveri and Carrizo APUs the Xtensa stream processor is more powerful and doesn't need GPU shaders.

 

[Peltz]

Reading rigby threads is like listening to a beautiful french woman. I understand nothing but it's still fascinating.

Yep. You and me both:

 

[jeff_rigby]

What this means is the XB1 is designed to support HEVC 4K streaming including two way for Skype and only the HDMI 2.0 port with HDCP 2.2 (see bottom two slides) is yet to be proved for 4K blu-ray support (the blu-ray drive is a non-issue as older drives can support the Panasonic tweak and can read 3 layers).

For the PS4 there is no proof but the same hardware AMD uses in Kaveri is in the XB1 and likely the PS4's Southbridge. The PS4 has a custom HDMI chip that likely allows the Southbridge to do the HDCP 2.2 which just requires the custom HDMI chip have a faster clock and to pass through the HDMI transpitter-receiver DRM negotiation to the PS4 southbridge.

The slides referring to Windows 10 are about Windows 10 supporting HEVC and Windows 10 having Playready 3.0 support for 4K media which requires hardware DRM like AMD's Trustzone processor managing the Xtensa stream processor. Last year a quote caught my eye in that Microsoft is AMD's Trustzone partner which makes the slide I posted a Duhhhh.

Again from BY3D showing Microsoft wanted HEVC for Skype support (two way encode -decode).

It's an add-on circuit = Xtensa DPU on the ARM buss as seen in the following breakdown. The Two Xtensa processors for decode/encode are labeled h.264 multi-stream codecs which is confusing everyone. They are software programmable accelerators and can support any current codec or compression scheme (See the XB1 DLNA file types supported). Everything tinted green below is on the ARM buss and is Cadense IP including the HDMI port which can support HDMI 2 and HDCP 2.2 via the Trustzone processor controlled Xtensa processors. There is a Pay view article for the XB1 stating that the blocks in green are all Cadense IP.

Cadense IP below in Grey and Xtensa stream processor in RED. Notice DP/HDMI port at bottom is the same description as DP/HDMI in green above.

 

[onQ123]

I think the PS4 Secondary chip might feature Mali-V500

http://www.arm.com/products/multimedia/mali-video/index.php

http://www.arm.com/products/multimedia/

Video processing from ARM
Mali Video processors provide advanced video technology combined with power-efficiency and one of the smallest chip sizes in the market to offer creators of mass market mobile devices support for a wide range of video technologies. Designed to scale from 1080p60 on a single core to 4k120 on the full eight cores, all variations offer stunning HD visual quality to end users.

 

[onQ123]

Seems to be what PS4 secondary chip & Xbox One offloading processors are for


http://www.armtechforum.com.cn/2014...ogethertoDriveDownSystemPowerandBandwidth.pdf

http://www.arm.com/products/multimedia/mali-display/mali-dp500.php

When ARM supplies an ARM Mali-DP500, we supply the intelligent ARM Mali Display DDK as well. The ARM Mali Display DDK utilizes unique architectural features in ARM Mali GPU and Video IP and makes smart decisions about when and how to offload tasks from the GPU in order to maximize the battery life of the device. When designing an ARM-based SoC with an ARM Cortex® CPU and ARM Mali GPU, Video and Display, each processor’s Android driver will work efficiently with any of the other drivers straight out of the box. This optimized Android multimedia software stack addresses universally resource-consuming challenges such as integrating and optimizing processors from different suppliers or supporting regular Android updates. By taking advantage of ARM’s pre-optimized software stack, partners have more time to differentiate their solutions and bring their product to market sooner.

The ARM Mali-DP500 enables the GPU or CPU to offload tasks such as composition, scaling, image and colour enhancement and does so in a single pass. It also implements AFBC. AFBC is an ARM technology which, when implemented across the SoC, results in 50% less system-wide bandwidth consumption than a SoC without AFBC. ARM Mali processors with support for AFBC include the ARM Mali-T760 GPU and the ARM Mali-V500.

The ARM Mali-DP500’s level of secure technology support is unique in today’s market. Consumers need confidence in their displays for situations such as mobile payments. For this reason the ARM Mali-DP500 has a secure display layer which is only programmable by a Trusted Execution Environment (TEE). It is compatible with ARM TrustZone Ready Client and GlobalPlatform™ Trusted User Interface which ensures that in payment situations the secure screen is always on top and protected from software attacks. In addition, hardware backed security is becoming a critical requirement of content vendors in order for consumers to watch high value content on their mobile devices. For secure video playback from download to display, the ARM Mali-DP500 is compatible with ARM TrustZone Media Protection security requirements which ensure that unencrypted content is not accessible by the high level OS and that it is only ever written to protected memory.

 

[jeff_rigby]

Seems to be what PS4 secondary chip & Xbox One offloading processors are for

That was my first guess more than a year ago until you found the Xtensa DPUs. If you investigate them and know what's coming for the Consoles; I.E. UHD Blu-ray & digital bridge, Xtensa IVP would be better at codecs and video processing than a GPGPU. Also if you use a Xtensa DPU for the audio portion of Codecs you need to use a Xtensa processor for the video. Timing/lip sync issues are a big part of a design as Video and audio take different paths and need different timing.

AMD and the XB1 are using Xtensa DPUs for both audio and video...they wouldn't do so if other hardware could do it better. Remember HSA literature saying a DSP is 20 times more efficient than a GPU and FPGA some 100X at some tasks. Many of the TVs and phones include Xtensa processors even though they have Mali GPUs

http://electronicdesign.com/microprocessors/tiny-ivp-plows-through-video-image-processing

Key phrase voice turn-on with a HiFi Xtensa DPU can run with very low power, much lower than a Mali GPU....that is a planned feature.

A Xtensa IVP can emulate a GPU...we don't know what's in the PS4 Southbridge but I suspect it's all Cadence IP just as the ARM block in AMD APUs, dGPUs and XB1 are.

If you look at timing, Xtensa IVP was available to partners in 2012. This could have caused a planned delay for the XB1 APU but the PS4 southbridge is a simpler lower clock and power IC that would have a high yield and require less lead time.

 

[onQ123]

That was my first guess more than a year ago until you found the Xtensa DPUs. If you investigate them and know what's coming for the Consoles; I.E. UHD Blu-ray & digital bridge, Xtensa IVP would be better at codecs and video processing than a GPGPU. Also if you use a Xtensa DPU for the audio portion of Codecs you need to use a Xtensa processor for the video. Timing/lip sync issues are a big part of a design as Video and audio take different paths and need different timing.

AMD and the XB1 are using Xtensa DPUs for both audio and video...they wouldn't do so if other hardware could do it better. Remember HSA literature saying a DSP is 20 times more efficient than a GPU and FPGA some 100X at some tasks. Many of the TVs and phones include Xtensa processors even though they have Mali GPUs

http://electronicdesign.com/microprocessors/tiny-ivp-plows-through-video-image-processing

Key phrase voice turn-on with a HiFi Xtensa DPU can run with very low power, much lower than a Mali GPU....that is a planned feature.

A Xtensa IVP can emulate a GPU...we don't know what's in the PS4 Southbridge but I suspect it's all Cadence IP just as the ARM block in AMD APUs, dGPUs and XB1 are.

If you look at timing, Xtensa IVP was available to partners in 2012. This could have caused a planned delay for the XB1 APU but the PS4 southbridge is a simpler lower clock and power IC that would have a high yield and require less lead time.

It's not just a GPGPU it also has a CPU, VPU , DPU (Display Processing Unit) Texture compression & decompression & so on


VPU

For encode: H.264, VP8 For decode: H.264, H.263, MPEG4, MPEG2, VC-1/WMV, Real, VP8

https://www.arm.com/products/multimedia/mali-video/mali-v500.php

The ARM® Mali™-V500 offers advanced video technology combined with power-efficiency and one of the smallest chip sizes in the market to creators of mass-market mobile devices. Designed to deliver a pixel throughput* of 1080p60 on a single core scaling linearly to 4k120 on the maximum eight cores, all Mali-V500 variations will offer stunning HD quality to end users. The Mali-V500 is the initial offering in the innovative Mali Video roadmap and offers multi-standard codec support.
* 'pixel throughput’ means the total amount of video content that can be processed on a core or group of cores: so a single core with 1080p60 capability could be made up by two 1080p30 video streams, or four 720p30 streams, etc. The streams can be handled independently, so can encode or decode any codecs supported.


Request More Information
Mali-V500 chip diagram
View larger Mali-V500 image

Find out more...

Why Mali-V500? Specifications Related Products Resources
Building on the strong momentum and expertise of ARM Mali’s graphics solutions, the Mali-V500 offers a scalable, secure solution to the video processor market capable of 4k resolution at 120fps on eight cores; thanks to this “trick-play,” video edit and slow motion capture in HD is therefore possible. With a single core, the Mali-V500 is capable of bringing 1080p60 performance to mass-market smartphones, meeting growing consumer expectations and changing video consumption patterns. The design is capable of managing multiple, simultaneous encode and decode HD streams and through this can support multi-party video conferencing and advanced user interfacing.

multi-stream video call via Skype

The Mali-V500 incorporates a novel lossless frame compression technique, ARM Frame Buffer Compression (AFBC). By coupling AFBC in the Mali-V500 with AFBC in either the display controller or Mali GPU a bandwidth saving of over 50% is achievable. This delivers a considerable system-wide power reduction, potentially allowing the use of lower cost DDR memory.

ARM always takes a system-wide approach to security starting with the deepest levels of hardware and software. In accordance with these values, the Mali-V500 was architected with support for TrustZone® enabling hardware-backed content security from download to display. As high value movie and TV content is consumed across a wider range of devices, creators and distributors of this content are looking for greater protection of their investments and all ARM video solutions will meet these demands.

For use cases such as gaming on Wi-Fi display it is important to users that latency from mobile device to display is as low as possible, at least <100ms in order to maintain an engaging experience. The Mali-V500 boasts latency of <10ms at 1080p30 for both encode and decode making gaming on Wi-Fi display a compelling experience.

Whether we are talking about graphics or video processing, visual quality is extremely important and Mali-V500 includes high bus latency tolerance whereby video processing can continue for over 5000 cycles without external memory access. This feature ensures that consumers will not see jerky playback due to dropped frames when other processes are happening on the device

DPU

https://www.arm.com/products/multimedia/mali-display/mali-dp500.php

Mali-DP500
The ARM® Mali&#8482;-DP500 offers an extensive range of functionality, including composition, rotation and scaling, within a very small die area. By processing tasks offloaded from the GPU and through its support for ARM Frame Buffer Compression (AFBC) it is able to considerably lower system-wide power consumption. With compatibility for ARM TrustZone® technology, the ARM Mali-DP500 secures the display path for use cases such as mobile payment and video playback. When an ARM Mali-DP500 is implemented in a complete ARM-based SoC, the Android&#8482; driver for each processor will, by design, work together extremely efficiently; this highly optimized multimedia software stack significantly reduces implementation effort while at the same time delivering best in class performance and maximized power efficiency.

 

[jeff_rigby]

It's not just a GPGPU it also has a CPU, VPU , DPU (Display Processing Unit) Texture compression & decompression & so on


VPU

https://www.arm.com/products/multimedia/mali-video/mali-v500.php


DPU

https://www.arm.com/products/multimedia/mali-display/mali-dp500.php

What this shows me is that ARM supported the same features by 2014 that AMD was putting in their APUs and dGPUs in 2014 and that Sony and Microsoft targeted for 2013. The Xtensa IVP was ready for partners by early 2012 and was in literature Feb 2013 because it taped out Oct 2012 for the XB1 and PS4 November 2013 release. Prior to that date (from 2010) AMD used a less powerful Xtensa DPU for their UVD and had to supplement it with GPGPU from the GPU. This would work for small GPUs but with larger GPUs would use too much power to meet IPTV power requirements. Thus the XB1 HAD to use DDR3 and the more powerful Xtensa IVP with the AMD GPU turned off. AMD's Carrizo does not need to use the GPGPU for HEVC as it has a UVD 6 using Xtensa IVP EP. The second version of the PS4 may have a Xtensa IVP EP which is more efficient than IVP.

Sony, AMD and Nvidia have been using Xtensa processors for years; Sony for blu-ray players. Sony, Microsoft, AMD and Nvidia also have ARM licenses which mean they can pay for and use ARM designs including what you uncovered. Or they can use the more efficient Xtensa DPU stream processors for codecs/encryption, Face and feature recognition, Voice recognition etc.

OK, look at it this way; Microsoft with Windows 10 and game developer engines want to support as few hardware configurations as possible. AMD wants PS4 and XB1 game engines supporting the same hardware that is in AMD APUs, Microsoft wants this also. True Audio using Xtensa processors is not being used by most game engines because there are so few PCs supporting True Audio. They are currently using generic CPU based audio engines even in the XB1 and PS4. That will or has changed and a reserved CPU will be released in favor of Xtensa True Audio hardware.

Key for AMD is to have ALL game consoles using True Audio and Xtensa IVP processors following AMD's lead, this makes AMD hardware not Intel and Nvidia, the best hardware to develop games, even for the PC. AMD also wants Xtensa IVP being used for VR and openVX not the GPGPU so that they can free up GPU cycles for games. This is currently not being done on the PS4 and XB1 because developer game engines are designed to support PCs also and use the GPGPU for the same reason True Audio is not or wasn't supported. This is a big hump to overcome and when Game engines start using AMD True audio and Xtensa IVP we will see a bump in performance and or reserved hardware being released to developers.

FYI Sony uses AMD APIs and names in their game developer presentations. For instance, they support VCE encoder APIs even though they have all ARM IP including emulated VCE, UVD and True audio in the Southbridge SoC. VCE in AMD APUs and dGPUs is a AMD developed fixed codec h.264 encoder while the emulated VCE in Southbridge is a Xtensa DPU stream processor. I.E. Sony's PS4 OS presents to game developers AMD APIs to make it easier for them to develop PS4 games on an engine that needs few tweaks to also work on PCs with AMD hardware.

The above is obvious if you think about it. Sony will present APIs using CPU and GPGPU (2014) and then move them to Xtensa True Audio and IVP hardware (late 2015) freeing resources to developers. Southbridge Xtensa processor APIs to game developers first and then to IPTV like Netflix. Game developers will choose to use those APIs if they are also available on other platforms except for exclusives. If the APIs are not widely available on other Platforms they will use their own generic engine running on a CPU or GPGPU...this is the hump AMD must overcome for their Hardware to be accepted as the standard to be used to develop games. Following Khronos standards for AMD APIs is part of this and ARM does the same so you can see the same OpenVX APIs on ARM that you see or soon see on AMD APUs and dGPUs supported by similar hardware.

October 2014

OpenVX defines a higher level of abstraction for execution and memory models than compute frameworks such as OpenCL&#8482;, enabling significant implementation innovation and efficient execution on a wide range of architectures while maintaining a consistent vision acceleration API for application portability

&#8220;AMD is an enthusiastic advocate for natural user interfaces enabled by computer vision,&#8221; said Greg Stoner, senior director of application engineering, Heterogeneous Applications and Solutions, AMD. &#8220;As a proponent of open standards and a provider of highly-parallelized architectures ideal for computer vision, we embrace the OpenVX standard and look forward to standardized proliferation of these experiences throughout the industry.&#8221; ALL AMD APUs from 2010 have Xtensa processors and AMD has been using them for vision (gesture recognition) and audio with ARM middleware exactly like that used in ARM phones.

&#8220;Cadence is integrating OpenVX into our Tensilica® Imaging/Vision Library software development kit (Xtensa processors) to enable higher performance and power optimization for our scalable and configurable IVP-EP DSP cores, which are widely adopted for imaging, computer vision and automotive drivers&#8217; assistance applications,&#8221; stated Dr. Chris Rowen, CTO of the IP Group, Cadence.

By 2016 openVX APIs using Xtensa DPUs should be used for vision processing (VR applications) in games developed for the PS4. Sony likely provided OpenVR APIs using CPU and GPGPU routines first.

 

[onQ123]

We already know that the media accelerators are a part of Starsha (GNB)

https://fail0verflow.com/media/32c3-slides/#/6


From the leak

PS4:

New Starsha GNB 28nm TSMC
Milos
Southern Islands

DX11
SM 5.0
Open CL 1.0
Quad Pixel pipes 4
SIMD&#8217;s 5
Texture Units 5TCP/2TCC
Render back ends 2
Scalar ALU&#8217;s 320

EDIT: Some of those were crossed, may be they were updated/changed at a later date, I have no idea.
Quote:
Couple of more updates

Graphic North Bridge(GNB) Highlights
Fusion 1.9 support
DCE 7.0
UVD 4.0
VCE
IOMMU
ACP
5x8 GPP PCIE cores
SCLK 800MHz/LCLK 800MHz

 

[jeff_rigby]

We already know that the media accelerators are a part of Starsha (GNB)

From the leak

The leak gave us the Liverpool GPU that was the basis of the PS4 APU before the custom design and the removal of the ARM blocks. It gave us some names (Starsha, Thebe) that spawned speculation on what the names might mean. The Media accelerator(s) (UVD) in Liverpool are/is Xtensa DPUs on an ARM bus. I've "guessed" that Sony would use the same Media accelerator families in the Southbridge TEE (modern Media requires a TEE) because AMD did and Sony is additionally using the same Xtensa HiFi DPU for True Audio that AMD uses. The 256 MB DDR3 memory connected to Southbridge confirms this.

Edit: Yes in a Liverpool GPU, the media accelerators are in the GNB which is Cadence ARM IP but that is the AMD GPU not the final Sony PS4 APU design. In later APUs Southbridge is moved into the APU and that is also Cadence ARM IP and they still use DDR3. They needed to do this for power reasons. If you are trying to say the PS4 APU has a UVD then why would it or Southbridge have a ARM Mali GPU?

The two render backends in Southbridge allow one for GPU and one for overlay for the Low power mode UI and Full screen video. This allows the GPU to be off when full screen video is displayed and the Xtensa acting as a small GPU for UI during this mode like for Closed caption etc. The video overlay buffer also has to be in the TEE to comply with DRM.

This is no longer speculation as TEE for the XB1 requires the Trustzone controlled ARM buss inside the APU to process all video and HDCP 2.2 encrypt it before it leaves the APU for the HDMI port and the PS4 requires ALL media to be processed in the Southbridge ARM Trustzone TEE and HDCP encrypted before it leaves southbridge for the HDMI port; I.E. HDCP 2.2 occurs in Southbridge.

Edit: any exploit of the PS4 to run Linux has two issues, 1) The Southbridge trustzone boots the APU's OS and 2) All DRM media is via the Southbridge trustzone. Both are designed to be secure and are separate from the APU OS. It's likely that only Sony can enable TEE DRM on the PS4 and an exploit would have to be run after the PS4 OS boots. It's so secure that Sony could support Linux on the PS4 and not worry about security. Southbridge it'self boots from an internal ROM and then it trustzone boots the PS4 OS from encrypted files on the Hard Disk that are hash checked by Trustzone routines..

 

[onQ123]

The leak gave us the Liverpool GPU that was the basis of the PS4 APU before the custom design and the removal of the ARM blocks. It gave us some names (Starsha, Thebe) that spawned speculation on what the names might mean. The Media accelerator(s) (UVD) in Liverpool are/is Xtensa DPUs on an ARM bus. I've "guessed" that Sony would use the same Media accelerator families in the Southbridge TEE (modern Media requires a TEE) because AMD did and Sony is additionally using the same Xtensa HiFi DPU for True Audio that AMD uses. The 256 MB DDR3 memory connected to Southbridge confirms this.

Edit: Yes in a Liverpool GPU, the media accelerators are in the GNB which is Cadence ARM IP but that is the AMD GPU not the final Sony PS4 APU design. In later APUs Southbridge is moved into the APU and that is also Cadence ARM IP and they still use DDR3. They needed to do this for power reasons. If you are trying to say the PS4 APU has a UVD then why would it or Southbridge have a ARM Mali GPU?

The two render backends in Southbridge allow one for GPU and one for overlay for the Low power mode UI and Full screen video. This allows the GPU to be off when full screen video is displayed and the Xtensa acting as a small GPU for UI during this mode like for Closed caption etc. The video overlay buffer also has to be in the TEE to comply with DRM.

This is no longer speculation as TEE for the XB1 requires the Trustzone controlled ARM buss inside the APU to process all video and HDCP 2.2 encrypt it before it leaves the APU for the HDMI port and the PS4 requires ALL media to be processed in the Southbridge ARM Trustzone TEE and HDCP encrypted before it leaves southbridge for the HDMI port; I.E. HDCP 2.2 occurs in Southbridge.

Edit: any exploit of the PS4 to run Linux has two issues, 1) The Southbridge trustzone boots the APU's OS and 2) All DRM media is via the Southbridge trustzone. Both are designed to be secure and are separate from the APU OS. It's likely that only Sony can enable TEE DRM on the PS4 and an exploit would have to be run after the PS4 OS boots. It's so secure that Sony could support Linux on the PS4 and not worry about security. Southbridge it'self boots from an internal ROM and then it trustzone boots the PS4 OS from encrypted files on the Hard Disk that are hash checked by Trustzone routines..

Something that I have noticed is that things like the web browser, capture gallery & live from PlayStation all seem to be loaded in with the OS & never have to load up like the apps, I have a feeling that all these things are ran by Aeolia even the video playback. The last FW update somehow broke playback of Twitch streams in Live From PlayStation but the Twitch app was unaffected.

 

[jeff_rigby]

Something that I have noticed is that things like the web browser, capture gallery & live from PlayStation all seem to be loaded in with the OS & never have to load up like the apps, I have a feeling that all these things are ran by Aeolia even the video playback. The last FW update somehow broke playback of Twitch streams in Live From PlayStation but the Twitch app was unaffected.

All that shows is that the OS uses a webkit/browser UI that is always loaded; I.E. a browser desktop using Zero Copy techniques. It does not prove where those routines as APIs are running. We can know that eventually many APIs will be via Southbridge and ALL APIs dealing with Media and DRM will be via Southbridge because of TEE requirements.

I've not changed speculation on this since 2013 beyond a small GPU in Southbridge or how possibly two GPUs might use the Sony patent on a OS sharing a low power and performance GPU.

 

[jeff_rigby]

Update of finds:

PS4 and XB1 are UHD Capable
Launch (2013) consoles are UHD Capable and will be firmware updated (ARM TEE for PS4 in Southbridge and XB1 in the APU)
BDXL drives with 2010 specs can read UHD disks, BD-ROM Blu-ray drives are firmware updateable to read UHD Disks. (AACS 2 and BD+ still needed and they run in the TEE along with the Player and HEVC profile 10 codec).

AMD just released OpenVX beta (.9) for AMD APUs and dGPUs This is vision processing using Xtensa accelerators and OpenCV/OpenCL/GPU Compute. Prior to OpenVX, everything uses OpenCV which uses GPU compute. With OpenVX properly supported by the vendor, Accelerators are added (Xtensa processors) and vision processing becomes more efficient.

Sony and Microsoft should be releasing OpenVX APIs for the PS4 and XB1. OpenVX is vision processing released by Khronos October 2014 and updated April 2016 to version 1.1.

Open VX extensions for HTML and tiling beginning of 2015. The description mentions Tiling extension deals with the CPU only.

As with conventional User Kernels, Tiling User Kernels will typically be loaded and executed on HLOS/CPU-compatible targets, not on remote processors or other accelerators &#8212; the intent of this extension is to allow efficient scheduling and memory transfer between accelerators and user functions, not to provide code to run on an accelerator, since this capability is necessarily vendor-specific.

Micorosoft and Sony must write the extensions to use the Xtensa accelerators on the same ARM bus the TEE uses. The Xtensa accelerator is also used for the Codec which must be run in the TEE protected mode while the OpenVX can be run on the same bus without Trustzone protection.

Note: I in error assumed OpenVX would be supported by the PS4 soon after the October 2014 Khronos release of OpenVX 1.0. I should have understood that Sony would only release OpenVX when Game developers could use it on other Platforms (PC and XB1). OpenVX will be used by VR later this year so this release is timely.

OPENVX THE NEW KHRONOS API FOR COMPUTER VISION AND AR

OpenVX designed to improve the efficiency of computer vision from the camera to the compute, which can easily be handled by OpenCL 2.0, the new open heterogeneous compute standard that was finalized at the same time as OpenVX was announced.

it will do a whole host of image processing which range from mulitichannel color and BitDepth extraction and conversion to image resizing and warping. It will also do some 2D filtering and morphological operations in addition to the standard arithmetic, logical and statistical operations. It will also do core computer vision including pyramid computation and integral image computation. In addition to these major features, it will also be capable of histogram computation and equalization as well as canny edge detection and sparse optical flow. Harris and FAST corner detection are the final features of what OpenVX is epxected to do,

=> Warping the video out for VR goggles, UHD Blu-ray digital bridge video conversion, VR.....

Nvidia is also introducing OpenVX 1.01 5/2016