Scott Shumaker
Lead Programmer, Naughty Dog

The views expressed above are my own, and do not necessarily reflect those
of Naughty Dog.

Click to expand...

Scott Shumaker sweng-gamedev@midnightryder.com
Mon, 8 Aug 2005 21:52:28 -0700

* Previous message: [Sweng-gamedev] Asset Management System
* Next message: [Sweng-gamedev] Higher Level Languages (Was: Next Gen Multiplatform Load Balancing)
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Sorry for the delayed reply - was out @ Siggraph last week.

In all honesty, the biggest reason we're not using GOAL for next-gen
development is because we're now part of Sony. I can only imagine Sony's
shock when they purchased Naughty Dog a few years back, hoping to be able to
leverage some of our technology across other Sony studios, and then realized
that there was no way anyone else would be able to use any of our codebase.


Sony wants us to be able to share code with other studios, and this works
both ways - both other studios using our code and vice versa. Add this to
the difficulty curve of learning a new language for new hires, lack of
support from external development tools (we had our own compiler, linker,
and debugger, and pretty much had to use Emacs as our IDE), etc, means that
there are clearly a lot of other factors involved. Note, however, that
these issues aren't really technical problems, they're social ones.

I can definitively say that the investment in GOAL was worth it for our PS2
titles, despite the initial setup time and maintenance. Our productivity
gains were huge, and were more than worth the time investment. This time
around, however, the circumstances aren't quite the same. If we were still
an independent studio, I'm almost positive we'd be extending GOAL for the
next-generation of development. As it is, we are looking into alternative
approaches (custom preprocessors, assemblers, linkers, you name it) - but
all of these approaches fall short in many ways of the unified language and
environment we had with GOAL.

That said, if there was a serious effort on the part of the game development
community to develop and standardize a language for game development,
everyone could gain the benefits without suffering the drawbacks (lack of
code-sharing, learning curve, etc). And if there's enough community
support, it would only be a matter of time before some really high-quality
commercial tools came out to work with the language.

Scott Shumaker
Lead Programmer, Naughty Dog
sshumaker@naughtydog.com

-----Original Message-----
From: sweng-gamedev-midnightryder.com-admin@lists.midnightryder.com
[mailto:sweng-gamedev-midnightryder.com-admin@lists.midnightryder.com] On
Behalf Of Tom Plunket
Sent: Saturday, July 30, 2005 12:28 AM
To: sweng-gamedev@midnightryder.com
Subject: Re: [Sweng-gamedev] Higher Level Languages (Was: Next Gen
Multiplatform Load Balancing)

Scott Shumaker wrote:

> You've probably all heard about the custom-designed LISP-flavored
> language we used here for the Jak series of games.
[...]
> In other words, one programmer, in one year, designed wrote a language
> that was better for game development than C++.
>
> I frankly don't see why there isn't a serious effort underway to
> develop a better language for game development.

Well, an outsider looking in might point exactly at GOOL and GOAL and say,
"here's why not."

In other words, the fact that Naughty Dog is abandoning all of that for C++
seems to indicate that at some level this "better"
language isn't worth it after all.

I've been curious for a while, and now the door is open. Exactly why is
Naughty Dog abandoning the Lisp-en and going for C++? Is it because the
main proponent is no longer around and it actually isn't worth spending the
time to write these tools for a new platform?

It seems pretty clear that /once the system is in place/ you get huge
productivity gains. The fact remains, though, that it appears that those
productivity gains are /not/ offset by the expense of building and
maintaining such a system.

Now, I hope that in general this is not the case, but it seems pretty clear
that in Naughty Dog's case, the people evaluating risks and costs think that
it isn't the win that we're being told that it is.

Comments?

> Perhaps it's because people don't know what they're missing?

Let us know when the PS3 comes around and you guys put out a game on it.

> Or perhaps there's a deep FUD regarding new language technologies, and
> people are comfortable with what they know?

Both of these are certainly the case, to be sure. What's your
(company's) excuse?

Click to expand...

Scott Shumaker sweng-gamedev@midnightryder.com
Thu, 11 Aug 2005 01:29:23 -0700

* Previous message: [Sweng-gamedev] Higher Level Languages (Was: Next Gen Multiplatform Load Balancing)
* Next message: [Sweng-gamedev] Higher Level Languages (Was: Next Gen Multiplatform Load Balancing)
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Responses inline:

>From what I gathered, what GOAL had that I'd like:
>- fast iteration (due to the listener)

Well, fast iteration times weren't merely due to the the listener - that was
a nice touch, but only the tip of the iceberg. We could basically
dynamically link per-function or variable. Effectively, you could hit a key
while working in the IDE, and whatever function the cursor was on would
instantly get compiled, sent across the network to the TOOL, linked and
dropped into the game while it was running. The whole process took a
fraction of a second. You could also do the same per-file. This feature
was sort of like Edit and Continue, but you didn't have to broken in the
debugger - it could be done while the game was running. This was insanely
useful for programming gameplay, physics, and fx, as well as prototyping,
visual debugging (just drop in some debug spheres or prints while you have
the game in some interesting state), etc. We also used it for dynamic code
streaming - so only a fraction of the executable code was loaded at any
given time (to conserve memory).

It's astonishing at how much this ability changes your coding style - you
gravitate to a much more iterative development process, and write small bits
of code that evolve the codebase while the game is running. It's obviously
also much better for robustness and reliability, because you're testing out
each little piece of code as it gets added (as opposed to writing a whole
mess of code and trying to get it compiled and debuggged). As for bugs,
often you can fix them while the game is running and test out your changes
instantly - this makes bugfix verification much easier.

>
>- more performant generated code (I honestly find it hard to believe that
any small group of people (at least the ones I know!) could write a better
compiler (as in generates faster code) than Microsoft or GCC4 at this point,
but, perhaps compared to an ancient/early version of GCC on PS2?). Or is it
more that if you expressed certain things in
C++ that they'd be inherently slower?
>

On the performance front, the actual optimizing compiler really wasn't that
great. In fact, in most cases it generated code that was clearly inferior
to GCC (it did take advantage of native register sizes, but overall very
little development time was spent on the optimizer). However, due to GOAL's
seamless and tight integration with assembly, this wasn't as much of a
problem as it may seem. On the PS2, most heavy, performance-critical
operations you're going to want to delegate to the VU anyway. Writing
assembly code was awesome in GOAL - it unified the assembly syntax across
all of the disparate PS2 processors (with the same syntax as GOAL), you
could freely intermix assembly from all of the processors and normal GOAL
code, and it could access standard GOAL member variables, plus you had all
of the macro facilities, etc, to do stuff like automatic loop unwinding,
code pasting, etc. It was pretty typical to write a bunch of GOAL code and
just replace the inner loop with some macro-mode VU code for nice
performance gains. This was a pretty easy task given the fact that all
assembly code had the same syntax (no MSVC "__asm" blocks or GCC's extremely
awkward intrinsics), and it took advantage of the register colorer).

As for the other stuff, like co-routines, etc - one of the salient points
here is that GOAL was extensible. Since we wrote the language in-house, we
were able to extend it when the demands of our game required it. This adds
a whole new dimension to problem solving - when you aren't constrained by
the limitations of the language, you find that many problems become a lot
easier to tackle.

It was this ability to shape GOAL to our needs that allowed us to develop
features like coroutines, state inheritance, variable overlays, unique IDs,
etc. This really was one of our key advantages in working with the
language.

That said, GOAL wasn't exactly designed with language extensibility in mind.
The macro syntax was quite powerful, but fell short of really allowing you
to add stuff to the core language, like new tags and keywords. This meant
that a lot of new features had to be implemented by extending the actual
compiler, which was unfortunate - it was simply too time consuming to do
later in the PS2's lifecycle.

In the future, I think it is absolutely critical that any game development
language is built with extensibility in mind. This would imply both a good
macro syntax (for extensible code generation), compile-time reflection,
syntax extensions, and also the ability to add new tags, qualifiers, and
keywords (facilities that tools like OpenC++ try to provide). After all, a
DS developer has a very different set of problems to solve than a PS3
developer, and a lot of these problems can be greatly aided by the
programming language (managing the many processors on the PS3, for example).



Scott Shumaker
Lead Programmer, Naughty Dog
sshumaker@naughtydog.com

The views expressed above are my own, and do not necessarily reflect those
of Naughty Dog.

Click to expand...

Scott Shumaker sweng-gamedev@midnightryder.com
Thu, 11 Aug 2005 11:36:43 -0700

* Previous message: [Sweng-gamedev] Higher Level Languages (Was: Next Gen Multiplatform Load Balancing)
* Next message: [Sweng-gamedev] Higher Level Languages (Was: Next Gen Multiplatform Load Balancing)
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

> I would be curious to see what a typical function that had assembler for
vu0/vu1/ee mixed together ?actually looks like. Anything
not-proprietary/giving-away-secrets that you could post?

Okay. Here's some old code I dug up from Jak3 that uses regular GOAL code,
EE assembly, and VU0 assembly in macro-mode.

A few words of warning:
The syntax takes a while to get used to (it is LISP, after all), and this
function is sort of weird (it's a particle system callback function - the
assembly is there because I needed to convert some data from fixed point).
Also, it's been over a year and a half since I wrote this, and it was a
quick one-off for a weapon effect, so it's a bit messy.

Most of the code here is GOAL code, except for the code in the rlet. ('let'
is the way you declare variables in LISP, and rlet specifically guarantees
that variables will be bound to registers).

Code:

(defun sparticle-red-2-converge ((sp-system sparticle-system) (particle
sparticle-cpuinfo) (spvec sprite-vec-data-2d))
  "Red gun 2 charge glow effect"
  (let ((control (the sparticle-launch-control (-> particle key)))
	 )
    
    ;; figure out the offset from the player's position
    (let ((origin-vec (stack-vector))
	  (offset-vec (stack-vector))
	  )
      
      ;; retrieve offset vec off of userdata, omega.  We only had 8 bytes
for data storage,
   	;; but needed to store a full vec3, so we packed it into fixed
point.
      (let ((combined-vec (new 'stack 'vector4w))
	    )
	(set! (-> combined-vec x) (-> particle user-uint32))
	(set! (-> combined-vec y) (-> particle data 0))
	
	(rlet ((offset :reg vf1)
	       (preconverted :reg vf2)
	       (iconv0)
	       (src)
	       )
	  (l.q src combined-vec)   ;; GOAL->EE
	  (mer.vh lo iconv0 src r0) ;; EE
	  (sr.vw iconv0 iconv0 16)  ;; EE
	  (m preconverted iconv0)   ;; EE->VU0

	  ;; (format *stderr* "Value is ~d~%" iconv0) ;; GOAL
	  
	  ;; VU ftoi15
	  (cvt.s.w offset preconverted :fixed 12) ;; VU0
	  
	  (s.q offset offset-vec)  ;; VU0->GOAL
	  )
	)
      
      
      ;; figure out distance
      (let ((time-remaining (-> particle timer))
	    (distance 0.0)
	    (tt 0.0)
	    )
	(set! tt (/ (the float (-> particle timer)) (the float (frame-time
90))))
	(*! tt tt )
	(set! tt (- 1.0 tt))
	
	(vector-normalize! offset-vec (lerp PARTICLE_RED_2_CHARGE_SIZE
(meters 0) tt))
	
	(vector-copy! origin-vec (-> *target* gun fire-point))
	
	(set! (-> spvec trans x) (+ (-> offset-vec x) (-> origin-vec x)))
	(set! (-> spvec trans y) (+ (-> offset-vec y) (-> origin-vec y)))
	(set! (-> spvec trans z) (+ (-> offset-vec z) (-> origin-vec z)))
	)
      )
    )
  )

Click to expand...

Scott Shumaker sweng-gamedev@midnightryder.com
Thu, 11 Aug 2005 01:42:27 -0700

* Previous message: [Sweng-gamedev] Higher Level Languages
* Next message: [Sweng-gamedev] Higher Level Languages
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

> Why "alternative" languages are not used by game developers as main
development tool?

I think there's a number of factors. It has always surprised me that while
game programmers are typically very forward looking and interested in
learning new technology, we tend to be incredibly set in our ways when it
comes to programming languages. I'm sure everyone has had the experience
where they've encountered a co-worker who insisted that 'C++ is pure evil'
(although there is SOME truth to that, perhaps .

On the other hand, let's put things in perspective:

If you look at the art side of game development, artists have gone from
using simple paint programs to incredibly sophisticated 3d modeling and
animation packages. We've invented whole new paradigms for the way we
create art content for our games, but programmers continue to rely on a tool
that was designed over 15 years ago for writing operating systems. I'm not
claiming there's an easy solution, but it certainly pales in comparison to
the complexity of a modern art tool like Maya. Can you imagine if artists
were stuck using the same tools that they had back in the 8-bit era? It's a
nonsenical question - we wouldn't be able to make the games we make today
without the advances in art technology. What types of games could we be
making today if we had a better programming language?

And what types of games won't we able to make tomorrow, because we need to
hire huge teams of programmers for next-gen development, and get
correspondingly huge budgets, and even then we're spending all of our effort
on tech and none on gameplay let alone innovative gameplay, and we end up
with something derivative and uninspiring?

I'm not claiming a better language is the silver bullet for all of our game
development woes, but I'm firmly convinced it's a major step in the right
direction.

Scott Shumaker
Lead Programmer, Naughty Dog
sshumaker@naughtydog.com

The views expressed above are my own, and do not necessarily reflect those
of Naughty Dog.

Click to expand...

Doom_Bringer said:

i posted this last year pana, see my tag for further details

Click to expand...

Panajev2001a said:

He makes some very interesting point about the need he feels regarding a new programming language geared for game development, such as G.O.A.L. was: they were very happy and productive using it and they would have extended G.O.A.L. for the next/generation platforms if they had not been bought.

Click to expand...

Wollan said:

Maybe Sony should make ND design a new one for the PS4 gen that the 1st party could use across the board? (With coding courses and tools and whatnot released way ahead of time so people can learn/take a look before they even begin development for that future platform).

Click to expand...

Doom_Bringer said:

Not every first party Sony developer is as smart as those crazy Naughty Dog guys Not to mention the learning curve would very steep. Next gen hardware + a new programming language = a no no

Click to expand...

Doom_Bringer said:

Not every first party Sony developer is as smart as those crazy Naughty Dog guys Not to mention the learning curve would very steep. Next gen hardware + a new programming language = a no no

Click to expand...

Blimblim said:

Wow, very nice read. Looks like they were going the right way with GOAL.

Click to expand...

Panajev2001a said:

I guess it was working very well for them. With that said, there are MANY bright folks in SCEA, SCEE and SCEI (SCE WWS) that have pushed the technical envelope (God of War I/II, Primal, ICO/SotC... just to name only one amazing title for each group [and there are many other amazing titles Sony has been pumping out since PlayStation's earliest days]) like very few of their peers have been capable of doing. While it has been a set-back for them to go back to more common programming languages and practices so to speak, it is IMHO very valuable to them and to the other SCE WWS developers as well (in equal manner IMHO) if they can all collaborate and share ideas and solutions very rapidly.

Click to expand...

Blimblim said:

Well, you've got to agree that a system that REQUIRES you to write code for 2 (or even more) different type of processors makes everything much harder for the coders. And since your usual C/C++ compilers and especially debuggers aren't always very happy with inline ASM (not even mentioning making your code look like shit) it's yet another thing where developing an entirely new, or a derivative like here, language makes sense.

Click to expand...

Being able to drop a new version of a function in realtime in the game, that's some really cool stuff though.

Click to expand...

Zen said:

I had no idea that Naughty Dog made GOAL or that it even existed. It's pretty remarkable when you think about it. Doesn't the God of War team have something that's sort of similar? I could have sworn I'd read a speech with their lead programmer adn he talked about programing an editor that let the animators and graphics artists and AI programmers tweak their work and drop it in while the system was running?

I probably remembered it incorrectly, but the real reason I made this post is to ask if Anyone knows if Ready at Dawn uses the GOAL? I'd assume since isn't their PSP engine based off Naughty Dogs PS2 engine?

Click to expand...

Zen said:

I had no idea that Naughty Dog made GOAL or that it even existed. It's pretty remarkable when you think about it. Doesn't the God of War team have something that's sort of similar? I could have sworn I'd read a speech with their lead programmer adn he talked about programing an editor that let the animators and graphics artists and AI programmers tweak their work and drop it in while the system was running?

I probably remembered it incorrectly, but the real reason I made this post is to ask if Anyone knows if Ready at Dawn uses the GOAL? I'd assume since isn't their PSP engine based off Naughty Dogs PS2 engine?

Click to expand...

Zen said:

I could have sworn I'd read a speech with their lead programmer adn he talked about programing an editor that let the animators and graphics artists and AI programmers tweak their work and drop it in while the system was running?

Click to expand...

Doom_Bringer said:

Click to expand...

Lathentar said:

That's just a scripting engine, which is seperate from the game code. What Naughty Dog did is incredible.

Click to expand...

mckmas8808 said:

This game uses the same engine that God of War 2 uses? WOW!

Click to expand...

Doom_Bringer said:

Goal died ... with the departure of Andy Gavin.

Click to expand...