Assertions in Game Development, Part 1

Assertions in game development pose a particular problem that is unique to our industry. In other software development industries, software is generally developed and thoroughly tested, before it is delivered to the users. But in game development we offer half finished, barely tested versions of our work to our users. No, not to the consumer (hopefully), but to the artists and designers on our team. The content team. They are users, too. In order to put awesome content into the game, they need to be able to run it every day, stable or not, so they can see their work in its proper context.

The original idea of assertions

“In order that the man who checks may not have too difficult a task the programmer should make a number of definite assertions which can be checked individually, and from which the correctness of the whole programme easily follows.”
Turing, A., Checking a Large Routine. in Conference on High Speed Automatic Calculating Machines, (Cambridge, UK, 1949), 67-69.

The C assert macro itself, which we all know and love, was introduced by Dennis Ritchie in 1978, as part of the first formal C language specification.

The purpose of the assert macro, as designed, is to verify preconditions in key parts of the code. When such a test fails, the intended sequence of events is:

  • execution is halted
  • the programmer is alerted
  • the programmer examines the current state of the program while halted
  • the programmer fixes the problem in the code, recompiles and restarts the program

In other words the assert macro is an important tool for the programmer, or “the man who checks”, but not for the user of the software. The general idea is that by the time the software is put in the hands of the user, all test failures have been eliminated. Some developers even remove the assertion test in production builds, others just silence them but leave fix-up code in place.

But in the game development industry, we hand off our poorly tested executables to our content team, with assertions and all.

So what happens, all too often, is that an assert macro detects a problem, halts execution, and confronts the content artist with a screen full of gobbledygook. These messages are meaningless to them. The problem is that in the original vision “the programmer is alerted”, but the programmer is not alerted at all, unless the artist gets up from his desk and tracks the programmer down. And which programmer should that be? Meanwhile the artist has an unusable game executable, and they are working in the dark. That is an unacceptable interruption in the artist’s workflow.

Over the years, game development teams have come up with a variety of modifications to the assert macro, to address this problem.

First attempt: “skippable” assertions

Skippable assertions allow the user to resume execution of the program, without needing to call over a programmer. Since each assertion test already involves a test for a problematic condition, it is often possible to prevent immediately failure, recover and continue execution. Such a fix-up is usually just covering up the symptom, not the underlying problem, and the program is now running with part of its functionality disabled or with corrupt data. The program is continuing to run “with its fingers crossed”, but it is better than stopping. Stopping at every failure could lead to significant loss of productivity.

The following snippet shows the general idea of a skippable assertion in its simplest form.

// A primitive skippable assertion system
// (for illustration only, not recommended)

#include "stdio.h"
#include "math.h"
#include <stdarg.h>
#include <cstring>

void HandleSkippableAssert( const char* file_name, int line_number, const char* expr, const char* fmt, ... )
{
  // This is of course just a placeholder for a real
  // assert handler. The real thing will display a
  // graphical interface and offer options such as
  // skip or debug, and include additional useful
  // data such as a call stack.

  va_list args;
  va_start( args, fmt );
  printf( "*** Assertion failed in %s, line %d\n", file_name, line_number );
  printf( "*** Expression: %s\n*** Message: ", expr );
  vprintf( fmt, args );
  printf( "\n*** Press return key to continue" );
  getchar();
  va_end( args );
}

#define ASSERT( expression, format, ... )\
(\
  ( ( expression ) ? \
    true :\
    (\
      HandleSkippableAssert( __FILE__, __LINE__, #expression, format, ##__VA_ARGS__ ),\
      false\
    )\
  )\
)

struct Vec3
{
  Vec3( float X, float Y, float Z ) : x( X ), y( Y ), z( Z ) {}
  float x, y, z;
};

Vec3 operator* ( const Vec3& v, float scalar )
{
  return Vec3( v.x * scalar, v.y * scalar, v.z * scalar );
}

float VecLength( const Vec3& v )
{
  return sqrtf( v.x * v.x + v.y * v.y + v.z * v.z );
}

Vec3 VecNormalize( const Vec3& v )
{
  float length = VecLength( v );
  if( ASSERT( length != 0.f, "Vector length must be > 0" ) )
  {
    return v * ( 1.f / length );
  }
  else
  {
    return v;
  }
}

#define ARRAY_SIZE( a ) ( sizeof( a ) / sizeof( *a ) )

int main (int argc, const char * argv[])
{
  Vec3 my_vecs[] =
  {
    Vec3( 1, 2, 3 ),
    Vec3( 4, 5, 6 ),
    Vec3( 0, 0, 0 ),
    Vec3( 7, 8, 9 ),
  };

  for( int i = 0; i < ARRAY_SIZE( my_vecs ); ++i )
  {
    Vec3 v = my_vecs[ i ];
    Vec3 n = VecNormalize( v );
    printf( "Vec %f,%f,%f is normalized %f,%f,%f\n", v.x, v.y, v.z, n.x, n.y, n.z );
  }
  return 0;
}

Skippable assertions get ignored

Of course the problem that the assertion detected still needs to be reported and looked at. And that is the skippable assertion’s weak point. Since the messages are just noise to the content artist, they are skipped and ignored. It is not fair to rely on the user of the software to report these messages back to the its creator. It is not their job.

A skipped assertion is a useless assertion. Why is it even there?

And in practice, many of these assertions are overlooked, even by engineers. Why would the animation engineer pay attention to the sound bank initialization assertion, if he can just test his animations without sound? And as a consequence, some errors remain in the software for a long time, accumulating more and more annoying assertion dialogs that promptly get dismissed. In the past, I have heard several proposed approaches to make skippable assertions more -ahem- assertive, more insistent, harder to ignore. As if we need to somehow force the developer to pay attention and fix their errors. But this is solving the wrong problem. The problem is not that assertions are too easy to ignore or that engineers are too lazy. The problem is that by and far, assertions are presented to the wrong audience. They pop up on anyone’s machine. The signal to noise ratio is very poor. If 80% or more of assertion dialogs contain information that is not intended for you, of course clicking the “skip” button becomes a reflex.

The solution is to build a system that accurately routes the assertion message to the person that can do something with the information.

Second attempt: “tagged” assertions

Tagged assertions are a more evolved form of the skippable assertion. This form of the ASSERT macro was introduced in an attempt to present the message to a more narrow audience, ideally the one engineer who is in a position to fix the problem. Each assert macro that is placed in the code is provided with a tag, in this example the user name of the author of this piece of code.

// Partial example of a tagged assertion

#define ASSERT( tag, expression, format, ... )\
(\
  ( ( expression ) ? \
    true :\
    (\
      HandleSkippableAssert( tag, __FILE__, __LINE__, #expression, format, ##__VA_ARGS__ ),\
      false\
    )\
  )\
)

Vec3 VecNormalize( const Vec3& v )
{
  float length = VecLength( v );
  if( ASSERT( "rpieket", length != 0.f, "Vector length must be > 0" ) )
  {
    return v * ( 1.f / length );
  }
  else
  {
    return v;
  }
}

When the error is detected, and the assertion handler is called, and it does one of two things.

  • If the tag matches the logged in user name, the normal assertion failure message is displayed. The idea in this case is that the author of the piece of code is the intended audience for the message.
  • If the tag does not match the logged in user name, the assertion message is dumped into a text file, and sent to the user who is identified in the tag. There are several ways this can be implemented. A simple way would be to copy the text file into a folder on the network, clearly identifying the intended recipient. A more sophisticated system may send an email, or update a bug database.

(The details of this the network/email/database system are beyond the scope of this article — perhaps another time)

Although this clearly is much more targeted than the “shotgun” approach of the skippable assertion, it still misses the mark in too many cases. I wouldn’t want to be the author of the libraries. Let’s assume that the VecNormalize() function in the above example is part of the math library, and that “rpieket” is the author of this part of the math library. And let’s assume that it is the animation system that calls VecNormalize() with a zero length vector. Clearly, the problem that was detected by the assert macro in VecNormalize() was caused by the animation system, and not the math library. The library author may does not have the knowledge (or the time) to debug the animation system. This assertion failure should be addressed to the author of the animation library.

And that’s exactly what we’ll do in Part 2.