December 13, 2012

Aim acceleration in console shooters - Part 2: Results

Following the measurement procedure described in part 1 of this article, the aim acceleration characteristics of some popular FPS games on the Xbox 360 are now presented and discussed. The list of games is:

Call of Duty: Modern Warfare 2
Call of Duty: Black Ops
Battlefield 3
Halo: Combat Evolved
Halo 2
Halo 3
Halo: Reach
Halo 4
Shadowrun
Rainbow Six: Vegas 2
Team Fortress 2
Quake Arena Arcade
Borderlands
Unreal Tournament 3


A few considerations that should help with interpreting the results:
  • The shooter begins in a stationary position. The first frame of the shooter's rotation is identified by the first observable reticle movement in the captured video data. As such, the experiments do not measure the input lag between controller and console or the response time of the video display.
  • Unless otherwise stated, the results presented are for a full speed rotation (full stick deflection) at typical look sensitivities, which means they illustrate the effect of aim smoothing on the shooter's ability to twitch rather than his ability to make fine aiming adjustments. As it turns out, almost all games tested do not use any aim smoothing for small stick deflections, so it should not have much impact on the shooter's ability to track the target after engaging it.
  • The estimates of rotation angle and rotation speed are good but not perfect. The estimation error for rotation angle (attributed mostly to my own errors in judgement) is a fraction of a degree under ideal circumstances, but this can increase when the shooter reaches a high rotation speed because of the motion blur in some games. What is important is the overall trend in rotation speed over time rather than any slight variations from frame to frame.
  • In FPS games it is quite common for the shooter to make short twitches of 45° or 90° when searching for or fixing on targets. It's much less common to see large sweeps of 180° or 360°. Accordingly, we should be most concerned with the aim acceleration behavior over the first quarter of a rotation than anything afterwards. On the other hand, it is still useful to look at more data, say a complete rotation or a full second of time, to gain an understanding of the particular aim smoothing algorithm being used.
  • For the most part, I think it's sufficient to look at rotation speed as a function of time in order to assess the aim smoothing behavior of a game. If you're insistent on quantifying acceleration, the slope of the rotation speed curve (2nd derivative of the rotation angle curve) yields acceleration in degrees per second squared.



Game Year Developer Frame rate Look sens range
Call of Duty: Modern Warfare 2 2009 Infinity Ward 60 fps 1-10

Despite the criticisms of the style of play it promotes and the skill required to play it, CoD is frequently praised for a fluid and responsive aiming system. The 60 fps frame rate, which few other FPS games on the Xbox 360 have been able to achieve, no doubt has something to do with this. What we can also see below is a straightforward aim smoothing algorithm - the shooter's turning speed increases linearly from zero to maximum speed over a span of just 0.2 seconds. According to the rotation angle plot (left), 0.2 seconds corresponds to a 27° turn on 2 sens or a 48° turn on 4 sens.


It's also instructive to see how CoD handles smaller analog stick deflections. This can be simulated by reducing the amplitude of the step voltage applied to the controller circuit. Shown below are the rotation angle and speed plots on 2 sens for various percentages of a full stick deflection. As can be seen, the slope of the speed curve (acceleration) is identical in all cases. In other words, CoD subjects the player to aim smoothing regardless of how fast they're attempting to turn, but the time window of the smoothing effect is naturally reduced for low rotation speeds such as when aiming.





Game Year Developer Frame rate Look sens range
Call of Duty: Black Ops 2010 Treyarch 60 fps 1-10

Comparing the Blops and MW2 results, it should be clear that nothing was changed with regards to aim acceleration between these two iterations of the CoD series. Note that there wasn't any special reason for testing on 3 and 5 sens rather than 2 and 4 sens.





Game Year Developer Frame rate Look sens range
Battlefield 3 2011 DICE 30 fps 1-33

Battlefield 3 is our first game that doesn't use any aim smoothing. It actually presents the feeling of gradual acceleration by shifting the weapon slightly during the turn animation, but if you follow the reticle position (which is what matters) you'll find that it moves at a constant rotation speed the whole time. On a different but related topic, I would give the game a strike against for adding an extra frame of unnecessary input lag. This was discovered when I noticed that the stick must be activated for at least 2 frames (67 msec) to register any change in the shooter's look direction (tested pre-patch, might have been fixed).





Game Year Developer Frame rate Look sens range
Halo: Combat Evolved 2001 Bungie 30 fps 1-10

There exists a group of Halo players who remain strong supporters of the first installment of the series, viewing everything that came afterwards with a varying degree of contempt. They're probably right about the overall competitive merits of Halo CE compared to its successors, but they also tend to overreach by making unfounded assertions about the game's mechanics. One such example is the oft repeated claim that aim smoothing was introduced in Halo 2 or Halo 3 (depending on whether or not the claimant liked Halo 2) as part of Bungie's continuing efforts to appeal to casual players. As can be seen from the plots below, this claim is false.

Halo's implementation of aim smoothing differs significantly from that of CoD - although the increase in rotation speed is also linear, the initial speed is higher and the acceleration is spread out over a much longer period of time. You can't quite see it but the speed curves do flatten out beyond 1 sec on the time axis. Focusing on the first 90° of the shooter's turn, this takes about 0.5 sec on 3 sens during which time the rotation speed approximately doubles (114°/sec to 234°/sec). On 5 sens, a 90° turn takes 0.4 sec and the speed increases by a factor of 1.78 (153 °/sec to 273°/sec). In other words, the change in rotation speed over typical turning angles is big enough that you'll definitely notice it when twitching.


A natural question to ask is whether or not the aim smoothing in Halo can be avoided. The answer is yes, but only at the cost of turning slower. Again by testing a range of reduced stick voltages, I found that the shooter could rotate at a maximum of 70°/sec on 3 sens or 90°/sec on 5 sens without experiencing any smoothing. The 3 sens results are shown below with the curves labeled according to the equivalent stick deflection percentage. To put these numbers in perspective, avoiding smoothing on 3 sens would require 1.3 sec to make a 90° turn compared to the 0.5 sec for a full stick deflection. That seems impractically slow for trying to locate targets and secure first shot in the battle.





Game Year Developer Frame rate Look sens range
Halo 2 2004 Bungie 30 fps 1-10

Halo 2 aim smoothing is very similar to that of Halo CE, the minor difference being that rotation speed reaches its maximum slightly earlier (at about 0.8 sec instead of 1.0 sec). This modification would not be felt by the player unless they were making some very long turns at high speed.





Game Year Developer Frame rate Look sens range
Halo 3 2007 Bungie 30 fps 1-10

The Halo 3 results are essentially the same as Halo 2. You might notice an anomaly on the first frame of the 3 sens speed curve where the initial speed is lower than it should be. A glitch? I'm not sure how significant this would be in practice. Inconsistencies in the aiming system could be a serious problem, but a low rotation on the first frame might not be easily noticeable in light of the slower stick movement of a real player.





Game Year Developer Frame rate Look sens range
Halo: Reach 2010 Bungie 30 fps 1-10

In Halo Reach we again see the familiar pattern of gradual aim acceleration over the span of nearly a second. It looks like there might have been a subtle change to the smoothing algorithm such that maximum rotation speed is reached after a specific angle (about 225°) rather than after a fixed amount of time. Again though, this change is superficial as it would only be noticeable for very large turns.





Game Year Developer Frame rate Look sens range
Halo 4 2012 343i 30 fps 1-10

Halo 4 is the most recent installment of the Halo series. Its aim acceleration characteristics are similar to its predecessors and, in particular, appear to be identical to Halo Reach.





Game Year Developer Frame rate Look sens range
Shadowrun 2007 FASA 30 fps 0-9

Shadowrun is a game that received high praise from the competitive community, more so for its strategic depth than its shooting mechanics. As can be seen from the plots, it is actually quite similar to Halo with regards to aim acceleration. The 2 sens rotation speed curve reaches its maximum at about 1.0 seconds (which is cut off on the graph) while the 4 sens curve maxes out at 0.7 seconds. The slope of the speed curves appears to increase slightly over time whereas for Halo it was perfectly linear. This change is relatively subtle, so it's doubtful it would produce a dramatically different feel to aiming.





Game Year Developer Frame rate Look sens range
Rainbow Six: Vegas 2 2008 Ubisoft 30 fps 1-10

The first thing you'll notice about R6V2 is the slow turning speeds even on high sensitivity, which I guess could be attributed to it being more of a tactical shooter. Nevertheless, it is still interesting to see how another developer approaches aim acceleration. The results show the typical linear increase in rotation speed after starting off at a low value (initial speed is about 28% of final speed). The smoothing effect lasts for 0.37 sec, which is somewhat of a middle ground between CoD (0.2 sec) and Halo (0.8-1.0 sec).





Game Year Developer Frame rate Look sens range
Team Fortress 2 2007 Valve 30 fps 1-44

Although the PC version of TF2 remains the most popular by a wide margin, the Xbox 360 version was moderately successful in its time. The Xbox version is quite heavy on aim smoothing. The shooter starts off at such a slow rotation speed (45°/sec on 12 sens, 66°/sec on 20 sens) that there really isn't any way to avoid a lengthy period of aim acceleration (90° turn takes 0.58 sec on 12 sens and 0.46 sec on 20 sens). There's actually a weird bug in the game that causes the rotation speed to plummet instantly by a factor of 2 after 1.0 sec on time axis, but this isn't of much practical interest.





Game Year Developer Frame rate Look sens range
Quake Arena Arcade 2010 id / Pi 60 fps 1-13

If a franchise with as much history as Quake can't even gain a respectable following on the xbox 360, it seems that fast paced shooters on the console are forever doomed. Quake Arena Arcade runs at a smooth 60 fps and it stands as only the second game in our list to forgo aim smoothing in favor of the natural stick behavior. Curiously, switching the look sensitivity from 4 to 7 results in a change of only 13% in rotation speed.





Game Year Developer Frame rate Look sens range
Borderlands 2009 Gearbox 30 fps 1-11

Aim acceleration in Borderlands is similar to that of the Halo series. Once the rotation speed threshold (about 90°/sec on 4 sens and 135°/sec on 6 sens) is exceeded, it requires 0.8 seconds to reach maximum speed. I included two separate trials on 4 sensitivity to illustrate some inconsistencies in the aiming system - occasionally there is a higher than expected jump in rotation speed (hard to notice when playing because it's corrected on the following frame) as well as a lower than expected response on the first frame.





Game Year Developer Frame rate Look sens range Aim accel range
Unreal Tournament 3 2008 Epic 30 fps 5-20 1-10

I saved UT3 for last because there's a lot to say about its approach to aim smoothing. Like its twitch shooter counterpart Quake, it failed to gain any serious traction on the Xbox 360. The interesting thing about UT3 is that it actually has an aim acceleration setting which can be tweaked from 1 to 10. I've heard people trumpeting this setting as if it's the cure for all aiming ills, allowing the player to customize the level of aim smoothing or remove it completely. Don't get me wrong, I think an aim acceleration setting could be a nice feature for console FPS games which might even become standard if console developers weren't so stubbornly minimalist (compare to the wealth of settings offered by most PC shooters), but UT3's interpretation of this setting is puzzling to say the least.

Before going into what the aim acceleration setting actually does, let's first look at our usual plot of reticle speed versus time with acceleration set to its highest value of 10. For the physics impaired, max acceleration implies the fastest increase in rotation speed, i.e., the least amount of aim smoothing, so this is where we might expect to see the single frame transition from zero to top speed as in Battlefield 3 and Quake. What we actually see is probably the most byzantine aim smoothing algorithm ever conceived. The shooter's rotation starts off at a moderate speed that remains constant for 4 frames (0.13 sec), jumps to a higher speed that is also held for 4 frames, then ramps up rapidly, eventually settling at a very high value (off the graph) after about 0.9 sec. What purpose is served by deliberately introducing a discontinuity in rotation speed? It seems like the sort of thing you'd want to avoid when developing an aiming system that feels comfortable and consistent to the user.


So how about that aim acceleration setting? Intuitively, what I'd expect to see by adjusting this setting would be a slower / faster transition from the initial rotation speed to the final rotation speed. Keep in mind that the initial and final speeds are typically determined by the look sensitivity setting. As can be seen from the plots below, this is not what happens at all. Although the slope of the rotation speed curve (acceleration) does change according to the acceleration setting, the shooter always takes the same amount of time to reach max speed and it's actually just the initial and final rotation speeds that are being modified. In other words, the aim acceleration setting is behaving just like a look sensitivity setting. Perplexing implementation.





Concluding Remarks


I was somewhat surprised at the disparity among implementations of aim smoothing in the games tested. There isn't any 'stock' algorithm that developers rely on, and there also doesn't appear to be any clear relationship between acceleration and style of play. For example, Battlefield 3 and Quake Arena Arcade were both identified as games that don't employ any smoothing, yet there isn't much common ground between the two games with respect to movement speed, pace of play or duration of engagements.

So what can be said about the various approaches? I think it's worthwhile revisiting Call of Duty and Halo, two popular franchises which take very different approaches to aiming. More specifically, consider the Halo 4 and CoD Black Ops rotation speed curves for 3 sensitivity with 100% and 82% analog stick deflection, as shown below.


Similar to the results presented earlier for CoD MW2 and Halo CE, the 100% stick deflection curves illustrate a big difference in the duration of aim smoothing in the two games. In H4 the shooter requires a long time (1.0 sec) to approach maximum speed whereas in CoD BO the transition to max speed is very quick (0.22 sec). This is why a feeling of sluggishness is much more apparent in Halo when attempting to turn rapidly.

The 82% stick deflection curves are even more revealing with regards to the aiming philosophies of the two games. Halo grants the player a relatively large analog stick range (about 90%) at a low rotation speed and without any smoothing. The idea, I think, is to give the player a large 'aiming range' where the stick position is honored immediately. Only with a full stick deflection of 90%-100% (the 'twitching range') does the shooter accelerate to the high speed required for target searching .

In contrast, CoD BO always exhibits a brief period of acceleration when turning. However, the duration of the acceleration window is reduced according to the desired turning speed, and in fact it lasts only a few frames at typical aiming speeds. I suspect this is short enough to avoid negatively impacting the player's aiming ability . In my opinion, the merits of this system are its consistent behavior over the entire range of the stick (and for all look sensitivities) as well as its agreement with the basic purpose of aim smoothing, i.e., to provide a slight tempering of rapid reticle swings without grossly limiting the twitching ability of the shooter.


Coincidentally, I believe this is also the reason that 'slow turn' controllers are a bigger problem in Halo than CoD. These controllers have a worn out or defective analog stick that, when fully deflected, produces a voltage indicative of <90% deflection. This prevents the shooter from entering the 'twitching range'.
It can be argued that awareness and reaction speed in CoD are more important than precision aiming. Shooting battles are frequently concluded in less than a second, often before the slower player even has a chance to return fire.

Aim acceleration in console shooters - Part 1: The Basics

What is aim acceleration?


In shooting games, aim acceleration is related to how a physical movement of the input device, such as an analog stick, is translated into changes in the shooter's look direction. More specifically, it refers to the way that many console FPS games limit abrupt increases in the shooter's rotation speed by slowly ramping up the speed over time. The concept is somewhat analogous to platform games where the character model's running speed accumulates before reaching its maximum, except applied to rotational movement instead of linear movement.

I prefer the term 'aim smoothing' to describe the filtering of twitchy analog stick movements into a more gradual turning motion. Acceleration, by definition, occurs whether the changes in rotation speed are instantaneous or very slow. The confusion arises when people refer to games as having "more acceleration" when what they actually mean is that the increase in turning speed requires more time, i.e., lower acceleration.

Presumably, rather than attempting to accurately simulate real world physics, the main purpose of aim smoothing is to prevent rapid swings in look direction which could make aiming or changing direction feel difficult to control for the player. It therefore might be viewed as a primitive form of assistance for manipulating the analog stick. However, smoothing stands apart from the more obvious forms of aim assistance (sticky aim, inflated hitboxes) in a couple of ways:
  1. It happens even when the shooter does not have an enemy targeted
  2. Its behavior is usually independent of the shooter's choice of weapon
You'll occasionally hear complaints that aim smoothing is too severe in some games. This can actually hamper the player by imposing a limit on how fast they can react and creating a feeling of disconnect between the input controls and the character model's actions. The vast majority of console shooters have a specific smoothing behavior hard-coded into the game and do not offer any settings for adjusting its severity.


Mouse vs Controller


If you ask why a mouse is superior to a controller for FPS games, the usual answer you'll get is that the wider range of motion of the mouse naturally provides finer control for aiming than an analog stick. While this is certainly part of the equation, I think it glosses over a fundamental difference in how mouse movements and analog stick movements are translated to changes in look direction.

A mouse provides the player with direct control of the angular position of the reticle - displace the mouse by X centimeters and your look direction changes by Y degrees, where Y is proportional to X. It also provides precise control of rotation speed according to how fast the mouse is moved. In contrast, an analog stick affords the player only direct control of angular velocity - deflecting the stick by X degrees calls for a rotation speed of Y degrees per second . It's fairly obvious why analog sticks are interpreted in this way - as the stick is physically attached to the controller, its range of motion is very limited and you can't 'reset' it to a convenient location on the desk after it has been moved too far like you can with a mouse.

Why does any of this matter? I would contend that an input device which converts displacement to reticle speed is inherently more difficult to control (in the sense of supporting both fine aim adjustments and fast twitches) than one that converts displacement to reticle position. Justification for this claim comes from the physics involved, more specifically the integral-derivative relationship between angular position and angular velocity. For example, whereas moving the mouse at constant speed causes the reticle to turn at constant speed, moving the thumbstick at constant speed would cause the reticle to turn at linearly increasing speed. This controller 'hypersensitivity' is tempered by lowering the maximum rotation speed and introducing artificial measures such as aim smoothing to limit the growth in rotation speed over time (acceleration). As a result, players with good control of the sticks have reported a sluggish or unresponsive feel to reticle twitching in many console games.

Irrespective of aim acceleration, the relationship between stick deflection angle and rotation speed is usually nonlinear. Basically, once you pass the 'dead zone' the stick becomes increasingly sensitive moving towards the limits of its range. The reason for this behavior might be to give the shooter better fine aiming control. This is a topic in itself which I'll leave for another day.


Measuring Aim Acceleration


To investigate how aim smoothing is implemented in some of the more popular console FPS titles, we can simulate a basic thumbstick movement and observe the reticle rotation over time. I say simulate because, rather than attempting to manually replicate a predefined stick movement, we can use the controller hack discussed in the previous blog entries to directly synthesize the stick voltage. This approach provides precise control of the aiming signal seen by the console with near perfect repeatability.

What sort of stick movement should be simulated? Preferably something simple and intuitive that does a decent job of exposing the aim smoothing implemented by the game. I chose to use what might be referred to as an 'ideal twitch' - a step function that transitions instantaneously from the default stick voltage to a higher voltage and then remains constant, thus simulating a quick twitch of the stick from 0 degrees (resting position) to X degrees on the horizontal. Of course, in the real world it takes the player a small amount of time to twitch the stick, but I don't think it's so important to mimic this behavior for the purposes of our investigation.

It helps to know what result we're expecting to see with this ideal twitch input. Based on the previous explanation of the analog stick as a device that converts angular displacement to rotation speed, we should, in the absence of aim smoothing, observe an immediate jump in rotation speed from 0 to Y and then the constant speed Y until the input is removed. In other words, the angular velocity of the reticle should track the stick voltage perfectly. In contrast, when aim smoothing is present we expect a more gradual rise in rotation speed from 0 to Y. In both cases the eventual rotation speed Y is governed by our choice of look sensitivity in the game's control settings. The basic relationship is summarized in the plot below .


Notice also the instantaneous deceleration of the reticle when the stick is returned to its resting position (just after 1.6 sec on the time axis). This behavior, which allows the player to quickly reverse the reticle direction, has been confirmed for many different FPS games, and it suggests that aim deceleration doesn't warrant any further attention.


The next question is how exactly to measure the angular velocity of the reticle based on what we see from a first person point of view (PoV) on screen? A video capture card can be used to record the turning sequence which can then be played back frame by frame. The basic idea is to estimate the reticle's angular position θ at every frame and then calculate angular velocity ω by numerical differentiation,
where k denotes frame index and T is the time between video frames. One point to keep in mind is that numerical differentiation is quite sensitive to noise, meaning any inaccuracies in our estimates of θ will be amplified in ω.

What's needed is a way to accurately determine the shooter's rotation angle θ over time based on the video data. My approach was to build a video 'template' of known rotation angles by recording a full 360° rotation at a slow, constant speed. This can be accomplished by choosing a low look sensitivity, applying a small increase in voltage to the stick (a slower turn means more frames in the template, i.e., better angular resolution), and analyzing the second rotation in the sequence (because aim smoothing typically wears off during the first rotation). Since the speed of rotation is constant, the resolution of the template can be calculated simply by counting the number of frames for a full rotation. Typically, I set the speed slow enough to obtain a resolution of better than 3° between successive frames.

Once the template has been recorded, it's just a matter of doing 'measurement' recordings with a few different sensitivities and stick voltages - this time looking at the shooter's first rotation because we want the aim acceleration to be captured - and comparing the results to the template. Background scenery in the video makes it easy to identify the two template frames that fall on either side of a given measurement frame, and we can estimate θ by visual interpolation. The three images shown below illustrate how this works. The top image is the 10th frame of a full speed rotation on 3 sensitivity with aim acceleration present (starting from rest). The bottom images are the two nearest template frames, which have known rotation angles of 20.7° and 22.2°. Clearly the measurement frame is closer to the 20.7° template frame, so the shooter's rotation angle can be judged to be about 21.1°. The big white rectangles in the center are just an overlay added by my image processing script to make it easier to judge the angles based on the background scenery.

Black Ops measurement frame

Black Ops template frame (20.7°)

Black Ops template frame (22.2°)

Repeating this process for every frame of the shooter's turn gives a complete record of θ which can then be used to compute the rotation speed ω as explained above. I had considered trying to automate the entire process with more sophisticated image processing code for the template comparisons, but this proved difficult and I ended up falling back to the visual interpolation technique. Here is a short video which summarizes the measurement procedure and shows the frame by frame results for CoD: Black Ops and Halo 4:


That's all for the basics of aim acceleration and how it can be measured. Check out part 2 for a comprehensive analysis and discussion of the aim acceleration characteristics of many well-known console shooters.


September 22, 2012

An analog hack of the Xbox 360 controller - Part 2: Applications

The controller hack and DAC configuration are discussed in Part 1 of the blog. Now let's look at how it can be applied to achieve some impressive results in The Impossible Game, Super Meat Boy, Guitar Hero 3 and Halo.



How I Learned to Stop Raging and Win The Impossible Game

The Impossible Game is a 2D platformer that is one of the more popular Xbox Live Indie games (XBLIG). Simply described, you're an orange square moving through the level at a constant horizontal speed and you must jump your way to the finish while avoiding the spikes and pits. Each level (there are 3 in total if you buy the level pack) is about a minute and a half in length. Sounds easy, but you're in for a sizeable helping of rage if you plan on finishing. Numerous sets of triple-spikes are scattered throughout the levels, and there is only a couple video frames worth of forgiveness when jumping over these things.

So how can we output the jump button presses with the correct timings to get all the way through a level? No simple solution comes to mind. One approach is to build up the voltage sequence gradually through trial and error, each time adding one or two more jumps to get a bit further through the level. Tedious, yes, but we make relatively steady progress this way; once we've reached a certain point in the level, that part is "solved" in the sense that we'll be able to reach it every time using the same sequence. 

Following the procedure outlined above, it wasn't long before another problem cropped up. How can we ensure that our automated sequence of jumps is started at just the right time? It's very easy to be off by a few frames by manually starting the DAC sequence (clicking a button in the software) while eyeballing the Xbox screen. My initial plan for tackling this synchronization problem was to include the Xbox menu button press to start the level as the first part of the sequence. However, this didn't solve the problem completely, the reason being that the time it takes the Xbox to load the level and actually start the gameplay varied by as much as 6 or 7 frames. A more flexible approach is to allow small adjustments to the DAC sequence timing (say, a rewind or advance of 1 sample) after it has been started. This also serves as a remedy for any drift that accumulates between the DAC and xbox clocks over the long term. What we end up with is a simple set of controls for starting, advancing or rewinding the sequence in real time.  It's also helpful to have stop and reset buttons for when things go wrong and the level needs to be restarted.


This basic set of controls served quite well for most games, so let's have a quick look at the C# code used to implement it. Two software threads are required - one to handle the GUI button clicks and a second to communicate with the DAC. In NI-DAQmx the basic steps for operating the DAC are putting our voltage sequence into an array, creating a task for digital to analog conversion, configuring the task parameters (selected channels, sample rate, buffer size, etc.), writing the data to the DAC buffer (an onboard FIFO), and finally starting the task. The key point is to transfer the voltage sequence to the DAC in small chunks rather than all at once, thereby allowing the small timing adjustments (achieved by skipping or repeating one sample) or the task to be stopped early as the sequence progresses. Signalling between the two threads is primitive enough to be handled with a few booleans, and we don't even need mutexes to protect these variables (see this article). Finally, note the use of the helper class AOdataseq, which supports creation of the voltage array based on the channel mappings and voltage levels we've already calculated.


In the main thread:
// click event for Play button
private void button_ImpossibleGamePlay_Click(object sender, EventArgs e)
{
 /* some code for determining which sequence to play and creating the AOdataseq object D */
       
 // Launch the writer thread
 IGw = new IGWriter(D, VoltLev);
 IGthread = new Thread(new ThreadStart(IGw.Write));
 IGthread.IsBackground = true;
 IGthread.Start();
}
    
// click event for Stop button
private void button_ImpossibleGameStop_Click(object sender, EventArgs e)
{
 IGw.RequestStop();
}

// click event for Rewind button
private void button_ImpossibleGameRewind_Click(object sender, EventArgs e)
{
 IGw.RequestRewind();
}

// click event for Advance button
private void button_ImpossibleGameAdvance_Click(object sender, EventArgs e)
{
 IGw.RequestAdvance();
}


The worker thread:
// ---------------------------------- Impossible Game Writer class ------------------------------------
public class IGWriter
{
 private AOdataseq Dseq;
 private VoltageLevels vlev;
 private volatile bool stop = false;
 private volatile bool rewind = false;
 private volatile bool advance = false;

 //- Constructor
 public IGWriter(AOdataseq D, VoltageLevels vl)
 {
  Dseq = D;
  vlev = vl;
 }

 //- Writes the data to the DAC while checking for commands (called when worker thread is started)
 public void Write()
 {
  Task myTask = new Task("IGWriterSequence");  // create the task

  try
  {
   // create the analog output channel
   myTask.AOChannels.CreateVoltageChannel(Dseq.ChanList, "", vlev.Vmin, vlev.Vmax, AOVoltageUnits.Volts);

   // configure timing and stream properties
   myTask.Timing.ConfigureSampleClock("", Dseq.Fsamp, SampleClockActiveEdge.Rising, SampleQuantityMode.ContinuousSamples);
   long BufSize = 1000;
   myTask.Stream.ConfigureOutputBuffer(BufSize);
   myTask.Stream.WriteRelativeTo = WriteRelativeTo.CurrentWritePosition;
   myTask.Stream.WriteRegenerationMode = WriteRegenerationMode.DoNotAllowRegeneration;

   // create the channel writer object
   AnalogMultiChannelWriter writer = new AnalogMultiChannelWriter(myTask.Stream);

   // create some variables to facilitate writing data in chunks
   int xsamp = 0;     // sample index (into the voltage array Dseq.dao)
   long Nsampwritten = 0;    // total number of samples written to the DAC buffer
   int Nchunkwritten = 0;    // total number of chunks written
   int Nsampchunk = 60; //120;   // default chunk size (samples)
   int Nd;      // size of current chunk
   double[,] d;     // holds the current chunk

   while (xsamp < Dseq.Nsamp)  // loop until the entire sequence is written
   {
    // check for stop
    if (stop)
    {
     myTask.Stop();
     myTask.Dispose();
     stop = false;
     return;
    }
     
    // check for advance
    if (advance)
    {
     xsamp += 1;    // increment sample index
     advance = false;
     if(xsamp >= Dseq.Nsamp)
      break;
    }
     
    // check for rewind
    if (rewind)
    {
     xsamp -= 1;    // decrement sample index
     rewind = false;
     if(xsamp < 0)
      xsamp = 0;
    }
     
    // determine current chunk size
    Nd = Nsampchunk;
    if (Dseq.Nsamp - xsamp <= Nsampchunk)   // last chunk is smaller than normal
    {
     Nd = Dseq.Nsamp - xsamp;
     // ugggggh, this should avoid the buffer underflow exception but it apparently isn't supported by the device
     //myTask.Stream.NextWriteIsLast = true;
    }
               
    // copy the current data chunk into a temp buffer
    d = new double[Dseq.NumChan, Nd];
    for (int xchan = 0; xchan < Dseq.NumChan; xchan++)
    {
     for (int m = 0; m < Nd; m++)
      d[xchan, m] = Dseq.dao[xchan, xsamp + m];
    }

    // write the current data chunk to the DAC buffer
    if (Nchunkwritten == 0)   // first chunk (write immediately and start the task)
    {
     writer.WriteMultiSample(false, d);
     myTask.Start();
    }
    else   // all other chunks (wait until DAC buffer is partially empty before writing)
    {
     while (Nsampwritten - myTask.Stream.TotalSamplesGeneratedPerChannel > (long)(Nsampchunk / 2))
     {
      Thread.Sleep(0);  // sleep if we're not yet ready to write this chunk
     }
     writer.WriteMultiSample(false, d);
    }
     
    xsamp += Nd;   // increment sample index
    Nsampwritten += Nd;  // increment samples written count
    Nchunkwritten++;  // increment chunk count
   }

   // wait until task is done
   myTask.WaitUntilDone();
   myTask.Dispose();
   return;
  }
  catch (DaqException ex)   // handle any exceptions
  {
   if(ex.Error != -200290)   // silently ignore the buffer underflow exception
   {
    //throw new DaqException("IGWriter", ex);  // pass the exception to the calling function
    MessageBox.Show(ex.Message);   // display the exception message
   }
   myTask.Dispose();
   return;
  }
 }
  
 //- Request stop
 public void RequestStop()
 {
  stop = true;
 }

 //- Request advance
 public void RequestAdvance()
 {
  advance = true;
 }

 //- Request rewind
 public void RequestRewind()
 {
  rewind = true;
 }
}

Enough rambling, here are the results in video form.




Super Meat Boy

Super Meat Boy is a popular Xbox Live Arcade (XBLA) platformer that's well worth the $10 price tag. This game features many short levels divided up into chapters, the singular goal being to save Bandage Girl from the villainous Dr. Fetus (a la Mario Brothers). Failure to do so implies death by gravity, incineration, blunt instrument trauma, impalement, frickin' laser beams, rotary saw, ... and you get the idea. The control scheme allows for running, jumping, latching onto walls, sliding on walls, and accelerated movement via a speed button.

One of the basic principles in moving Meat Boy around the levels quickly is to maintain momentum wherever possible and rely heavily on wall jumping for direction changes. Often times this involves taking a slightly different path than the crow would fly. After playing the game for many hours and getting a look at just how ridiculously good some players are (check out the youtube links at the bottom of this post), I'd say the level of skill required to master Meat Boy physics is surprisingly high.

Following the same strategy of building up the 'solution' to a level one step at a time, which is quite feasible for Meat Boy due to the shorter levels, I focused on:
  • 2-8 The Sabbath
  • 6-3 Echoes
  • 6-5 Omega
  • 7-13x Bleach
  • Warp Zone 5-7 (The Kid)
The first 4 levels above were picked because they're good for speed running. The warp zone is just an annoying set of levels that demands some precision to avoid dying a hundred times.





Guitar Hero 3

Does this franchise need any introduction? To be honest, I almost dismissed the entire Guitar Hero (GH) series at the outset due to the sheer number of controller actions required to finish a song - we're talking a few thousand button presses for the more challenging songs played on Expert difficulty. The task of manually figuring out all these note timings seemed insurmountable. However, the simplicity of GH's graphical presentation offers a tantalizing possibility - what if the analysis could be automated using optical recognition? As it turns out, the game supports a 'no fail' cheat that allows you to play all the way through a song without hitting a single button. This means we can capture a video of the entire song and analyze it frame by frame in software to determine the note timings.

For this idea to work, a video capture card that's capable of a solid 60 fps is extremely helpful. The comparatively low time resolution of a 30 fps capture might be worked around by jumping through some hoops in the optical processing algorithm, but barring that we end up with many instances of closely spaced notes being resolved as a single longer note. I use the Hauppauge HD-PVR which is capable of a stable 720p60 through component inputs. This card features a hardware h.264 encoder which does a pretty good job with the quality vs compression tradeoff.

Problem solved, right? Not quite. I had decided to use the Matlab video processing tools for optical recognition because the ffmpeg library was looking like too much effort. Unfortunately, my version of Matlab did not have proper support for the MP4 format output by the HD-PVR, and many of the video converters out there are downright useless for format shifting h.264 MP4s at 60 fps. Despite its dubious name (AVS 4 ME? TY! <3), the AVS Video Converter supports conversion from h.264 to WMV or Microsoft MPEG4 AVI (both of which could be read by Matlab) at the source frame rate and with a good amount of options for controlling quality. Still, there were a lot of headaches due to dropped frames whenever Guitar Hero's light show was especially seizure inducing, but these were eventually resolved by using variable bit rate encoding and raising the bit rate of the output file.

The optical recognition requirements are not very demanding - we need only identify the presence or absence of colored notes moving down the fret board, a far cry from OCR of text or handwriting. For each video frame, presence or absence of a note can be determined by comparing the average color inside a small, statically located rectangle to a threshold based on the known note color. See the illustration below.


Going from left to right, define c1, c2, c3, c4, c5 as the average color inside the rectangles on an RGB scale of 0 to 255. Detection was based on:

Note color Criteria (note is present)
Green c1.green > 120
Red c2.red > 120
Yellow (c3.red > 120) && (c3.green > 120)
Blue (c4.green > 70) && (c4.blue > 120)
Orange (c5.red > 120) && (c5.green > 70)

These criteria were set according to the body color of the notes, but they also give a positive result for the white base and top of the note. The dark background of the fret board gives a negative result. For example, in the frame shown above, only a red note would be identified. The raw detection results over the entire song are then translated into a sequence of button presses (start time and duration of each press) and stored in a text file, which is subsequently loaded and converted to a voltage sequence by the DAC software.

That's the basis of a simple optical recognition algorithm that performs reasonably well on most songs. However, to achieve a really good score there are some other issues that must be addressed:
  • Insufficient resolution - Even with 60 fps video captures, closely spaced notes of the same color are occasionally resolved as a single note.
  • Duration too long - With the simple detection rule described above, regular notes usually work out to a button press duration of 4 frames. When this is long enough to overlap with the start of the next note of a different color, it will be interpreted as a mistake by Guitar Hero.
  • Timing offset - Multiple simultaneous notes comprising a chord will sometimes be detected with a 1 frame offset. This tends to happen in frames where the (statically located) rectangles land just on the edges of the notes.
  • Sustained notes - In 'no fail' mode the colored tracer on a sustained note disappears after about 10 frames. This leaves us without any knowledge of how long the sustained note actually is.
  • Star power phrases - These are sequences of star-shaped notes that increase the player's reserve of star power when executed correctly. Whammying sustained notes gains even more star power, but without a way to automatically identify these parts of the song, we don't know when to whammy.
  • Star power activation - Star power gives a big score multiplier on every note hit over its duration, so activating it at just the right points in the song is critical. There's also a technique known as 'squeezing' that abuses the timing window to get a extra few notes into an activation.
Most of the resolution, timing, and duration errors could be flagged automatically during optical recognition and manually corrected in the text file. The sustained note durations and whammies were added after reviewing the relevant sections of the captured video. As for the star power activations, fortunately a rather dedicated group of GH3 players has already systematically worked out the optimal activation points (star power paths) for most songs. So, all things considered, it still took me several hours of work on each song to get a nearly optimal result after the first cut of optical recognition.

Finally it's time for the results. The three songs shown here - One, Raining Blood and Through the Fire and Flames - played on Expert difficulty are widely considered to be the most challenging in the game.






Halo

It would have been an unforgivable oversight to finish without visiting the Halo series. More specifically our old friend Halo 2, a game that was rife with button glitches. If you listen hard enough you can still hear the echo of high-pitched screams of "cheater!" reverberating over the XBL mic. There has been many a heated debate on the topic of Halo 2 glitches over the years, and it's enough to say that the spectrum of opinions ranges from 'game-breaking' all the way up to 'game-saving' blunder by Bungie.

So, I put together a few short clips that focus mainly on Halo 2. There's also a Halo Reach golf hole that illustrates the sort of aiming and timing precision that can be achieved with the controller hack. A few random bits of MLG Halo history are sprinkled in for good measure. It's a video that will be hard to appreciate unless you're a Halo player, and even then the highest compliment you might find is 'nerdy'.




Wrap-Up

Thanks for reading. I pursued this project because I find it interesting from a technical standpoint, not to promote cheating in video games. Quite a lot of effort was needed to refine the Super Meat Boy and GH3 results to the point of being first rate, which I felt was necessary to show the capability of the controller 'hack'. All games were played offline so as not to pollute the leaderboards with tool assisted scores.

Thanks to:
  • CodeCogs for the online LaTeX equation editor.
  • YCOURIEL for the C# to HTML formatting utility.
  • xIMunchyIx for his Impossible Game video, which I shamelessly frame counted in parts of levels 2 and 3 to lessen the workload.
  • telesniperXBL, Takujixz, and ExoSDA for their Super Meat Boy speed runs.
  • ScoreHero, in particular debr and Barbaloot, for the optimal star power paths and note charts.



An analog hack of the Xbox 360 controller - Part 1: The Hack

What sort of 'hack' is possible on a video game controller that lacks a general purpose processor?

The first idea that came to mind was to mimic the USB communications between the controller and the Xbox. If we could record the signals transmitted and received using a scope or a more specialized USB signal analyzer, we could spoof the controller by manufacturing our own packets. Sounds feasible as long as the packet structure is simple and consistent, i.e., no encryption or scrambling, and we're able to convince the console that it's talking to a controller. I'm no expert on USB, so I discarded this idea as too much effort and prone to failure.

A simpler approach is to use a digital to analog converter (DAC), wired directly into the controller circuit, to simulate the button presses and stick movements. This might be thought of as an exploit of the 'analog hole' of an input device, the dual of its more well-known form in output devices such as speakers and displays. Of course, we need to know the points in the circuit to connect to and the required voltage levels. Armed with a multimeter, I was able to figure out the essentials of the controller circuit, as shown below.



The controller buttons should be familiar: A, B, X, Y, Back, Start, LB and RB (left and right bumpers). Note that the Guide button, stick clicks and D-pad buttons aren't shown because none of them were used. The left (LS) and right (RS) analog sticks are each comprised of separate horizontal (h) and vertical (v) potentiometers to allow for two axis movement. The left (LT) and right (RT) triggers are also pots, though they're effectively treated as buttons in most games (more on this later). All input signals are routed to an ASIC which presumably handles most of the the controller processing chores - debouncing, A/D conversion for the pots, forming the data packets, USB communications, etc. As shown in the diagram, all inputs except the bumpers are conveniently accessible via test points that can be easily soldered to. It's also worth mentioning that the controller board used for this project is a wired model marked as year 2007. Earlier or later models may differ depending on any revisions to the controller circuit, although I did inspect a 2010 model which appeared to be identical.

A quick word about the DAC. Our basic requirements are a high number of channels with hardware-synchronized simultaneous output on all channels. Sample rate is a non-factor as pretty much any DAC will support at least 60 samples per second, which is, as far as I know, the highest rate at which any Xbox game processes input. We do however need DC coupled outputs, and this requirement ruined my initial plan of repurposing a cheap sound card DAC. Rather than going for a fully home brew solution, I eventually settled on the National Instruments PCI-6713 which I was able to find used for a reasonable price. This is a PCI card with 12-bit resolution, 8 output channels, a voltage range of ±10 V, current drive capability of ±5 mA per channel, and an easy to use software API.

Buttons

Buttons are the easiest type of input to handle. They are active high on the Xbox controller and can be asserted by applying a voltage of 1.0 V or higher. The addition of a series resistor RS is just a safety precaution to avoid directly driving the ASIC when the controller is turned off and also to avoid driving Vcc if we accidentally press one of the buttons while the DAC is connected. Choosing RS = 2.0 kΩ results in a voltage drop of about 6% of the DAC output voltage VA (suggesting an input impedance of around 30 kΩ for the buttons), so we should not increase RS too far beyond this value.


Sticks and Triggers

The most straightforward approach for dealing with the analog sticks and triggers is to desolder the potentiometers and apply the DAC voltage directly. However, I've found it useful at times to be able to manually operate the sticks and triggers in the usual way without relying on the DAC, so I decided to leave them on the controller. In this configuration we can use a series resistor R2 and an appropriate VA to control the wiper voltage VX. The total pot resistance of 10 kΩ is divided into its two parts, R1a and R1b, which represent the resistance between the corresponding end terminal and the wiper.


From the diagram:


Sticks
  
With the stick in its resting position and without connecting the DAC, I measure a typical voltage of VX = 0.80 V = Vcc / 2. This suggests that the pot has the same range of rotation in either direction, R1a = R1b = 5.0 kΩ. Solving for VX,


Pushing the stick all the way left / up results in VX = 1.60 V = Vcc on the horizontal / vertical pot whereas pushing it all the way right / down results in VX = 0 V. This tells us the range of voltages that we must be able to produce for VX, thereby guiding our choice of R2 and VA. From the equation we can see that a large R2 will lend too small an influence to the DAC output, so let's try R2 = 1.0 kΩ:


The required values of VA are well within the range of the DAC and the current draw is only I2 = ±0.32 mA.

In practice, the voltages required to simulate a full stick deflection are smaller than those listed above. This is because the holes cut in the controller shell limit the stick's range of movement to about 60% of its actual mechanical range, and obviously the system is designed with this factor in mind. Full stick deflection can be achieved with VX = 1.30 V for left / up and VX = 0.30 V for right / down, which, if we keep R2 at 1.0 kΩ, corresponds to VA = 1.50 V and VA = 0.10 V respectively. There is also a 'dead zone' of about ±0.1 V centered around the stick's resting position which must be exceeded to generate any action. The purpose of the dead zone is to mitigate small voltage offsets due to mechanical imperfections in the potentiometers. An illustration of the critical VX values and their relationship to physical movement of the stick is provided below.



Triggers

With the trigger in its resting position and without connecting the DAC, I measure a typical voltage of VX = 1.40 V = 0.875 Vcc. This means that R1a = 1.25 kΩ and R1b = 8.75 kΩ when the trigger is inactive.


Pressing the trigger results in VX = 0.35 V. Thus, the triggers are active low, and it turns out that VX = 1.30 V or lower is sufficient for a trigger press. This can be realized with the following choices of R2 and VA:


Again these voltages are well within the DAC specs, as is the max current draw of I2 = -1.14 / 4.7 k = -0.24 mA. There are a couple subtle reasons for not attempting to pull VX any lower - it's unclear how much extra current (I1) can be sourced from the power supply, and we'd also like to avoid burning out the pot.

Hardware Setup

Nothing fancy - just the controller, DAC breakout board, and a breadboard for the resistors and signal connections. Following an unfortunate incident that involved me snagging my foot on the controller cord, duct tape was gratuitously added.



Software

Before we can do anything interesting we need some basic software for controlling the DAC. The Windows version of the DAC driver (known as NI-DAQmx) has support for .NET, so I decided to work in C#.  I wrote a GUI based program to accommodate the basic controller functions as well as some more advanced features. The main sections of the program can be seen in the screenshot below:
  • Voltage Levels - The DAC output voltages we've already calculated for operating the buttons, sticks, and triggers. Obviously, the sticks can accept a continuous range of voltages, but we can specify the minimum and maximum of that range as well as a default voltage that represents no movement.
  • Channel Map - Sets the mapping of DAC output channels to the various controller inputs. These will change depending on what game we're playing.
  • Controls - Replicates the basic functionality of the Xbox controller and provides access to a list of predefined control sequences (macros). These control sequences can be very short, such as a glitch requiring a couple well-timed button presses, or much longer and more complex sequences involving many controller actions.




Individual button and trigger presses can be implemented by outputting the active voltage for a few frames followed by one sample of the inactive voltage. This is because we're actually simulating pressing and releasing the button, and the DAC latches and holds the final sample in the sequence. Panel controls adorned with arrow images were used as a rudimentary means of simulating the analog sticks. A left mouse click on a panel is translated into a brief stick press (such as when navigating a menu) whereas a right mouse click starts a continuous sequence of voltages, applied to the horizontal and vertical pots, corresponding to the mouse position within the panel (such as when rotating or moving in a game). In all cases the DAC sample rate is set to match the typical Xbox frame rate of 59.94 fps. While this is not absolutely essential, nothing is gained by using a higher sample rate and doing so actually makes things more awkward when dealing with complex sequences.

At this point you might be wondering if our modified controller could be used to play Xbox games with a keyboard and mouse like in PC games. It's possible, but the extra lag introduced between the mouse actions / key presses and the DAC output would be intolerable for the discriminating gamer. What we're left with is basically a dumb record player whose main ability is synthesizing prearranged controller sequences with precise timing and good repeatability. Fortunately there are a variety of games, primarily from the platformer and rhythm genres, in which these two attributes are enough to become an excellent player.

That's all for the first part of this blog. It's long enough that I decided to break it up into two parts. If you haven't fallen asleep yet, give Part 2 a read to see what we can actually do with this contraption.