Performance framework proof-of-concept
Performance is a very important factor for JavaScript games. It becomes
a crucial factor when you bring mobile devices into the mix.
Luckily we have some great tools for testing different methods of coding
such as JSPerf. This lets developers test code snippets
across multiple browsers to compare.
On the same vein, JSGameBench stress tests rendering methods including WebGL, Canvas and DOM.
At Game Closure we want to look deeper and more thorough so I’ve been thinking
about different ways to design a framework that will allow us to run tests
with an assortment of different methods and combinations. With JSPerf one has
to write individual test cases for every combination. Here is my proposal:
Templates
Code to test should have some templates that form the basis of what we’re testing.
It is similar to startup and teardown methods but finer grained.
You may want to move some startup code outside of the template to avoid it being
benchmarked unnecessarily.
Placeholders
This is the key component to the power of this testing framework. Add
placeholders in the template code that will be replaced based on
the combinations being tested. The placeholders are denoted by a
comment followed by a colon and the name of the placeholder.
function renderDOM() {
dom.style.left = /*:x*/;
dom.style.top = /*:y*/;
}
function renderCanvas() {
ctx.drawImage(
/*:type*/,
/*:x*/,
/*:y*/
);
}
This looks like gibberish now but starts to make sense when you create
a list of different values for these placeholders in the form of an
object where the key is the name of the placeholder and the value
is an array of possible combinations to test:
{
x: [
"~~x",
"x",
"Math.floor(x)"
],
y: [
"~~y",
"y",
"Math.floor(y)"
],
type: [
"cnv",
"img"
]
}
The test framework will use all the possible combinations of these values
testing to see which performs the best. You can also specify which functions
to test against. In our case the two functions are “renderDOM” and “renderCanvas”.
This means it will generate 36 combinations of the placeholders and functions.
Test it!
Running a test requires the startup code, a template defined with relevant placeholders
and then an object with the combinations of values. You can optionally specify an
amount of iterations to run (it defaults to 1000).
The above test gives us this:
~~x, ~~y, cnv, renderDOM | 28ms
~~x, ~~y, cnv, renderCanvas | 1222ms
~~x, ~~y, img, renderDOM | 18ms
~~x, ~~y, img, renderCanvas | 8ms
~~x, y, cnv, renderDOM | 33ms
~~x, y, cnv, renderCanvas | 1234ms
~~x, y, img, renderDOM | 33ms
~~x, y, img, renderCanvas | 8ms
~~x, Math.floor(y), cnv, renderDOM | 18ms
~~x, Math.floor(y), cnv, renderCanvas | 1237ms
~~x, Math.floor(y), img, renderDOM | 18ms
~~x, Math.floor(y), img, renderCanvas | 8ms
x, ~~y, cnv, renderDOM | 25ms
x, ~~y, cnv, renderCanvas | 1305ms
x, ~~y, img, renderDOM | 24ms
x, ~~y, img, renderCanvas | 7ms
x, y, cnv, renderDOM | 54ms
x, y, cnv, renderCanvas | 1322ms
x, y, img, renderDOM | 55ms
x, y, img, renderCanvas | 7ms
x, Math.floor(y), cnv, renderDOM | 25ms
x, Math.floor(y), cnv, renderCanvas | 1301ms
x, Math.floor(y), img, renderDOM | 26ms
x, Math.floor(y), img, renderCanvas | 8ms
Math.floor(x), ~~y, cnv, renderDOM | 18ms
Math.floor(x), ~~y, cnv, renderCanvas | 1238ms
Math.floor(x), ~~y, img, renderDOM | 18ms
Math.floor(x), ~~y, img, renderCanvas | 8ms
Math.floor(x), y, cnv, renderDOM | 36ms
Math.floor(x), y, cnv, renderCanvas | 1235ms
Math.floor(x), y, img, renderDOM | 34ms
Math.floor(x), y, img, renderCanvas | 7ms
Math.floor(x), Math.floor(y), cnv, renderDOM | 18ms
Math.floor(x), Math.floor(y), cnv, renderCanvas | 1235ms
Math.floor(x), Math.floor(y), img, renderDOM | 18ms
Math.floor(x), Math.floor(y), img, renderCanvas | 7ms
A few things to note is my method of benchmarking is currently
very naive as this is a proof-of-concept. I will be looking into
more solid forms detailed in this great post.
The other thing to note is some of these tests are essentially duplicates. The
renderDOM method doesn’t use the type placeholder so it generates combinations
that have already been run. This isn’t really a problem and is more to
do with how you define your templates and placeholders.
Analysing the data
Once we have this data we need a way to understand it and derive facts. I imagine a big table with the placeholders, functions and result as the columns and the combinations as the rows. You will be able to filter by placeholder values and sort ascending or descending on the result column.
Browsers
Running these tests against different browsers can be useful to determine cross-browser optimization or things to avoid. It is therefore important to be able to view and compare all the results and how they performed on which browsers. To display this in a table, the execution time column would have to include the result from different browsers. This could be done by adding a column for each browser result and keeping the main result as an average.
These ideas are still very early stage and I have only recently managed to get code executing from placeholders and generating combinations. I will do more research to determine how it can be more useful for existing projects and how to best display the data. Maybe even a GUI along the lines of JSPerf could make it more helpful.