PropellerEngine postmortem

What is the PropellerEngine?

PropellerEngine is a cross-platform game engine written in C++. It has an experimental integrated editor written in C#. I started the project in January of 2014, over the next years I developed the engine by myself.

The project has always been educational, so instead of aiming to certain set of features I kept implementing and testing features that I would like to know how to implement.

Source code

Source code for the PropellerEngine has now been released under BSD license. You can find the code on my Github repository.

Please note that this code does not contain the full Git history, or any code that has been licensed by Sony, so do not expect to find any PS4 related code in there.

The release is as-is, without extra documentation added for possible future developers. I might add some if there is interest for using this engine.

Original target

The original plan was not to make an engine, but to make a 2D game. The game was supposed to be top down shooter. (Inspiration was from “Hotline: Miami” and “Tapan Kaikki”) I had created similar types of prototypes for this kind of games before, so technically the original scope should have been really easy.

Because I had made the same kind of games multiple times, I was sick of the basic things I had to do every time, this time I decided I could make a “framework”, so that making the next game would be faster. Little did I know that this would open the endless hole of engine development…

Before this project I have tried to make game engines couple of times, but all of them crashed due to too large scope or not enough experience.

Development process

As the project was an educational one, I was learning while I was implementing the features. Because of this I had to do some refactoring to keep things under control. Fortunately I was able to finish up almost all features without refactoring them.

During this project I ended up with almost 2000 commits. I went through the commit log to see the full history of the project.

Start of the project

January 2014 I started the project. During the first month of the development was fast and I was able to test out a lot of small features.

After the first month I already had a lot of base systems up and running for the engine.

  • Basic scene system
  • ResourceManager
  • Input system using signals
  • CVar/Console system reused form previous project
  • Entity component system

There were some signs already of me starting to go away from the “small framework” idea, to a full game engine. There was simple implementation of the C#-editor already in place. At this point it was mostly planned to be a level-editor, not a fully integrated editor.

Graphics were done with SFML, as I was not planning to implement a custom renderer at all during this point. If I knew that I would not have used SFML: I would not have included it at all at this point of the project. All features that I had implemented using SFML had to be remade. (Input, 2D-graphics, etc)

There were some plans to have a custom text format for creating entities from them, later in the process the entities were saved in binary format.

Base system implementation

During the second month I heavily improved the ResourceManager and implemented a cross-platform filesystem support. This was mostly the final version, as no more major changes were made to the ResourceManager/Filesystem.

First base system for the serialization started to take shape. There were some factories to create objects and custom serialization data implemented into the objects.

C# to C++ interoperability was heavily improved, mostly complete. The editor was able to receive a list of game-entities on the scene.

(Note that the plan was for future weeks, not containing the first two weeks)

I had planned the following 12 weeks of work, during this time I would have finished up the framework and created a little prototype for a game.

Completely losing the original focus

Third month was the point where I seemed to have completely given up with the original idea of making a game and small framework along with it. I was aiming for a full engine. The editor was getting a lot of nice features, the entity-component system got a lot of attention and the week planning shown on the above screenshot was removed from the repository.

I was doomed to finish the original plan, but this new direction was much more exciting and interesting. In the end this was the right path for this project.

Interestingly I still had the GameInstance running under the game, not under the engine. It was one of the few remains when I was still working on a framework, not on a full game-engine. The game project still had more control over the runtime than the engine project.

Editor progress

During the following months I had great progress with the integrated editor. I implemented RCC++, full undo support, multiple engine instances and many more features to the editor. dynamic_cast usage was also changed in most places to use my own RTTI system.

Most of the features can be seen on a video I published around the time.

By June of 2014 I had started my first cross-platform experiments. The engine was successfully compiled for Linux.

SFML removed & Android added

By September of 2014, shortly after tagging the version “0.9” I started two big changes. Remove SFML and add Android support. Removal of SFML was going to happen anyway, but for Android-support it was necessary right away.

PropellerEngine was added under Travis at this time. It compiled the Linux versions, so when I coded on Windows I could easily see when I accidently broke some cross-platform related stuff. I should have added the project into CI earlier to the project. Later to the project I also used Jenkins.

CI is really helpful when working on project with multiple different platforms. The CI was running on Linux, so I was able to see when the Linux builds were failing without manually building them. When Linux builds fail, almost always most of the platforms failed. Now, I would try to extend the CI to compile all the platforms that were targets, not only one of them.

The engine was developed for over 1 month without a renderer at all. By this time I ported a lot of stuff to Android, implemented a new input support on Windows, etc. (As I no longer had input from SFML.)

About one month into the D3D11 support, I changed the scope of the project again. I decided to also support 3D on top of the 2D. Part of this decision was me wanting to do 3D-graphics, part of it was that I had already solved problems with 2D-graphics and wanted more challenge.

Conclusion for the year 2014

November of the 2014 was most likely the most productive month for the whole project. Even if I didn’t have as many commits as during the start of the project, I had more features implemented.

During this time I implemented my own 3D-math classes. I got crossplatform 3D-rendering working on both Android and Windows, using GLES2 and D3D11.

Lots of small features were also implemented. First version of a scoped profiler was introduced, it was later improved and released as a ProfilerPropeller. HTML5 version was also released using the Emscripten. (Works best on Firefox)

For 3D-graphics I implemented my own simple 3D-mesh format. (.PropellerModel) User would import a model using the editor, the engine itself would use the internal format only. This simplified the model loading on the engine side, as the more time consuming converting process was happening in the editor.

All thirdparty dependencies were moved into the repository itself, so now I was able to compile the project without any issues.

Here is a screenshot from a version that was 1 year old.

Note the integrated profiler in the editor. This was later replaced with the PropellerProfiler.

Second year of development

By 2015 I had started this blog so most of the development process can be seen on my previous blog posts, some of the most interesting ones are listed here:

To summarize the year of 2015. I started the year by working a lot on the graphics features, as well as implementing physics system using Box2D and Bullet physics. (Also some experimenting with PhysX).

Closer to the summer I started to port the engine to PS4. At this point I decided to completely rewrite the rendering backend in order to make the PS4 version working. During summer I took a break from the project again. After the summer I started the work on the new rendering backend. The D3D11 support was implemented in a couple of months.

Once I was done porting the engine to PS4 and finished the D3D11 rendering the least interesting 3 months of the project started. Fixing all the small bugs across all the features.

What could have been improved

For a project of this scale and size, it is not unusual to have some issues. Fortunately most of these issues were minor and under control. With some more development time most of these could be fixed.

There were some features in the project that could have been improved. Usually these features had to be cut because of lack of time or to make room for other features to be implemented and bug fixes.

RCC++

While runtime compiled C++ worked really well, keeping it working had some issues. Big changes to the low level engine interfaces usually broke this feature. Any changes to the build configuration could also break the system. Also, the serialization had to have special care to make sure the state is not messed up during the reload.

RCC++ was heavily tied to the editor interface. This meant that if the editor wasn’t working, or there were big changes to editor, RCC++ would break.

If I would now go and implement RCC++, I would probably move it away from the editor into the base game itself. Instead, making the engine more modular would make this feature even more effective.

Editor

While the editor itself was a great success, it was also one of the most “work-in-progress” features.

Having an integrated editor is one of those parts of an engine you have to almost completely finish before it starts to be usable. For example I could add couple of small features into the rendering part of the engine and they would not feel “incomplete”, but with the editor even with multiple big features, it was far from a production level editor.

In the end this was not a technical failure, but rather lack of time to finish it up to the state I would like it to be.

Build configuration

While the automated build system worked really nicely for building and releasing the engine, the build configuration lacked a lot. I targeted PS4, Android and Windows from VS, while other platforms were targeted using premake4. For Emscripten the VS tools were old and using Emscripten with premake4 turned out to be inefficient.

When targeting multiple platforms with multiple different toolsets, a custom build system is usually going to be really useful in the long run.

Hacks…

During the end of the development cycle, deadlines and lack of time caused some drop on the quality of the code, some hacks were introduced to the engine. Fortunately these hacks were mostly minor and could be fixed in couple of a days. The scope of the project was a perhaps a little bit too large for one person to handle perfectly.

Features that worked well

Cross-platform support

Cross platform support is one of the best features in the PropellerEngine. The engine has been run successfully on the following platforms. (Some have had full rendering support, but headless support)

  • Windows (DX11, full support)
  • Linux (Headless server, GL was planned)
  • Mac OS (Headless)
  • Android (GL, full support)
  • PS4 (GNM, graphics in progress)
  • HTML5 (GL, full support, some optimization issues)

Windows has been the primary development platform, so naturally that platform had the most support. Android was the second platform that had fully working support and feature parity with the Windows version.

The Linux version is really solid, but it got never got full graphics. This was not a problem because I used Linux on my continuous integration systems. However, implementing graphics for Linux would be trivial, as there was already a working OpenGL backend written for Android.

Mac version is mostly identical to Linux, it has mostly the same systems running, just some headers changed. OpenGL would also be trivial to implement for that platform.

PS4 is the most interesting platform to port the engine into. It was also the most pleasant platform to develop for. The first rendering backend had some design issues that made it difficult to get the rendering to work on this platform. Unfortunately after the rendering refactor, I haven’t had time to get the rendering back to working on PS4.

The biggest issue with the HTML5 version was not the coding/porting, but the tools. When using Emscripten, the code behaves like it would be running on Linux, with GLES rendering. This made the task easier for me, as I already had working Linux version and rendering running on GLES.

Visual studio of Emscripten required VS2010, that I stopped using a while ago. Building stuff from commandline with Emscripten is easy, but my command line builds were done with premake4, a system which was not easy to build Emscripten support for.

For a project of this scale I would really consider writing my own build systems. If you want to target multiple different platforms, it is rare that you can find a tool which supports building to all of them.

Rendering system

Second version of the rendering backend was really successful. Big change from the first version was to separate the low-level rendering and the high-level rendering. This allowed me to work on the rendering features without touching the API specific code most of the time.

Shader cross-compiling is also something that worked out really well. Instead of rewriting all the shaders, I just automatically compile the .hlsl shaders into .glsl shaders.

Looking back, I should have perhaps cut Android out from the project completely and focused more on the 3D-graphics.

Integrated editor

While the integrated editor was never fully finished, it was still one of the big achievements in the PropellerEngine. It had a lot of features that would be expected from a full integrated editor.

Multiple instances of engine running under the same process for example is one of those features that is not so easy to implement, but is not that visible for the user.

Undo-system is something that people expect from an editor. The implementation that I used for undo-system creates a list of all operations made for the active scene. All operations always have an invert-operation. This way, if you add an entity to the scene, the undo-list is going to get operation of removing that said entity.

Play-in-editor is a highlighted feature of the modern game engines like Unity and UE4. I was able to also get this feature to work nicely.

Serialization

Serialization was enabling me to create a lot of other features in the system. All of the following features used serialization one way or another:

  • Save / Load game (quick saving!)
  • Save / Load scene in editor
  • RCC++
  • Editor
    • Entity list
    • Component lists
  • Experimental networking
  • Experimental LUA support

Automated release

During the final parts of the development, automating the release builds saved a lot of time and helped me focus on the code and not releasing things.

I wrote about the system more in depth on my previous post.

Summary

In the end, one full game was successfully created with the PropellerEngine by a team of other students. I was providing engine support, and did not work directly on the game.

Screenshot of the Foam Raider: Arena, running under the PropellerEngine

The PropellerEngine has been a wonderful project for me, I learned a lot during the development. If I could now decide other use for the time I spend on the project, I would not change it.

Posted in C++, PropellerEngine | Tagged , , | Leave a comment

Rendering refactor finished

I have now finished PropellerEngine’s renderer refactoring. The rendering is now ported from DX11 to GL2. DX11 has been the primary rendering backend for the engine for a while now. GL2 is done because the engine also targets Android, making it possible to run it on as many devices as possible.

Before the refactor the rendering supported GL2, GNM and DX11, I completely rewrote the rendering during the summer to DX11 and after that started the porting process to GL2. Most of the porting went without a hitch.

There are couple of big differences to the previous rendering engine after this rewrite.

Better API design

The API is much better in design and can support multithreading. It is easier to maintain different shaders and pipelines effectively. Low and high-level renderers are more separated, making supporting multiple backends easier.

Context-recreation support

The first iteration of the rendering engine didn’t support context recreation. On PC that is not a problem, as many games even still request user to restart the game in order to change graphics options. However, on mobile it’s important to be able to recreate the context at any given time.

If one of the API’s has to support the context-recreation, it’s easier to just enable it for all the API’s. Now all the API’s support recreation on demand.

Shader cross-compiling

On the previous project I had a lot of duplicated work on the shaders. All shaders had to be written 3 times. As hlsl, pssl and glsl. This was usually a boring work to get the graphics features to work on all platforms.

Now I have HLSLCrossCompiler in use to compile hlsl-shaders to glsl. It reads the cso-bytecode and builds glsl-shaders from that.

On mobile targets I came up with a problem in the HLSLCrossCompiler. It was using GLSL extension that was not supported by the mobile devices. Fortunately, it was easy to hack the compiler to remove this extension from use.

Now, all shaders are written once as HLSL, this makes working with graphics features a lot easier.

Posted in C++, PropellerEngine, Rendering | Tagged , , , , , , , | Leave a comment

PropellerEngine 1.0 and some automation

When I started the PropellerEngine project, I decided that when I mark the engine 1.0, it would be possible to make games with it. That day has finally arrived. One game team is now using the PropellerEngine to develop a game.

So far the engine has been in development for little over 1.5 years. It’s refreshing to see some results for the work.

Automating building and releasing the engine

Now that there is users for the engine, I had to make sure updating and patching is as smooth as possible.

I did not want to release the whole source for the engine. There is couple of reasons for that:

  • To prevent modifications to engine that could break future patches
  • There is some code that’s under Sony NDA (PS4..) (some still in the headers, so NDA’s are necessary anyway…)
  • Shorter build times for the user

Currently the engine and the thirdparty libraries are build into .lib files. Relevant files are copied out of the repository. Then all irrelevant files (engine .cpp, etc.) are removed from the package. For a patch release the game/ and data/ folder is also removed, to prevent overwriting users changes for the game.

In the end the release procedure is fairly simple and doesn’t take a lot. Its usually couple of minutes of compiling, copying files over and creating a zip-archive. Perfect place to apply some automation to prevent human errors!

PropellerEngine's release scripts output

PropellerEngine’s release scripts output

publishEngine.bat automates everything behind one command, it does the following tasks:

  • Remove old release versions/temporary directories
  • Rebuild all projects trough msbuild
  • Copy all relevant files to propellerengine_release/ (making a clean copy of the repository tree)
  • Remove files that should not be released (engine/*.cpp, etc)
  • Add hostname and timestamp to timestamp.txt file
  • Copy the propellerengine_release/ to propellerengine_patch/
  • Remove data/ src/game/ from the patch version
  • Zip both directories
  • Print time spent working on the build and wait for input
Posted in PropellerEngine | Tagged , | 1 Comment

Resuming devblogging

Its been about 3 months since my last blog post. I haven’t had any major progress in that time anyway..

Here are some progress updates / project ideas from the summer:

PropellerEngine

For the PropellerEngine I have done more planning than actual coding. On the coding side I have mostly just fixed some minor issues there and here. For example the multithreading/scheduler stuff now works on Linux and the 2D-light effects are actually working. Some sprite sorting issues are also fixed.

Future for the PropellerEngine is still open. I have worked with this projects for a bit over 1.5 years and currently its not even close to a finish. Currently I am considering dropping the 3D stuff from the engine and simplifying the editor a lot. This way I could get a solid 2D-engine and actually release it. This is still just a plan, so time will show what path I will take with the engine.

DX12

If I remove the 3D-stuff from the PropellerEngine, I would lose my “3D-playground”. For that purpose I have been planning to create some kind of graphics-demo for DX12. (+ any other API that I happen to port the demo)

Windows Phone development

During the summer I switched from Android to Windows phone. The decision was simple: finding a phone under 100€ with basic communication app support (WhatsApp, etc..). Windows phone is just that, with much smoother interface than Android.

By getting the new phone, of course I had to test how could I develop some applications for it. After downloading the dev tools, compiling applications to WinPhone is simple. Especially on the native side, WinPhone’s development tools are far superior when comparing to Android side.

I have some plans for a simple application, no idea if I have the time to actually implement it…

 

Posted in PropellerEngine | Tagged , , | Leave a comment

New RendererSystem and multithreading

New RendererSystem

The whole RendererSystem has now been written from scratch, with actually some sense in the design. Instead of just blindly wrapping commands and structures, I have created interface that is easier to use and is easier to maintain in the future.

The new API was supposed to support more API’s easily, but I came up with some problems while designing the API. The wrapper is designed based on modern low-level API’s, like D3D12, including stuff like multithreading, etc.

With DX11 I could use the commandlists to batch up the commands/draw calls in different threads, multithreading was not a problem here. However it seems like OpenGL doesn’t support this without thirdparty extensions. OpenGL versions of the engine were mainly running on low-end platforms (Android/WebGL), that have really ancient versions of OpenGL available.

I may implement my own emulation of this feature to support devices with old OpenGL versions, but first I will focus on the wrapper to get it working under one API correctly. Or maybe the Vulkan comes out before that and  I can just write support for it.

Scheduler

PropellerEngine is now multithreaded! The design ideas for the interface/implementations are from the Parallelizing the Naughty Dog Engine Using Fibers-GDC speak. The idea is to split everything in the engine into a small jobs that are then run by worker threads, the jobs are run so that they are not going to run into memory-access issues. For example gameplay related jobs should be done before the rendering jobs are started.

The interface still has some issues and the code example has clear problems in it, those are going to be fixed in the near future, when the overall design of the scheduler-system progresses.

I jobified some loops and on average I gained 25%-50% performance boost compared to the original singlethreaded tasks. With some optimization I could get even more performance out from this.

Multithreaded profiler

As I am moving the engine to be multithreaded, many things failed at first and the profiler is one of them. I designed the profiler at first to work only on single thread, but that didn’t work at all in the multithreaded context.

It was simple to fix this issue, by moving the profiler to thread local storage and keeping a global list of all the profilers. This way the profilers collected information by thread and I could export the data easily. Its still going to generate per thread reports, instead of merging the results, but for now thats good enough for my profiler needs.

Posted in C++, PropellerEngine, Rendering | Tagged , , , , , | Leave a comment

Future plans for the PropellerEngine

Its been couple of weeks since the last update on the engine project. Here are the major improvements since the last post.

High-precision timers & deltatime

Deltatime support is something that has always been in the engine, the timer just has never been implemented and the value provided as deltatime was constant.

Now I have implemented the per-platform high-precision timers and I am providing the correct deltatimings to all the subsystems requiring it. It was surprising how easy it was to get it to work. I had not tested the deltatime before.

PropellerProfiler

The engine has had a profiler for a while now, but until now it wasn’t usable. Without the high-precision timers the profiler is useless and now that I implemented the timers, the profiler became useful.

The profiler is implemented using scope based classes that start timing on construction and stop the timer during destruction of the object. The profilerManager collects the timings and creates a tree out of the functions.

Usage of the profiler couldn’t really be easier. To profile a scope/function, all the developer needs to do is to add “PROFILE;” to the beginning of the scope.

ProfiledPropeller

Screenshot of the PropellerProfiler Analyzer

PS4 version

PS4 version is up and running! (kinda)

I have managed to port most of the engine to the PS4, there are still some issues, but issues are known and will be resolved later.

PropellerEngine on PS4, first renderings

PropellerEngine on PS4, first versions

The issues with the graphics are fairly simple, however the fix is a bit longer. I managed to code simple working renderings of textures for the PS4 already, so I know mostly where the issues are.

Problems came up with my graphics wrapper implementation that now wraps D3D11, GLES2 and GNM.

Future plans

So, the engine is currently in pretty good state. The core systems are stable and free from hacks, it runs on multiple platforms, etc. However, to continue supporting all platforms I get my hands on, I need to do major changes to the renderer wrapper.

As I started the engine, my target graphics-API’s were D3D11 and GLES2. While they have lots of differences, it was fairly easy to wrap those under the same wrapper. However, now that I added GNM, the issues with the design of the wrapper came up. Fixing these issues is highly relevant if I ever want to target more API’s than the two core API’s.

Currently the wrapper is designed so that I wrap commands and ‘objects’. Buffers, Shaders, etc have their own classes. GraphicsDevice has set of commands that are wrapper for each API. So when I have “drawIndexed” on my wrapper, it calls “glDrawElements” and “device->drawIndexed”. So basically just wrap the commands for each API.

This works for API’s that are similar to each other. D3D and OpenGL, while different API’s, still share a lot of common in terms of what to call and when. With low-level API’s I am not able to implement them without hacks to this kind of wrapper.

I am going to plan and recreate the wrapper during the following months. I am going to wrap “concepts” instead of “commands”. This way I would have things like “Pipeline”, etc. User of the wrapper would just set state of pipelines and devices, while the wrapper does all the API-specific stuff correctly.

Posted in C++, PropellerEngine, Rendering | Tagged , , , | 1 Comment

Rendering fixes and more platforms

Earlier this week I looked at the old rendering and compared it to other hellknights rendered by different engines. My hellknight was way too brown. After some analyzing, I realized that the whole shading model was invalid.

Now the models look as they should and lights works as expected. Lights also now have a range, only illuminating objects that are close enough.

Fixed the shading model. Two hellknights and a pointlight.

Fixed the shading model. Two hellknights and a pointlight.

More platforms

Lately I got access to a PS4 dev kit and I started to port the engine to it. Currently the engine runs on the platform, but without rendering. With this addition I currently have six target platforms. (Win, Linux, Mac, PS4, Android, HTML5) If everything goes as planned, I should get the rendering working in the near future.

Posted in C++, PropellerEngine, Rendering | Tagged , , | Leave a comment

PropellerEditor: Play in editor

PropellerEditor now supports Play-in-editor. To be more clear, the feature has available for a long time, but now its possible to pause and stop the engine. Its now possible to modify scenes and test out the changes easily, demonstrated by the following video.

Other changes since the last blog post includes custom allocators for the RTTI objects. (Some debug messages can be seen in the previous video!) The custom allocator makes less system calls by providing a memory pool.

Posted in C++, PropellerEngine | Tagged , , | Leave a comment

Closing to engine 1.0 release

After over one year, the engine is closing its 1.0 release. When I started the project, I decided that with the version 1.0, you could actually make a game. There is still a lot of stuff to develop, but making a 2D game with the engine is possible right now, with few minor issues.

Major features where the engine is currently lacking is the audio and the editor. Editor should be ready for 2D stuff later this year and audio should be done in during the following month. However, the 3D stuff is going to stay in the engine and I will continue to develop that along with the 2D gameplay systems.

HTML5 version

One of the major features I got up and running is the HTML5 version, its still work in progress, but so far it features 2D-rendering and input. The HTML5 build is created from the same code thats used to build the Android and Windows versions of the game/engine.

You can test the first playable HTML5 build here. (Some performance issues with Chrome, frame rates should improve to future releases)

Android version

Android version received major improvements during the last weeks. Framerate issues related to input are now fixed and the engine runs smoothly on modern hardware.

Some work is also done to support the joystick/button inputs on the Tegra Shield device.

Posted in C++, PropellerEngine | Tagged , , , , , | Leave a comment

Rendering update and physics

Its been a while since my last update on the engine project. In short I have updated the 3D mesh rendering and implemented working 2D physics.

Hellknight_rendererd_with_propeller

Hellknight rendered with the PropellerEngine

The rendering has been reworked in multiple ways during the last couple of weeks:

  • Better wrapping of the graphics API’s
  • Normal/Specular/Diffuse mapping for meshes
  • Brushes have been removed, I may add them later
  • Loading of the resources for components refactored
Box2D physics running inside the PropellerEditor

Box2D physics running inside the PropellerEditor

Box2D physics have been mostly integrated with the engine now. The system can automatically fetch the size of the physics object from the current sprite and use that. It also supports custom parameters for objects. (friction, density, restitution..) Applying impulses is also possible.

The physics still need some work. For example, I need to add support for other shapes than boxes. Some way to detect/notify other systems when specific objects hit each other would be nice.

3D-physics are still work in progress, but should be implemented in the near future.

Tiled map imported into the PropellerEngine

Tiled map imported into the PropellerEngine

One minor feature that I also implemented was the Tiled importer. Its still a bit work in progress. (see how it imports the map upside down.) It maps different tile types into EntityTypes, making it possible to spawn different actors to the scene just using the Tiled-editor.

This is just a temporary solution. I have planned to include grid-snapping and other features, making it possible to edit the scene like it was a tilemap.

Posted in C++, PropellerEngine, Rendering | 1 Comment