From BlenderWiki

Save converted BGE data out to a file

Introduction

Whenever a scene is added at the blender game engine, it must be converted. By now this is done at runtime and each time the scene is added. This requires time, and thus with big scenes it can slower the start up of the game(both from player and viewport). My aim is to allow BGE to load converted scenes from file. This will bring several advantages: 1)speed up in loading scenes 2)protect the scene data (this file won't be readable directly by blender) The converted scene export should be done automatically(giving an option to the user to disable or enable) or with a function in the UI (like save as runtime). No need to add a python function in the api, since the check for the converted file will be always be done automatically at loading/adding of the scene.

Implementation details

With this design i tried to minimize the read and writes to disk, in addition the classes will write all the data at once(sequential write is faster)

Since DNA can be used only by editors, the bge will need its own part that defines how to save/load files. I think of creating a new KX_KetsjiFileConverter.

This file will contain all the classes used to convert bge data to use in files. Since we can't make a dump of the memory of the class KX_Scene, i think of creating a struct that contains all the data to store. This struct will be used as an intermediate between the file and the actual KX_Scene. The class will be something like:

KX_KetsjiFileConverter.h

class KX_SceneInterface {
 
 typedef struct {
        RAS_BucketManager*	        m_bucketmanager;
	CListValue*			m_tempObjectList;
	CListValue*	                m_euthanasyobjects;
 
	CListValue*			m_objectlist;
	CListValue*			m_parentlist;
	CListValue*			m_lightlist;
	CListValue*			m_inactivelist;
 
	SG_QList			m_sghead;
        [... etc,etc,etc....]
 } KX_SceneFileInterface;
 
 protected:
        char* m_filepath;
        KX_Scene m_sceneToConvert;
        KX_SceneFileInterface m_currentConvertedScene;
 
 public:
        KX_SceneInterface(class KX_Scene);
        void setFilePath(char* filepath);
        void KX_SceneInterface::FromSceneToStruct()
        void KX_SceneInterface::FromStructToScene()
        void KX_SceneInterface::SaveScene() //maybe can return an error code, or can be created a variable with the purpose of storing errors in the class.
        void KX_Scene KX_SceneInterface::LoadScene()
 }

KX_KetsjiFileConverter.cpp

 KX_SceneInterface::KX_SceneInterface(KX_Scene* passedScene) : m_sceneToConvert(*passedScene) {}
 
 void KX_SceneInterface::setFilePath(char* filepath){
      m_filepath = filepath;
 }
 void KX_SceneInterface::FromSceneToStruct(){
 /*Here the code that will take the data from the scene and put it in the  struct. In this part there will be the instantiation of the other converting classes(eg KX_GameObjectInterface, that will call RAS_MeshObject, that will call....) that will convert all the data in structs easy to save.*/
 }
 void KX_SceneInterface::FromStructToScene(){
 /* This function will do the opposite of FromSceneToStruct. Once the scene struct is created, this will create all the needed instences of the classes, and use the data in the structs to set them properly.*/
 }
 void KX_SceneInterface::SaveScene(){
  /*This class will call FromSceneToStruct().
  Than, as done in blender DNA, create a file, with an header for the blender  version, and header for the data block, and also store the struct, so that in  future versions blender will know how to read the data block.
After that, it can do a dump of the struct in the file.*/
 }
 void KX_Scene KX_SceneInterface::LoadScene(){
  /*This class will read the file and load the data. It must be able to read even files from older versions, using the struct bundled with the file. Once it knows how to read the data, it can "populate" the various structs needed for FromStructToScene to create the actual KX_Scene*/
 }
 }

The file structure will be similar to the blend one. The file header will be identical to the blender one:

The endianess switch code can be copied from dna. The file will be divided in file-block-header and file-block, that will contain file-data-header and file-data for each data to save.

When a scene is converted, the file struct will be something like

BGE file header
 Scene1 Header
  Scene1 struct
 GameObject1 header
  GameObject1 struct
 RAS poly1 header
  [...all the stuff recursively inside the object..]
 Scene2 Header
  Scene2 struct
 Game object...

Each time a a new struct have to be stored, it check in the already saved file for headers with the same old memory address, and if one is found, the struct doesn't get saved(since it is already in the file). On load time, when inside a struct is found an old pointer, all the file is searched for an header with that value(it should be fast since the headers contain the data dimension, so we can jump from an header to another)

Still to be defined

Current working blend

I currently don't know how to find what blend file is currently used. Since the bge should output a converted file of that blend, i need to know what and where the blend is(and if it exist. For example if i open a new scene and run the ge, the file is not saved jet..)

Saving method

I think is better to add an option in the ui for defining a location and a button for converting all the scenes inside the current file, rather than converting on runtime (because it can happens that someone run the bge, is sure to have used all the scene, but one left out, and than when he will distribute only the converted one, the game won't work.) And this may eliminate special cases that would need to be handled.

Pointers to pointers

Since i think of using old pointers as unique identifiers for the structs, it a struct contains a pointer to pointer(an array of pointer) i need to store all this pointers. Should i create a special struct for this? If so, how much pointers should i store?(since in struct i need to define exactly how much to store)

Single class instance for multiple objects vs. multiple class instances for single objects

Right now the design expects that a class is create for each object to convert. Should instead be created a class for each object type, and than this class converts all the object of that type?

Possible References

In addition to Blender, there are other projects that use this system to save data(i think all of them were inspired by blender):

1)Bullet

2)Gamekit

Since they are written in C++, i think it can be useful to look at their source code. Gamekit should also have a generalized system ready to be reused. I think it is worth a look.