From BlenderWiki

Creating a custom modifier

Here is a tutorial to make a custom modifier that does nothing but printf its existence! I figured this out by copying the SoftBody modifier and scouring the source code for all references to it. I called my modifier Enja, so just replace any instance of Enja in this post with your own chosen name.

Preparing the files

First we want to create our main modifier code file.

Let's create blender/modifiers/intern/MOD_enja.c (all paths start in the source folder):

cp blender/modifiers/intern/MOD_softbody.c blender/modifiers/intern/MOD_enja.c

In here you will add functionality and of course replace all instances of SoftBody with Enja. Have a look at my modified MOD_enja.c

Modifying the files

The rest of the instructions are just modifying files (if you are confused about where things go, just look at Softbody!

At line 184 of blender/modifiers/intern/MOD_util.c add

    INIT_TYPE(Enja);

At line 70 of blender/modifiers/MOD_modifiertypes.h add

    extern ModifierTypeInfo modifierType_Enja;

At line 564 of: blender/makesrna/RNA_access.h add

    extern StructRNA RNA_EnjaModifier;

At line 303 of: blender/blenkernel/BKE_modifier.h add

    int modifiers_isEnjaEnabled(struct Object *ob);

We need to edit blender/makesdna/DNA_modifier_types.h in a couple places:

• on line 69, just after eModifierType_Screw and before NUM_MODIFIER_TYPES add

    eModifierType_Enja

• on line 441, after the SoftbodyModifierData struct, add

    typedef struct EnjaModifierData {
        ModifierData modifier;
    } EnjaModifierData;

Finally on line 85 of blender/makesrna/intern/rna_modifier.c (before the {0,NULL,0,NULL,NULL} entry:

    {eModifierType_Enja, "ENJA", ICON_MOD_SOFT, "Enja", ""}

Note that I didn't change the icon to ICON_MOD_ENJA because I didn't take the time to figure out making a new icon yet.

To declare the User Interface for our new modifier, we need to edit one last file ../release/scripts/startup/bl_ui/properties_data_modifier.py around line 624

    def ENJA(self, layout, ob, md, wide_ui):
        layout.label(text="Coming soon.")

This should build and give you a modifier under the Simulate header that does nothing but print out a message when you add it.