From BlenderWiki

Project Overview / Progress

Percentages show current progress (July 17, 2011)

Midterm Progress Summary / Future Plan

I've updated the above diagram to reflect my progress. What follows is what is left to be done, and a general plan for the next 5 weeks:

• Footplants - Refactor and optimize code to allow more modular use - the ability to reverse the function, to determine footspeed based on root translation.
• Object Interaction - Change or add a two-way distance constraint - Lerp both bones to be closer/farther from one another in equal measure, by precalculating each bone's path.
• Point Constraint - implement using another bone's space.

Curve Simplication - Needs bugfixing, occasionally fails, and has a strange issue where it places more keyframes towards the start of the animation for no visible reason.

• Path Editing
• Limition Preprocessing
• Animation Stitching
• Displacement Mapping - Just needs a small Operator in the UI to add the correct NLA Layer
• Control Bones - Reverse retarget the rig's control bones from the FK/Deformer bones. This remains an open issue, without a clear approach. The only thing I can think of is to deal with the simpler constraints such as copy location/rotation by determining the constraint's matrix and inverting it.
• Hierachy "Guess" Heuristics

Timeline

July 17-24

Footplants, Displacement Mapping, Guess Heuristics, 2-Way Distance Constraint, Point Constraint improvement

July 24-31

Path editing, Animation Stitching

August 1-7

Fix issues with Preprocessing tools, add limitation, Control Bones

August 7-21

Control Bones, Bugfixing, Docs

Environment Interaction

This is the class of utilities and scripts (mainly constraint based) that can be used to fix a the way an animation (often after retargeting) interacts with its environment – the floor, other objects, etc. IK constraints are used to fix/change the end effectors (hands, footplants). Related to this is path editing, to have the animation follow a user given curve.

Implementation:

• During retargeting, the root bone locations are modified according to the change in gait/scale of the new rig, in such a way that the original footplants are maintained properly without skid. #Retarget
• However, these footplants will not be accurate if we wish to determine new positions for the feet – for path editing or other uses. We use IK constraints on the chain to have the end effector reach the desired position. By using the stiffness values/limit rotation across the chain, determined from the information contained in the original animation, we can insure that the rotations given by the IK constraint maintain the character and realism of the animation. #IK
• Determining the new end effector positions automatically should be resolved by the other components, for example path editing or object interaction. Within the scene, these positions will be encoded in empties.

Preprocessing

An array of semi-related tools to assist in the early stages of the mocap workflow.

• Denoising the animation can be necessary when dealing with the raw sensor data from the mocap.
• Curve simplification is a convenient to make the animation more workable and more alike human keyframe animation, and can also assist with denoising, as it generally smooths out the animation.
• Looping can be a desired transformation of the original motion capture, where we take a long animation that loops and trim it down to a single section that loops in a visually pleasing way.
• Limitation constraints is an idea to analyze the given animation, determine the range that each bone rotates in, and place damping or hard limits in this range, to assist the animator with keyframe animation or the IK that other tools use.

Implementation

• Denoising: I believe that a local, adaptive filter (from the signal processing field) is the best option, probably a median filter. [1]
• Curve simplification: I have written and completed code to convert the sampled data for each bone to keyframes, where keyframes are placed on the same frame for all 3 axises of the bone.
• Looping: Via a script, we perform autocorrelation on the animation to determine the best start and end frame to loop given animation. We then make sure the seam is not noticeable by slighting changing the data in that section, using a basic falloff algorithm (needs testing as to which type of falloff can give the best visual look – Gaussian, square, root, etc.).
• Limitation constraints: A simple script can analyze the given motion and determine the range in which each bone moves, and to set IK stiffness, limit rotation, or dampening accordingly. Also, an option will be given to set these according to a different given animation, as many mocap sessions contain a “range of motion” clip when calibrating the sensors or beginning the session. #IK

Base Retargeting

The ability to copy over the original animation from the mocap “Performer” rig to the end user rig.

Implementation

• A script has been written that performs this base retargeting very well, regardless of changes in the hierarchy, following a user determined mapping. In addition to copying the rotation, we can modify the root location transform to match the changes in the rig – especially for footplants.
• This is performed in the following way:
• We can identify the frames where footplants occur by marking those frames, in the original animation, where the world position of the foot bone does not move. We then look at those frames on the retargeted rig, and determine how much the foot bone moves in that location. From that we can determine the change in scale needed to copy over the root translation correctly, so the footplants are maintained.
• An open problem that I have yet to solve is how to copy the performer animation to a rig controlled by control bones, and not simple FK rigs. A script is needed to backtrack from the FK rotation wanted for a bone/bones and determine the correct transform of the control bone (based on the driver/constraint).

Post Processing

• Displacement mapping is a simple technique to add, via a layered system, small user tweaks to add a small delta to bones, to manually fix the retargeted animation in a non-destructive way.
• Animation stitching is simply combining two or more animations seamlessly to appear one after the other.
• Qualitative is an idea, which I feel is optional to the project, to use techniques from signal processing to change the animation's character in a visual way and change its emotional content. [2]

Implementation

• Displacement mapping, and the above environment interaction components, would be implemented via Blender existing NLA system. For example, adding an “additive” layer to the retargeted action is an elegant way to implement user tweaks across the animation, utilizing the existing UI, with real-time graphical feedback. #NLA
• Animation stitching is similar to the looping algorithm, which could use correlation to determine the best frames to mix (if timing is not critical), and to modify those frames so the end product is visually pleasing, with no bone skipping or feet skid.
• Qualitative changes, if included (dependent on development time available and usefulness), to use a system similar to sound equalizers, to increase or decrease certain frequencies in the animation, that would modify the animation in such a way to increase tempo and character via a “equalizer” interface.

Motion Capture Constraint Proposal / Implementation plan

• After retargeting, a system is needed to correct certain artifacts that arise from the retargeting process. Footskate, interaction with an object are examples of such artifacts. The system I plan on implementing deals with this subject. Via a new panel, users will be able to easily add “Mocap constraints” that target the specific issues. Programmaticly, the system wraps the existing Blender constraint system, using existing bone constraints to implement the system.
• When retargeting, the user can choose to simply create IK chains, thus allowing the constraints to deal with the end-effectors, or bone locations, directly, while the IK solves the rotations of the rest of the chains.
• These constraints, by giving the user slightly different options than offered by the constrait, calculate the correct settings for the Blender Constraint when updated. This is done via the UI and a custom property type (MocapConstraint - a CollectionProperty) which keeps everything in sync.
• The constraints all have a specified frame range, and the influence is keyframed to “ramp” in and out, to ensure a smooth transition from unconstrained poses to constrained ones. Once configured by the users and tweaked, they can be baked to a seperate NLA layer. The custom data is stored, so it can easily be “unbaked” and changed.

The different types:

• Point constraint:
  • Constrain the bone to a specific place, either in global space or local to the armature object.
  • Implemented via the Limit Location of the IK target.
  • Location Vector itself is received from the input of the m_Constraint, and the evaluation space as well.

• Freeze constraint:
  • Constrain the bone to the specific place it was at a certain frame, in global space. Very useful for maintaining footplants.
  • Implemented via the Limit Location of the IK target. Evaluate target is set to world.
  • Vector itself is global location of target at specified frame.

• Floor constraint:
  • Constrain the bone not to pass a certain mesh object. The Blender constraint “Floor” could be used, but it has the disadvantage of only supporting 2D walls, and not more complicated “ground” geometry. Ideally, we could project (raycast) the current position onto the mesh, to determine the correct position.
  • Initially will be implemented via the Floor constraint. It’s possible that I will use a different constraint, utilizing a path, that is recalculated when the user changes something. However, this requires some calculations that might not be trivial in terms of real-time editing.
  • Mesh is passed from mocap ui to the actual constraint.

• Distance constraint:
  • Maintain distance between 2 different bones.
  • Implemented via the distance constraint.

UI Screenshot

Initial Code Snippets

Initial code of MocapConstraint class

class MocapConstraint(bpy.types.PropertyGroup):
    name = bpy.props.StringProperty(name = "Name",
        default = "Mocap Constraint",
        description = "Name of Mocap Constraint",
        update=updateConstraint)
    boneA = bpy.props.StringProperty(name = "Bone",
        default = "",
        description = "Constrained Bone",
        update=updateConstraint)
    boneB = bpy.props.StringProperty(name = "Bone (2)",
        default = "",
        description = "Other Constrained Bone (optional, depends on type)",
        update=updateConstraint)
    s_frame = bpy.props.IntProperty(name = "S",
        default = 1,
        description = "Start frame of constraint",
        update=updateConstraint)
    e_frame = bpy.props.IntProperty(name = "E",
        default = 500,
        description = "End frame of constrain",
        update=updateConstraint)
    targetMesh = bpy.props.StringProperty(name = "Mesh",
        default = "",
        description = "Target of Constraint - Mesh (optional, depends on type)",
        update=updateConstraint)
    active = bpy.props.BoolProperty(name = "Active",
        default = True,
        description = "Constraint is active",
        update=updateConstraint)
    baked = bpy.props.BoolProperty(name = "Baked / Applied",
        default = False,
        description = "Constraint has been baked to NLA layer",
        update=updateConstraint)
    targetFrame = bpy.props.IntProperty(name = "Frame",
        default = 1,
        description = "Target of Constraint - Frame (optional, depends on type)",
        update=updateConstraint)
    targetPoint = bpy.props.FloatVectorProperty(name = "Point", size = 3,
        subtype = "XYZ", default = (0.0, 0.0, 0.0),
        description = "Target of Constraint - Point",
        update=updateConstraint)
    targetSpace = bpy.props.EnumProperty(
        items = [("world", "World Space", "Evaluate target in global space"),
            ("object", "Object space", "Evaluate target in object space"),
            ("boneb", "Other Bone Space", "Evaluate target in specified other bone space")],
        name = "Space",
        description = "In which space should Point type target be evaluated",
        update=updateConstraint)
    type = bpy.props.EnumProperty(name="Type of constraint",
        items = [("point", "Maintain Position", "Bone is at a specific point"),
            ("freeze", "Maintain Position at frame", "Bone does not move from location specified in target frame"),
            ("floor", "Stay above", "Bone does not cross specified mesh object eg floor"),
            ("distance", "Maintain distance", "Target bones maintained specified distance")],
        description = "Type of constraint",
        update=updateConstraint)
    realConstraint = bpy.props.StringProperty()

Operator example

class OBJECT_OT_AddMocapConstraint(bpy.types.Operator):
    bl_idname = "mocap.addconstraint"
    bl_label = "Add constraint to target armature"
 
    def execute(self, context):
        enduser_obj = bpy.context.active_object
        enduser_arm = enduser_obj.data
        newCon = enduser_arm.mocap_constraints.add()
        addNewConstraint(newCon)
        return {"FINISHED"}
 
class OBJECT_OT_RemoveMocapConstraint(bpy.types.Operator):
    bl_idname = "mocap.removeconstraint"
    bl_label = "Removes constraints from target armature"
    constraint = bpy.props.IntProperty()
 
    def execute(self, context):
        enduser_obj = bpy.context.active_object
        enduser_arm = enduser_obj.data
        constraints = enduser_arm.mocap_constraints
        removeConstraint(constraints[self.constraint])
        constraints.remove(self.constraint)
        return {"FINISHED"}

Initial code of constraint module

def addNewConstraint(m_constraint):
    print(m_constraint.boneA)
    obj = bpy.context.active_object
    bones = obj.pose.bones
    bone = bones[m_constraint.boneA]
    if m_constraint.type == "point" or m_constraint.type == "freeze":
        c_type = "LIMIT_LOCATION"
    if m_constraint.type == "distance":
        c_type = "LIMIT_DISTANCE"
    if m_constraint.type == "floor":
        c_type = "FLOOR"
    real_constraint = bone.constraints.new(c_type)
    real_constraint.name = "Mocap constraint " + str(len(bone.constraints))
    m_constraint.realConstraint = real_constraint.name
    setConstraint(constraint,m_constraint)
 
 
def removeConstraint(m_constraint):
    obj = bpy.context.active_object
    bones = obj.pose.bones
    bone = bones[m_constraint.boneA]
    real_constraint = bone.constraints[m.constraint_realConstraint]
    bone.remove(real_constraint)
 
def updateConstraint(self,context):
    print(type(self))
 
# Function that copies all settings from m_constraint to the real Blender constraints
def setConstraint(constraint,m_constraint):
    if m_constraint.type == "point":
        real_constraint.target_space = "WORLD" #temporary for now, just World is supported
        x, y, z = m_constraint.targetPoint
        real_constraint.max_x = x
        real_constraint.max_y = y
        real_constraint.max_z = z
        real_constraint.min_x = x
        real_constraint.min_y = y
        real_constraint.min_z = z
        real_constraint.use_max_x = True
        real_constraint.use_max_y = True
        real_constraint.use_max_z = True
        real_constraint.use_min_x = True
        real_constraint.use_min_y = True
        real_constraint.use_min_z = True

Screenshots, Vidcasts, and References

Example of IK use with stiffness values determined from original motion

Screenshot of currently implemeted UI panel

Use of NLA for Postprocessing Retargeted Animation This vid shows the way I plan to use NLA to allow the user to teak retargeted mocap animation in a non linear, non destructive way, using existing code and features.

Showcasing the base retargeting functionality already coded, allowing to copy animation from one hierarchy to another, via one-to-one mappings, one-to-many, or many-to-one.

A more detailed retargeting demo/tutorial, with annotations.