Doc:Tutorials/Animation/Advanced/MoCap/1:Life-Cycle

From BlenderWiki

This section presents the life-cycle process for Motion Capture. A process is described which encompasses the creation and use of Blender Actions, Poses, and IK Targets. The start of the process is figuring out what motions to capture. While not a present capability, code is under consideration that would enable Blender to stream in MoCap data real-time. Also, Project Petunia is using Blender, and the video producer is doing his own MoCap. Therefore, while not done now specifically using Blender, the steps are very relevant to successfully using Blender with MoCap. As a typical Blender user, your entry into the process is probably in the middle, where you want to find MoCap files that you can use for your project. The last step of the process provides a feedback loop that can drive further research and capture, which results in long-term expansion and improvement in the overall product.

Terms

Action: A series of bone movements that moves an Armature from one pose to another

Armature: A series of bones that represent the skeleton of a figure, and is used to deform the mesh skin.

Pose: A particular orientation of an Armature

IK Target: an Inverse Kinematic location used by Blender to point a bone to a specific spot in space

Problem Domain: This tutorial focuses on the problem domain of isolating primitive actions from a Motion Capture datastream, and the subsequent integration and re-use process in order to improve both product quality and reduce video/film development time. This whole issue (big picture) is using Blender to visualize some real-world motion. The larger conceptual topic is also known as Geo-Spatial Coordinate Real-Time Data-streams. We are applying that data to produce a localized, specific implementation of a computer graphics visualization and device called an Armature. The Armature can then be used to deform a mesh object which resembles a human, called an Avatar. The concepts presented in this paper are applicable to other datastreams, domains and tool-sets.

[edit] CG 3D Animation Life-Cycle

Animation of the Armature is the main focus of the majority of the body of MoCap data available, although recent MoCap studies have placed markers on actor's faces, enabling the capture of facial deformations during dialog. In Blender, this has been the domain of shape keys, and while this area has not been fully explored, Blender provides hooks that can be used to deform a mesh. A "rig" refers to all the bones, drivers, representations, custom shapes, and reference data used to make an armature move. Depending on marker placement during MoCap, some or all of this information is available to the animator. If markers are placed on a human's sternum and back, breathing action is easily discerned. If facial marker are present, the artist can use that data to guide facial deformation during phoneme formulation.

Exploring the creative process as it relates to animation, we see two major efforts emerge: Character Development and Animation. Development refers to creating the character and their attitude and role, and how that affects their motion. A happy robot and a killer robot are each robots, but they move very differently. A happy robot will walk upright with arms swinging wide, whereas a killer robot may walk in a hunch and menacing manner with arms tucked in. Animation is where an animator makes the rig move in a manner consistent with its character as called for in the screenplay and indicated in the storyboard. The two come together when the animated rig is skinned and rendered as called for by the shots in the shooting script. This tutorial focuses no Character Animation.

When animation occurs depends on the kind of video that is being produced, and is driven by a shooting script. A shooting script re-arranges the screenplay into economical shots filmed on location to minimize travel, logistics and works around scheduling conflicts. It tries to shoot the film in-sequence, so that the actors and animators can develop their character and have a sense of progression. In Pure CG, all the shots are virtual, so it becomes a matter of building up skills and shooting the simple scenes first. In Blended CG, much revolves around the real-world shooting schedule, so that the CG and the real-world film are ready at the same time so that CG lighting and compositing can begin as soon as possible. While MoCap has been performed on horses and possibly other creatures, this paper focuses on the use of human MoCap to create/animate human characters.

[edit] MoCap Life-Cycle

The result of this exercise is the rendering of CG animation featuring realistic creature movement. The film or video is the end product (artifact) of a very complicated process involving people spread out across space and time:

1. Research Field: First, someone has to come up with ideas on subjects and motions that can be captured. For example, two people shaking hands is a complex interaction, both psychologically and mechanically,
2. Capture Motion: Second, a number of recording sessions have to be conducted, using available actors.
3. Digitize Session: Third, the images from the camera have to be digitized into a datastream of 3D coordinates, with channels of each stream assigned to each marker.
4. Import MoCap: Using this data file, the user imports the data into the animation package.
5. Constrain Rig: An armature or skeleton is defined which is constrained to follow the motions of the markers, thus transferring the motion to bones.
6. Bake Armature: However, the actions and motions of the bones needs to be isolated for future scaleable and quantizeable use, through a process called baking.
7. Isolate Actions: Once baked, the data stream and empty cloud is no longer needed, as the bones now “know” how to move synchronously to achieve the overall motion. The animator can now isolate each individual action from the complex motion that was captured.
8. Integrate Actions: The animator can assemble these action components into new motions that are called for in the script.

In developing a Life-cycle, it is important to include a feedback mechanism so that the end result can be used to improve or help drive the next life-cycle iteration. This link is provided by the connection of Integrate Actions to Research Field. For example, in trying integrate actions from the library, the artist may discover that some action is not there, and that there is no MoCap sessions which cover the topic. The artist would then submit a request to the person performing field research, and thus closes the loop and ensures that "demand drives supply".

[edit] Research Field

First, a professor or director must come up with the idea or topics to be captured. He or she must select those action which have a high probability of utility or use in a final animation, or are of some scientific or artistic value. As an additional consideration, they must be able to capture the motion accurately using the equipment available. Also, the motions must be realistically achievable by the actors and talent available. Some examples include:

• Tennis Serve to examine how racket design affects (helps/hinders) technique
• Shoveling dirt to examine ways to reduce work-related injuries and proper lifting techniques
• Shaking hands for a CG movie

For example, studying the motion mechanics of a tennis serve will not accomplished if the frame rate of the recording apparatus is low, or if none of the actors knows how to play the sport well. Studying stroke mechanics and water flow for swimming is sorely needed, but would be difficult out of the water. As a Blender example, Project Petunia members have devised their own rig and are researching motions and motion capture to be used in their independent film.

The artifacts of this process include grant proposals and a screen play. OpenOffice Writer and Celtx make excellent tools for this stage. Blender is a general purpose modeling and animation program, and can be used to model different engineering designs, prototypes and host the storyboard. Blender can contribute to the pre-visualization and animatic used in pitching the project.

[edit] Capture Motion

MoCap data is obtained by a human acting in a special suit. The suit has markers on it, which are recorded by a special camera. The positions of these markers are then digitized and the results are saved in a file. The file contains the known location of each marker at each frame. Frame rates can be low or high in terms of frames per second. Higher frame-rates provide more marker location information, and can provide smoother and more accurate motion capture, depending on resolution error. For more information about motion capture, visit the [Carnegie Mellon University] for a discussion and definition of terms, techniques, and equipment regarding MoCap.

Presently, Blender provides a nearly infinite 3D space, exceeding 10k units volume. The number of data points that can be simultaneously streamed is also nearly infinite, as is the number of different actors that can be simultaneously discerned. Animations can exceed an hour in length. Thus, Blender does not provide any creative restrictions on the concept.

For this tutorial, we want to use MoCap to drive our character as they stand and shake hands with someone that walks up to them.

Presently, Blender can process up to 120 frames per second of animation, and the number of frames is limited to 65000. Therefore, at maximum framrerate, Blender restricts a single session to 9 minutes at maximum framerate. In practical terms, Blender does not present any restriction on recording sessions.

Perusing the Internet, we find the library at Carnegie Mellon and their 24-camera rig. Several grad students have capture several sessions of various activities. The suit and marker system they use is shown in the images to the right

[edit] Digitize Session

The video and information about each marker captured by each camera must be assigned 3D space coordinates, and each marker must be properly discriminated. The resolution of each position is crucial in avoiding jitter due to rounding. It is at this point where the software may become confused, and switch markers, or lose markers. It is possible with some hardware to feed the datastream in real-time to visualization software. Most of the time the datastream is saved to a file for later analysis and use.

The data file standard may change, and meta data (such as marker name) may be omitted or inaccurate. The result of the digitization process is a compressed data file. Each of these files are different formats and store the data differently. Some kinds of MoCap file formats are:

• C3D
• BVH
• ASF
• AMC

Huge libraries of these MoCap files are available, cataloged often by type of action and then specific action, and then by recording session. For example, libraries of MoCap files are hosted by:

• [the C3D organization]
• [Carnegie Mellon University]

These libraries may contain public domain information that you can use without fee or royalty; other sites may vary. These libraries and catalogs are organized by Motion Categories. Perusing the library at CMU, you will find file 18_02, a handshke between two actors as a good starter file. Additionally, there are public domain utilities to convert ASF and AMC files into C3D or BVH, at least one of which is called AMC2BVH.

Presently, there are Python scripts to parse and process C3D and BVH files for Blender. By using the converter therefore, Blender has the ability to process any MoCap file. Blender provides channels of information for each object, and allows full naming of properties and objects and actions throughout. It provides visibility and editing down to the datapoint level, exposing information through the user interface or accessible via the scripting language, Python via its API.

[edit] Selecting Motion Capture File Workflow

The high-level workflow is to

1. . Determine what you want your character to do
2. . Find application motion capture libraries
3. . Browse the catalog, view samples of real video captures and animation pre-visualizations
4. . Download the files most applicable; if necessary converting them into BVH or C3D format
5. . Import each into a Blender scene using the applicable File->Import function.
6. . Review and Visualize bone / armature movement and resulting mesh deformation
7. . Assess the file's quality, relevance to your immediate need, and possible future uses.
8. . Save or discard the MoCap file.

[edit] Establish Taxonomy

When surveying the field of motion capture, with an eye toward making a library of actions, I have had to create a standard way of organizing the hundreds of files and motions in outline fashion:

<creature><class>-<noun>-<verblist>

The creatures in your library include those you have developed and those you have obtained, for example the animals from Big Buck Bunny. This tutorial focuses on the Human creature. At the highest level, I have identified the following <class> values:

• play - sports, exercise, enjoyment (climbing, jumping, sports)
• walk - biped locomotion
• dance - all forms of expressive movement: ballet, modern, jazz.
• fight - battle, by hand or with implements
• work - activities involved in production (hammer, sweep)
• team - interactions between individuals (hand shake, consolation, pushing)
• move - all other normal daily movements, like standing around, or sitting down in a chair

Nouns include the name of the sport, such as basketball, or predominant action. Verbs include the name of the movement executed, such as break-left, or direction. Examples include:

• team-hand-shake: two people shake hands
• play-climb-jump: playing around on a climbing (jungle gym) apparatus, climbing and jumping down
• play-basketball-dribble-forward
• dance-ballet-pirouette
• work-clean-windows
• move-chair-sit-stand

Hopefully, this taxonomy will be useful in organizing your files, and will be used when discussing specific files available from Internet libraries. I had an issue with tai-chi. While it is a martial art in slow motion, it is not fighting in the strict sense of the word, but more of a play activity for exercise and meditation, and looks like a dance form. Because the poses are identical to those used in fighting, I classified it there. How you classify motions is at your discretion. What you need to do is establish a navigable file structure and initialize a series of blend files for each <class>.

[edit] Import MoCap

During this step, you obtain the file and the rights to use the file as you wish. You will need to mirror a local copy on your network or hard drive, accessible to your computer. You will then run a script that parses the file and extracts the motion data from it. This stage results in a .blend file which contains an "Empty Cloud", a set of data points (called Empties) which travel in the 3D space over time. Each Empty represents the digitized location of a marker. Workflow step include:

1. Install of Blender3D and instruction on its use
2. Obtain and run the Import script
3. Load the MoCap datastream file
4. Confirm proper interpretation of the datastream
5. Save the resulting Empty Cloud in a graphics animation file

The graphics animation file can be formatted to be interchangeable with other animation packages. Once such exmaple of an interchange file format is the .obj file format, which Blender can export to. Natively, Blender saves animation information in a .blend file format, which is an open-format file used by and specific to Blender for saving animation information.

Up to this point, the workflow in interoperable with a variety of packages. However, past this point, at this time, localized versions of animation information proceed; there is no interchangeable format for armature information across the leading animation packages.

[edit] Constrain the Rig

You now build or adapt a rig that uses the location of these empties as reference points. A "rig" is an armature that has actions and drivers. An armature is a set of connected or parented bones. Actions are a frame sequence of locations, rotations, and scales that move the bone. Drivers are objects that, when they move or rotate, cause other objects to move or rotate.

Rigs can be easily built from scratch, or an existing rig can be adapated to the MoCap data. There are several Blender rigs available in the public domain, for example, Emo and Proog (from Elephants Dream), Mancandy, and Ludwig. For C3D files,

The Blender Import script supports a few marker sets. If the marker set is supported, Blender will automatically build a constrained rig for you. In a C3D file, the marker set is named, and then the markers themselves are named specific names and placed on the body at standard points.

The way the rig uses the MoCap Empties is a process called constraining. Once we have a rig constrained to the markers (empties), it will perform the Motion. We can re-use the rig and make it do other motions simply by importing the new MoCap file, and assigning each MoCap Empty to the new motion curve. Essentially, the rig represents the suit. As long as that same MoCap suit is used (by anyone doing anything in any MoCap studio), we can use this same rig to make the MoCap motion by changing the motion of the constraining empty movements. Therefore, once you have the rig defined and constrained, you want to save it or have some means of reliably reproducing it.

[edit] Bake Armatures

To transition this constrained rig into a form that is more usable in general situations, we bake the derived actions into specific motions of each bone within the armature. In this way, we can now directly adjust and manipulate (edit) the bone's movements and orientation to suit our particular need. For example, the MoCap Empties may cause the character to dance facing East, but we want our character to dance facing West, or possibly to spin around while dancing. By Baking the motion and making the motion known to each bone, we can then orient the entire Armature however we want by adjusting only one curve, and not having to adjust all bones. The overall user workflow for this step is to:

1. Select the Armature
2. Run the baking program, which creates a shadow armature with the motions known to the bones
3. Adjust this motion for any data transcription, recording or other errors
4. Save this overall motion for further study

[edit] Isolate Actions

Ultimately, we want to build a pose library of discrete actions, separate and distinct from other actions. In our example, if the motion capture file consists of the actor taking a step forward, shaking hands, and stepping backward, we want to isolate those three sets of bone movements into three separate Actions. We now have a library of realistic motion primitives which can be used as building blocks. The user workflow for this step is to:

1. Observe the motion and isolate bones and time-space, developing a list of discrete actions
2. Catalog and assign these IPO curves to actions, copying or modifying motion curves as needed
3. Test the action under different conditions (orientations, scales)
4. Test the action with different mesh deformations
5. Identify the maximum values which the range of motion can be exaggerated
6. Create drivers for the action
7. Add the primitive action to the repository

[edit] Integrate Actions

With our library of discrete actions, we can then integrate them together into a shot which calls for different combinations of simultaneous movement, allowing us to combine those components in an infinite variety of ways. If we had Actions called wave, walk, run, wait, and shake hands, we could make an animation where our character starts running, slows to a walk and waves to someone, then stops and shakes their hand. Those component actions came from different MoCap files, but can be easily integrated and strung together. Just as easily, we can make an animation where the character is standing and waves, then walks forward and shakes their hand. This philosophy of component re-use is essential in engineering (hardware re-uses standard integrated circuit components and software re-uses objects), increasing reliability and productivity. The user workflow for this step is to:

1. Browse the repository for available/applicable primitive actions
2. Assemble these action to occur at appropriate times during the animation
3. Integrate these actions together into a seamless animation
4. Assign the character's mesh to the armature
5. Render the animation for test and acceptance.