Post processing
Everything that this page will talk about happens in a viewport tab for a given scene. You did select a scene when recording, if you don't have the tab open you can load it again from Recent scenes on the quickstart tab or using menu bar (File -> New-> Scene).
Scene tab consists of multiple panels, here is the default layout:
In this part of tutorial we will focus on the lower part of screen as this is where most of postprocessing happens. Of course you can move the panels around if you want.
First take a look at the reconstruction list on the right side. This contains all the recordings done for the current scene. Selecting (left-click) a recording will load its animation data from the disk, if available. Right-click on a recording to open the context menu to access management options.
This is where most of data management happens, after selecting a recording you will see the processing options, playback controls and the list of available tracks. Each track is a snippet of data representing a skeletal animation, at the beginning there is just a single track for every camera used in recording. If you did not select 'estimate right away' during the recording there will be no tracks as no processing was done yet, your recording data is for now stored as frames on disk.
To preview a track in the viewport use the eye icon, this will display the playback controls that are self-explainatory. Now we will run through the processing options from left to right.
First you have the save button, this will light up in yellow if there are some unsaved changes in the recording. Right to it are the estimation options:
- The three-button selector for estimation mode: left hand, right hand, main body only or full model. To select full model double click the human button.
- The chip button to start the estimation, you should use it if you didn't select 'estimate right away' when recording. You can also rerun the estimation as long as the frame images are on disk.
- The triangle button to run triangulation (multi-camera setup only). This will produce new track containing data composed from all cameras according to their extrinsic data.
Next we have the antenna button, which lets you control streaming the preview to connected integrations (like Unreal Engine). It is followed by the confidence score display, this tells how much the AI was sure about detections in this frame.
Scale/rotate options control the preview of data. You can use Auto Scale to adjust scaling basing on known height of the actor. Rotating may be useful depending on how the camera was oriented. And finally we get to the modyfing operations.
A track is a block of data representing a single, unique skeletal animation. It allows us to distinguish between different stages of processing or various versions of the animation obtained from a recording. A single recording usually contains multiple tracks - first the tracks obtained from estimation of images, then smoothed or cropped data. You can copy data between recordings in the right-click menu in recording list panel.
When hovering your mouse over a track you can see some extra information. You can notice some tracks are marked as "raw", which means the data includes pixel positions of detected bones. Tracks added through estimation always begin as "raw". Some of tasks including Detect Z Distance and Detect Leg Movement require this pixel data to be run. We try to preserve that data as long as it is possible, hovewer operations like smoothing break the correlation between image and world coordinates and resulting track is no longer raw.
Working with tracks
- Use the left mouse button to select a track. Hold Shift to select multiple tracks. All trackbar tasks depend on track selection.
- Right-click on a track to open its context menu.
- Use the eye button on the left side of a track to preview it in the viewport
Tasks displayed in the trackbar can only be used when their selection conditions are met. Most tasks require one track to be selected, but there are a few exceptions:
- Body-Hands Join (requires body, left hand, and right hand tracks to be selected)
- Merge Tracks (requires two tracks to be selected)
Smoothing tasks that can be ran on a single track can be accessed from the right-click context menu of a track:
- Polynomial smoothing
- Spline filling
- Exponential smoothing
- Low pass filtering
The tasks that are available from the top of trackbar panel include:
- Split and cut - divide a track into smaller tracks by cutting out the frames for which bone detection failed;
- Cut track - separate some frames into a new tract;
- Body-hands split - divide a track containing full model data into separate tracks for body, left and right hand;
- Body-hands join - rejoin the tree separate tracks for body and hands back together;
- Merge - glue one track at the end of another, with optional linear interpolation to smooth out the transition;
- Anchor bones - adjust translation of the model so that particular body part stays as still as possible for a given set of frames;
- Detect leg movement - based on image data detects which leg is on the ground and then adjusts model translation accordingly. This is useful for when actor was walking in front of camera.
- Detect Z distance - useful for single-camera setups. Tries to estimate distance between camera and actor to recover front-back movement.
In either mode we always want to start by cutting out the parts where we weren't ready for recording. Preview the track using the 'eye' icon and skim through the unfiltered animation to figure out our region of interest. Use 'Cut track' task to produce a new track, first argument is a starting frame and second argument is ending frame. If parts of your recording do not include an actor you can also use Split and cut to remove them quickly. If your camera was not looking straight when recording you may want to adjust preview rotation now.
Single vs. multi-camera workflows
The workflow differs depending on the mode. In a multi-camera system, we usually skip anchoring and Z detection because we work with triangulated tracks, which already have accurate 3D positions, so anchoring is not required.
In single-camera mode, there are extra steps before moving on to smoothing the data.
Single camera mode:
As discussed earlier, in single-camera mode, we start by running pixel data tasks. Select the track and use the 'Detect Z Distance' task to analyze bone movement in camera view.
This will apply root motion to the animation.
If you're creating an in-place animation (where the character stays in one spot), you can skip this task.
Next, we need to analyze the animation and find stable points that we can anchor the animation to.
What is anchoring in motion capture?
Anchoring involves identifying fixed points in your animation where certain body parts stay in contact with a surface (like the ground) or remain relatively still. These anchor points help keep the animation stable, especially for parts of the body that need to stay put during specific movements.
For instance, when recording a high kick, the supporting leg stays on the ground while the other leg kicks. By anchoring the supporting leg, you ensure it remains in place while the rest of the body moves naturally.
To set anchors for specific sections of your recording, use the 'Anchor Bone' task. You can run this task multiple times, which allows you to place several anchors throughout your animation as needed.
Multi-camera recordings can use those tasks as well, but they are typically not necessary since the points are already triangulated.
Copying tracks between recordings
For some workflows you would like to compbine results of multiple recordings. For example, you can record a finger or hand animation separately and then use it in a full-body animation. To do that right-click your source recording in the recording list and select Copy all tracks which will open a modal window. After that you can merge the data using cut and merge tasks.
Smoothing the animation
You have several algorithms at your disposal, each with its own advantages and disadvantages. We've designed that way to give you the flexibility to choose the best smoothing method for your specific scenario.
For example, if your animation has been recorded with finger tracking enabled, start by splitting the body from the hands using the Body-Hands Split task. Then, apply an exponential smoothing task to the finger tracks while using a low-pass filter on the body track then join all the tracks together using Body-hands join.
Or combine multiple smoothing tasks, you could start by using one task to smooth out only the biggest biggest offenders, and then follow up with a another task to smooth the entire animation.
More about tasks, use cases and how they work at the bottom of the page.
To access the smoothing algorithms, right-click on a track and locate the tasks section.
Note: After running a smoothing algorithm, the track will lose its 'raw' status, and using pixel data tasks will no longer be possible.
Example workflow summary
Assuming we're working on the high-kick animation we've mentioned previously
1. Cut out the animation part from the unfiltered track (region of interest)
2. Run Detect Z distance task if root motion is desired
3. Anchor one of the legs as it is mostly immovable using Anchor Bone task
4. Analyze if we can place more anchors, for example in my case across many frames one of my hands is mostly immovable so i am going to anchor it as well
5. Smooth the output by right-clicking on a task, optionally cutting it in parts (Cut track task) to use different smoothing methods on different parts.
A combination of smoothing algorithms is also an option, we can run Exponential smoothing algorithm before e.g. spline filling
6. Stream it to the engine or export it as file
Exporting the Animation
Before exporting, it's important to note that AImation Studio currently does not support any form of retargeting and consuming engine needs to support that.
Whether you choose to stream the animation or export it as an FBX or BVH file, your engine will need to process the bone data through inverse kinematics (IK) to map it onto your own model. We recommend checking out our UE5 project for a demonstration of how we drive Metahuman models. You can find the details here installing unreal engine plugin
For Unreal Engine users, we recommend using the LiveLink project alongside Unreal’s Take Recorder to capture the incoming animation. That way animation is already created for metahumans, all the files are automatically created inside unreal and you can easily retarget onto any other model using UE powerful tools.
We plan to add support for retargeting in the near future.
To export a track as a file, select it for preview and then right-click on the track and select 'Export as file'
Smoothing algorithms
Since each task by default creates a new track and does not affect the original one, we recommend experimenting with different parameters to see how they affect the results.
The possible use cases provided are only suggestions; in some animations, certain algorithms will perform better than others.
Polynomial Smoothing
How it works
Smooths out data by approximating it with a curve (polynomial) that runs over a window of frames.
The degree of the polynomial determines how complex the curve is, and the window size defines how many frames are included at once in this smoothing process.
Parameters
Degree: A higher degree means a more complex curve, which can smooth out more complicated motion patterns but may take longer to compute.
Window: A larger window means the algorithm will consider more frames at once, giving a more gradual smoothing effect, but can make the process slower.
Possible use case
Use this method when you need to smooth out motion that has small, irregular noise or jitter.
Low-pass filter
How it works
Removes high-frequency noise, which means it focuses on keeping slower, smoother movements and filters out fast, jerky motions that are likely unwanted noise.
Parameters
Cutoff Frequency (band): A lower cutoff frequency causes more smoothing because it filters out faster movements more aggressively. Higher values will let more movement through, focusing only on the really fast noise.
Possible use case
This filter is useful when there’s a lot of jittery or jerky movement in your motion capture, and you want to focus on the slow, natural movements.
Spline Filling
Use Case: Spline filling is used to fill in gaps where data might be missing. It estimates what the missing frames should be by looking at the frames around them and fitting a curve (spline) through the gap.
Parameters
Degree: Higher-degree splines are more complex curves that provide more accurate fills for missing data, but take longer to compute.
Window: The window size determines how many neighboring frames the algorithm will look at to fill the missing ones. A larger window gives more context, but increases processing time.
Possible use case
This is ideal when your estimated data has missing or bad frames, and you want to create smooth transitions across those frames.
Exponential smoothing
How it works
Method that reduces noise by giving more importance to recent frames and gradually reducing the influence of older ones.
Parameters
Alpha: A smaller alpha value means stronger smoothing, making the data appear more stable, but too small a value might make the movement too sluggish. A higher value will make the data more responsive but less smooth.
Possible use case
Often used after some processing has been done on a track already and most frames are looking okay, smooths out the data and adds fluidity to it.
