Rotoscoping in After Effects creates a moving cutout so you can place a person or object over another background. Roto Brush provides the initial selection; your job is to correct the frames where it loses the subject and check the exported alpha channel.
This guide uses Roto Brush 3 terminology. Older projects may use a different version of the effect. Start with a short section containing the hardest movement in your shot before committing to the full clip.
Editorial scope: The workflow was reviewed on September 22, 2026 against Adobe’s current Roto Brush and output documentation. Menu names and available codecs can vary by After Effects version and operating system, so confirm the selected effect version and the receiving editor’s delivery requirements.
Choose the right method for the shot
| Your footage or deliverable | A useful starting point | What to check first |
|---|---|---|
| A presenter against a busy room | Roto Brush | Hands crossing the face and hair against similar colors |
| A clean, evenly lit green screen | Chroma key | Color spill and shadows; see the green screen workflow |
| A rigid object with a simple outline | Animated masks | Corners, rotation, and any objects passing in front |
| A quick cutout for an existing edit | An AI background-removal preview | Whether the difficult frames survive without manual cleanup |
These are starting points, not quality rankings. A ten-second test can reveal that a simple-looking clip needs substantial correction. Keep the original footage available if you try more than one method.
Select and track the subject with Roto Brush
- Match the composition frame rate to the source and work at Full resolution. Double-click the footage layer to open the Layer panel; Roto Brush selection happens there rather than directly in the Composition panel.
- In Effect Controls, confirm the Roto Brush version used by the project. Select version 3.0 when you intend to use Next-Gen Roto Brush; keep an older version when an existing project must reproduce its earlier matte.
- Choose a base frame where the complete subject is visible and not hidden behind another object. Avoid beginning on the hardest occlusion merely because it occurs first in the clip.
- Draw a small foreground stroke through representative areas inside the subject. Do not trace the outline. Use as few strokes as necessary, then hold Alt on Windows or Option on macOS to paint a background stroke over an unwanted region.
- Advance through adjacent frames and stop at the first visible error. Correct that frame, then check how the correction propagates. Work forward and backward from the base frame and keep the Roto Brush span inside the intended shot.
- Use Refine Edge only across boundaries that need partial transparency, such as hair or motion-softened fabric. Review the alpha view as well as the composite over contrasting backgrounds.
- Adjust Feather for boundary softness, Shift Edges for boundary size, and Reduce Chatter for erratic movement. Change one control at a time; a setting that steadies a stationary edge can make a moving edge lag.
- Click Freeze in the Layer panel only after reviewing the whole required span. Save the project after freezing. Unfreeze before changing selection strokes or propagation controls.
Adobe’s Roto Brush reference explains these controls and version differences. Freeze locks segmentation; it does not prove every frame is correct.
Understand what Freeze does
Freeze caches and locks the propagated segmentation information for the span. Matte controls can still affect the result, but new foreground or background strokes do not change the frozen propagation until you unfreeze it. Treat Freeze as a reproducibility step after review, not as an automatic quality check.
Save a version before unfreezing a long approved span. If a later correction makes an earlier section worse, you then have a known-good project state to compare instead of relying on memory.
Review a difficult movement before doing the whole clip
Consider a presenter raising a hand across their hair. Use that movement as your test rather than an easy frame with arms at their sides. This is an illustrative review exercise, not a measured performance result.
First, look at the approach: does the space between the fingers remain background? Next, inspect the overlap: does part of the face disappear when the hand passes it? Finally, inspect the separation: does a patch of the old wall follow the hand away from the head?
Write down the first failing frame for each problem. Correct the selection and review the surrounding frames again. If the correction fixes the hand but damages the face, consider isolating that short problem area in a separate layer or mask pass. Judge the result at normal playback speed as well as frame by frame.
This gives you a practical scope estimate: if one short gesture requires many corrections, budget time for similar gestures throughout the shot. Do not extrapolate from a still frame alone.
Diagnose the artifact before changing settings
| What you see | What to inspect | Next action |
|---|---|---|
| A hand or accessory disappears | The first frame where the selection loses it | Correct that frame, then review the following movement |
| Background remains between fingers | The gap in the alpha view | Remove the unwanted region and check the gap as the hand moves |
| The outline looks fine on black but bad on white | The same frames on both backgrounds | Separate a selection error from a color fringe before changing the matte |
| A still image looks clean but playback flickers | Several consecutive frames at normal speed | Check selection consistency; review temporal smoothing without hiding real motion |
| A soft moving edge looks unnaturally sharp | The original footage beside the cutout | Preserve plausible motion softness instead of judging every edge as a hard silhouette |
Make one change at a time and compare the same range. Otherwise, it becomes difficult to tell which adjustment helped. Keep a saved version before an extensive cleanup pass so you can compare rather than rely on memory.
Export a file with transparency
A transparent composition needs an output format and channel setting that preserve alpha. For an editing handoff, add the composition to the After Effects Render Queue, open Output Module, and select an alpha-capable format and codec, such as QuickTime with ProRes 4444 where available. Set Channels to RGB + Alpha. A PNG sequence with alpha is another option when the receiving workflow accepts image sequences.
Check the receiving application’s requirements for straight versus premultiplied alpha. Adobe documents the relevant output module and alpha settings. A filename ending in .mov alone does not establish that the file contains transparency.
For a finished video with a replacement background, composite that background before exporting. The usual H.264 MP4 delivery is a flattened video, not a reusable transparent cutout. See our transparent video guide for delivery choices.
Test the exported file, not just the composition
Import a short export into the application that will actually receive the clip. Put a bright solid under it, then replace that solid with a dark one. Watch the troublesome hand or hair movement again. If a solid rectangle appears, review the output channels and codec. If the edge develops a fringe only after import, check alpha interpretation in the receiving application.
Also confirm the frame count, dimensions, frame rate, and any required audio. A clean cutout is not ready for handoff if the export is missing the last gesture or the editor cannot use its format.
When an automated cutout is enough
If the job only needs a background replacement, compare a short RemoveBGVideo preview with your required result before investing in manual work. Use identical source footage and inspect the same hard frames. Choose based on corrections needed and export compatibility, rather than a promise that one method is always faster or cleaner.
For a client handoff, agree on the acceptance criteria: which background the clip will sit over, whether individual hair detail matters at the final display size, and whether an alpha file is required. Those decisions determine how much edge cleanup is useful. The complete background-removal guide covers the broader choice between automated removal, keying, and manual work.
Questions before delivery
Can I fix everything with more feather?
No. Softening the outline does not restore a missing finger or remove a large patch of background. Correct the selection first, then judge edge softness against the final composition.
How long should I allow for a shot?
Time a representative difficult section and count similar movements in the rest of the clip. Include review and export checks. A fixed estimate based only on clip duration can miss most of the work.
Do I need to redo the entire shot if one section fails?
Identify the first bad frame and the extent of the problem before restarting. Keep the acceptable result saved, then decide whether a local correction, a separate mask pass, or a new selection span is easier to review.
What should I send to the next editor?
Send the agreed alpha-capable file or image sequence, a reference composite, and the frame rate and alpha interpretation used. Keep the source and project until the receiving editor has confirmed the handoff.
