What does it take to make a superhero fly across the cityscape in a movie? How do filmmakers make something that is as impossible as flying look realistic?

Well, it’s “movie magic,” which is nothing but video compositing.

Video compositing is the process of combining multiple visual elements into a single, cohesive sequence. It is almost like creating a collage and is incredibly important in post-production, as it helps to seamlessly integrate different visual elements for a realistic scene.

What is video compositing?

Compositing is a video editing technique where visual elements from various sources are put together to form single images/frames, creating the illusion that they’re all in the same scene. It is not just putting them together in layers but involves blending, manipulating, and integrating in a way that makes the scene believable.

It is not simply putting different clips and elements on top of each other in layers, but much more. The various elements (some, if not all) have to blend together and interact with each other in a realistic way so the whole thing is believable. Movies with CGI characters are a good example of this kind of video blending.

In the below image, you can see how the child actor from Disney’s The Jungle Book is interacting with a dummy, while on screen it looks like he is actually holding a real wolf. Here, digital compositing is used to blend together clips of the actor with CGI elements.

jungle book actor with cgi wolf behind the scenes

Copyright: Cartoon Brew

As you can clearly see, through digital compositing, compositors aim to create a believable sequence or scene from different sources.

The Core Techniques of Compositing

Masking and Rotoscoping

Masking is a useful tool in both photo and video editing that lets you select a specific part of an image/frame to edit. Changes will apply to the selected parts only. Some examples of things you can do with masking are isolating moving objects, removing backgrounds, cutting out a person, etc.

Rotoscoping

Rotoscoping, meanwhile, is a technique used to make animated characters’ movements look more realistic and believable. Here you trace over a live actor’s movements and apply them to your character. Famously, rotoscoping was used in Star Wars, where prop lightsabers were rotoscoped to add the colored glow and sparks over the live-action footage.

Chroma Key (using Green or Blue Screens)

Chroma key is a technique where the block of color in the footage is replaced by another color, image, or another clip. For scenes like a superhero flying across the city or a boy hugging a wolf (like in The Jungle Book), we use a green or blue screen and chroma key instead of filming it with a background. This ensures a seamless image integration, therefore a believable visual experience.

Apart from films and TV shows, chroma key is also used for weather reports, where you see animated maps and weather graphics behind the reporter.

Luma Key

Luma key is where the brightness or “luma” values (instead of a color) are used to separate the foreground from the background. It’s useful for removing black or white backgrounds, isolating elements (like logos, text, or certain parts of a scene), and creating various VFX (picture-in-picture, transitions, blending), especially in grayscale or low-color frames.

Here, you don’t keying doesn’t need a green or blue screen and it can work with any type of video footage.

Tracking

In video compositing, tracking is the process of analyzing/tracking the movement of an object or camera position within a video and then applying said movement to another object or layer. Tracking can also be for the movement of a specific point within a frame.

The types of tracking include

  • 2D Tracking: Tracks the motion of points, edges, or features only within a two-dimensional plane (X and Y axes).
  • Planar Tracking: An advanced 2D tracking method that tracks a flat surface or a “plane” within footage.
  • 3D Camera Tracking: A complex method that analyzes the complete footage to reconstruct the original camera’s movement and position in a 3D space to create a virtual 3D camera that perfectly matches the motion of the real-world camera.
  • Object Tracking: Specialized trackers follow only the motion of a single object within a scene so effects can be masked or applied directly to it.
  • Specialized Tracking: Includes facial tracking (tracking the intricate movements and markings of a human face for de-aging, face swapping, etc.), body tracking (tracking the movement of an entire human body for motion capture, character animation, or applying effects to an actor’s limbs), and spline tracking (tracking and adjusting the movement of areas defined by complex shapes).

Blending Modes

Blending modes (also called transfer modes) are effects added to one layer of a visual effects composition that affect the colors of the below layers. It could be a color effect, color keying, or a silhouette.

Some of the commonly used blending modes are

  • Screen: Removes the dark parts of a new layer.
  • Multiply: Removes the bright parts of a new layer, the opposite of Screen.
  • Color Burn: Darkens the overall image while blending into the background and retaining highlights for a vintage, burned look.
  • Add: Like Screen, it removes the dark parts on an image but adds its own color values to the image below. The final composite will be brighter than the original.
  • Overlay: Changes the color of the mid-tones.

You can adjust the effect of a blending mode by manipulating its opacity.

Examples of the effects created by blending modes include glows, shadows, reflections, and more.

Physical Compositing

All the above techniques we mentioned above are digital compositing methods done with the help of powerful video editing software. Then there is physical compositing, where camera tricks and multiple real-world visual elements (like props, sets, etc.) are used to create composites.

While digital compositing has taken over the filmmaking world, physical compositing is still done to get very specific, unique creative effects. Some of these include partial models (miniature replicas of objects, locations, sets, etc.), front and rear projections (where foreground action is combined with pre-recorded backgrounds), and glass paintings (visual elements and images painted onto glass panels placed between the camera and the background).

Multiple Exposure

It is a compositing technique used to create surreal or artistic visual effects. Depending on how you are doing it, it can be a digital or physical compositing method. In the physical method, it involves exposing a single frame of film multiple times to different light sources or scenes. In the digital method, various software and tools can be used to create multiple exposure effects.

Essential Software for Compositing

You can master video blending and complex VFX with tools like

  • Adobe After Effects: The industry standard for video compositing and motion graphics, widely used for a vast array of visual effects tasks. It has been used on many feature films like Iron Man, World of Tomorrow, Sky Captain, Stranger Than Fiction, etc.
  • Nuke: Free for non-commercial purposes, Nuke is a high-end, node-based compositing and VFX software favored by major visual effects studios for complex feature film, animation, and television projects. It has been used on several poplar film projects, including Thor: Love and Thunder.
  • DaVinci Resolve Fusion: Built into DaVinci Resolve, Fusion offers powerful compositing capabilities with a node-based workflow. Some of the major projects it was used on include the Avengers, Hunger Games, and Terminator Genisys.
  • Blackmagic Fusion: Standalone software that offers a comprehensive set of tools for advanced compositing. Some major projects it has been used on include movies like Red Sparrow, Ant-Man, and London Has Fallen.

Why Compositing is Crucial in Modern Video Production?

Compositing in video editing, contrary to popular belief, is not just about adding flashy effects to films and videos. It serves a higher purpose: storytelling and efficient production. But, to elaborate, here are some of the reasons why video compositing is crucial.

  • VFX in filmmaking brings stories to life. It empowers filmmakers to stretch the possibilities of a story, make impossible visuals possible, and create imaginative worlds.
  • It is cost-effective in the sense that they give you an alternative to building elaborate sets or traveling to exotic locations, all of which require lots of money.
  • It’s a powerful post-production technique for correcting imperfections, as you can use it to fix issues in footage, like removing unwanted objects or wires or even replacing skies.
  • It produces realistic visual effects by making unrelated elements look like they genuinely belong together.

Common Video Compositing Challenges (And How to Overcome Them)

Lighting Mismatches

Getting consistent lighting is one of the biggest challenges in VFX. Compositing combines multiple visual elements taken from different sources, which means that there can be inconsistencies in light direction, intensity, tint, color temperature, etc. Lighting mismatches between visual elements can shatter the realism and ruin the viewing experience.

To solve this issue, first understand that light interaction is key to realistic compositing.

Perspective and Scale Issues

Elements have to be of the correct perspective and scale in relation to each other and the background for smooth image integration. If there are any incorrect angles or sizes, the elements will look like they have been copy-pasted rather than integrated into the scene.

When compositing, ensure that your foreground and background elements align correctly in terms of size and perspective. It involves understanding vanishing points, adjusting focal lengths, and matching lighting and shadows. You can get precise scaling and perspective adjustments when you use the right tools, like Adobe Photoshop, DaVinci Resolve Fusion, etc.

Grain/Noise Discrepancies

Every camera will produce some amount of grain depending on the lighting conditions and other factors. When you’re working with different cameras and shooting conditions, you will get a range in the levels of digital noise or film grain. If you don’t match the noise/grain levels between elements, they will stand out.

To overcome this challenge:

  • Capture clean footage (low ISO, adequate lighting; shooting stills is possible)
  • Use post-production techniques like noise reduction, grain effects, and blur and sharpen.
  • Use the right compositing tools and techniques, like masking, color matching, and blending modes.
  • Use external plugins (if needed) for additional noise reduction, like Noise Video, Magic Bullet, Denoiser, and Boris FX plugins.

Edge Artifacts from Keying

Poor chroma keying can leave visible colored fringes or “halos” around the keyed subject.

Perform clean keying, fine-tune and smooth the edges, address spill and other artifacts, and ensure good background quality to create cleaner edges for the visual elements.

Tips for Successful Video Compositing

  • Plan and film your shots with compositing in mind. Understand what elements will be added in post and how they will interact with the live actors, props, sets, and each other.
  • Capture clean plates, use consistent lighting, and take accurate measurements on set to make compositing easier and hassle-free.
  • Use real-world lighting, shadow, and texture references so your digital elements look much more authentic and realistic.
  • It never hurts to be attentive. So, pay close attention to the subtle details (lens distortion, depth of field, and environmental reflections) when creating your composites.

The Future of Video Compositing

The world of VFX is always changing, always evolving. Video compositing is no exception. Here are some of the trends you can expect to see in the near future:

AI and Automation

Artificial intelligence (AI) is already being used to automate complicated and time consuming tasks like rotoscoping, content-aware filling, and denoising, making the workflow faster and more efficient. It is also being used to create hyperrealistic digital characters and complex animated backgrounds.

By the end of 2037, it is estimated that Artificial intelligence (AI) will dominate the VFX market with a 75.2% share.

Real-time Compositing

Real-time compositing allows immediate blending of various visual elements (CGI, videos, and live footage) into a single image. In this way, the compositors gets a real-time preview of the final outcome can manipulate the composite accordingly till they get it right. It is far more efficient than traditional compositing, where each frame is rendered separately.

Virtual Production

Virtual production is an advanced compositing technique where real-time digital elements are blended together with real-world visuals through the use of tech like LED walls, CGI, and AR. This way, compositing often happens in-camera or with minimal post-production.

And There You Have It…

Video compositing is an incredible storytelling tool scores of filmmakers and content creators use to bring the boldest of visions to life.

For compositing, there are numerous techniques, like chroma key, rotoscoping, motion tracking, etc. And there are just as many challenges, like lighting mismatches, differences in noise/grain levels, improper keying, etc.

Whether you want to be a video editor or filmmaker, mastering video compositing is a vital step. So, start exploring the tools and techniques today, and begin your journey to learn visual effects. Once you master the techniques and take the right precautions, you can finally create the most extraordinary visual experiences like the greats before you.

Frequently Asked Questions (FAQ’s)

When should I use a green screen or a blue screen?

Green screens are much brighter than blue ones. So, green screens are better for when you want bright, well-lit scenes, like daytime scenes. Blue screens are better for scenes that don’t need so much light.

What is the main difference between video editing and compositing?

Video editing and compositing have different purposes. In editing, the primary focus is on trimming, cutting, arranging, adding transitions, and making color adjustments (color correction and grading). Compositing, on the other hand, is about combining multiple visual elements (like different video clips, still images, 3D renders, or motion graphics) into a single frame. Not all of these elements will be present on set at the time of filming.

Is compositing difficult to learn for beginners?

Compositing, like any specialized skill, has a learning curve. Green screen removal and other basic techniques are learned more easily and quickly. Realistic and complex composites, however, require a deep understanding of lighting, perspective, color theory, and VFX software like Adobe After Effects or DaVinci Resolve Fusion.

Can I do compositing with free software?

Yes, there are free options available for video compositing. For example, the free version of DaVinci Resolve includes the Fusion tab. There are other open-source alternatives, but they might be difficult to learn or may have fewer features compared to premium software.

What are the career opportunities in video compositing?

Compositors are in high demand for movies, TV shows, music videos, animation, and advertising. Presently the roles in video compositing include junior compositor, lead compositor, and VFX artist.