Massively Multimodal
Masked Modeling

4M is a multimodal and multitask training scheme, enabling the training of versatile any-to-any models, i.e. capable of predicting / generating any modality from any subset of other modalities. 4M models are capable of performing a wide variety of vision tasks out-of-the-box, and excel when fine-tuned to novel downstream tasks.

By tokenizing modalities into sequences of discrete tokens, we can train a single unified Transformer encoder-decoder on a diverse set of modalities, including text, images, geometric, and semantic modalities, as well as neural network feature maps. 4M is trained by mapping one random subset of tokens to another. Please see the animated visualization below for an overview of the 4M training scheme.

The trained 4M models can be used to generate any modality from any combination of other modalities, and are able to perform prediction from partial inputs. When predicting multiple modalities from one, rather than predicting each individually, 4M can be used to predict them one-by-one, always looping fully generated modalities back into the input and conditioning the generation of subsequent modalities on them. The consequence is that all training modalities can be predicted in a self-consistent manner. Please see the animation below for an illustration on how multimodal chained generation is performed with 4M.

Using the 4M training objective, we can train a single model on tens of highly diverse modalities including several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata and color palettes. To access the pseudo labeled and tokenized data, please contact us.

Tokenization consists of converting modalities and tasks into sequences or sets of discrete tokens, thereby unifying the representation space of all modalities. This is critical for training large multimodal models. We use different tokenization approaches to tokenize modalities with different characteristics, see the figure below for an overview.

One of the consequences of 4M's multimodal masked modeling objective is that it enables steerable multimodal generation of any modality given any subset of (partial) other modalities. This not only enables an exciting level of control over the generation process, but also allows for predicting multiple tasks in a consistent manner through chained generation (as demonstrated above).

We invite you to explore the capabilities of 4M models by interacting with the interactive visualizations below. They are best viewed on a desktop computer.

4M allows for generating any modality from any other subset of modalities in a self-consistent manner. We can achieve this level of consistency by looping back predicted modalities into the input when generating subsequent ones. Remarkably, 4M is able to perform this feat without needing any loss-balancing or architectural modifications commonly used in multitask learning. Below, we showcase the generation of all modalities given one input modality, but as we show in other visualizations further below, 4M can also effectively integrate information from multiple inputs.

Hint: Click here for a zoomable version of the figure below, or click on the individual entries to enlarge them.

In this example, we illustrate the generation of several modalities from an RGB image. As demonstrated 4M is able to generate all modalities in a self-consistent manner.

Through the above shown any-to-any capabilities and the fact that 4M can perform generation from partial inputs, we can do fine-grained multimodal generation and editing tasks, as shown in the examples below. Key to this is that certain modalities such as semantic segmentation or depth maps can serve as intermediate steps in the generation process, grounding the generation of subsequent modalities, while allowing to perform high-level semantic edits on them.

In the following example, we showcase the generation of an RGB image conditioned on captions and bounding boxes, and vary the position and shape of only one bounding box. Besides giving users a higher degree of control, notice how the model is able to make sense of unusual bounding box inputs (e.g. the bicycle above a bed is turned into a painting).

In this example, we take a semantic segmentation map extracted from a reference image, and show how 4M resolves edits of a single class in the segmentation map. Notice how semantically stable the fixed parts are, but how they can change in appearance (e.g. the mountains in the first image become snow too when the class of the ground changes to snow).

We are able to perform compositional generation by weighting different conditions by different amounts. This allows users to control precisely how strongly or weakly a generated output should follow each condition, as shown in the example below. It can further be used to avoid the generation of certain undesired concepts via negative weighting.

4M enables performing multimodal retrievals by predicting the global embeddings of DINOv2 and ImageBind models from any subset of the input modalities. This unlocks new retrieval capabilities that were not possible with the vanilla DINOv2 and ImageBind models. Below we exemplify this for two cases, namely predicting RGB from any modality (Any-to-RGB) and predicting any modality from RGB (RGB-to-any).

As shown below, we can retrieve RGB images from distinctly different query modalities. Note that each query modality constrains the retrieved RGBs differently (e.g. semantically or geometrically).

Likewise, given an RGB image as a query input, we can retrieve any other modality.

4M demonstrates strong out-of-the-box capabilities across different tasks, often matching or surpassing the performance of the pseudo labelers and specialist baselines, while being a single model for all tasks. The large performance gap with other multitask models like Unified-IO and Unified-IO-2 is notable.

We evaluate 4M out-of-the-box performance on a variety of tasks including surface normals and depth estimation on DIODE, semantic and instance segmentation on COCO, k-NN retrieval on ImageNet-1K, and 3D human keypoint estimation on 3DPW. As demonstrated 4M-21 matches performance of 4M-7 on common tasks, while being able to solve 3x more tasks and modalities. Note that 4M also outperforms Unified-IO in both the number of modalities it supports and performance on various tasks.

We further provide a visual comparison of 4M-21's predictions with single task experts (pseudo labelers) across different tasks. As demonstrated, 4M-21 is able to predict different modalities with a quality matching or surpassing the experts.

4M-21 also surpasses multimodal models like Unified-IO 1 and 2 on tasks such as DIODE surface normal and depth estimation, as well as COCO semantic segmentation.

We perform an extensive set of ablations and transfer experiments to understand the impact of different design choices of 4M training and to showcase its transfer capabilities. We show that 4M is a scalable and versatile training method that transfers well to a wide range of downstream tasks.

We transfer 4M models of different sizes to ImageNet-1K classification, COCO object detection and instance segmentation, ADE20K semantic segmentation, NYUv2 depth estimation, and ARKitScenes 3D object detection. 4M outperforms or stays competetive with the baselines on all tasks. We observe that 4M-21 has no loss in performance for the transfer tasks that are similar to 4M-7, while solving 3x more tasks. In contrast to 4M, all of the baselines employed data augmentations to achieve their results.

We perform multimodal transfers on NYUv2, Hypersim semantic segmentation, and 3D object detection on ARKitScenes. We compare transfers using RGB images only, and RGB pixels + tokenized sensory depth as inputs. As shown below, 4M makes strong use of optionally available depth inputs and significantly improves upon RGB-only input.

We perform an extensive ablation study to understand key design choices of 4M training, including what modalities to pre-train on, how to sample from them for the multimodal masking scheme, and how many input and output tokens should be used. We measure the effect of different design choices by transferring the trained models to a diverse set of downstream tasks and reporting the average loss. We note that training 4M on all modalities as both inputs and targets results in a generalist model that transfers best to novel tasks and modalities on average. Please see our NeurIPS paper appendix for further ablations.

We further show promising scaling trends in terms of dataset size, training duration, and model size.