CVPR 2026 - 基础视觉大模型 x 三维视觉

Core Projects

3D Vision Foundation Models

精选 4 个 CVPR 2026 开源项目，聚焦基础视觉大模型在三维重建、空间推理和几何学习上的突破。

★ 291

G2VLM

Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning

统一几何感知专家（3D 重建）和语义感知专家（多模态理解）。输入图片即可生成 3D 点云，并支持交错推理的空间问答。2B 参数 MoT 架构。

3D Reconstruction Spatial Reasoning VLM Point Cloud MoT

Parameters

Min VRAM

~8 GB

Institution

Shanghai AI Lab / UCLA / ZJU

License

Apache 2.0

GitHub ↗

★ 363

VLM-3R

Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction

统一 VLM 框架，整合 3D 重建指令微调，实现单目视频的深度空间理解。无需外部深度传感器或预构建 3D 地图，基于 LLaVA-NeXT-Video-7B。

Monocular Video Instruction Tuning VLM Depth-Free LLaVA

Parameters

Min VRAM

~16 GB

Institution

UT Austin / Meta / UCLA

Backbone

LLaVA-NeXT + CUT3R

GitHub ↗

★ 56

TALO

Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction

即插即用框架，修正 3D 视觉基础模型中的空间几何不一致性，实现全局一致的在线 3D 重建。兼容 VGGT、Pi3、MapAnything 等主流骨干网络。

Plug-and-Play Global Consistency Online Recon VGGT SLAM

Type

Framework

Min VRAM

~12 GB

Compatible

VGGT / Pi3 / MapAnything

Platform

Linux + CUDA

GitHub ↗

★ 60

Flow3r

Factored Flow Prediction for Scalable Visual Geometry Learning

通过分解化光流预测增强视觉几何学习，利用无标注单目视频的密集 2D 对应关系作为监督信号。在 8 个 benchmark 上达到 SOTA。

Optical Flow 8x SOTA Geometry Learning Self-Supervised Scalable

Type

Model

Min VRAM

~8 GB

Supervision

Unlabeled Video

Benchmarks

8 SOTA

GitHub ↗

Typical Pipeline

3D Vision Foundation Model Pipeline

这些项目的共同架构模式：视觉编码 + 几何感知 + 语义理解。

RGB Image
/ Video

➔

Visual Encoder
ViT / DINOv2

➔

Geometry Expert
3D Reconstruction

➔

3D Output
Point Cloud / GS

Text Query
Spatial Q&A

➔

LLM Backbone
Qwen2 / LLaMA

➔

Semantic Expert
Spatial Reasoning

➔

Answer
+ 3D Grounding

Project	Params	Min VRAM	Recommended	Usage
G2VLM	2B	~8 GB	16 GB+	RTX 3090 / 4070Ti
VLM-3R	7B	~16 GB	24 GB	RTX 4090 / A100
TALO	-	~12 GB	24 GB	RTX 4090 / A100
Flow3r	-	~8 GB	16 GB+	RTX 3090 / 4070Ti

Solutions

Local Hardware Options

macOS 不支持 NVIDIA eGPU 驱动。以下是可行的本地运行方案。

☁

Cloud GPU (Zero Hardware)

Google Colab Pro+ / RunPod / Vast.ai
A100 40GB 或 4090 24GB 按需租用
适合偶尔跑实验，无前期投入

~¥3/hour (4090) | ~¥150/month (Colab Pro+)

🔌

eGPU + Linux (Intel Mac Only)

雷电 eGPU 盒子 + RTX 4090
外置 SSD 装 Ubuntu，开机选系统启动
仅限 Intel Mac，Apple Silicon 不支持

~¥13,500 (盒子 1.5K + 4090 12K)

Quick Start

Environment Setup Guide

显卡到位后的通用环境搭建步骤（以 Ubuntu + RTX 4090 为例）。

Install NVIDIA Driver + CUDA 12.1

安装 NVIDIA 驱动和 CUDA Toolkit。

# Install NVIDIA driver
sudo apt update && sudo apt install -y nvidia-driver-550
sudo reboot

# Verify GPU
nvidia-smi

# Install CUDA 12.1
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda_12.1.0_530.30.02_linux.run
sudo sh cuda_12.1.0_530.30.02_linux.run
          

Setup Conda + PyTorch

创建 Python 环境并安装 CUDA 版 PyTorch。

# Install Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

# Create env
conda create -n cvpr3d python=3.10 -y
conda activate cvpr3d

# PyTorch + CUDA 12.1
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
          

Clone & Run Projects

克隆项目并按照各自 README 安装依赖。

# G2VLM (2B, ~8GB VRAM)
git clone https://github.com/InternRobotics/G2VLM && cd G2VLM
pip install -r requirements.txt
python inference_recon.py  # 3D reconstruction
python inference_chat.py   # spatial reasoning

# VLM-3R (7B, ~16GB VRAM)
git clone https://github.com/VITA-Group/VLM-3R && cd VLM-3R
pip install -r requirements.txt

# TALO (plug-and-play, ~12GB VRAM)
git clone https://github.com/Xian-Bei/TALO && cd TALO

# Flow3r (~8GB VRAM, 8x SOTA)
git clone https://github.com/Kidrauh/flow3r && cd flow3r
          

Roadmap

Personal Progress

从调研到本地运行的计划时间线。

2026-03-19

Project Survey Complete

完成 CVPR 2026 3D Vision Foundation Model 调研，筛选出 G2VLM / VLM-3R / TALO / Flow3r 四个项目。

2026-03-19

Research Page Built

研究展示页面上线，记录项目详情、硬件需求、环境搭建指南。

Pending

GPU Hardware Arrives

等待 NVIDIA 显卡到位。推荐 RTX 4090 (24GB)，可覆盖全部 4 个项目。

TBD

Environment Setup

Ubuntu + CUDA 12.1 + PyTorch 2.5.1 环境搭建。

TBD

Run Experiments

逐个运行 G2VLM → Flow3r → TALO → VLM-3R，从低显存到高显存依次验证。

Foundation Vision Models
x 3D Vision

3D Vision Foundation Models

3D Vision Foundation Model Pipeline

VRAM Requirements Comparison

Local Hardware Options

Cloud GPU (Zero Hardware)

eGPU + Linux (Intel Mac Only)

Environment Setup Guide

Install NVIDIA Driver + CUDA 12.1

Setup Conda + PyTorch

Clone & Run Projects

Personal Progress

Project Survey Complete

Research Page Built

GPU Hardware Arrives

Environment Setup

Run Experiments

Reference Links

Foundation Vision Modelsx 3D Vision

3D Vision Foundation Models

3D Vision Foundation Model Pipeline

VRAM Requirements Comparison

Local Hardware Options

Cloud GPU (Zero Hardware)

Linux Mini-ITX + RTX 4090

eGPU + Linux (Intel Mac Only)

Environment Setup Guide

Install NVIDIA Driver + CUDA 12.1

Setup Conda + PyTorch

Clone & Run Projects

Personal Progress

Project Survey Complete

Research Page Built

GPU Hardware Arrives

Environment Setup

Run Experiments

Reference Links

Foundation Vision Models
x 3D Vision