文章标题NViST: In the Wild New View Synthesis from a Single Image with Transformers1. 环境配置创建环境conda create -n nvist python3.9进入环境conda activate nvist安装torch、torchvision、torchaudiopip install torch2.1.2 torchvision0.16.2 torchaudio2.1.2 --index-url https://download.pytorch.org/whl/cu121安装其它依赖pip install tqdm scikit-image opencv-python configargparse lpips imageio-ffmpeg lpips tensorboard torch_efficient_distlosspip install easydict timm plyfile matplotlib kornia acceleratepip install tensorflow pandas pip install githttps://github.com/google/nerfies.gitv2 pip install githttps://github.com/google/nerfies.git#eggpycolmapsubdirectorythird_party/pycolmap2. 数据下载与预处理2.1. 获取下载地址和密码点击链接 https://docs.google.com/forms/d/e/1FAIpQLSfU9BkV1hY3r75n5rc37IvlzaK2VFYbdsvohqPGAjb2YWIbUg/viewform填写所有的必填项得到下载地址和密码点击链接并输入密码2.2. 使用chrome下载进入开发者模式Windows和Linux快捷键CtrlShiftIMacOS快捷键commandoptionJ进入Network tab选择若干文件点击下载如果是下载到桌面客户端则等待下载完成即可如果想下载到远端则需要继续下面的步骤。看到一个类似download.aspx?...的条目右键点击→Copy→Copy as cURL在复制的内容后面加入--output mvi_xxx.zip然后粘贴到终端运行2.3. 数据预处理2.3.1. 解压数据包然后进行下采样python preprocess/downsample_images.py --data_dir [data directory]2.3.2. 计算相机位姿修改read_colmap_results_mvimgnet.py中的data_dirpython preprocess/read_colmap_results_mvimgnet.py2.3.3. 生成cache文件python preprocess/make_cache.py --data_dir [data directory] python preprocess/make_cache.py --data_dir [data directory] --split test2.4. 问题记录pycolmap自带bugTraceback (most recent call last):File /workspace/xuehtxiaopeng.com/code/nvist_official/preprocess/read_colmap_results.py, line 3, in moduleimport pycolmapFile /opt/conda/envs/nvist/lib/python3.9/site-packages/pycolmap/__init__.py, line 4, in modulefrom .scene_manager import SceneManagerFile /opt/conda/envs/nvist/lib/python3.9/site-packages/pycolmap/scene_manager.py, line 22, in moduleclass SceneManager:File /opt/conda/envs/nvist/lib/python3.9/site-packages/pycolmap/scene_manager.py, line 23, in SceneManagerINVALID_POINT3D np.uint64(-1)这是一个明显bug把-1转换为无符号整型改为INVALID_POINT3D np.int64(-1)3. 训练3.1. 精调MAE3.1.1. 下载预训练模型mkdir pretrained cd pretrained wget -nc https://dl.fbaipublicfiles.com/mae/visualize/mae_visualize_vit_base.pth3.1.2. 修改配置文件修改文件configs/mvimgnet_mae.txt的data_dir和base_dirdataset_namemvimgnet # dataset name - mvimgnet or shapenet data_dir/xxx/MVImgNet_test/ # dataset directory img_size[160,90] vis_every5000 # how you often visualize intermediate results batch_size84 vis_every5000 n_iters30001 # number of iterations for training base_dir../../output/mae_finetuned # output parent directory expnamemae_mvimgnet # output directory using_mae_pretrainedFalse # whether you would use the pretrained model as initialization lr_encoder_init0.0001 # start lr rate (after warm up) lr_minimum0.000001 # final lr rate encoder_warmup_iters1000 # lr warmup until this iteration - to lr_encoder_init ckptFalse # encoder encoder_patch_size5 encoder_depth12 apply_minus_one_to_one_normFalse encoder_embed_dim768 encoder_num_heads12 # mae decoder mae_decoder_embed_dim512 mae_decoder_depth8 mae_decoder_num_heads16 masking_ratio0.75 # masking ratio for MAE using_mae_pretrainedTrue3.1.3. 训练accelerate launch --mixed_precisionfp16 scripts/train_mae.py --config configs/mvimgnet_mae.txt --apply_minus_one_to_one_norm False --expname mae_mvimgnet_imgnet实际测试单卡内存占用约为15GB3.2. 训练NViST支持多GPU训练。以下两条命令分别针对单卡和双卡训练其中设置的batch size输入到编码器的图像数量和 batch pixel sizes用于渲染的像素数量占用 40GB A100 GPUs。如果把batch size和batch pixel size增加到N倍则需把学习率增加到倍。3.2.1. 单卡训练CUDA_VISIBLE_DEVICES0 accelerate launch --mixed_precisionfp16 scripts/train_nvist.py --config configs/mvimgnet_nvist.txt\ --batch_size 11 --batch_pixel_size 165000 --expname nvist_mvimgnet_1gpu3.2.2. 双卡训练accelerate launch --mixed_precisionfp16 scripts/train_nvist.py --config configs/mvimgnet_nvist.txt\ --batch_size 22 --batch_pixel_size 330000 --expname nvist_mvimgnet_2gpus --lr_encoder_init 0.00006 --lr_decoder_init 0.0003 --lr_renderer_init 0.00033.3. 问题记录发生崩溃 报这样的错误AttributeError: AcceleratorState object has no attribute use_fp16这应该是代码中的bug把报错的行注释掉就行了。4. 推理CUDA_VISIBLE_DEVICES0 python scripts/eval_nvist.py --config config_path --ckpt_dir ckpt_path参考文献GitHub - wbjang/nvist_official: (CVPR 2024) NViST: In the wild New View Synthesis from a Single Image with Transformers