NerfBaselines

Nerf
Baselines

GitHub Paper Docs

NerfBaselines is a framework for evaluating and comparing existing NeRF methods. Currently, most official implementations use different dataset loaders, evaluation protocols, and metrics which renders the comparison of methods difficult. Therefore, this project aims to provide a unified interface for running and evaluating methods on different datasets in a consistent way using the same metrics. But instead of reimplementing the methods, we use the official implementations and wrap them so that they can be run easily using the same interface.

Mip-NeRF 360 

Mip-NeRF 360 is a collection of four indoor and five outdoor object-centric scenes. The camera trajectory is an orbit around the object with fixed elevation and radius. The test set takes each n-th frame of the trajectory as test views.

Method PSNR SSIM LPIPS (VGG) Time GPU mem.
COLMAP 16.67 0.445 0.590 2h 52m 55s 0.00 MB
garden 18.87 0.468 0.477 1h 35m 12s 0.00 MB
bicycle 18.29 0.352 0.644 3h 19m 55s 0.00 MB
flowers 14.50 0.257 0.634 1h 51m 21s 0.00 MB
treehill 15.73 0.340 0.726 1h 6m 41s 0.00 MB
stump 19.65 0.366 0.646 3h 6m 50s 0.00 MB
kitchen 17.35 0.497 0.575 3h 31m 29s 0.00 MB
bonsai 14.06 0.548 0.586 3h 45m 27s 0.00 MB
counter 15.04 0.549 0.520 3h 45m 10s 0.00 MB
room 16.55 0.628 0.505 3h 54m 6s 0.00 MB
Instant NGP 25.51 0.684 0.398 3m 54s 7.79 GB
garden 24.78 0.654 0.346 3m 57s 6.86 GB
bicycle 23.00 0.526 0.489 3m 37s 6.29 GB
flowers 20.54 0.462 0.478 4m 2s 6.26 GB
treehill 22.36 0.528 0.526 3m 40s 6.51 GB
stump 23.84 0.590 0.439 3m 52s 6.07 GB
kitchen 29.02 0.844 0.255 4m 19s 9.52 GB
bonsai 30.30 0.890 0.295 3m 33s 9.61 GB
counter 26.56 0.812 0.373 4m 10s 9.25 GB
room 29.16 0.850 0.383 3m 59s 9.73 GB
NerfStudio 26.39 0.731 0.343 19m 30s 5.86 GB
garden 25.89
Paper's PSNR: 26.47
0.752
Paper's SSIM: 0.774
0.254
Paper's LPIPS (VGG): 0.235
19m 31s 5.30 GB
bicycle 23.58
Paper's PSNR: 24.08
0.567
Paper's SSIM: 0.599
0.456
Paper's LPIPS (VGG): 0.422
19m 12s 5.15 GB
flowers 21.16 0.511 0.434 19m 6s 5.22 GB
treehill 22.85 0.549 0.488 19m 31s 5.24 GB
stump 25.81
Paper's PSNR: 24.78
0.697
Paper's SSIM: 0.662
0.353
Paper's LPIPS (VGG): 0.38
18m 53s 5.15 GB
kitchen 29.92
Paper's PSNR: 30.29
0.883
Paper's SSIM: 0.89
0.200
Paper's LPIPS (VGG): 0.19
20m 5s 6.66 GB
bonsai 30.59
Paper's PSNR: 32.16
0.907
Paper's SSIM: 0.933
0.249
Paper's LPIPS (VGG): 0.197
19m 46s 6.66 GB
counter 27.09
Paper's PSNR: 27.2
0.830
Paper's SSIM: 0.843
0.336
Paper's LPIPS (VGG): 0.314
19m 28s 6.66 GB
room 30.61
Paper's PSNR: 30.89
0.880
Paper's SSIM: 0.896
0.315
Paper's LPIPS (VGG): 0.296
19m 53s 6.66 GB
2D Gaussian Splatting 26.81
Paper's PSNR: 27.04

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.796
Paper's SSIM: 0.805

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.297 31m 10s 13.16 GB
garden 26.69
Paper's PSNR: 26.95

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.843
Paper's SSIM: 0.852

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.166 27m 56s 8.58 GB
bicycle 24.77
Paper's PSNR: 24.87

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.733
Paper's SSIM: 0.752

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.302 33m 18s 11.96 GB
flowers 21.14
Paper's PSNR: 21.15

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.572
Paper's SSIM: 0.588

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.403 26m 31s 7.64 GB
treehill 22.36
Paper's PSNR: 22.27

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.616
Paper's SSIM: 0.627

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.433 28m 44s 8.81 GB
stump 26.20
Paper's PSNR: 26.47

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.758
Paper's SSIM: 0.765

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.299 29m 12s 28.56 GB
kitchen 30.41
Paper's PSNR: 30.5

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.916
Paper's SSIM: 0.919

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.179 35m 31s 15.79 GB
bonsai 31.30
Paper's PSNR: 31.52

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.931
Paper's SSIM: 0.933

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.280 33m 16s 15.39 GB
counter 28.10
Paper's PSNR: 28.55

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.892
Paper's SSIM: 0.9

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.292 32m 38s 9.95 GB
room 30.37
Paper's PSNR: 31.06

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.906
Paper's SSIM: 0.912

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.317 33m 20s 11.74 GB
gsplat 27.41 0.815 0.256 29m 19s 8.30 GB
garden 27.49
Paper's PSNR: 27.39
0.868
Paper's SSIM: 0.87
0.120 39m 56s 12.52 GB
bicycle 25.26
Paper's PSNR: 25.29
0.766
Paper's SSIM: 0.77
0.237 35m 41s 11.75 GB
flowers 21.59 0.603 0.368 26m 2s 8.64 GB
treehill 22.57 0.635 0.377 24m 28s 7.79 GB
stump 26.57
Paper's PSNR: 26.51
0.772
Paper's SSIM: 0.77
0.248 26m 40s 9.63 GB
kitchen 30.86
Paper's PSNR: 31.37
0.923
Paper's SSIM: 0.93
0.158 30m 54s 6.79 GB
bonsai 32.13
Paper's PSNR: 32.21
0.941
Paper's SSIM: 0.94
0.254 23m 13s 5.69 GB
counter 28.94
Paper's PSNR: 29.01
0.906
Paper's SSIM: 0.91
0.257 28m 24s 5.79 GB
room 31.30
Paper's PSNR: 31.23
0.916
Paper's SSIM: 0.92
0.286 28m 30s 6.11 GB
Gaussian Opacity Fields 27.42 0.826 0.234 1h 3m 54s 28.44 GB
garden 27.55 0.874 0.115 1h 13m 32s 34.59 GB
bicycle 25.52 0.792 0.197 1h 11m 12s 37.49 GB
flowers 21.71 0.639 0.308 53m 26s 33.63 GB
treehill 22.44 0.643 0.320 57m 48s 33.73 GB
stump 26.99 0.798 0.223 52m 14s 33.82 GB
kitchen 31.23 0.925 0.158 1h 12m 30s 22.96 GB
bonsai 31.94 0.940 0.247 59m 18s 23.38 GB
counter 28.83 0.907 0.258 1h 5m 60s 12.64 GB
room 30.59 0.913 0.282 1h 9m 1s 23.70 GB
Gaussian Splatting 27.43
Paper's PSNR: 27.20
0.814
Paper's SSIM: 0.815
0.257
Paper's LPIPS (VGG): 0.214
23m 25s 11.14 GB
garden 27.34
Paper's PSNR: 27.41
0.866
Paper's SSIM: 0.868
0.124
Paper's LPIPS (VGG): 0.103
28m 58s 14.75 GB
bicycle 25.21
Paper's PSNR: 25.246
0.764
Paper's SSIM: 0.771
0.240
Paper's LPIPS (VGG): 0.205
28m 25s 14.90 GB
flowers 21.60
Paper's PSNR: 21.52
0.604
Paper's SSIM: 0.605
0.367
Paper's LPIPS (VGG): 0.336
20m 53s 10.01 GB
treehill 22.44
Paper's PSNR: 22.49
0.631
Paper's SSIM: 0.638
0.377
Paper's LPIPS (VGG): 0.317
20m 51s 11.36 GB
stump 26.58
Paper's PSNR: 26.55
0.771
Paper's SSIM: 0.775
0.250
Paper's LPIPS (VGG): 0.21
22m 30s 12.53 GB
kitchen 31.14
Paper's PSNR: 30.317
0.926
Paper's SSIM: 0.922
0.155
Paper's LPIPS (VGG): 0.129
25m 35s 9.75 GB
bonsai 32.20
Paper's PSNR: 31.98
0.941
Paper's SSIM: 0.938
0.254
Paper's LPIPS (VGG): 0.205
19m 15s 8.35 GB
counter 28.96
Paper's PSNR: 28.7
0.907
Paper's SSIM: 0.905
0.258
Paper's LPIPS (VGG): 0.204
21m 53s 8.55 GB
room 31.43
Paper's PSNR: 30.632
0.917
Paper's SSIM: 0.914
0.287
Paper's LPIPS (VGG): 0.22
22m 23s 10.05 GB
Mip-Splatting 27.49
Paper's PSNR: 27.79

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.815
Paper's SSIM: 0.827

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.258
Paper's LPIPS (VGG): 0.203

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 25m 37s 11.01 GB
garden 27.47
Paper's PSNR: 27.76

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.869
Paper's SSIM: 0.875

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.124
Paper's LPIPS (VGG): 0.103

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 30m 6s 13.45 GB
bicycle 25.25
Paper's PSNR: 25.72

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.765
Paper's SSIM: 0.78

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.243
Paper's LPIPS (VGG): 0.206

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 33m 19s 15.07 GB
flowers 21.60
Paper's PSNR: 21.93

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.605
Paper's SSIM: 0.623

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.371
Paper's LPIPS (VGG): 0.331

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 22m 47s 9.90 GB
treehill 22.65
Paper's PSNR: 22.98

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.633
Paper's SSIM: 0.655

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.381
Paper's LPIPS (VGG): 0.32

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 22m 49s 9.79 GB
stump 26.64
Paper's PSNR: 26.94

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.774
Paper's SSIM: 0.786

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.251
Paper's LPIPS (VGG): 0.209

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 25m 8s 12.49 GB
kitchen 31.25
Paper's PSNR: 31.55

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.926
Paper's SSIM: 0.933

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.155
Paper's LPIPS (VGG): 0.113

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 27m 47s 9.50 GB
bonsai 31.96
Paper's PSNR: 32.31

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.941
Paper's SSIM: 0.948

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.254
Paper's LPIPS (VGG): 0.173

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 20m 49s 8.98 GB
counter 29.04
Paper's PSNR: 29.16

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.907
Paper's SSIM: 0.916

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.258
Paper's LPIPS (VGG): 0.179

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 23m 21s 9.03 GB
room 31.54
Paper's PSNR: 31.74

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.918
Paper's SSIM: 0.928

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.286
Paper's LPIPS (VGG): 0.192

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 24m 25s 10.85 GB
3DGS-MCMC 27.57 0.798 0.281 35m 8s 21.58 GB
garden 27.81
Paper's PSNR: 28.16

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.877
Paper's SSIM: 0.89

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.109 48m 5s 35.23 GB
bicycle 25.69
Paper's PSNR: 26.15

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.799
Paper's SSIM: 0.81

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.194 49m 28s 38.87 GB
flowers 19.38 0.420 0.572 10m 20s 5.53 GB
treehill 21.94 0.550 0.567 10m 16s 5.58 GB
stump 27.38
Paper's PSNR: 27.8

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.811
Paper's SSIM: 0.82

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.200 40m 48s 34.43 GB
kitchen 31.91
Paper's PSNR: 32.27

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.933
Paper's SSIM: 0.94

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.148 42m 32s 20.35 GB
bonsai 32.66
Paper's PSNR: 32.88

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.947
Paper's SSIM: 0.95

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.236 35m 14s 14.88 GB
counter 29.32
Paper's PSNR: 29.51

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.916
Paper's SSIM: 0.92

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.238 38m 44s 20.42 GB
room 32.05
Paper's PSNR: 32.48

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.927
Paper's SSIM: 0.94

Authors evaluated on larger images which were downscaled to the target size (avoiding JPEG compression artifacts) instead of using the official provided downscaled images. As mentioned in the 3DGS paper, this increases results slightly ~0.5 dB PSNR.

 0.262 40m 43s 18.96 GB
Mip-NeRF 360 27.68
Paper's PSNR: 27.69
0.792
Paper's SSIM: 0.792
0.272
Paper's LPIPS (VGG): 0.237
30h 14m 36s 33.61 GB
garden 27.00
Paper's PSNR: 26.98
0.814
Paper's SSIM: 0.813
0.189
Paper's LPIPS (VGG): 0.17
30h 13m 41s 33.06 GB
bicycle 24.28
Paper's PSNR: 24.37
0.686
Paper's SSIM: 0.685
0.329
Paper's LPIPS (VGG): 0.301
30h 14m 19s 32.86 GB
flowers 21.74
Paper's PSNR: 21.73
0.583
Paper's SSIM: 0.583
0.371
Paper's LPIPS (VGG): 0.344
30h 13m 53s 32.78 GB
treehill 22.89
Paper's PSNR: 22.87
0.633
Paper's SSIM: 0.632
0.378
Paper's LPIPS (VGG): 0.339
30h 13m 9s 32.98 GB
stump 26.50
Paper's PSNR: 26.4
0.749
Paper's SSIM: 0.744
0.298
Paper's LPIPS (VGG): 0.261
30h 15m 5s 32.88 GB
kitchen 32.10
Paper's PSNR: 32.23
0.919
Paper's SSIM: 0.92
0.155
Paper's LPIPS (VGG): 0.127
30h 15m 15s 34.48 GB
bonsai 33.48
Paper's PSNR: 33.46
0.940
Paper's SSIM: 0.941
0.211
Paper's LPIPS (VGG): 0.176
30h 13m 6s 34.48 GB
counter 29.51
Paper's PSNR: 29.55
0.894
Paper's SSIM: 0.894
0.252
Paper's LPIPS (VGG): 0.204
30h 14m 4s 34.47 GB
room 31.64
Paper's PSNR: 31.63
0.912
Paper's SSIM: 0.913
0.267
Paper's LPIPS (VGG): 0.211
30h 18m 49s 34.47 GB
Scaffold-GS 27.71 0.813 0.262 23m 28s 8.72 GB
garden 27.50
Paper's PSNR: 27.17
0.863
Paper's SSIM: 0.842
0.136 21m 46s 7.91 GB
bicycle 25.19
Paper's PSNR: 24.5
0.759
Paper's SSIM: 0.705
0.259 20m 38s 7.37 GB
flowers 21.44 0.592 0.382 20m 23s 6.90 GB
treehill 23.15 0.640 0.373 20m 17s 6.48 GB
stump 26.59
Paper's PSNR: 26.27
0.766
Paper's SSIM: 0.784
0.277 17m 1s 6.22 GB
kitchen 31.59
Paper's PSNR: 31.3
0.927
Paper's SSIM: 0.928
0.156 33m 25s 10.97 GB
bonsai 32.58
Paper's PSNR: 32.7
0.943
Paper's SSIM: 0.946
0.249 24m 33s 10.94 GB
counter 29.48
Paper's PSNR: 29.34
0.910
Paper's SSIM: 0.914
0.256 28m 34s 10.38 GB
room 31.89
Paper's PSNR: 31.93
0.922
Paper's SSIM: 0.925
0.275 24m 37s 11.28 GB
Zip-NeRF 28.55
Paper's PSNR: 28.54
0.829
Paper's SSIM: 0.828
0.218
Paper's LPIPS (VGG): 0.189
5h 30m 20s 26.82 GB
garden 28.19
Paper's PSNR: 28.2
0.864
Paper's SSIM: 0.86
0.127
Paper's LPIPS (VGG): 0.118
5h 19m 14s 26.77 GB
bicycle 25.89
Paper's PSNR: 25.8
0.775
Paper's SSIM: 0.769
0.226
Paper's LPIPS (VGG): 0.208
5h 22m 47s 26.74 GB
flowers 22.33
Paper's PSNR: 22.4
0.637
Paper's SSIM: 0.642
0.310
Paper's LPIPS (VGG): 0.273
5h 16m 23s 26.75 GB
treehill 24.01
Paper's PSNR: 23.89
0.677
Paper's SSIM: 0.681
0.279
Paper's LPIPS (VGG): 0.242
5h 48m 6s 26.75 GB
stump 27.41
Paper's PSNR: 27.55
0.792
Paper's SSIM: 0.8
0.233
Paper's LPIPS (VGG): 0.193
5h 25m 16s 26.74 GB
kitchen 32.35
Paper's PSNR: 32.5
0.929
Paper's SSIM: 0.928
0.134
Paper's LPIPS (VGG): 0.116
5h 39m 34s 26.92 GB
bonsai 34.68
Paper's PSNR: 34.46
0.951
Paper's SSIM: 0.949
0.195
Paper's LPIPS (VGG): 0.173
5h 35m 52s 26.92 GB
counter 29.13
Paper's PSNR: 29.38
0.905
Paper's SSIM: 0.902
0.223
Paper's LPIPS (VGG): 0.185
5h 39m 53s 26.91 GB
room 32.99
Paper's PSNR: 32.65
0.928
Paper's SSIM: 0.925
0.237
Paper's LPIPS (VGG): 0.196
5h 25m 53s 26.91 GB

Blender 

Blender (nerf-synthetic) is a synthetic dataset used to benchmark NeRF methods. It consists of 8 scenes of an object placed on a white background. Cameras are placed on a semi-sphere around the object. Scenes are licensed under various CC licenses.

Method PSNR SSIM LPIPS (VGG) Time GPU mem.
COLMAP 12.12 0.766 0.214 1h 20m 34s 0.00 MB
lego 12.31 0.776 0.220 2h 5m 25s 0.00 MB
drums 8.06 0.657 0.296 2h 14m 38s 0.00 MB
ficus 15.12 0.838 0.143 39m 23s 0.00 MB
hotdog 12.21 0.831 0.195 1h 13m 26s 0.00 MB
materials 14.56 0.802 0.193 1h 42m 37s 0.00 MB
mic 9.29 0.765 0.182 1h 13m 24s 0.00 MB
ship 10.02 0.616 0.335 48m 3s 0.00 MB
chair 15.42 0.847 0.148 47m 35s 0.00 MB
NeRF 28.72
Paper's PSNR: 31.00
0.936
Paper's SSIM: 0.947
0.092
Paper's LPIPS (VGG): 0.081
23h 26m 30s 10.25 GB
lego 32.86
Paper's PSNR: 32.54
0.964
Paper's SSIM: 0.961
0.049
Paper's LPIPS (VGG): 0.05
23h 32m 41s 10.25 GB
drums 25.12
Paper's PSNR: 25.01
0.926
Paper's SSIM: 0.925
0.087
Paper's LPIPS (VGG): 0.091
22h 56m 29s 10.25 GB
ficus 30.31
Paper's PSNR: 30.13
0.964
Paper's SSIM: 0.964
0.042
Paper's LPIPS (VGG): 0.044
23h 38m 35s 10.25 GB
hotdog 36.39
Paper's PSNR: 36.18
0.975
Paper's SSIM: 0.974
0.095
Paper's LPIPS (VGG): 0.121
23h 14m 14s 10.25 GB
materials 29.84
Paper's PSNR: 29.62
0.950
Paper's SSIM: 0.949
0.062
Paper's LPIPS (VGG): 0.063
23h 58m 42s 10.25 GB
mic 13.03
Paper's PSNR: 32.91
0.881
Paper's SSIM: 0.98
0.154
Paper's LPIPS (VGG): 0.028
23h 21m 46s 10.25 GB
ship 28.80
Paper's PSNR: 28.65
0.859
Paper's SSIM: 0.856
0.210
Paper's LPIPS (VGG): 0.206
23h 30m 51s 10.25 GB
chair 33.43
Paper's PSNR: 33.0
0.970
Paper's SSIM: 0.967
0.041
Paper's LPIPS (VGG): 0.046
23h 18m 37s 10.25 GB
NerfStudio 29.19 0.941 0.095 9m 38s 3.65 GB
lego 31.37 0.967 0.069 9m 44s 3.65 GB
drums 22.48 0.897 0.139 9m 12s 3.65 GB
ficus 27.82 0.957 0.087 9m 22s 3.65 GB
hotdog 31.09 0.963 0.104 9m 44s 3.65 GB
materials 25.38 0.903 0.121 9m 23s 3.65 GB
mic 33.74 0.984 0.029 9m 47s 3.65 GB
ship 28.71 0.881 0.166 10m 2s 3.64 GB
chair 32.94 0.977 0.044 9m 48s 3.65 GB
Mip-NeRF 360 30.34 0.951 0.060 3h 29m 39s 114.80 GB
lego 33.20 0.975 0.028 2h 43m 18s 126.91 GB
drums 24.36 0.923 0.083 8h 51m 32s 30.02 GB
ficus 26.66 0.952 0.048 2h 43m 27s 126.91 GB
hotdog 36.44 0.979 0.039 2h 43m 11s 126.91 GB
materials 27.91 0.944 0.067 2h 43m 22s 126.91 GB
mic 31.50 0.984 0.021 2h 43m 9s 126.91 GB
ship 28.66 0.875 0.164 2h 45m 55s 126.91 GB
chair 34.01 0.977 0.032 2h 43m 17s 126.91 GB
gsplat 31.47 0.966 0.054 14m 45s 2.80 GB
lego 32.13 0.977 0.043 13m 31s 2.94 GB
drums 24.50 0.948 0.076 11m 15s 2.74 GB
ficus 34.44 0.986 0.014 11m 48s 2.75 GB
hotdog 35.39 0.983 0.041 20m 32s 2.67 GB
materials 29.93 0.959 0.044 11m 37s 2.76 GB
mic 33.73 0.991 0.023 11m 30s 2.84 GB
ship 28.77 0.899 0.155 17m 56s 2.86 GB
chair 32.88 0.986 0.034 19m 53s 2.82 GB
Tetra-NeRF 31.95
Paper's PSNR: 32.52
0.957
Paper's SSIM: 0.982
0.056 6h 53m 20s 29.57 GB
lego 33.93
Paper's PSNR: 34.75
0.972
Paper's SSIM: 0.987
0.036 7h 8m 12s 25.21 GB
drums 24.99
Paper's PSNR: 25.01
0.927
Paper's SSIM: 0.947
0.087 8h 34m 5s 36.26 GB
ficus 32.37
Paper's PSNR: 33.31
0.977
Paper's SSIM: 0.989
0.032 4h 19m 24s 21.84 GB
hotdog 35.80
Paper's PSNR: 36.16
0.978
Paper's SSIM: 0.989
0.040 6h 41m 21s 23.06 GB
materials 28.75
Paper's PSNR: 29.3
0.941
Paper's SSIM: 0.968
0.076 6h 23m 31s 37.98 GB
mic 34.54
Paper's PSNR: 35.49
0.987
Paper's SSIM: 0.993
0.022 6h 51m 58s 31.04 GB
ship 31.06
Paper's PSNR: 31.13
0.896
Paper's SSIM: 0.994
0.129 9h 4m 53s 27.19 GB
chair 34.17
Paper's PSNR: 35.05
0.977
Paper's SSIM: 0.99
0.029 6h 3m 14s 34.01 GB
Instant NGP 32.20
Paper's PSNR: 33.18

Instant-NGP trained and evaluated on black background instead of white.

 0.959 0.055 2m 23s 2.57 GB
lego 35.65
Paper's PSNR: 36.39

Instant-NGP trained and evaluated on black background instead of white.

 0.981 0.020 2m 41s 2.59 GB
drums 24.57
Paper's PSNR: 26.02

Instant-NGP trained and evaluated on black background instead of white.

 0.930 0.109 2m 6s 2.58 GB
ficus 30.29
Paper's PSNR: 33.51

Instant-NGP trained and evaluated on black background instead of white.

 0.972 0.031 1m 58s 2.58 GB
hotdog 37.02
Paper's PSNR: 37.4

Instant-NGP trained and evaluated on black background instead of white.

 0.982 0.037 2m 33s 2.58 GB
materials 28.96
Paper's PSNR: 29.78

Instant-NGP trained and evaluated on black background instead of white.

 0.944 0.069 2m 11s 2.58 GB
mic 35.41
Paper's PSNR: 36.22

Instant-NGP trained and evaluated on black background instead of white.

 0.989 0.016 2m 25s 2.47 GB
ship 30.61
Paper's PSNR: 31.1

Instant-NGP trained and evaluated on black background instead of white.

 0.892 0.136 2m 24s 2.58 GB
chair 35.07
Paper's PSNR: 35.0

Instant-NGP trained and evaluated on black background instead of white.

 0.984 0.023 2m 49s 2.57 GB
K-Planes 32.27 0.961 0.062 23m 58s 4.64 GB
lego 35.73 0.981 0.047 23m 54s 4.64 GB
drums 25.68 0.938 0.096 24m 10s 4.64 GB
ficus 31.31 0.974 0.052 23m 37s 4.64 GB
hotdog 36.46 0.981 0.033 23m 38s 4.64 GB
materials 29.41 0.949 0.070 24m 1s 4.64 GB
mic 33.95 0.988 0.019 24m 6s 4.64 GB
ship 30.72 0.897 0.141 24m 8s 4.64 GB
chair 34.85 0.983 0.036 24m 9s 4.64 GB
3DGS-MCMC 33.07
Paper's PSNR: 33.80

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.969
Paper's SSIM: 0.970

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.040 6m 13s 3.89 GB
lego 34.40
Paper's PSNR: 36.01

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.979
Paper's SSIM: 0.98

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.026 5m 44s 3.82 GB
drums 26.03
Paper's PSNR: 26.29

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.953
Paper's SSIM: 0.95

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.049 5m 41s 3.80 GB
ficus 34.54
Paper's PSNR: 35.07

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.987
Paper's SSIM: 0.99

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.014 5m 9s 3.71 GB
hotdog 37.35
Paper's PSNR: 37.82

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.986
Paper's SSIM: 0.99

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.026 8m 26s 4.18 GB
materials 30.09
Paper's PSNR: 30.59

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.962
Paper's SSIM: 0.96

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.044 6m 12s 3.89 GB
mic 36.10
Paper's PSNR: 37.29

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.992
Paper's SSIM: 0.99

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.008 5m 43s 3.92 GB
ship 30.59
Paper's PSNR: 30.82

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.906
Paper's SSIM: 0.91

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.134 7m 11s 3.93 GB
chair 35.45
Paper's PSNR: 36.51

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.986
Paper's SSIM: 0.99

Exact hyperparameters for Blender dataset are not provided in the released source code. The default parameters were used in NerfBaselines likely leading to worse results.

 0.020 5m 41s 3.86 GB
Scaffold-GS 33.08
Paper's PSNR: 33.68
0.966 0.048 7m 4s 3.72 GB
lego 34.96
Paper's PSNR: 35.69
0.980 0.024 6m 40s 3.72 GB
drums 26.28
Paper's PSNR: 26.44
0.949 0.054 6m 46s 3.73 GB
ficus 34.38
Paper's PSNR: 35.21
0.986 0.015 6m 9s 3.69 GB
hotdog 37.62
Paper's PSNR: 37.73
0.983 0.075 8m 26s 3.71 GB
materials 30.29
Paper's PSNR: 30.65
0.962 0.045 6m 49s 3.73 GB
mic 35.99
Paper's PSNR: 37.25
0.991 0.010 6m 31s 3.71 GB
ship 29.97
Paper's PSNR: 31.17
0.895 0.139 8m 30s 3.74 GB
chair 35.16
Paper's PSNR: 35.28
0.984 0.019 6m 38s 3.71 GB
TensoRF 33.17
Paper's PSNR: 33.14
0.963
Paper's SSIM: 0.963
0.051
Paper's LPIPS (VGG): 0.047
10m 47s 16.37 GB
lego 36.49
Paper's PSNR: 36.46
0.983
Paper's SSIM: 0.983
0.022
Paper's LPIPS (VGG): 0.018
10m 1s 20.53 GB
drums 26.01
Paper's PSNR: 26.01
0.936
Paper's SSIM: 0.937
0.076
Paper's LPIPS (VGG): 0.073
8m 56s 19.67 GB
ficus 34.06
Paper's PSNR: 33.99
0.982
Paper's SSIM: 0.982
0.029
Paper's LPIPS (VGG): 0.022
10m 45s 7.28 GB
hotdog 37.49
Paper's PSNR: 37.41
0.982
Paper's SSIM: 0.982
0.033
Paper's LPIPS (VGG): 0.032
11m 18s 19.77 GB
materials 30.08
Paper's PSNR: 30.12
0.952
Paper's SSIM: 0.952
0.059
Paper's LPIPS (VGG): 0.058
14m 55s 21.09 GB
mic 34.85
Paper's PSNR: 34.61
0.988
Paper's SSIM: 0.988
0.021
Paper's LPIPS (VGG): 0.015
8m 8s 11.21 GB
ship 30.69
Paper's PSNR: 30.77
0.894
Paper's SSIM: 0.895
0.141
Paper's LPIPS (VGG): 0.138
13m 19s 18.80 GB
chair 35.72
Paper's PSNR: 35.76
0.984
Paper's SSIM: 0.985
0.027
Paper's LPIPS (VGG): 0.022
8m 56s 12.60 GB
Gaussian Splatting 33.31
Paper's PSNR: 33.31
0.969 0.037 6m 6s 3.08 GB
lego 35.70
Paper's PSNR: 35.78
0.982 0.019 5m 59s 3.07 GB
drums 26.15
Paper's PSNR: 26.15
0.953 0.044 6m 3s 3.07 GB
ficus 34.79
Paper's PSNR: 34.87
0.987 0.013 5m 22s 2.82 GB
hotdog 37.64
Paper's PSNR: 37.72
0.985 0.026 5m 49s 2.85 GB
materials 30.01
Paper's PSNR: 30.0
0.959 0.043 5m 28s 2.91 GB
mic 35.49
Paper's PSNR: 35.36
0.991 0.008 6m 11s 3.02 GB
ship 30.85
Paper's PSNR: 30.8
0.904 0.130 8m 13s 3.99 GB
chair 35.84
Paper's PSNR: 35.83
0.987 0.015 5m 45s 2.89 GB
Mip-Splatting 33.33 0.969 0.039 6m 49s 2.65 GB
lego 35.45 0.982 0.021 6m 30s 2.51 GB
drums 26.14 0.953 0.046 6m 36s 2.53 GB
ficus 35.12 0.988 0.013 5m 8s 2.43 GB
hotdog 37.78 0.985 0.028 6m 44s 2.58 GB
materials 30.12 0.960 0.044 5m 49s 2.46 GB
mic 35.55 0.991 0.008 8m 8s 2.75 GB
ship 30.78 0.904 0.132 9m 19s 3.54 GB
chair 35.70 0.986 0.018 6m 16s 2.40 GB
Gaussian Opacity Fields 33.45 0.969 0.038 18m 26s 3.15 GB
lego 35.56 0.982 0.021 19m 4s 2.88 GB
drums 26.17 0.955 0.045 17m 50s 2.84 GB
ficus 35.19 0.988 0.013 11m 10s 2.71 GB
hotdog 37.46 0.985 0.028 17m 52s 2.61 GB
materials 30.20 0.961 0.043 14m 48s 2.72 GB
mic 36.06 0.992 0.008 21m 56s 2.99 GB
ship 30.68 0.901 0.131 29m 60s 5.64 GB
chair 36.28 0.988 0.017 14m 46s 2.79 GB
Zip-NeRF 33.67
Paper's PSNR: 33.09
0.973
Paper's SSIM: 0.971
0.036
Paper's LPIPS (VGG): 0.031
5h 21m 57s 26.20 GB
lego 35.81
Paper's PSNR: 34.84
0.983
Paper's SSIM: 0.98
0.019
Paper's LPIPS (VGG): 0.019
5h 20m 18s 26.20 GB
drums 25.90
Paper's PSNR: 25.84
0.948
Paper's SSIM: 0.944
0.054
Paper's LPIPS (VGG): 0.05
5h 20m 22s 26.20 GB
ficus 34.73
Paper's PSNR: 33.9
0.987
Paper's SSIM: 0.985
0.014
Paper's LPIPS (VGG): 0.015
5h 25m 57s 26.20 GB
hotdog 37.98
Paper's PSNR: 37.14
0.987
Paper's SSIM: 0.984
0.023
Paper's LPIPS (VGG): 0.02
5h 19m 2s 26.20 GB
materials 30.98
Paper's PSNR: 31.66
0.967
Paper's SSIM: 0.969
0.040
Paper's LPIPS (VGG): 0.032
5h 23m 48s 26.20 GB
mic 35.90
Paper's PSNR: 35.15
0.992
Paper's SSIM: 0.991
0.008
Paper's LPIPS (VGG): 0.007
5h 26m 19s 26.20 GB
ship 32.30
Paper's PSNR: 31.38
0.937
Paper's SSIM: 0.929
0.114
Paper's LPIPS (VGG): 0.091
5h 14m 50s 26.20 GB
chair 35.75
Paper's PSNR: 34.84
0.987
Paper's SSIM: 0.983
0.017
Paper's LPIPS (VGG): 0.017
5h 24m 56s 26.20 GB

Tanks and Temples 

Tanks and Temples is a benchmark for image-based 3D reconstruction. The benchmark sequences were acquired outside the lab, in realistic conditions. Ground-truth data was captured using an industrial laser scanner. The benchmark includes both outdoor scenes and indoor environments. The dataset is split into three subsets: training, intermediate, and advanced.

Method PSNR SSIM LPIPS Time GPU mem.
COLMAP 11.92 0.436 0.606 5h 16m 11s 0.00 MB
auditorium 14.55 0.390 0.637 3h 17m 31s 0.00 MB
ballroom 14.09 0.408 0.466 3h 34m 34s 0.00 MB
courtroom 14.48 0.371 0.537 3h 24m 16s 0.00 MB
museum 14.16 0.418 0.495 3h 18m 47s 0.00 MB
palace 9.08 0.407 0.784 3h 8m 22s 0.00 MB
temple 7.32 0.408 0.617 3h 5m 14s 0.00 MB
family 10.28 0.440 0.550 2h 33m 22s 0.00 MB
francis 8.33 0.302 0.686 2h 40m 27s 0.00 MB
horse 6.35 0.404 0.563 2h 41m 33s 0.00 MB
lighthouse 10.68 0.514 0.665 2h 46m 8s 0.00 MB
m60 14.19 0.547 0.550 10h 54m 40s 0.00 MB
panther 14.78 0.594 0.524 10h 32m 2s 0.00 MB
playground 13.30 0.455 0.615 10h 52m 56s 0.00 MB
train 11.83 0.396 0.813 9h 18m 35s 0.00 MB
barn 8.97 0.479 0.520 8h 15m 20s 0.00 MB
caterpillar 12.67 0.376 0.771 3h 20m 10s 0.00 MB
church 15.67 0.425 0.514 9h 2m 32s 0.00 MB
courthouse 9.09 0.397 0.737 9h 30m 56s 0.00 MB
ignatius 12.74 0.400 0.551 2h 54m 26s 0.00 MB
meetingroom 14.38 0.509 0.591 2h 48m 23s 0.00 MB
truck 13.35 0.507 0.545 2h 39m 37s 0.00 MB
2D Gaussian Splatting 21.54 0.768 0.281 15m 47s 7.23 GB
auditorium 22.86 0.835 0.304 13m 45s 6.55 GB
ballroom 23.40 0.808 0.120 19m 53s 6.12 GB
courtroom 18.92 0.660 0.422 15m 12s 9.39 GB
museum 21.11 0.764 0.170 26m 10s 8.09 GB
palace 16.91 0.668 0.512 13m 40s 5.50 GB
temple 19.74 0.763 0.305 13m 46s 4.06 GB
family 24.14 0.854 0.111 16m 16s 4.32 GB
francis 26.50 0.884 0.202 14m 56s 9.51 GB
horse 23.21 0.867 0.122 15m 15s 7.43 GB
lighthouse 18.00 0.707 0.438 13m 3s 8.47 GB
m60 22.83 0.825 0.249 15m 2s 8.98 GB
panther 21.70 0.771 0.358 13m 40s 6.39 GB
playground 21.45 0.745 0.316 16m 51s 7.16 GB
train 16.41 0.583 0.529 11m 43s 4.77 GB
barn 25.42
Paper's PSNR: 28.79

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.816 0.213 13m 36s 6.29 GB
caterpillar 22.24
Paper's PSNR: 24.23

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.761 0.227 14m 3s 7.15 GB
church 18.50 0.658 0.443 16m 11s 14.63 GB
courthouse 19.48
Paper's PSNR: 23.51

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.707 0.398 13m 18s 8.98 GB
ignatius 21.34
Paper's PSNR: 23.82

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.762 0.176 18m 57s 5.54 GB
meetingroom 24.26
Paper's PSNR: 26.15

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.849 0.168 20m 5s 7.41 GB
truck 23.83
Paper's PSNR: 26.85

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.843 0.123 15m 54s 4.99 GB
Instant NGP 21.62 0.712 0.340 4m 27s 4.13 GB
auditorium 20.67 0.761 0.429 4m 31s 3.92 GB
ballroom 21.62 0.652 0.352 4m 5s 4.02 GB
courtroom 19.44 0.640 0.448 4m 35s 3.93 GB
museum 15.19 0.471 0.606 5m 53s 3.93 GB
palace 19.09 0.668 0.440 4m 46s 4.64 GB
temple 17.84 0.689 0.424 4m 40s 3.94 GB
family 22.59 0.761 0.235 3m 42s 3.42 GB
francis 24.38 0.824 0.265 4m 10s 3.94 GB
horse 21.82 0.784 0.225 3m 47s 3.42 GB
lighthouse 21.65 0.765 0.281 4m 35s 4.02 GB
m60 25.82 0.832 0.202 4m 34s 4.04 GB
panther 26.81 0.844 0.208 4m 29s 4.04 GB
playground 23.33 0.696 0.344 4m 14s 3.97 GB
train 20.01 0.658 0.334 4m 39s 3.94 GB
barn 25.90 0.772 0.271 4m 41s 4.31 GB
caterpillar 21.72 0.633 0.360 4m 30s 4.22 GB
church 19.92 0.650 0.419 4m 21s 4.64 GB
courthouse 20.80 0.681 0.414 4m 55s 6.72 GB
ignatius 19.40 0.613 0.343 3m 58s 3.81 GB
meetingroom 23.24 0.783 0.326 3m 59s 4.18 GB
truck 22.85 0.770 0.216 4m 20s 3.77 GB
NerfStudio 22.04 0.743 0.270 19m 27s 3.74 GB
auditorium 20.77 0.771 0.330 19m 46s 3.88 GB
ballroom 22.68 0.705 0.261 19m 44s 3.88 GB
courtroom 20.24 0.673 0.336 19m 17s 3.88 GB
museum 17.84 0.648 0.311 18m 49s 3.88 GB
palace 17.68 0.640 0.452 20m 9s 3.64 GB
temple 17.06 0.678 0.392 19m 37s 3.88 GB
family 24.32 0.822 0.158 18m 54s 3.63 GB
francis 24.60 0.851 0.190 18m 48s 3.63 GB
horse 24.31 0.847 0.139 18m 53s 3.88 GB
lighthouse 20.85 0.768 0.245 19m 33s 3.89 GB
m60 26.54 0.843 0.179 19m 47s 3.64 GB
panther 27.57 0.858 0.174 20m 10s 3.89 GB
playground 24.69 0.755 0.249 19m 33s 3.64 GB
train 20.43 0.693 0.261 19m 42s 3.88 GB
barn 26.40 0.794 0.215 19m 16s 3.63 GB
caterpillar 21.71 0.666 0.302 19m 52s 3.63 GB
church 20.06 0.671 0.338 19m 29s 3.63 GB
courthouse 18.11 0.632 0.465 19m 37s 3.63 GB
ignatius 20.44 0.689 0.251 19m 5s 3.63 GB
meetingroom 23.21 0.793 0.261 19m 3s 3.63 GB
truck 23.37 0.797 0.167 19m 28s 3.63 GB
Gaussian Opacity Fields 22.39 0.825 0.172 N/A N/A
auditorium 23.20 0.871 0.194 - -
ballroom 22.84 0.818 0.107 - -
courtroom 21.15 0.781 0.168 - -
museum 19.92 0.761 0.152 - -
palace 16.46 0.683 0.443 - -
temple 20.29 0.794 0.234 - -
family 22.31 0.875 0.084 41m 32s 35.16 GB
francis 24.76 0.901 0.158 35m 52s 12.30 GB
horse 23.73 0.881 0.092 38m 29s 24.41 GB
lighthouse 21.80 0.833 0.181 - -
m60 28.04 0.906 0.104 40m 9s 26.18 GB
panther 28.47 0.910 0.102 40m 13s 30.99 GB
playground 23.89 0.869 0.142 - -
train 19.69 0.796 0.164 - -
barn 25.72 0.866 0.140 - -
caterpillar 21.78 0.791 0.187 - -
church 19.65 0.775 0.208 48m 2s 36.45 GB
courthouse 19.60 0.726 0.346 35m 12s 12.82 GB
ignatius 20.34 0.769 0.162 43m 50s 34.97 GB
meetingroom 24.31 0.862 0.140 40m 29s 23.30 GB
truck 22.33 0.860 0.099 - -
Gaussian Splatting 23.83 0.831 0.165 13m 48s 6.95 GB
auditorium 24.13 0.871 0.193 10m 54s 4.98 GB
ballroom 24.07 0.824 0.101 19m 55s 9.08 GB
courtroom 23.12 0.790 0.165 17m 12s 9.02 GB
museum 20.92 0.764 0.160 20m 55s 11.38 GB
palace 19.63 0.736 0.350 11m 21s 5.85 GB
temple 20.85 0.806 0.222 10m 56s 5.13 GB
family 24.43 0.865 0.095 14m 60s 6.31 GB
francis 27.22 0.897 0.169 10m 20s 4.28 GB
horse 23.82 0.875 0.103 11m 35s 4.50 GB
lighthouse 22.11 0.843 0.156 12m 3s 6.09 GB
m60 27.84 0.901 0.113 13m 18s 6.80 GB
panther 28.32 0.908 0.107 13m 19s 7.01 GB
playground 25.37 0.848 0.170 15m 33s 7.44 GB
train 21.67 0.791 0.171 11m 56s 5.39 GB
barn 27.51 0.852 0.160 11m 9s 6.22 GB
caterpillar 23.38 0.791 0.190 11m 32s 5.89 GB
church 22.79 0.811 0.177 17m 15s 8.36 GB
courthouse 22.22 0.779 0.266 11m 8s 10.34 GB
ignatius 21.53 0.776 0.153 16m 36s 8.82 GB
meetingroom 25.19 0.866 0.141 12m 39s 5.50 GB
truck 24.25 0.853 0.108 15m 18s 7.47 GB
Mip-Splatting 23.93 0.833 0.166 15m 56s 7.27 GB
auditorium 24.41 0.872 0.196 11m 47s 4.82 GB
ballroom 24.15 0.826 0.098 25m 4s 9.76 GB
courtroom 23.00 0.791 0.165 20m 46s 9.27 GB
museum 20.88 0.768 0.158 24m 55s 12.57 GB
palace 19.63 0.731 0.354 12m 53s 6.48 GB
temple 20.55 0.805 0.226 12m 3s 5.93 GB
family 24.55 0.872 0.095 15m 44s 6.99 GB
francis 27.61 0.899 0.172 11m 47s 4.40 GB
horse 23.94 0.879 0.104 12m 47s 4.58 GB
lighthouse 22.25 0.844 0.159 13m 47s 7.01 GB
m60 27.98 0.904 0.112 15m 13s 7.13 GB
panther 28.27 0.908 0.109 15m 43s 7.38 GB
playground 25.87 0.861 0.155 18m 26s 8.30 GB
train 21.82 0.795 0.172 13m 26s 5.65 GB
barn 27.75 0.855 0.161 12m 42s 6.12 GB
caterpillar 23.42 0.790 0.197 13m 16s 5.69 GB
church 22.76 0.812 0.176 19m 53s 8.37 GB
courthouse 22.15 0.779 0.265 13m 57s 10.28 GB
ignatius 21.73 0.780 0.159 18m 44s 8.46 GB
meetingroom 25.46 0.870 0.137 14m 13s 5.73 GB
truck 24.36 0.857 0.108 17m 32s 7.68 GB
Zip-NeRF 24.63 0.840 0.131 5h 44m 9s 26.61 GB
auditorium 24.52 0.877 0.153 5h 24m 16s 26.61 GB
ballroom 25.45 0.835 0.113 5h 25m 1s 26.61 GB
courtroom 22.17 0.790 0.153 5h 17m 54s 26.61 GB
museum 19.34 0.746 0.159 5h 24m 17s 26.61 GB
palace 19.11 0.718 0.317 6h 42m 11s 26.61 GB
temple 20.58 0.805 0.183 5h 37m 11s 26.61 GB
family 27.10 0.889 0.067 5h 32m 15s 26.61 GB
francis 29.10 0.915 0.106 5h 34m 11s 26.61 GB
horse 26.82 0.897 0.069 5h 55m 60s 26.61 GB
lighthouse 23.07 0.849 0.131 5h 52m 55s 26.63 GB
m60 29.01 0.912 0.076 6h 3m 19s 26.63 GB
panther 28.76 0.909 0.081 5h 36m 45s 26.63 GB
playground 27.13 0.880 0.095 5h 29m 4s 26.61 GB
train 22.19 0.814 0.119 6h 14m 13s 26.61 GB
barn 29.26 0.884 0.083 5h 28m 43s 26.61 GB
caterpillar 23.94 0.802 0.152 5h 53m 16s 26.61 GB
church 23.14 0.807 0.153 6h 4m 32s 26.61 GB
courthouse 22.88 0.780 0.218 6h 4m 6s 26.61 GB
ignatius 22.61 0.789 0.127 5h 48m 32s 26.61 GB
meetingroom 25.93 0.875 0.121 5h 21m 9s 26.61 GB
truck 25.09 0.864 0.081 5h 37m 11s 26.61 GB

LLFF 

LLFF is a dataset of forward-facing scenes with a small variation in camera pose. NeRF methods usually use NDC-space parametrization for the scene representation.

Method PSNR SSIM LPIPS (VGG) Time GPU mem.
Mip-NeRF 360 26.59 0.846 0.168 9h 55m 23s 104.98 GB
Fern 24.59 0.820 0.210 7h 29m 8s 129.06 GB
Flower 27.56 0.867 0.140 14h 2m 26s 64.85 GB
Fortress 31.34 0.900 0.117 14h 2m 6s 64.85 GB
Horns 28.51 0.909 0.124 7h 26m 41s 129.06 GB
Leaves 19.84 0.721 0.231 7h 26m 52s 129.06 GB
Orchids 19.51 0.660 0.246 7h 27m 12s 129.06 GB
Room 33.49 0.965 0.115 7h 26m 22s 129.06 GB
Trex 27.86 0.927 0.158 14h 2m 15s 64.85 GB
TensoRF 26.68
Paper's PSNR: 26.73
0.834
Paper's SSIM: 0.839
0.202
Paper's LPIPS (VGG): 0.204
31m 32s 20.38 GB
Fern 25.08
Paper's PSNR: 25.27
0.801
Paper's SSIM: 0.814
0.246
Paper's LPIPS (VGG): 0.237
36m 6s 18.28 GB
Flower 28.38
Paper's PSNR: 28.6
0.860
Paper's SSIM: 0.871
0.171
Paper's LPIPS (VGG): 0.169
33m 33s 23.74 GB
Fortress 31.48
Paper's PSNR: 31.36
0.898
Paper's SSIM: 0.897
0.141
Paper's LPIPS (VGG): 0.148
32m 41s 18.39 GB
Horns 28.33
Paper's PSNR: 28.14
0.882
Paper's SSIM: 0.877
0.180
Paper's LPIPS (VGG): 0.196
33m 26s 18.27 GB
Leaves 20.96
Paper's PSNR: 21.3
0.730
Paper's SSIM: 0.752
0.239
Paper's LPIPS (VGG): 0.217
28m 21s 20.11 GB
Orchids 19.85
Paper's PSNR: 19.87
0.644
Paper's SSIM: 0.649
0.279
Paper's LPIPS (VGG): 0.278
32m 3s 20.69 GB
Room 31.86
Paper's PSNR: 32.35
0.950
Paper's SSIM: 0.952
0.159
Paper's LPIPS (VGG): 0.167
27m 20s 16.38 GB
Trex 27.53
Paper's PSNR: 26.97
0.909
Paper's SSIM: 0.9
0.199
Paper's LPIPS (VGG): 0.221
28m 46s 27.20 GB

SeaThru-NeRF 

SeaThru-NeRF dataset contains four underwater forward-facing scenes.

Method PSNR SSIM LPIPS Time GPU mem.
Gaussian Splatting 21.31 0.729 0.300 20m 9s 6.49 GB
Curasao 24.15 0.738 0.318 22m 26s 9.30 GB
Panama 23.68 0.749 0.250 22m 41s 8.43 GB
IUI3 18.86 0.677 0.390 18m 36s 4.26 GB
Japanese Gradens 18.54 0.752 0.242 16m 55s 3.96 GB
Mip-Splatting 21.33 0.730 0.304 20m 53s 6.19 GB
Curasao 24.41 0.739 0.316 23m 38s 9.54 GB
Panama 23.99 0.752 0.237 22m 42s 6.85 GB
IUI3 19.18 0.683 0.412 19m 32s 4.26 GB
Japanese Gradens 17.75 0.747 0.253 17m 43s 4.10 GB
SeaThru-NeRF 26.37 0.815 0.245 2h 49m 9s 127.55 GB
Curasao 30.00
Paper's PSNR: 30.48
0.870
Paper's SSIM: 0.87
0.215
Paper's LPIPS: 0.2
2h 48m 47s 127.66 GB
Panama 27.82
Paper's PSNR: 27.89
0.834
Paper's SSIM: 0.83
0.226
Paper's LPIPS: 0.22
2h 54m 57s 127.66 GB
IUI3 25.92 0.787 0.294 2h 46m 50s 127.44 GB
Japanese Gradens 21.73
Paper's PSNR: 21.83
0.768
Paper's SSIM: 0.77
0.246
Paper's LPIPS: 0.25
2h 46m 1s 127.44 GB

Photo Tourism 

Photo Tourism is a dataset of images of famous landmarks, such as the Sacre Coeur, the Trevi Fountain, and the Brandenburg Gate. The images were captured by tourist at different times of the day and year, images have varying lighting conditions and occlusions. The evaluation protocol is based on NeRF-W, where the image appearance embeddings are optimized on the left side of the image and the metrics are computed on the right side of the image.

Method PSNR SSIM LPIPS Time GPU mem.
K-Planes 21.10 0.761 0.313 24m 37s 3.59 GB
Sacre Coeur 19.96 0.762 0.299 24m 26s 3.62 GB
Trevi Fountain 19.70 0.662 0.388 24m 44s 3.59 GB
Brandenburg Gate 23.65 0.859 0.253 24m 40s 3.56 GB
GS-W 21.38
Paper's PSNR: 24.70

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.817
Paper's SSIM: 0.865

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.213
Paper's LPIPS: 0.124

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 1h 13m 50s 21.93 GB
Sacre Coeur 19.73
Paper's PSNR: 23.24

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.824
Paper's SSIM: 0.8632

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.210
Paper's LPIPS: 0.13

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 1h 8m 56s 18.80 GB
Trevi Fountain 20.06
Paper's PSNR: 22.91

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.724
Paper's SSIM: 0.8014

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.272
Paper's LPIPS: 0.1563

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 1h 21m 34s 27.92 GB
Brandenburg Gate 24.35
Paper's PSNR: 27.96

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.903
Paper's SSIM: 0.9319

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 0.155
Paper's LPIPS: 0.0862

The original paper reports metrics for test images where the appearance embedding is estimated from the full test image, not just the left half as in the official evaluation protocol. The reported numbers are computed using the official evaluation protocol and are, therefore, lower than the numbers reported in the paper.

 1h 10m 60s 19.06 GB
NeRF-W (reimplementation) 21.75 0.790 0.268 44h 23m 46s 98.80 GB
Sacre Coeur 19.56 0.795 0.260 41h 41m 36s 98.80 GB
Trevi Fountain 21.48 0.693 0.331 49h 43m 22s 98.81 GB
Brandenburg Gate 24.22 0.884 0.213 41h 46m 21s 98.80 GB
Scaffold-GS 23.50 0.854 0.170 1h 27m 49s 18.34 GB
Sacre Coeur 21.85 0.871 0.157 1h 38m 45s 11.38 GB
Trevi Fountain 23.21 0.768 0.228 1h 21m 34s 21.84 GB
Brandenburg Gate 25.45 0.923 0.127 1h 23m 7s 21.80 GB
gsplat 23.66 0.857 0.162 1h 44m 24s 4.68 GB
Sacre Coeur 22.05 0.876 0.154 1h 30m 15s 3.33 GB
Trevi Fountain 22.56 0.765 0.213 2h 11m 60s 7.05 GB
Brandenburg Gate 26.36 0.931 0.118 1h 30m 56s 3.68 GB
WildGaussians 24.65 0.851 0.179 10h 18m 16s 18.24 GB
Sacre Coeur 22.56 0.859 0.177 8h 41m 57s 8.66 GB
Trevi Fountain 23.63 0.766 0.228 13h 14m 51s 38.25 GB
Brandenburg Gate 27.76 0.927 0.133 8h 57m 60s 7.83 GB

Citation

If you use this code in your research, please cite the following paper:

@article{kulhanek2024nerfbaselines,
  title={NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods},
  author={Jonas Kulhanek and Torsten Sattler},
  year={2024},
  journal={arXiv},
}

Acknowledgements

We want to thank Brent Yi and the NerfStudio Team for helpful discussions regarding the NerfStudio codebase and for releasing the Viser platform. This work was supported by the Czech Science Foundation (GAČR) EXPRO (grant no. 23-07973X), the Grant Agency of the Czech Technical University in Prague (grant no. SGS24/095/OHK3/2T/13), and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).

License

The NerfBaselines project is licensed under the MIT license. Each implemented method is licensed under the license provided by the authors of the method. For the currently implemented methods, the following licenses apply: