Tanks and Temples

Tanks and Temples is a benchmark for image-based 3D reconstruction. The benchmark sequences were acquired outside the lab, in realistic conditions. Ground-truth data was captured using an industrial laser scanner. The benchmark includes both outdoor scenes and indoor environments. The dataset is split into three subsets: training, intermediate, and advanced.

Web: https://www.tanksandtemples.org/
Paper: Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction
Authors: Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, Vladlen Koltun
Method PSNR SSIM LPIPS Time GPU mem.
COLMAP 11.92 0.436 0.606 5h 16m 11s 0.00 MB
auditorium 14.55 0.390 0.637 3h 17m 31s 0.00 MB
ballroom 14.09 0.408 0.466 3h 34m 34s 0.00 MB
courtroom 14.48 0.371 0.537 3h 24m 16s 0.00 MB
museum 14.16 0.418 0.495 3h 18m 47s 0.00 MB
palace 9.08 0.407 0.784 3h 8m 22s 0.00 MB
temple 7.32 0.408 0.617 3h 5m 14s 0.00 MB
family 10.28 0.440 0.550 2h 33m 22s 0.00 MB
francis 8.33 0.302 0.686 2h 40m 27s 0.00 MB
horse 6.35 0.404 0.563 2h 41m 33s 0.00 MB
lighthouse 10.68 0.514 0.665 2h 46m 8s 0.00 MB
m60 14.19 0.547 0.550 10h 54m 40s 0.00 MB
panther 14.78 0.594 0.524 10h 32m 2s 0.00 MB
playground 13.30 0.455 0.615 10h 52m 56s 0.00 MB
train 11.83 0.396 0.813 9h 18m 35s 0.00 MB
barn 8.97 0.479 0.520 8h 15m 20s 0.00 MB
caterpillar 12.67 0.376 0.771 3h 20m 10s 0.00 MB
church 15.67 0.425 0.514 9h 2m 32s 0.00 MB
courthouse 9.09 0.397 0.737 9h 30m 56s 0.00 MB
ignatius 12.74 0.400 0.551 2h 54m 26s 0.00 MB
meetingroom 14.38 0.509 0.591 2h 48m 23s 0.00 MB
truck 13.35 0.507 0.545 2h 39m 37s 0.00 MB
2D Gaussian Splatting 21.54 0.768 0.281 15m 47s 7.23 GB
auditorium 22.86 0.835 0.304 13m 45s 6.55 GB
ballroom 23.40 0.808 0.120 19m 53s 6.12 GB
courtroom 18.92 0.660 0.422 15m 12s 9.39 GB
museum 21.11 0.764 0.170 26m 10s 8.09 GB
palace 16.91 0.668 0.512 13m 40s 5.50 GB
temple 19.74 0.763 0.305 13m 46s 4.06 GB
family 24.14 0.854 0.111 16m 16s 4.32 GB
francis 26.50 0.884 0.202 14m 56s 9.51 GB
horse 23.21 0.867 0.122 15m 15s 7.43 GB
lighthouse 18.00 0.707 0.438 13m 3s 8.47 GB
m60 22.83 0.825 0.249 15m 2s 8.98 GB
panther 21.70 0.771 0.358 13m 40s 6.39 GB
playground 21.45 0.745 0.316 16m 51s 7.16 GB
train 16.41 0.583 0.529 11m 43s 4.77 GB
barn 25.42
Paper's PSNR: 28.79

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.816 0.213 13m 36s 6.29 GB
caterpillar 22.24
Paper's PSNR: 24.23

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.761 0.227 14m 3s 7.15 GB
church 18.50 0.658 0.443 16m 11s 14.63 GB
courthouse 19.48
Paper's PSNR: 23.51

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.707 0.398 13m 18s 8.98 GB
ignatius 21.34
Paper's PSNR: 23.82

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.762 0.176 18m 57s 5.54 GB
meetingroom 24.26
Paper's PSNR: 26.15

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.849 0.168 20m 5s 7.41 GB
truck 23.83
Paper's PSNR: 26.85

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 0.843 0.123 15m 54s 4.99 GB
Instant NGP 21.62 0.712 0.340 4m 27s 4.13 GB
auditorium 20.67 0.761 0.429 4m 31s 3.92 GB
ballroom 21.62 0.652 0.352 4m 5s 4.02 GB
courtroom 19.44 0.640 0.448 4m 35s 3.93 GB
museum 15.19 0.471 0.606 5m 53s 3.93 GB
palace 19.09 0.668 0.440 4m 46s 4.64 GB
temple 17.84 0.689 0.424 4m 40s 3.94 GB
family 22.59 0.761 0.235 3m 42s 3.42 GB
francis 24.38 0.824 0.265 4m 10s 3.94 GB
horse 21.82 0.784 0.225 3m 47s 3.42 GB
lighthouse 21.65 0.765 0.281 4m 35s 4.02 GB
m60 25.82 0.832 0.202 4m 34s 4.04 GB
panther 26.81 0.844 0.208 4m 29s 4.04 GB
playground 23.33 0.696 0.344 4m 14s 3.97 GB
train 20.01 0.658 0.334 4m 39s 3.94 GB
barn 25.90 0.772 0.271 4m 41s 4.31 GB
caterpillar 21.72 0.633 0.360 4m 30s 4.22 GB
church 19.92 0.650 0.419 4m 21s 4.64 GB
courthouse 20.80 0.681 0.414 4m 55s 6.72 GB
ignatius 19.40 0.613 0.343 3m 58s 3.81 GB
meetingroom 23.24 0.783 0.326 3m 59s 4.18 GB
truck 22.85 0.770 0.216 4m 20s 3.77 GB
NerfStudio 22.04 0.743 0.270 19m 27s 3.74 GB
auditorium 20.77 0.771 0.330 19m 46s 3.88 GB
ballroom 22.68 0.705 0.261 19m 44s 3.88 GB
courtroom 20.24 0.673 0.336 19m 17s 3.88 GB
museum 17.84 0.648 0.311 18m 49s 3.88 GB
palace 17.68 0.640 0.452 20m 9s 3.64 GB
temple 17.06 0.678 0.392 19m 37s 3.88 GB
family 24.32 0.822 0.158 18m 54s 3.63 GB
francis 24.60 0.851 0.190 18m 48s 3.63 GB
horse 24.31 0.847 0.139 18m 53s 3.88 GB
lighthouse 20.85 0.768 0.245 19m 33s 3.89 GB
m60 26.54 0.843 0.179 19m 47s 3.64 GB
panther 27.57 0.858 0.174 20m 10s 3.89 GB
playground 24.69 0.755 0.249 19m 33s 3.64 GB
train 20.43 0.693 0.261 19m 42s 3.88 GB
barn 26.40 0.794 0.215 19m 16s 3.63 GB
caterpillar 21.71 0.666 0.302 19m 52s 3.63 GB
church 20.06 0.671 0.338 19m 29s 3.63 GB
courthouse 18.11 0.632 0.465 19m 37s 3.63 GB
ignatius 20.44 0.689 0.251 19m 5s 3.63 GB
meetingroom 23.21 0.793 0.261 19m 3s 3.63 GB
truck 23.37 0.797 0.167 19m 28s 3.63 GB
Gaussian Opacity Fields 22.39 0.825 0.172 41m 21s 24.07 GB
auditorium 23.20 0.871 0.194 35m 17s 8.44 GB
ballroom 22.84 0.818 0.107 1h 0m 31s 38.17 GB
courtroom 21.15 0.781 0.168 47m 1s 31.55 GB
museum 19.92 0.761 0.152 50m 33s 34.90 GB
palace 16.46 0.683 0.443 35m 6s 10.30 GB
temple 20.29 0.794 0.234 34m 55s 7.02 GB
family 22.31 0.875 0.084 41m 32s 35.16 GB
francis 24.76 0.901 0.158 35m 52s 12.30 GB
horse 23.73 0.881 0.092 38m 29s 24.41 GB
lighthouse 21.80 0.833 0.181 40m 8s 7.85 GB
m60 28.04 0.906 0.104 40m 9s 26.18 GB
panther 28.47 0.910 0.102 40m 13s 30.99 GB
playground 23.89 0.869 0.142 45m 37s 37.62 GB
train 19.69 0.796 0.164 37m 11s 19.04 GB
barn 25.72 0.866 0.140 37m 4s 15.67 GB
caterpillar 21.78 0.791 0.187 36m 21s 23.78 GB
church 19.65 0.775 0.208 48m 2s 36.45 GB
courthouse 19.60 0.726 0.346 35m 12s 12.82 GB
ignatius 20.34 0.769 0.162 43m 50s 34.97 GB
meetingroom 24.31 0.862 0.140 40m 29s 23.30 GB
truck 22.33 0.860 0.099 44m 56s 34.46 GB
Gaussian Splatting 23.83 0.831 0.165 13m 48s 6.95 GB
auditorium 24.13 0.871 0.193 10m 54s 4.98 GB
ballroom 24.07 0.824 0.101 19m 55s 9.08 GB
courtroom 23.12 0.790 0.165 17m 12s 9.02 GB
museum 20.92 0.764 0.160 20m 55s 11.38 GB
palace 19.63 0.736 0.350 11m 21s 5.85 GB
temple 20.85 0.806 0.222 10m 56s 5.13 GB
family 24.43 0.865 0.095 14m 60s 6.31 GB
francis 27.22 0.897 0.169 10m 20s 4.28 GB
horse 23.82 0.875 0.103 11m 35s 4.50 GB
lighthouse 22.11 0.843 0.156 12m 3s 6.09 GB
m60 27.84 0.901 0.113 13m 18s 6.80 GB
panther 28.32 0.908 0.107 13m 19s 7.01 GB
playground 25.37 0.848 0.170 15m 33s 7.44 GB
train 21.67 0.791 0.171 11m 56s 5.39 GB
barn 27.51 0.852 0.160 11m 9s 6.22 GB
caterpillar 23.38 0.791 0.190 11m 32s 5.89 GB
church 22.79 0.811 0.177 17m 15s 8.36 GB
courthouse 22.22 0.779 0.266 11m 8s 10.34 GB
ignatius 21.53 0.776 0.153 16m 36s 8.82 GB
meetingroom 25.19 0.866 0.141 12m 39s 5.50 GB
truck 24.25 0.853 0.108 15m 18s 7.47 GB
Mip-Splatting 23.93 0.833 0.166 15m 56s 7.27 GB
auditorium 24.41 0.872 0.196 11m 47s 4.82 GB
ballroom 24.15 0.826 0.098 25m 4s 9.76 GB
courtroom 23.00 0.791 0.165 20m 46s 9.27 GB
museum 20.88 0.768 0.158 24m 55s 12.57 GB
palace 19.63 0.731 0.354 12m 53s 6.48 GB
temple 20.55 0.805 0.226 12m 3s 5.93 GB
family 24.55 0.872 0.095 15m 44s 6.99 GB
francis 27.61 0.899 0.172 11m 47s 4.40 GB
horse 23.94 0.879 0.104 12m 47s 4.58 GB
lighthouse 22.25 0.844 0.159 13m 47s 7.01 GB
m60 27.98 0.904 0.112 15m 13s 7.13 GB
panther 28.27 0.908 0.109 15m 43s 7.38 GB
playground 25.87 0.861 0.155 18m 26s 8.30 GB
train 21.82 0.795 0.172 13m 26s 5.65 GB
barn 27.75 0.855 0.161 12m 42s 6.12 GB
caterpillar 23.42 0.790 0.197 13m 16s 5.69 GB
church 22.76 0.812 0.176 19m 53s 8.37 GB
courthouse 22.15 0.779 0.265 13m 57s 10.28 GB
ignatius 21.73 0.780 0.159 18m 44s 8.46 GB
meetingroom 25.46 0.870 0.137 14m 13s 5.73 GB
truck 24.36 0.857 0.108 17m 32s 7.68 GB
Zip-NeRF 24.63 0.840 0.131 5h 44m 9s 26.61 GB
auditorium 24.52 0.877 0.153 5h 24m 16s 26.61 GB
ballroom 25.45 0.835 0.113 5h 25m 1s 26.61 GB
courtroom 22.17 0.790 0.153 5h 17m 54s 26.61 GB
museum 19.34 0.746 0.159 5h 24m 17s 26.61 GB
palace 19.11 0.718 0.317 6h 42m 11s 26.61 GB
temple 20.58 0.805 0.183 5h 37m 11s 26.61 GB
family 27.10 0.889 0.067 5h 32m 15s 26.61 GB
francis 29.10 0.915 0.106 5h 34m 11s 26.61 GB
horse 26.82 0.897 0.069 5h 55m 60s 26.61 GB
lighthouse 23.07 0.849 0.131 5h 52m 55s 26.63 GB
m60 29.01 0.912 0.076 6h 3m 19s 26.63 GB
panther 28.76 0.909 0.081 5h 36m 45s 26.63 GB
playground 27.13 0.880 0.095 5h 29m 4s 26.61 GB
train 22.19 0.814 0.119 6h 14m 13s 26.61 GB
barn 29.26 0.884 0.083 5h 28m 43s 26.61 GB
caterpillar 23.94 0.802 0.152 5h 53m 16s 26.61 GB
church 23.14 0.807 0.153 6h 4m 32s 26.61 GB
courthouse 22.88 0.780 0.218 6h 4m 6s 26.61 GB
ignatius 22.61 0.789 0.127 5h 48m 32s 26.61 GB
meetingroom 25.93 0.875 0.121 5h 21m 9s 26.61 GB
truck 25.09 0.864 0.081 5h 37m 11s 26.61 GB

PSNR

Peak Signal to Noise Ratio. The higher the better.

Method auditorium ballroom courtroom museum palace temple family francis horse lighthouse m60 panther playground train barn caterpillar church courthouse ignatius meetingroom truck
COLMAP 14.55 14.09 14.48 14.16 9.08 7.32 10.28 8.33 6.35 10.68 14.19 14.78 13.30 11.83 8.97 12.67 15.67 9.09 12.74 14.38 13.35
2D Gaussian Splatting 22.86 23.40 18.92 21.11 16.91 19.74 24.14 26.50 23.21 18.00 22.83 21.70 21.45 16.41 25.42
Paper's PSNR: 28.79

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 22.24
Paper's PSNR: 24.23

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 18.50 19.48
Paper's PSNR: 23.51

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 21.34
Paper's PSNR: 23.82

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 24.26
Paper's PSNR: 26.15

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.

 23.83
Paper's PSNR: 26.85

2DGS used different data pre-processing and train/test split for Tanks and Temples. It sets specific hyperparameters for each scene which may not be suitable with the public Tanks and Temples released by NerfBaselines. The results are not directly comparable and a hyperparameter tuning is needed to improve the results.
Instant NGP 20.67 21.62 19.44 15.19 19.09 17.84 22.59 24.38 21.82 21.65 25.82 26.81 23.33 20.01 25.90 21.72 19.92 20.80 19.40 23.24 22.85
NerfStudio 20.77 22.68 20.24 17.84 17.68 17.06 24.32 24.60 24.31 20.85 26.54 27.57 24.69 20.43 26.40 21.71 20.06 18.11 20.44 23.21 23.37
Gaussian Opacity Fields 23.20 22.84 21.15 19.92 16.46 20.29 22.31 24.76 23.73 21.80 28.04 28.47 23.89 19.69 25.72 21.78 19.65 19.60 20.34 24.31 22.33
Gaussian Splatting 24.13 24.07 23.12 20.92 19.63 20.85 24.43 27.22 23.82 22.11 27.84 28.32 25.37 21.67 27.51 23.38 22.79 22.22 21.53 25.19 24.25
Mip-Splatting 24.41 24.15 23.00 20.88 19.63 20.55 24.55 27.61 23.94 22.25 27.98 28.27 25.87 21.82 27.75 23.42 22.76 22.15 21.73 25.46 24.36
Zip-NeRF 24.52 25.45 22.17 19.34 19.11 20.58 27.10 29.10 26.82 23.07 29.01 28.76 27.13 22.19 29.26 23.94 23.14 22.88 22.61 25.93 25.09

SSIM

Structural Similarity Index. The higher the better. The implementation matches JAX's SSIM and torchmetrics's SSIM (with default parameters).

Method auditorium ballroom courtroom museum palace temple family francis horse lighthouse m60 panther playground train barn caterpillar church courthouse ignatius meetingroom truck
COLMAP 0.390 0.408 0.371 0.418 0.407 0.408 0.440 0.302 0.404 0.514 0.547 0.594 0.455 0.396 0.479 0.376 0.425 0.397 0.400 0.509 0.507
2D Gaussian Splatting 0.835 0.808 0.660 0.764 0.668 0.763 0.854 0.884 0.867 0.707 0.825 0.771 0.745 0.583 0.816 0.761 0.658 0.707 0.762 0.849 0.843
Instant NGP 0.761 0.652 0.640 0.471 0.668 0.689 0.761 0.824 0.784 0.765 0.832 0.844 0.696 0.658 0.772 0.633 0.650 0.681 0.613 0.783 0.770
NerfStudio 0.771 0.705 0.673 0.648 0.640 0.678 0.822 0.851 0.847 0.768 0.843 0.858 0.755 0.693 0.794 0.666 0.671 0.632 0.689 0.793 0.797
Gaussian Opacity Fields 0.871 0.818 0.781 0.761 0.683 0.794 0.875 0.901 0.881 0.833 0.906 0.910 0.869 0.796 0.866 0.791 0.775 0.726 0.769 0.862 0.860
Gaussian Splatting 0.871 0.824 0.790 0.764 0.736 0.806 0.865 0.897 0.875 0.843 0.901 0.908 0.848 0.791 0.852 0.791 0.811 0.779 0.776 0.866 0.853
Mip-Splatting 0.872 0.826 0.791 0.768 0.731 0.805 0.872 0.899 0.879 0.844 0.904 0.908 0.861 0.795 0.855 0.790 0.812 0.779 0.780 0.870 0.857
Zip-NeRF 0.877 0.835 0.790 0.746 0.718 0.805 0.889 0.915 0.897 0.849 0.912 0.909 0.880 0.814 0.884 0.802 0.807 0.780 0.789 0.875 0.864

LPIPS

Learned Perceptual Image Patch Similarity. The lower the better. The implementation uses AlexNet backbone and matches lpips pip package with checkpoint version 0.1

Method auditorium ballroom courtroom museum palace temple family francis horse lighthouse m60 panther playground train barn caterpillar church courthouse ignatius meetingroom truck
COLMAP 0.637 0.466 0.537 0.495 0.784 0.617 0.550 0.686 0.563 0.665 0.550 0.524 0.615 0.813 0.520 0.771 0.514 0.737 0.551 0.591 0.545
2D Gaussian Splatting 0.304 0.120 0.422 0.170 0.512 0.305 0.111 0.202 0.122 0.438 0.249 0.358 0.316 0.529 0.213 0.227 0.443 0.398 0.176 0.168 0.123
Instant NGP 0.429 0.352 0.448 0.606 0.440 0.424 0.235 0.265 0.225 0.281 0.202 0.208 0.344 0.334 0.271 0.360 0.419 0.414 0.343 0.326 0.216
NerfStudio 0.330 0.261 0.336 0.311 0.452 0.392 0.158 0.190 0.139 0.245 0.179 0.174 0.249 0.261 0.215 0.302 0.338 0.465 0.251 0.261 0.167
Gaussian Opacity Fields 0.194 0.107 0.168 0.152 0.443 0.234 0.084 0.158 0.092 0.181 0.104 0.102 0.142 0.164 0.140 0.187 0.208 0.346 0.162 0.140 0.099
Gaussian Splatting 0.193 0.101 0.165 0.160 0.350 0.222 0.095 0.169 0.103 0.156 0.113 0.107 0.170 0.171 0.160 0.190 0.177 0.266 0.153 0.141 0.108
Mip-Splatting 0.196 0.098 0.165 0.158 0.354 0.226 0.095 0.172 0.104 0.159 0.112 0.109 0.155 0.172 0.161 0.197 0.176 0.265 0.159 0.137 0.108
Zip-NeRF 0.153 0.113 0.153 0.159 0.317 0.183 0.067 0.106 0.069 0.131 0.076 0.081 0.095 0.119 0.083 0.152 0.153 0.218 0.127 0.121 0.081