Exporting results tables

Open in Colab

NerfBaselines allows you to export the results either as a markdown or a latex table. In this tutorial, we assume you have trained several methods on multiple datasets and obtained a folder with results.json files. We assume the structure looks something like the follows:

gaussian-splatting/
    mipnerf360/
        flowers/
            results.json
        garden/
            results.json
zipnerf/
    mipnerf360/
        flowers/
            results.json
        garden/
            results.json

Note

How you structure the results directory is up to you and there are no strong requirements. The only requirement is that the results.json files have to be somewhere in that directory.

However, if you haven’t trained your model yet, you can try rendering the official NerfBaselines results. In that case, you don’t need any result files.

First, take a look at the generate-dataset-results command and it’s options:

!nerfbaselines generate-dataset-results --help

Usage: nerfbaselines generate-dataset-results [OPTIONS]

Options:
  --results PATH
  --dataset TEXT
  --output-type [markdown|json|latex]
  --method-links [paper|website|results|none]
  --output FILE
  --scenes TEXT                   Comma-separated list of scenes to include in
                                  the results.
  -v, --verbose                   Enable verbose logging.
  --help                          Show this message and exit.

We can see, that we can add --results PATH argument to point it towards our results directory. We specify which dataset to render by adding --dataset DATASET argument. If we add --output FILE path, we output the results to a file instead of printing it to the console output. We control the type output py adding --output-type markdown|latex argument.

Generating markdown tables

First, we will show you how to export your results as a markdown table. If you want to render your own data, use --data {path to your data argument} to all commands. If you don’t specify this argument, the official NerfBaselines results will be rendered. The following command renders the results table of all methods on Mip-NeRF~360 dataset.

!nerfbaselines generate-dataset-results --output-type markdown --dataset mipnerf360

| Method                  |       PSNR |      SSIM | LPIPS (VGG) |        Time | GPU mem. |
|:------------------------|-----------:|----------:|------------:|------------:|---------:|
| Zip-NeRF                | **28.553** | **0.829** |   **0.218** |  5h 30m 20s |  26.8 GB |
| Scaffold-GS             |   *27.714* |     0.813 |       0.262 |     23m 28s |   8.7 GB |
| Mip-NeRF 360            |     27.681 |     0.792 |       0.272 | 30h 14m 36s |  33.6 GB |
| 3DGS-MCMC               |     27.571 |     0.798 |       0.281 |      35m 8s |  21.6 GB |
| Mip-Splatting           |     27.492 |     0.815 |       0.258 |     25m 37s |  11.0 GB |
| Gaussian Splatting      |     27.434 |     0.814 |       0.257 |     23m 25s |  11.1 GB |
| Gaussian Opacity Fields |     27.421 |   *0.826* |     *0.234* |   1h 3m 54s |  28.4 GB |
| gsplat                  |     27.412 |     0.815 |       0.256 |     29m 19s |   8.3 GB |
| 2D Gaussian Splatting   |     26.815 |     0.796 |       0.297 |     31m 10s |  13.2 GB |
| NerfStudio              |     26.388 |     0.731 |       0.343 |   *19m 30s* | *5.9 GB* |
| Instant NGP             |     25.507 |     0.684 |       0.398 |  **3m 54s** |   7.8 GB |
| COLMAP                  |     16.670 |     0.445 |       0.590 |  2h 52m 55s | **0 MB** |

You change the scenes over which the average is computed:

!nerfbaselines generate-dataset-results --output-type markdown --dataset mipnerf360 --scenes garden,bonsai

| Method                  |       PSNR |      SSIM | LPIPS (VGG) |        Time | GPU mem. |
|:------------------------|-----------:|----------:|------------:|------------:|---------:|
| Zip-NeRF                | **31.434** |   *0.907* |   **0.161** |  5h 27m 33s |  26.8 GB |
| Mip-NeRF 360            |   *30.239* |     0.877 |       0.200 | 30h 13m 23s |  33.8 GB |
| 3DGS-MCMC               |     30.237 | **0.912** |     *0.173* |     41m 39s |  25.1 GB |
| Scaffold-GS             |     30.044 |     0.903 |       0.192 |      23m 9s |   9.4 GB |
| gsplat                  |     29.808 |     0.904 |       0.187 |     31m 34s |   9.1 GB |
| Gaussian Splatting      |     29.769 |     0.904 |       0.189 |      24m 7s |  11.5 GB |
| Gaussian Opacity Fields |     29.743 |   *0.907* |       0.181 |   1h 6m 25s |  29.0 GB |
| Mip-Splatting           |     29.718 |     0.905 |       0.189 |     25m 28s |  11.2 GB |
| 2D Gaussian Splatting   |     28.994 |     0.887 |       0.223 |     30m 36s |  12.0 GB |
| NerfStudio              |     28.238 |     0.829 |       0.252 |   *19m 38s* | *6.0 GB* |
| Instant NGP             |     27.541 |     0.772 |       0.320 |  **3m 45s** |   8.2 GB |
| COLMAP                  |     16.464 |     0.508 |       0.532 |  2h 40m 19s | **0 MB** |

Furthermore, the links to the official results, paper webpages, or original papers can be added by using --method-links option.

!nerfbaselines generate-dataset-results --output-type markdown --dataset mipnerf360 --method-links website

| Method                                                                               |       PSNR |      SSIM | LPIPS (VGG) |        Time | GPU mem. |
|:-------------------------------------------------------------------------------------|-----------:|----------:|------------:|------------:|---------:|
| [Zip-NeRF](https://jonbarron.info/zipnerf/)                                          | **28.553** | **0.829** |   **0.218** |  5h 30m 20s |  26.8 GB |
| [Scaffold-GS](https://city-super.github.io/scaffold-gs/)                             |   *27.714* |     0.813 |       0.262 |     23m 28s |   8.7 GB |
| [Mip-NeRF 360](https://jonbarron.info/mipnerf360/)                                   |     27.681 |     0.792 |       0.272 | 30h 14m 36s |  33.6 GB |
| [3DGS-MCMC](https://ubc-vision.github.io/3dgs-mcmc/)                                 |     27.571 |     0.798 |       0.281 |      35m 8s |  21.6 GB |
| [Mip-Splatting](https://niujinshuchong.github.io/mip-splatting/)                     |     27.492 |     0.815 |       0.258 |     25m 37s |  11.0 GB |
| [Gaussian Splatting](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/)      |     27.434 |     0.814 |       0.257 |     23m 25s |  11.1 GB |
| [Gaussian Opacity Fields](https://niujinshuchong.github.io/gaussian-opacity-fields/) |     27.421 |   *0.826* |     *0.234* |   1h 3m 54s |  28.4 GB |
| [gsplat](https://docs.gsplat.studio/main/)                                           |     27.412 |     0.815 |       0.256 |     29m 19s |   8.3 GB |
| [2D Gaussian Splatting](https://surfsplatting.github.io/)                            |     26.815 |     0.796 |       0.297 |     31m 10s |  13.2 GB |
| [NerfStudio](https://docs.nerf.studio/)                                              |     26.388 |     0.731 |       0.343 |   *19m 30s* | *5.9 GB* |
| [Instant NGP](https://nvlabs.github.io/instant-ngp/)                                 |     25.507 |     0.684 |       0.398 |  **3m 54s** |   7.8 GB |
| [COLMAP](https://colmap.github.io/)                                                  |     16.670 |     0.445 |       0.590 |  2h 52m 55s | **0 MB** |

Generating latex tables

Instead of markdown, we can generate LaTeX tables by changing the --output-type to latex:

!nerfbaselines generate-dataset-results --output-type latex --dataset mipnerf360

\providecommand{\pf}[1]{\textbf{#1}}
\providecommand{\ps}[1]{\underline{#1}}
\providecommand{\pt}[1]{#1}
\begin{tabular}{lrrrrr}\hline
Method                  &        PSNR &       SSIM & LPIPS (VGG) &         Time &    GPU mem. \\
Zip-NeRF                & \pf{28.553} & \pf{0.829} &  \pf{0.218} &   5h 30m 20s &     26.8 GB \\
Scaffold-GS             & \ps{27.714} &      0.813 &       0.262 &      23m 28s &      8.7 GB \\
Mip-NeRF 360            & \pt{27.681} &      0.792 &       0.272 &  30h 14m 36s &     33.6 GB \\
3DGS-MCMC               &      27.571 &      0.798 &       0.281 &       35m 8s &     21.6 GB \\
Mip-Splatting           &      27.492 & \pt{0.815} &       0.258 &      25m 37s &     11.0 GB \\
Gaussian Splatting      &      27.434 &      0.814 &       0.257 & \pt{23m 25s} &     11.1 GB \\
Gaussian Opacity Fields &      27.421 & \ps{0.826} &  \ps{0.234} &    1h 3m 54s &     28.4 GB \\
gsplat                  &      27.412 & \pt{0.815} &  \pt{0.256} &      29m 19s &      8.3 GB \\
2D Gaussian Splatting   &      26.815 &      0.796 &       0.297 &      31m 10s &     13.2 GB \\
NerfStudio              &      26.388 &      0.731 &       0.343 & \ps{19m 30s} & \ps{5.9 GB} \\
Instant NGP             &      25.507 &      0.684 &       0.398 &  \pf{3m 54s} & \pt{7.8 GB} \\
COLMAP                  &      16.670 &      0.445 &       0.590 &   2h 52m 55s &   \pf{0 MB} \\
\end{tabular}

In this caase, we provide an easi way to format the first, second, and third place by redefining the \pf, \ps, and \pt command. An example could be to define them as follows:

\newcommand{\pf}[1]{\cellcolor{red!30}#1}
\newcommand{\ps}[1]{\cellcolor{orange!30}#1}
\newcommand{\pt}[1]{\cellcolor{yellow!30}#1}

Same as in the markdown case, you can also specify which scenes to use:

!nerfbaselines generate-dataset-results --output-type latex --dataset mipnerf360 --scenes garden,bonsai

\providecommand{\pf}[1]{\textbf{#1}}
\providecommand{\ps}[1]{\underline{#1}}
\providecommand{\pt}[1]{#1}
\begin{tabular}{lrrrrr}\hline
Method                  &        PSNR &       SSIM & LPIPS (VGG) &         Time &    GPU mem. \\
Zip-NeRF                & \pf{31.434} & \ps{0.907} &  \pf{0.161} &   5h 27m 33s &     26.8 GB \\
Mip-NeRF 360            & \ps{30.239} &      0.877 &       0.200 &  30h 13m 23s &     33.8 GB \\
3DGS-MCMC               & \pt{30.237} & \pf{0.912} &  \ps{0.173} &      41m 39s &     25.1 GB \\
Scaffold-GS             &      30.044 &      0.903 &       0.192 &  \pt{23m 9s} &      9.4 GB \\
gsplat                  &      29.808 &      0.904 &       0.187 &      31m 34s &      9.1 GB \\
Gaussian Splatting      &      29.769 &      0.904 &       0.189 &       24m 7s &     11.5 GB \\
Gaussian Opacity Fields &      29.743 & \ps{0.907} &  \pt{0.181} &    1h 6m 25s &     29.0 GB \\
Mip-Splatting           &      29.718 &      0.905 &       0.189 &      25m 28s &     11.2 GB \\
2D Gaussian Splatting   &      28.994 &      0.887 &       0.223 &      30m 36s &     12.0 GB \\
NerfStudio              &      28.238 &      0.829 &       0.252 & \ps{19m 38s} & \ps{6.0 GB} \\
Instant NGP             &      27.541 &      0.772 &       0.320 &  \pf{3m 45s} & \pt{8.2 GB} \\
COLMAP                  &      16.464 &      0.508 &       0.532 &   2h 40m 19s &   \pf{0 MB} \\
\end{tabular}