Reproducible validation and replication studies in nanoscale physics

Result from our paper on "Reproducible Validation and Replication Studies in Nanoscale Physics," showing a validation case that compares with experiments from Ellis et al. (2016, doi:10/f83zcb)

Philosophical Transactions of the Royal Society A, 379(2197):20200068 (March 2021). https://doi.org/10.1098/rsta.2020.0068

Theme issue: ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico’ – compiled and edited by P. V. Coveney, D. Groen and A. G. Hoekstra.

Abstract

Credibility building activities in computational research include verification and validation, reproducibility and replication, and uncertainty quantification. Though orthogonal to each other, they are related. This paper presents validation and replication studies in electromagnetic excitations on nanoscale structures, where the quantity of interest is the wavelength at which resonance peaks occur. The study uses the open-source software PyGBe: a boundary element solver with trecode acceleration and GPU capability. We replicate a result by Rockstuhl et al. (2005, doi:10/dsxw9d) with a two-dimensional boundary element method on silicon carbide particles, despite differences in our method. The second replication case from Ellis et al. (2016, doi:10/f83zcb) looks at aspect ratio effects on high-order modes of localized surface phonon-polariton nanostructures. The results partially replicate: the wavenumber position of some mode match, but for other modes they differ. With virtually no information about the original simulations, explaining the discrepancies is not possible. A comparison with experiments that measured polarized reflectance of silicon carbide nano pillars provides a validation case. The wavenumber of the dominant mode and two more do match, but differences remain in other minor modes. Results in this paper were produced with strict reproducibility practices, and we share reproducibility packages for all, including input files, execution scripts, secondary data, post-processing code and plotting scripts, and the figures (deposited in Zenodo). In view of the many challenges faced, we propose that reproducible practices make replication and validation more feasible.

Reproducibility packages

As in our previous papers, readers can reproduce all the figures in this paper using the repro-packs shared in the manuscript GitHub repository: https://github.com/barbagroup/pygbe_validation_paper (and archived separately in Zenodo, as described in the paper). They include Jupyter notebooks with all the plotting code, and also the manually digitized data from the figures in the source articles for our replication cases.

Take-aways from the replication

• We needed private communications with the authors to complete comparisons.
• Reproducible practices make replication more feasible.
• The process of replication/validation is non-linear and iterative.
• Many intermediate results have to be thrown away.
• If it weren't for a figure in the supplementary materials, we couldn't have produced the final result.

Reference

• "Reproducible validation and replication studies in nanoscale physics", Natalia C. Clementi, Lorena A. Barba. Philosophical Transactions of the Royal Society A, 379(2197):20200068 (March 2021). 10.1098/rsta.2020.0068 // Preprint arXiv:2008.05414 // manuscript repository
  

  Published online 29/03/2021. Published in print 17/05/2021.