Reproducible validation and replication studies in nanoscale physicsPreprint: arXiv 2008.05414, 12 Aug 2020. Submitted: July 27, 2020. Accepted: September 23, 2020. Published online: 29/03/2021
Published in print: 17/05/2021
Philosophical Transactions of the Royal Society A, 379(2197):20200068 (March 2021). https://doi.org/10.1098/rsta.2020.0068
Theme issue: ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico’ – compiled and edited by P. V. Coveney, D. Groen and A. G. Hoekstra.
Credibility building activities in computational research include verification and validation, reproducibility and replication, and uncertainty quantification. Though orthogonal to each other, they are related. This paper presents validation and replication studies in electromagnetic excitations on nanoscale structures, where the quantity of interest is the wavelength at which resonance peaks occur. The study uses the open-source software PyGBe: a boundary element solver with trecode acceleration and GPU capability. We replicate a result by Rockstuhl et al. (2005, doi:10/dsxw9d) with a two-dimensional boundary element method on silicon carbide particles, despite differences in our method. The second replication case from Ellis et al. (2016, doi:10/f83zcb) looks at aspect ratio effects on high-order modes of localized surface phonon-polariton nanostructures. The results partially replicate: the wavenumber position of some mode match, but for other modes they differ. With virtually no information about the original simulations, explaining the discrepancies is not possible. A comparison with experiments that measured polarized reflectance of silicon carbide nano pillars provides a validation case. The wavenumber of the dominant mode and two more do match, but differences remain in other minor modes. Results in this paper were produced with strict reproducibility practices, and we share reproducibility packages for all, including input files, execution scripts, secondary data, post-processing code and plotting scripts, and the figures (deposited in Zenodo). In view of the many challenges faced, we propose that reproducible practices make replication and validation more feasible.
As in our previous papers, readers can reproduce all the figures in this paper using the repro-packs shared in the manuscript GitHub repository: https://github.com/barbagroup/pygbe_validation_paper (and archived separately in Zenodo, as described in the paper). They include Jupyter notebooks with all the plotting code, and also the manually digitized data from the figures in the source articles for our replication cases.
Take-aways from the replication
- We needed private communications with the authors to complete comparisons.
- Reproducible practices make replication more feasible.
- The process of replication/validation is non-linear and iterative.
- Many intermediate results have to be thrown away.
- If it weren't for a figure in the supplementary materials, we couldn't have produced the final result.
- "Reproducible validation and replication studies in nanoscale physics", Natalia C. Clementi, Lorena A. Barba. Philosophical Transactions of the Royal Society A, 379(2197):20200068 (March 2021). 10.1098/rsta.2020.0068 // Preprint arXiv:2008.05414 // manuscript repository
Published online 29/03/2021. Published in print 17/05/2021.