Motivation: Marine scientific software, both ocean models and data analytics, should correctly implement the underlying scientific theories. Changes to the software may introduce faults and thus invalidate previously correct behavior. Regression tests that are automatically executed after changes help to uphold correctness. There is a large potential for automating tests of marine scientific software, because the underlying system and the underlying scientific theory are full of symmetries and constraints that can be leveraged to discern correct from incorrect behavior even if it is not possible to build a test oracle predicting the correct response of the software to all possible boundary conditions.
The state of testing of software developed and used in the marine sciences is two-fold: On the one hand, there are monolithic code bases mainly used for ocean and climate simulations. On the other hand there is a stack of often highly modularized scientific software used for the analysis of observations of the marine domain (satellite images, in-situ measurements) and of data produced by simulations of the ocean and earth system. While the cut is not perfect, it can be said that software testing is more widely used in the development of the analysis stack.
Challenges: Applying state-of-the-art practices from software engineering to scientific software poses a challenge that is partly due to cultural differences between the scientific-computing and the software-engineering community, and partly due to the very special nature of scientific software [JH2018]. An important part of the software-development process is testing (for validation and verification) of software.
Aim: The proposed PhD project will invent new automated metamorphic testing methods for scientific software in the marine domain, explore the use of machine learning for automatic test generation [KB2013], and possibly implement continuous integration / testing of an existing software package. The following figure illustrates the process of metamorphic testing without test oracles.
Objectives: (1) Develop new automated metamorphic testing methods, (2) Apply these new methods for continuous integration processes of an existing idealized ocean model [BC2017], (3) Empirical evaluation of the new automated metamorphic testing methods in the marine science domain.
References:
[BC2017] P. Brandt, M. Claus, R.J.. Greatbatch, R. Kopte, J.M. Toole, W.E. Johns, C.W. Böning, “Annual and Semiannual Cycle of Equatorial Atlantic Circulation Associated with Basin-Mode Resonance,” Journal of Physical Oceanography, 46 (10), 2018, pp.3011-3019, doi: 10.1175/JPO-D-15-0248.1.
[JH2018] A. Johanson and W. Hasselbring, “Software Engineering for Computational Science: Past, Present, Future,” Computing in Science & Engineering, 20 (2). 2018, pp. 90-109. doi: 10.1109/MCSE.2018.108162940.
[KB2013] U. Kanewala and J. M. Bieman, "Using machine learning techniques to detect metamorphic relations for programs without test oracles," 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), Pasadena, CA, 2013, pp. 1-10. doi: 10.1109/ISSRE.2013.6698899
