Design patterns and software techniques for large-scale, open and reproducible data reduction
- Authors: Molenaar, Gijs Jan
- Date: 2021
- Subjects: Radio astronomy -- Data processing , Radio astronomy -- Data processing -- Software , Radio astronomy -- South Africa , ASTRODECONV2019 dataset , Radio telescopes -- South Africa , KERN (omputer software)
- Language: English
- Type: text , Thesis , Doctoral , PhD
- Identifier: http://hdl.handle.net/10962/172169 , vital:42172 , 10.21504/10962/172169
- Description: The preparation for the construction of the Square Kilometre Array, and the introduction of its operational precursors, such as LOFAR and MeerKAT, mark the beginning of an exciting era for astronomy. Impressive new data containing valuable science just waiting for discovery is already being generated, and these devices will produce far more data than has ever been collected before. However, with every new data instrument, the data rates grow to unprecedented quantities of data, requiring novel new data-processing tools. In addition, creating science grade data from the raw data still requires significant expert knowledge for processing this data. The software used is often developed by a scientist who lacks proper training in software development skills, resulting in the software not progressing beyond a prototype stage in quality. In the first chapter, we explore various organisational and technical approaches to address these issues by providing a historical overview of the development of radioastronomy pipelines since the inception of the field in the 1940s. In that, the steps required to create a radio image are investigated. We used the lessons-learned to identify patterns in the challenges experienced, and the solutions created to address these over the years. The second chapter describes the mathematical foundations that are essential for radio imaging. In the third chapter, we discuss the production of the KERN Linux distribution, which is a set of software packages containing most radio astronomy software currently in use. Considerable effort was put into making sure that the contained software installs appropriately, all items next to one other on the same system. Where required and possible, bugs and portability fixes were solved and reported with the upstream maintainers. The KERN project also has a website, and issue tracker, where users can report bugs and maintainers can coordinate the packaging effort and new releases. The software packages can be used inside Docker and Singularity containers, enabling the installation of these packages on a wide variety of platforms. In the fourth and fifth chapters, we discuss methods and frameworks for combining the available data reduction tools into recomposable pipelines and introduce the Kliko specification and software. This framework was created to enable end-user astronomers to chain and containerise operations of software in KERN packages. Next, we discuss the Common Workflow Language (CommonWL), a similar but more advanced and mature pipeline framework invented by bio-informatics scientists. CommonWL is supported by a wide range of tools already; among other schedulers, visualisers and editors. Consequently, when a pipeline is made with CommonWL, it can be deployed and manipulated with a wide range of tools. In the final chapter, we attempt something unconventional, applying a generative adversarial network based on deep learning techniques to perform the task of sky brightness reconstruction. Since deep learning methods often require a large number of training samples, we constructed a CommonWL simulation pipeline for creating dirty images and corresponding sky models. This simulated dataset has been made publicly available as the ASTRODECONV2019 dataset. It is shown that this method is useful to perform the restoration and matches the performance of a single clean cycle. In addition, we incorporated domain knowledge by adding the point spread function to the network and by utilising a custom loss function during training. Although it was not possible to improve the cleaning performance of commonly used existing tools, the computational time performance of the approach looks very promising. We suggest that a smaller scope should be the starting point for further studies and optimising of the training of the neural network could produce the desired results.
- Full Text:
- Date Issued: 2021
Ecological assessment of a temperate river system using biomonitoring techniques: a case study of the Bloukrans River system, South Africa
- Authors: Magadze, Tinotenda
- Date: 2020
- Subjects: Bloukrans River , Ecological assessment (Biology) -- South Africa , Stream ecology -- South Africa , Rivers -- Environmental aspects -- South Africa , Benthic animals -- Effect of human beings on -- South Africa , Water quality biological assessment -- South Africa , Diatoms -- South Africa -- Data processing
- Language: English
- Type: text , Thesis , Doctoral , PhD
- Identifier: http://hdl.handle.net/10962/142933 , vital:38177
- Description: Escalating pressures from growing human populations and environmental impacts increasingly imperil freshwater ecosystems. The Bloukrans River, which drains an urbanised and agricultural catchment in the Eastern Cape province of South Africa, is no exception. Detailed knowledge of the structure and function of the aquatic ecosystems is required in order to create models and matrices that predict, guide assessment and direct intervention on ecological integrity and water quality management in these systems. The main objectives of this research were to: examine the effects of urbanization on the benthic macroinvertebrate functional feeding guild structure among different stream orders; determine if benthic diatoms can be used as effective and reliable indicators of ionic composition and conductivity in different stream order categories and finally; to evaluate the applicability of the South African Diatom Index (SADI) and other indices in the Eastern Cape region of South Africa. Field studies were carried out to explore temporal patterns in community structure (macroinvertebrates and diatoms) and ecosystem function related to land-use patterns, instream habitat availability, and water quality parameters in the Bloukrans River system across four study periods: February (summer) and July (winter) 2016 and February (summer) and May/July (winter) 2018. The study was conducted along a gradient of impacts from less impacted agricultural headwaters to highly impacted urban sites located immediately downstream of the city of Makhanda. Macroinvertebrates were separated into functional feeding groups (FFGs) (i.e. collector-gatherer, collector-filterer, scraper, shredder, and predator) which were then used to assess the effects of selected physico-chemical variables and riparian zone condition on FFG organization. Collector-gatherers were the most abundant in the Bloukrans River and represented 71.3 % of the macroinvertebrate assemblages. Stream order 1species such as Nitzschia palea, Gomphonema parvulum, Tryblionella apiculata, Diploneis vulgaris and Staurosira elliptica. Multivariate analysis (Canonical correspondence analysis (CCA)) indicated that differences in diatom community assemblages were best explained by calcium, magnesium, pH, phosphate, nitrate, dissolved oxygen, sediment nitrate, conductivity and salinity. These results indicate that diatoms can be used as bioindicators for monitoring highly impacted river systems and to also further examine pollution gradients and impacts of specific/point pollution sources. In order to further test the application of diatom indices, nine sites with contrasting water quality were sampled along the length river system in February, May and July 2018. Diatom-based indices incorporated in OMNIDIA software were applied to assess the integrity of the water quality as indicated by diatom communities. For comparative purposes, several foreign indices (e.g. the trophic diatom index (TDI), the percentage pollution-tolerant valves (%PTV), biological diatom index (BDI)) and the South African Diatom Index (SADI) were used in the study. From the results, the Percentage Pollution-Tolerant Valves (%PTV) of most urban sites in the Bloukrans River was above the 20% limit indicating the presence of organic pollutants. Although the foreign diatom indices were applicable in the study, the SADI had significant correlations with most water quality variables (p < 0.05) compared to other indices such as Watanabe Index (WAT), Biological Index of Water Quality Trophic Index (BIWQ) and Trophic Index (TI)). These results support wider use of the SADI as an indicator of water quality conditions in South African river systems. Finally, the observed variations in diatom community structure and composition as a result of changes in water quality were broadly in agreement with the results of macroinvertebrate FFG structure indicating that the two biological indicators can, and should, be used as complementary techniques in the biomonitoring of rivers and streams in South Africa.
- Full Text:
- Date Issued: 2020