Scipedia: Open Access Repository of the ExaQUte project: Deliverables

D3.3 Report of ensemble based parallelism for turbulent flows and release of solvers

Jesús Sánchez Pinedo — Thu, 17 Mar 2022 11:46:03 +0100

In this work we focus on reducing the wall clock time required to compute statistical estimators of highly chaotic incompressible flows on high performance computing systems. Our approach consists of replacing a single long-term simulation by an ensemble of multiple independent realizations, which are run in parallel with different initial conditions. A failure probability convergence criteria must be satisfied by the statistical estimator of interest to assess convergence. Its error analysis leads to the identification of two error contributions: the initialization bias and the statistical error. We propose an approach to systematically detect the burn-in time in order to minimize the initialization bias, accompanied by strategies to reduce simulation cost. The framework is validated on two very high Reynolds number obstacle problems of wind engineering interest in a high performance computing environment.

D4.5 Framework development and release

Jesús Sánchez Pinedo — Thu, 17 Mar 2022 11:42:03 +0100

This deliverable presents the nal release of the ExaQUte framework as result of task 4.6 of the project focused on the framework development and optimization. The rst part of the document presents an overview of the dierent parts of the ExaQUte framework providing the links to the repositories where the code of the dierent components can be found as well as the installation and usage guidelines. These repositories will include the nal version of the ExaQUte API and its implementation for the runtimes provided in the project (PyCOMPSs/COMPSs and Quake).

The second part of the document presents a performance analysis of the framework by performing strong and weak scaling experiments. In this case, we have focused on the analysis of the new features introduced during the last part of the project to support and optimize the execution of MPI solvers inside the framework. The support for OpenMP was already reported in Deliverable D4.3 [21]. The results of the experiments demonstrate that the proposed framework allow to reach very good scalability for the analysed Monte Carlo problems.

D6.5 Report on stochastic optimisation for wind engineering

Jesús Sánchez Pinedo — Thu, 17 Mar 2022 11:20:04 +0100

This report presents the latest methods of optimisation under uncertainties investigated in the ExaQUte project, and their applications to problems related to civil and wind engineering. The measure of risk throughout the report is the conditional value at risk.

First, the reference method is presented: the derivation of sensitivities of the risk measure; their accurate computation; and lastly, a practical optimisation algorithm with adaptive statistical estimation. Second, this method is directly applied to a nonlinear relaxation oscillator (FitzHugh–Nagumo model) with numerical experiments to demonstrate its performance. Third, the optimisation method is adapted to the shape optimisation of an airfoil and illustrated by a large-scale experiment on a computing cluster. Finally, the benchmark of the shape optimisation of a tall building under a turbulent flow is presented, followed by an adaptation of the optimisation method.

All numerical experiments showcase the open-source software stack of the ExaQUte project for large-scale computing in a distributed environment.

D5.5 Report on the application of multi-level Monte Carlo to wind engineering

Jesús Sánchez Pinedo — Thu, 17 Mar 2022 11:15:03 +0100

We study the use of multi-level Monte Carlo methods for wind engineering. This report brings together methodological research on uncertainty quantification and work on target applications of the ExaQUte project in wind and civil engineering.
First, a multi-level Monte Carlo for the estimation of the conditional value at risk and an adaptive algorithm are presented. Their reliability and performance are shown on the time-average of a non-linear oscillator and on the lift coefficient of an airfoil, with both preset and adaptively refined meshes. Then, we propose an adaptive multi-fidelity Monte Carlo algorithm for turbulent fluid flows where multilevel Monte Carlo methods were found to be inefficient. Its efficiency is studied and demonstrated on the benchmark problem of quantifying the uncertainty on the drag force of a tall building under random turbulent wind conditions.
All numerical experiments showcase the open-source software stack of the ExaQUte project for large-scale computing in a distributed environment.

D7.4 Final report on Stochastic Optimization results

Jesús Sánchez Pinedo — Thu, 17 Mar 2022 11:06:04 +0100

This deliverable report focuses on the final stochastic optimization results obtained within the EXAscale Quantification of Uncertainties for Technology and Science Simulation (ExaQUte) project. Details on a novel wind inlet generator that is able to incorporate local wind-field data through a deep-learned rapid distortion model and generates the turbulent wind data during run-time is presented in section 2. Section 3 presents the results of the overall stochastic optimization procedure applied to a twisted tapered tower with multiple design parameters within an uncertain synthetic wind field. Thereby, the significance of the developed methods and the obtained results are discussed and their integration in industrial wind-engineering workflows is outlined in section 4.

D8.4 Report on dissemination activities

Jesús Sánchez Pinedo — Thu, 17 Mar 2022 11:02:03 +0100

In Deliverable D8.1 (M18) we presented a first version of the Dissemination Plan for the ExaQUte project. The present document, prepared during the 2nd (and last) review period of the project, represents the updated version of the Dissemination Plan of ExaQUte, and therefore builds on the aforementioned deliverable. This document, thus, focuses on the new activities that have been undertaken from M18 to M42 regarding the dissemination activities.
It should be mentioned that the pandemic situation that started in Mach 2020 (actually, in Spain we were sent to confine to our houses the day after the celebration of the First Review meeting of this project) has had an impact in our life, and in our project, particularly in the dissemination actions.
The lockdown made impossible most of the activities related to dissemination (travels, conferences, consortium meetings…). But we reinvented our work and found different ways to do things and fulfil pour obligations. Still, in this deliverable, you will find indeed a difference between the type of action that we carried on up to M18 and our activities after M18.

D7.3 Report on UQ results and overall user experience

Jesús Sánchez Pinedo — Tue, 07 Sep 2021 12:10:22 +0200

This deliverable report focuses on the main Uncertainty Quanti cation (UQ) results obtained within the EXAscale Quanti cation of Uncertainties for Technology and Science Simulation (ExaQUte) project. Details on the turbulent wind inlet generator, that enables the supply of random, yet steady, wind velocity boundary conditions during run-time, are given in section 2. This enables the developed UQ workflow, whose results are presented on the basis of the Commonwealth Advisory Aeronautical Council (CAARC) as described in Deliverable 7.1. Finally, the completed UQ workflow and the results are evaluated from an application-driven wind engineering point of view. Thereby, the significance of the developed methods and the obtained results are discussed and their applicability in practical wind-engineering applications is tested through a complete test-run of the UQ workflow.

D4.4 API and runtime (complete with documentation and basic unit testing) for IO employing fast local storage

Jesús Sánchez Pinedo — Tue, 07 Sep 2021 11:58:37 +0200

This deliverable presents the activities performed on the ExaQUte project task 4.5 Development of interface to fast local storage. The activities have been focused in two aspects: reduction of the storage space used by applications and design and implementation of an interface that optimizes the use of fast local storage by MPI simulations involved in the project applications.

In the rst case, for one of the environments involved in the project (PyCOMPSs) the default behavior is to keep all intermediate les until the end of the execution, in case these les are reused later by any additional task. In the case of the other environment (HyperLoom), all les are deleted by default. To unify these two behaviours, the calls \delete object" and \detele le"have been added to the API and a ag \keep" that can be set to true to keep the les and objects that maybe needed later on. We are reporting results on the optimization of the storage needed by a small case of the project application that reduces the storage needed from 25GB to 350MB.

The second focus has been on the de nition of an interface that enables the optimization of the use of local storage disk. This optimization focuses on MPI simulations that may be executed across multiple nodes. The added annotation enables to de ne access patters of the processes in the MPI simulations, with the objective of giving hints to the runtime of where to allocate the di erent MPI processes and reduce the data transfers, as well as the storage usage.

D6.3 Report on stochastic optimisation for simple problems

Jesús Sánchez Pinedo — Thu, 14 Jan 2021 11:53:56 +0100

This report addresses the general matter of optimisation under uncertainties, following a previous report on stochastic sensitivities (deliverable 6.2). It describes several theoretical methods, as well their application into implementable algorithms. The specific case of the conditional value at risk chosen as risk measure, with its challenges, is prominently discussed. In particular, the issue of smoothness – or lack thereof – is addressed through several possible approaches. The whole report is written in the context of high-performance computing, with concern for parallelisation and cost-efficiency.

D5.3 Report on theoretical work to allow the use of MLMC with adaptive mesh refinement

Jesús Sánchez Pinedo — Wed, 13 Jan 2021 14:26:09 +0100

This documents describes several studies undertaken to assess the applicability of MultiLevel Monte Carlo (MLMC) methods to problems of interest; namely in turbulent fluid flow over civil engineering structures. Several numerical experiments are presented wherein the convergence of quantities of interest with mesh parameters are studied at different Reynolds’ numbers and geometries.

It was found that MLMC methods could be used successfully for low Reynolds’ number flows when combined with appropriate Adaptive Mesh Refinement (AMR) strategies. However, the hypotheses for optimal MLMC performance were found to not be satisfied at higher turbulent Reynolds’ numbers despite the use of AMR strategies.

Recommendations are made for future research directions based on these studies. A tentative outline for an MLMC algorithm with adapted meshes is made, as well as recommendations for alternatives to MLMC methods for cases where the underlying assumptions for optimal MLMC performance are not satisfied.

D6.4 Report on stochastic optimisation for unsteady problems

Jesús Sánchez Pinedo — Wed, 13 Jan 2021 13:57:10 +0100

This report brings together methodological research on stochastic optimisation and work on benchmark and target applications of the ExaQute project, with a focus on unsteady problems. A practical, general method for the optimisation of the conditional value at risk is proposed. Three different optimisation problems are described: an oscillator problem selected as a suitable trial and illustration case; the shape optimisation of an airfoil, chosen as a benchmark application in the project; the shape optimisation of a tall building, which is the challenging target application set for ExaQUte. For each problem, the current developments and results are presented, the application of the proposed method is discussed, and the work to be done until the end of the project is laid out.

D3.2 Report on parallel in time methods and release of the solvers

Jesús Sánchez Pinedo — Wed, 13 Jan 2021 13:51:27 +0100

In this deliverable we provide the details related to the design, implementation, and scalability analysis of Space Time Balancing Domain Decomposition by Constraints (STBDDC) preconditioners that have been implemented in the FEMPAR project [8]. First, we describe the state of the art of space-time methods in Sect. 2 and we then provide some details of our particular implementation in Sect. 3. Next, in Sect. 4, we present a detailed description of the numerical experiments performed during the project showing the excellent scalability results that these algorithms permit to achieve. At the same time, we show the limitations of these algorithms when dealing with nonlinear problems. Finally we draw some concluding remarks in Sect. 5.

D5.4 Report on MLMC for time dependent problems

Jesús Sánchez Pinedo — Wed, 13 Jan 2021 13:19:05 +0100

In this report, we study the use of Multi-Level Monte Carlo (MLMC) methods for time dependent problems. It was found that the usability of MLMC methods depends strongly on whether or not the underlying time dependent problem is chaotic in nature. Numerical experiments are conducted on both simple problems, as well as fluid flow problems of practical interest to the ExaQUte project, to demonstrate this. For the non-chaotic cases, the hypotheses that enable the use of MLMC methods were found to be satisfied. For the chaotic cases, especially the case of high Reynolds’ number fluid flow, the hypotheses were not satisfied. However, it was found that correlations between the different levels were high enough to merit the use of multi-fidelity or control-variate approaches. It was also noted that MLMC methods could work for chaotic problems if the time window of analysis were chosen to be small enough. Future studies are proposed to examine this possibility.

D7.2 Finalization of "deterministic" verification and validation tests

Jesús Sánchez Pinedo — Wed, 13 Jan 2021 12:30:47 +0100

This deliverable focus on the verification and validation of the solvers of Kratos Multiphysics which are used within ExaQUte. These solvers comprise standard body-fitted approaches and novel embedded approaches for the Computational Fluid Dynamics (CFD) simulations carried out within ExaQUte. Firstly, the standard body-fitted CFD solver is validated on a benchmark problem of high rise building - CAARC benchmark and subsequently the novel embedded CFD solver is verified against the solution of the body-fitted solver. Especially for the novel embedded approach, a workflow is presented on which the exact parameterized Computer-Aided Design (CAD) model is used in an efficient manner for the underlying CFD simulations.

It includes:

A note on the space-time methods
Verification results for the body-fitted solver based on the CAARC benchmark
Workflow consisting of importing an exact CAD model, tessellating it and performing embedded CFD on it
Verification results for the embedded solver based on a high-rise building
API definition and usage

D1.3 First public Release of the solver

Jesús Sánchez Pinedo — Tue, 12 Jan 2021 13:46:56 +0100

This deliverable presents the software release of Kratos Multiphysics, together with the XMC library, Hyperloom and PyCOMPSs API definition [8]. This report is meant to serve as a supplement to the public release of the software. Kratos is “a framework for building parallel, multi-disciplinary simulation software, aiming at modularity, extensibility, and high performance. Kratos is written in C++, and counts with an extensive Python interface”. XMC is a python library for hierarchical Monte Carlo algorithms. Hyperloom and PyCOMPSs are environments for enabling parallel and distributed computation.

D1.1 Solvers "stub" implementation of the capabilities to be delivered

Jesús Sánchez Pinedo — Tue, 12 Jan 2021 12:59:31 +0100

The current deliverable describes the initial API available for the solvers. The API is intended to be based on the Kratos Multiphysics fraemework and will evolve during the project. The current deliverable describes the essential features of the interface and provides an initial working implementation to be used as a basis for the future developements.

The initial implementation described here is currently operative on the master branch of Kratos. As of the end of July 2018 (moment of handing in of current deliverable) the interface is operative and is being used “in production”. Nevertheless, it still does not fully support the model serialization capabilities that are needed for pyCompSS and HyperLoom.

The interface is documented in the project wiki page [wiki]. The same documentation is also presented in the current deliverable.

D1.4 Final public Release of the solver

Jesús Sánchez Pinedo — Tue, 12 Jan 2021 12:48:00 +0100

This deliverable presents the final software release of Kratos Multiphysics, together with the XMC library, Hyperloom and PyCOMPSs API definitions [13]. This release also contains the latest developements on MPI parallel remeshing in ParMmg. This report is meant to serve as a supplement to the public release of the software. Kratos is “a framework for building parallel, multi-disciplinary simulation software, aiming at modularity, extensibility, and high performance.

Kratos is written in C++, and counts with an extensive Python interface”. XMC is “a Python library for parallel, adaptive, hierarchical Monte Carlo algorithms, aiming at reliability, modularity, extensibility and high performance“. Hyperloom and PyCOMPSs are environments for enabling parallel and distributed computation. ParMmg is an open source software which offers the parallel mesh adaptation of three dimensional volume meshes.

D2.2 First release of the octree mesh-generation capabilities and of the parallel mesh adaptation kernel

Jesús Sánchez Pinedo — Tue, 12 Jan 2021 12:35:18 +0100

This document presents a description of the octree mesh-generation capabilities and of the parallel mesh adaptation kernel. As it is discussed in Section 1.3.2 of part B of the project proposal there are two parallel research lines aimed at developing scalable adaptive mesh refinement (AMR) algorithms and implementations. The first one is based on using octree-based mesh generation and adaptation for the whole simulation in combination with unfitted finite element methods (FEMs) and the use of algebraic constraints to deal with non-conformity of spaces. On the other hand the second strategy is based on the use of an initial octree mesh that, after make it conforming through the addition of templatebased tetrahedral refinements, is adapted anisotropically during the calculation. Regarding the first strategy the following items are included:

Description of the octree-based AMR kernel with space-filling curves.
Description of a outer, wrapping AMR layer supporting FEM needs.
Numerical results in a distributed environment.

Regarding the second strategy the following items are included:

An outline of the anisotropic mesh adaptation algorithm.
A description of the state of the art of the implementation.
A discussion on the ongoing and future work.

D1.2 First realease of the softwares

Jesús Sánchez Pinedo — Tue, 12 Jan 2021 11:49:23 +0100

This deliverable presents the software release of the Kratos Multiphysics software [3], ”a framework for building parallel, multi-disciplinary simulation software, aiming at modularity, extensibility, and high performance. Kratos is written in C++, and counts with an extensive Python interface”. In this deliverable we focus on the development of Uncertainty Quantification inside Kratos. This takes place in the MultilevelMonteCarloApplication, a recent development inside the software that allows to deal with uncertainty quantification.

D4.3 Benchmarking report as tested on the available infrastructure

Cecilia Soriano — Thu, 27 Feb 2020 16:31:02 +0100

The main focus of this deliverable is testing and benchmarking the available infrastructure using the execution frameworks PyCOMPSs and HyperLoom. A selected benchmark employing the Multi Level Monte Carlo (MLMC) algorithm was run on two systems: TIER-0 (MareNostrum4) and TIER-1 (Salomon) supercomputers. In both systems, good performance scalability was achieved.