### Our Goal #

For EXN/Aero to be the number one CFD software choice for engineers in small and medium firms.

### Why Try Something New? #

The biggest supercomputing facilities are continually pushing boundaries and adopting new technologies as they approach the goal of exa-scale computing.

The adoption and rapid increase in multi-CPU cluster size arose because individual processor clock-speeds stopped getting faster in the early 2000s.

By 2010 in order to develop petascale supercomputers, and to minimize power consumption, muti-CPU clusters  began  to adopt energy-efficient co-processing technologies such as GPUs.

These innovations drive the world’s fastest supercomputers and they inevitably impact the work of everyday scientists and engineers.

General-purpose CFD now has an excellent opportunity to capitalize on the co-processor trend to increase simulation performance and dramatically reduce costs.

We also see an entrenched licensing model that limits what engineers can do with CFD and hinders the ability for users to scale their business.

The current trend toward increased power and lower cost for computing resources gives us the ability to try new things with licensing and offer an alternative high-performance CFD product built on a whole new set of assumptions.

### How we Get There #

Our unique parallel computing design enables engineers to operate economically on the cloud and to build inexpensive, powerful systems that boost CFD solution speed by an order of magnitude.

EXN/Aero is built from the ground up to take full advantage of hybrid parallel computing architectures like multi-core CPU and massively-parallel GPU. It exposes every aspect of the simulation to parallelism, removing serial tasks and maximizing speed-up.

We’re also innovating on the distribution model, enabling engineers to access a large amount of compute power on-demand and with minimal risk to their operations.

### EXN Concept #

The EXN manycore computing platform was built to do engineering calculations quickly and economically on massively parallel processors. This approach is the future for engineering simulation; as industry demands greater fidelity, speed, and complexity from its simulation tools, software designers need to optimize — and parallelize — to keep up.

### Cell - Interface Desgn #

Our patented design subdivides simulation data and associated computing tasks into two types of objects: cells and interfaces.

Cell Objects contain CFD mesh and represent the bulk of the computational load. Cells might contain structured or unstructured mesh, and use single or double precision arithemetics.

Interface Objects are multi-function data transformation utilities with many uses, including boundary condition input, communication between cells, reduction of output data, addition of source terms and interpretation of sensor inputs.

In the schematic below, cells ( grey mesh blocks) are connected to each other and the outside world via interfaces (red lines).

The number and location of Cell and Interface objects are largely aligned with the block/interface specification at the meshing stage. Cells and interfaces are tagged with attributes such as data structure and precision arithmetics that help to identify the type of computation required.

### Cell-Based Mapping Module #

Distribution and assignment of Cell and Interface objects to compute resources depends on a parallel load balancing module we call the CBMM.

Cell and Interface attributes are used to automate cell splitting and automatic interface generation as part of off-line load-balancing operations. In practical terms it means some Cell and Interface compute tasks are assigned to GPU resources, while others are sent to the CPU resources, all with goal of maximizing parallelism and minimizing computing time.

The CBMM effectively separates the engineering problem from the underlying compute hardware. It therefore plays an important role in enabling EXN-based software to adapt quickly to new architectures in an environment where technology is changing very rapidly.

### Flexible Manycore Computing #

The Cell and Interface design, working in concert with the CBMM allows for greater creativity and flexibility in the way a CFD run is parallelized. Optimal use of manycore compute resources requires that:

1. Cell and interface objects are subdivided by task and data type attributes
2. The computing environment is automatically interrogated to determine its layout
3. Tasks can be dynamically assigned to resources depending on predicted load

These requirements exist because the manycore computing environment consists of diverse compute architectures, and not all architectures are equally good at every computing task.

Expensive Cell tasks are computed on GPU devices much of the time, where the thousands of simple processors work in concert to accelerate physics calculations

Interface tasks are computed based on the location of the Cell objects in order to minimize communication overhead.  For example if two Cell objects that communicate both reside on GPUs then the interface data and tasks will also reside on GPUs.  However, if one of the Cells is located on a host CPU, and its tasks are executed there, then the connecting Interface will reside there also.

EXN efficiently load balances a single simulation over manycore computing resources, but co-processors such as GPUs also allow multiple simulations to access the same device. This provides further opportunities for speeding up the engineering workflow by allowing multiple simulations to run concurrently on the same shared resource.

EXN simulations are maintained in a higher level object called a Propagator.  To take advantage of the large amounts of parallelism becoming available on manycore systems, it is possible to solve problems involving multiple Propagators working together in a coupled manner. We are demonstrating this capability today with our prototype multi-propagator parallel-in-time (aka para-real) solution technique. Para-real is designed to accelerate large steady-state runs, and also to speed up the start-up phase for transient simulations.

### Solver Description #

EXN/Aero is a pressure-based, finite volume, fully implicit CFD solver based on the SIMPLEC method with Algebraic Multigrid (AMG).

The EXN/Aero solver utilizes a hybrid multiblock approach where structured and unstructured mesh regions are concurrently solved in their native data format, preserving the inherent benefits of each type.  This allows EXN/Aero to better exploit CPU and GPU resources, playing to the strengths of a heterogeneous compute architecture layout.

EXN/Aero’s true multiblock approach also allows the user to specify the computational precision of each block. In this way, double precision can be applied only where it is needed (e.g. boundary layers), while other areas of the mesh operate in single precision (e.g. far-field mesh). This approach minimizes the memory footprint and increases performance relative to an all-or-nothing double precision approach.

### Solver Features & Models #

The following solver modes are available in release version 1.0x

• Steady State & Transient solution modes
• Structured & unstructured data (mesh) support
• Block-wise mixed precision

The following engineering models are available in the current release

• RANS – SST & k-$\omega$ models
• LES – Smagorinsky-Lily
• Detached eddy – SST formulation
• Synthetic turbulence generation for LES/DES
• Heat transfer
• Pressure-based compressible flow
• Buoyancy-driven flows
• Conjugate Heat Transfer
• Many 2 many block connectivity
• Multiphase (DQMOM) particle transport
• Passive scalar transport
• Prescribed motion
• Porous media

### EXN/Aero Walkthrough #

Welcome to the EXN/Aero walkthrough, and thanks for deciding to evaluate our product.

This tutorial is designed to get you up and running quickly and will help you understand the EXN/Aero workflow. We’ll cover:

1. Preparations
2. Importing a mesh
3. Setting up physics
4. Running the simulation
5. Monitoring the simulation in progress
6. Post analysis
7. Summary of the project file structure

The walkthrough takes an hour or so and you’ll probably notice some similarities to the live walkthrough we did with you during your trial.

### Connecting to EXN/Aero on the NIMBIX cloud #

This part of the walkthrough instructs you how to set up an account on our cloud providers.

Computing time on cloud resources is billed at an hourly rate and instances do not shut down automatically.

If you want to avoid unnecessary charges for compute hours make sure your job is shut down at the end of the compute task.

2. From the dashboard page, select ‘Compute’ on the right and type EXN into the search bar. You will see three applications

• EXN/Aero Meshing – opens an instance with a full installation of Pointwise, billed at an hourly rate.
• EXN/Aero Setup – opens an instance with the EXNView user interface on a low-powered compute system, useful for setting up runs without paying for big compute resources.
• EXN/Aero Solve – opens up an instance with the EXNView user interface with the solver accessible.

Select any of these applications to launch a compute instance.

3. For EXN/Aero meshing and for EXN/Aero Setup, resource allocation is fixed. For EXN/Aero solve, you will need to select the resource allocation. You can choose one of four compute node hardware layouts:

• 4 CPU + 1 Nvidia K40
• 4CPU + 1 Nvidia K80
• 16 CPU + 2 Nvidia K40
• 16 CPU + 2 Nvidia K80

Roughly speaking, two K40 cards are about equal to 1 K80 card in terms of memory and parallel core.

GPU computations are usually memory-limited; for an incompressible isothermal flow, you can expect to fit about 50-60 million control volumes on a dual K80 instance.

4. After selecting resources, click ‘Submit’.

The ‘Dashboard’ tab will now show that an EXN/Aero job has started. This display updates automatically. The session takes 30 – 60 seconds to boot up. You will know it is ready when a thumbnail image of our desktop appears beneath the job information.

The connect button opens a browser-based remote connection to the EXN/Aero desktop.

You can also connect with TigerVNC client, which has less display latency. The small “i” next to the session’s power button provides information on how to run through TigerVNC.

5. Click the desktop thumbnail and a new browser tab will connect you to the EXN/Aero desktop. There several icons available here:

• Demo cases – links anonymously to our FTP server containing links to many of our demo cases
• User Manual – opens a browser window within the session that links to this manual
• Release Notes – contains information on the latest release including bug fixes an major new capabilities
• Report a Bug – links to our zendesk support portal
• EXN/Solve – available only in solve sessions, launches the EXNView user interface
• EXN/Setup – available only in setup sessions, launches the EXNView user interface with no connection to the solver
• EXN/Meshing by Pointwise – opens a full version of Pointwise for the duration of the session.

EXNView and Pointwise take 20-30 seconds to launch. Double click once and wait please!

Your account-bound persistent storage directory is /data. This is the only directory that persists when the server is shut down. You must store all simulation files and results files in ‘/data’ if you wish to keep them.

6. When the run is complete and data is downloaded or safely stored in ‘/data’, remember to shut down the server.

Closing the browser-based remote desktop does not kill the session. You must click the power button in a current session and confirm shutdown to end a session otherwise compute hours will continue counting and you will be billed for them.

Please re-read the above and make it part of standard procedures…we don’t want our users to be billed for hours they do not use.

7. To double check that the job was terminated, the job should appear as “recent” on the dashboard. You will also receive a ‘job shutdown’ email from Nimbix.

If you encounter an error shutting down a job, please reach out to Nimbix directly. They are very responsive and can make sure you are not accidentally billed for a job that was left running because of the error

### A first look at EXNView #

EXN/View is our lightweight UI designed to get you up and running quickly.

In addition to the menu & toolbar, it has two main areas:

1.  Tree viewer: This is the area where you specify physics, model settings, boundary conditions and solver settings. Menu selections are dynamic and become available only when the relevant model is activated. For example, the turbulent kinetic energy and turbulent dissipation variables are only available when a turbulence model is active.
2. Mesh window: This area displays a wireframe of the model, output from the solver and indicators to help identify the mesh axes and scale. Left click to rotate, right click to pan, scroll to zoom. Boundaries and blocks in the mesh are highlighted when they are selected
in the tree viewer.

### Preparations #

You can use your own mesh, or one of our other example meshes if you like, but for this tutorial we will assume you are studying a cylinder in cross-flow.

Open a Firefox browser by hitting the Envenio logo in the bottom left corner of the the EXN/Aero desktop. Navigate to and launch Firefox.

Go to our public FTP site: ftp://acenetm01.cs.unb.ca and navigate to

/nimbix_mesh/2Dcylinder/Cylinder.proj/

Alternatively, you can double click on the Demo Cases folder on the the desktop and connect to our public FTP site directly. There you can find many meshes to work with.

### Create a Project #

Click File to open the File Menu, and then select Create New Project.
This will create a new project folder with the name you assign. If you are on cloud resources, make sure that the save location is in the persistent storage directory (/data).
For nimbix users, the persistent storage directory is /data. Everything not saved in the data directory will be deleted when you terminate the compute job.
If successfully created, the Envenio logo disappears from the mesh outline view and the project tree area populates with some default entries.

### Import a mesh into EXN/View #

On the menu bar, click Mesh and select Import CGNS File. This will open another file dialog, where you need to navigate to your CGNS file.
Behind the scenes, EXN/View makes a copy of the CGNS file to your project directory and will give it the same name as the project, plus the .cgns extension. For example, the project name ‘test.proj’ will copy and rename the CGNS file to ‘test.cgns’. EXN/View also creates a .json file, which contains data to generate the wireframe object in the mesh outline view.
A new tab will appear in the render view, and shortly after, the mesh geometry will appear. This may take a few seconds or several minutes, depending on the size and complexity of the mesh.
An output ticker in the bottom left of the tree view displays that the mesh has successfully loaded

### Models, Properties & Initial Conditions #

Within the Cell_families branch of the tree view, you can specify:

• Model settings
• Fluid properties
• Initial conditions for the fluid volume
• Set sub-domains
• Source terms
• Running statistics

### A bit about Cells & Cell Families #

When we say ‘Cell families’ we do not refer to individual control volumes.

Cells are mesh blocks and Cell families are groups of mesh blocks within the mesh. In EXN/Aero, blocks and families are given attributes, and they do not need to be consistent throughout the mesh.

Some attributes are inherited and cannot be modified. For example, unstructured and structured mesh data are baked into the mesh blocks by the meshing tool. Mesh generators can also enforce single precision.

Other attributes are malleable. If the mesh is exported as double precision, EXN/View can set individual cells as single or double precision in a zonal fashion. Cells can also be designated as sub-domains that will inherit special source terms or models (e.g. porous media).

### Editing Cell Families #

Rather than using static images, a dynamic version of the EXN/View UI is available here: EXN/View Live section. This shows the complete and correct problem set up ready for execution.

If you are following this guide as part of a trial, you can refer to the example UI to get the exact values needed to reproduce our results.

This will give you a general idea about the set up procedure, but during your trial you can select any settings you like.

1. Create a fluid domain

Right click on the Cell Families and select ‘Add Cell Family’, then select ‘FluidDomain’. A new fluid domain will appear beneath Cell Families.

Next, select Cell Blocks and open the drop-down menu to add blocks to the fluid domain. If your mesher gave the blocks a family name (‘unspecified’ is the default here) then select the family. If not, then select multiple mesh blocks by holding the CTRL key or CTRL+SHIFT and click on the blocks you wish, just like you would in a file manager.

2. Turn on turbulence and energy models

Expand Model Parameters and then Turbulence. Set Turbulent Flow to on. Selection drop-downs appear that allow you to select from multiple LES and RANS models. For this example, select RANS and SST.

Expand Energy and set Energy Equation to on.

3. Set Initial Conditions and Properties

Expand both of these branches and copy the values over to your trial run.

4. From the File menu, select ‘Save Project’.

### Editing BC Families #

Rather than using static images, a dynamic version of the EXN/View UI is available here: EXN/View Live section. This shows the complete and correct problem set up ready for execution.

If you are following this guide as part of a trial, you can refer to the example UI to get the exact values needed to reproduce our results.
This will give you a general idea about the set up procedure, but during your trial you can select any settings you like.

1. Insert an inlet condition

Right click BC Families and select Add BC Family…, then select InletBC.

Next, select BC Location and open the drop-down menu to add interfaces to the inlet condition. If you mesher gave the interface a family name (‘inlet’ is used here) then select that family. If not, then select multiple mesh blocks by holding the CTRL key or CTRL+SHIFT and click on the blocks you wish, just like you would in a file manager.

Copy inlet settings form the EXN/View live section. In your trial, you can modify these settings any way you like to generate different results

2. Insert an outlet

Same procedure as for the inlet. Select the ‘outlet’ family at the BC Location.

3. Insert wall boundaries

Same procedure as for the inlet. Select the ‘wall’ family at the BC Location.

4. Insert symmetry planes

Same procedure as for the inlet. Select both the ‘symmetry’ and ‘sides’ families at the BC Location using the CTRL key.

5. From the File menu, select ‘Save Project’.

### Editing Solver Control Settings #

In the Solver Control branch the settings for the flow solver are set. There are a lot of settings under this branch, but for the purposes of the present tutorial, the more important ones will be shown. Refer to the EXN/Live section to see options under Solver Control.

Under the Simulation branch:

Total Run Time sets the total physical time for an unsteady simulation. When the simulation time reaches this value, EXN/Aero writes output files as per save/output settings, writes a restart solution to the CGNS file and exits.

Delta Time sets the time step size.

Initial time from Solution has two options: On and Off. Upon restart, setting this variable to ‘On’ causes EXN/Aero to use the final simulation time from the previous run as the start time of the current run. If set to ‘Off’ then the initial time is set to zero.

Initial Values from Solution enables the user to pick which solution data arrays are initialized from a previous solution in a restart run. If an array is required but not included in this list on restart, EXN/Aero defaults to the initial value settings in Cell Families.

Transient Output Frequency sets the interval in time steps at which solution output files (in VTK format) are written to the solution directory. A value of ‘1’ outputs every time step, while a value of ‘0’ disables VTK output entirely. Writing VTK files to the hard drive is very time consuming and can slow the solution time significantly if the output frequency is small.

Solution Backup Frequency sets the interval in time steps at which EXN/Aero writes a restart-able solution to the CGNS file. A value of ‘1’ writes a solution restart at every time step, while a value of ‘0’ disables backups entirely. Writing solution arrays to the CGNS file is very time consuming and can slow the solution time significantly if backup frequency is small.

Transient Output Variables controls which solution variables are included in the VTK output files for fluid volumes. Some output variables are vectors (e.g. Velocity) while others are scalars or individual components of vectors (e.g. Temperature, Velocity_X). Picking a vector in this field is sufficient to include its components — for example, picking Velocity means the user does not also need to select ‘Velocity_X’, ‘Velocity_Y’ and ‘Velocity Z’.

Transient Output Variables BC controls which solution variables at the domain boundaries are included in the VTK output files, simultaneously with the VTK output for fluid volumes. Vector & scalar picking rules are the same as for Transient Output Variables.

### Assigning Resources #

Resource assignment works differently from CPU-only parallel computing. CPU/GPU resources are not reserved for the task and parallelization is handled automatically through a combination of OpenMP and CUDA. There are 3 important settings for resources set at the Resources branch:

CPU Threads sets the number of  CPUs allocated to the EXN/Aero launch. This number cannot be greater than the available cores on the user’s system.  The bare minimum is to assign one CPU for every GPU device. It is usually advisable to set CPU count to the number of GPU devices plus one (e.g. 2 GPU + 3CPU).  More CPUs should be allocated if the multi-block mesh has a large number of internal block-to-block interfaces.

GPU Devices sets the number of GPU cards allocated to the EXN/Aero launch. This number cannot be greater than the available GPU cards installed on the user’s system. Keep in mind that GPUs do not exhibit strong scaling in all cases. Small runs are often faster on a single GPU rather than multiple GPUs.

Memory Size (in the Advanced Mode) is a multiplier for the default memory allocation for data arrays. Sometimes, very large models require this limit to be increased. If it is too large, the system will report that it has insufficient memory at run time.

### EXN/View Live #

No valid security key found. Please use at least the following shortcode:
Please also check in the html mode that your shortcode does only contain normal spaces and not a &nbsp; instead. It is also possible that you use wrong quotes like “ or ”. Only " is valid!

### Running a simulation #

Run EXN/Aero launches EXN/Aero on the specified compute resources. After running, the status bar (lower left corner at the GUI) will indicate the process ID number of the session. Shortly thereafter, an output window should appear, detailing the solver output during solution.

When an EXN/Aero session is active, it will appear in the Run in the menu bar with a three digit suffix, indicating the name of the output directory in the project directory, along with a completion percentage. Hovering over this option will display a sub menu with options specific to that run session.

Show displays data from the EXN/Aero session in an output tab.

Backup causes EXN/Aero to write a solution file to a new CGNS file having the same name as the original CGNS file, with a BAK suffix and sequential number. EXN/Aero continues to run after a backup.

Update causes EXN/Aero to re-read selected fields in the CGNS file (e.g. time step) at the end of the next time step. No solutions are written and EXN/Aero continues to run.

Stop causes EXN/Aero to complete the current iteration, write a solution file for restart to the CGNS file, and then exit.

Kill stops the process without writing solution files.

Properties opens a small dialog window with information about the EXN/Aero session, including the duration of the current session.

### Project file structure #

The project directory contains all data related to a simulation in EXN/Aero, including setup, solver control and results files.  For the sample file structure shown here, the simulation has already been set-up and several simulations have been performed. Some files may not be present in the user’s directory at early stages of the set up process.

When the user creates a new project in EXN/View, the project parent folder is created. The image below shows the contents of the Example.proj folder.

#### Parent Directory

CGNS file (.cgns) contains mesh information and is created by EXN/View when the user imports a mesh. Upon import, the CGNS file contains only mesh and block connectivity information, but eventually will contain custom folders after the problem is set up and ready to run on EXN/Aero.

JSON file (.json) is a reduced version of the mesh information in the CGNS file. It is used by the render view in EXN/View to visualize the outline of the mesh blocks.

CVC directories and Results Directories are explained below.

#### CVC directories (_cvc#.##)

XML structure files are housed in the CVC directories, where #.## represents the version number. CVC stands for ‘CGNS Version Control”. These are updated to the latest version each time a project is saved in EXN/View. Old CVC files are kept for reference, but can be deleted once the user is satisfied that they are no longer needed.

An example of a CVC directory contents is shown below:

XML structure files contain all project settings. These are created by EXN/View when the user creates a new project. They can also be imported by other projects that share similar settings. The formatted text contained in XML files drives the appearance of the tree viewer in EXN/View, and can be edited manually in any text editor if the user so desires.

CVC.log is a record of version changes.

Info.txt provides some basic information for manual editing of XML files and cvc.log. This might be of interest for advanced users wishing to automate XML structures using scripts for running large parameter sets

#### Results directories (-###)

Simulation output data files are contained in the results directories. These are created each time EXN/Aero is launched, and are given a unique name, where ### is the sequential number of the run.

An example of the results directory contents is shown below

output_stream_prop-0.txt contains condensed simulation health data such as residuals and global mass imbalance. This is the file read by the output view in EXN/View.

error_stream_prop-0.txt contains warnings and errors encountered throughout the run that an help to diagnose any issues that might be encountered.

SQLite Output (-it_1-ts_1.sqldb) is a SQLite database file that contains an array of solver outputs. By default it contains equation residuals, CPU timing information. Monitor points and integrated quantities also end up in the SQLite file as additional tables.

The SQLite database can be appended and queried in real time, and it is the primary input to the simulation monitoring tool EXN/Vizer. There are several open-source tools available for viewing, editing and exporting data from SQL databases.

XML directory contains an exact copy of the structures used at run time. These are useful if a user wishes to re-create the exact conditions of a run, and can be imported directly into EXN/View.

Memory Map  directory contains memory usage summaries. This data is used mostly for debugging and will not be of interest to the average user.

Parareal Profile Status (parareal_profile_win-1.txt) contains diagnostic information from Envenio’s parallel-in-time solver. This capability is in alpha and the contents of this directory will not be of interest to the average user.

#### VTK sub-directory

This contains all field data outputs from the simulation, saved in VTK database format. This format is native to Paraview but is supported by several other visualization tools.

Drilling down through this file structure to the Iteration-1 directory, and then a step further into the Example_1 directory shows the following:

VTK Multiblock file (.vtm) contains information on how the various data blocks in the results folder are connected. Its name will always be identical to its corresponding results folder.

VTK blocks folder (-###) contains the data blocks that are connected to the multi-block file. In the above image there are four blocks. One each for wall boundaries, outlet plane, inlet plane, and the mesh block.

### Finding solution files #

The file management section talks about the various files within the project directory in detail. For the purposes of the walkthrough, it suffices to know that files are written in 3 main areas:

Restartable solutions are written directly to the CGNS file in the main project directory. The CGNS file contains only one solution file at a time. Backup solutions are copies of the main CGNS file written within a solution directory.

Monitor & probe data are written to a SQL database that can be viewed from the EXN/View UI. If a simulation is in progress, they can be displayed and updated as new data are appended to the SQL file by the solver. If a simulation is complete, the SQL file is static.

VTK files contain complete solutions, including cell volumes and boundaries, and are written to the solution directory. Each one of them represents an instant in time. If the solution is transient, the files are written out with sequential numbers equal to the time step count at the time of writing. If the solution is steady-state, there is one solution written at the end of the run.

### Meshing Best Practices #

This section explains how to label blocks and boundaries during the meshing process to optimize the workflow and take advantage of subdomain designation in EXN/View.

We provide examples from the Pointwise meshing tool, but equivalent operations are available in other meshing tools.

A simple hybrid structured / unstructured mesh of a square channel flow shown below is used to explain the concept.

### Mesh Requirements #

Please adhere to these requirements to ensure a fault-free run on the EXN/Aero solver.

• Export in CGNS format.
• Export mesh in a single file.
• Subject to the single file requirement, the mesh can be structured, unstructured, or hybrid.
• Minimum included angle throughout the mesh should be greater than 15 degrees.
• Aspect ratios less than 1000 in single precision, or else plan to use double precision.
• Minimize skewness.
• Check for negative mesh volumes.
• Label all boundary conditions.
• As much as possible, group blocks into block families and boundary patches into BC families.

If you are exporting CGNS from Ansys:

1. Set up blocks, associations and sizing, then compute the pre-mesh.
2. In the Pre-Mesh context menu choose “Convert to Muliblock Mesh”. Choose “Volume” when asked which kind of domains you want to convert.
3. You’ll get the multiblock mesh (the tree entry for it looks different from the one for unstructured mesh, it doesn’t have Shell/Volume nodes, instead it has Domains/Subfaces and etc.).
4. ICEM Parts do not automatically convert to BC locations during the export. Set up BCs via “Boundary Conditions” button at the “Output Mesh” tab. If you hadn’t selected an output format already, you will receive a prompt for it when you’ll press this button. Choose CGNS.
5. Expand the “Mixed/unknown” branch; this lists all the Parts that are present. Expand a surface part, click “Create new”. Choose “BCType” entry. Don’t worry about the type because this is set in EXN/Aero setup. Repeat for all other boundary parts. This is where we group the individual boundary patches into families.
6. Press “Write Input” button. You’ll be prompted for the BC file, then for Multiblock mesh file, then, finally, which domains you want to export – all or just some of them. In the CGNS export dialog do not choose “Yes” for “Create default BC patches?”. Doing so will create a BC location called “DEFAULT_BOUNDARY” that contains all block-to-block connections. This is not needed to set up problems in EXN/Aero.
7. Don’t forget to specify the proper scale factor.
8. CGNS version “3.1 ADF” is a good format choice.

### Labeling #

The figure below shows a section of the control panel in Pointwise that is used to alter boundary (Domains) and volume (Blocks) names. In the figure below, three blocks: blk-1, blk-2 and a third labeled “blk-porous_media” are shown, along with a list of domains.

In EXN/View, the same block names are listed one-for-one under the Cell Families branch, where they can be selected and assigned to a FluidDomain.  The same convention holds for BC Families.

Notice that the first two entries in the drop-down list under Cell Blocks are not present in the block list from the meshing tool. In many meshing tools, blocks and block boundaries can be grouped using ‘family names’. Selecting blocks by family name automatically includes all mesh blocks bearing that name. In a mesh with a continuous fluid domain for example, it is sufficient to label all blocks as the same fluid family name and select only the family name in EXN/View to include them in the same fluid domain specification. The same holds true for boundaries.

### Family Names #

The figure below shows another Pointwise example of family labeling during the meshing process in Pointwise. Here, we see nine boundary domains combined to one “symmetry” boundary family (yellow), one ‘outlet’ boundary family (orange), and two blocks are combined to one “Fluid_str” cell family. Family names can be chosen arbitrarily by the user.

### Subdomains #

If required, the user may identify certain blocks as subdomains within a cell family. For example, to treat the blue block as porous media, the corresponding cell is selected under the Cell Blocks field of the sub-domain.

### Building a FAST mesh #

Manycore compute resources (CPU & GPU)  dramatically improve speed and reduce costs of running a CFD solver. On GPU resources especially, parallel comptuations on EXN/Aero are handled automatically with minimal user-intervention.

There are a few caveats when operating in a manycore CPU/GPU environment. To realize their maximum potential however, it’s helpful to understand how the devices work, and to be aware of their general strengths and weaknesses.

The following is a list of optimizations that users can apply in most situations.

### Use Structured Blocks Where Possible #

EXN/Aero supports hybrid multiblock meshing and computes both structured and unstructured blocks (Cells) in their native format. EXN/Aero does this to improve performance, in particular with GPU processing — GPUs have a very high compute-to-memory bandwidth ratio and structured data is extremely cache-efficient due to its aligned arrangement. In testing, developers at Envenio see about 2.4x speedup operating on structured data vs. the exact same mesh written as unstructured hex data.

### Be Mindful of Interface Count #

Multiblock meshing allows a lot of flexibility, but block-to-block interfaces introduce computational overhead. EXN/Aero mitigates this overhead by processing many of the interface operations on CPUs, while mesh blocks are computed concurrently on GPUs.  An ideal mesh is broken into enough blocks, which can be easily spread out across compute resources, but without many interfaces, as they become a bottleneck in the computation.

It is good practice to avoid using very small blocks.  GPU resources are very good at handling large amounts of data, but if a block becomes too small the performance gains of a GPU are reduced. If a user is generating large numbers of small structured blocks solely to avoid unstructured mesh it is often better to go with the larger unstructured block.  Like many things, balance is key and Envenio support engineers can help users strike that balance.

### Minimize Double Precision Regions #

High Reynolds number flows, and high aspect ratio meshes often require double precision calculations near wall regions, but not in far-field areas. EXN/Aero’s multiblock approach enables users to specify single or double precision arithmetic block-wise.

In this way, the speed penalty and increased memory footprint associated with double precision calculations can be minimized. This is especially important on GPU resources where the double-precision penalties are greater.

### The Right Blend of CPU & GPU #

Users can specify the number of CPU cores and GPUs to use during the simulation.  EXN/Aero distributes the workload across the resources to optimize performance.  The most efficient resource selection depends on the layout of a user’s system, but a few general rules apply:

1. The number of CPU threads (cores) should always be at least one greater than the number of GPUs.
2. Increase the number of CPU threads if using a mesh with a large number of block-to-block interfaces.
3. Two GPUs is not always faster. Often, it’s better to ‘fill’ a single card and use a smaller cloud resource to minimize the cost of the simulation.

### U.I. Overview #

EXN/View is a lightweight simulation set up utility intended to get you up and running quickly with your CGNS formatted mesh.

In EXN/View you can:

• Load and render a wireframe mesh.
• Set up physical models for the fluid domains.
• Set up boundary and initial conditions.
• Edit solver control variables.
• Launch EXN/Aero runs

### File #

Create New Project opens a file dialog and asks the user to designate a project directory. This creates a new directory with a .proj extension. A sub-directory structure is also created, containing default XML structures and the current CVC version number.

Open Existing Project opens a file dialog where you can navigate to your project and open it. Single-click on the ‘.proj’ directory and then click open.

Open Recent Projects displays a list of recently-opened files. Selecting one will open that project.

Open Running Projects displays a list of projects attached to currently running EXN/Aero sessions. Selecting one will open that project.

Save Project commits any changes made in the tree viewer to the XML structure files in the CVC folder for that project.

Save Project As combines ‘create new project’ and ‘save project’ simultaneously. It opens a file dialog and asks for a new project name & location. It saves changes without affecting the original project.

Import Structures (advanced mode) enables you populate the CVC folder in a project with pre-existing XML structures from a previous version of the project, or from another project altogether. Multiple XML files can be selected simultaneously and imported at once.

Update Structures (advanced mode) updates XML structures from an older version to match the current EXN/Aero installation. Versions must match for the Run EXN/Aero command to be successful. Update structures is applied automatically in basic mode.

Revert Project reloads the current project from the last saved version.

Close Project closes the current project, after a save prompt.

Exit closes the program, after a save prompt.

### Mesh #

Import CGNS file opens a file dialog where the user can select a CGNS mesh file to associate with the project. During the import, EXN/View copies the CGNS file to the project directory and renames it to match the project name. CGNS files created by some meshers are also ‘converted’ so that labeling of nodes within the CGNS file match the expected standard used by EXN/Aero.

A 3D wireframe of the mesh will appear* in the render view when conversion is complete. During the conversion process, an output will display the conversion progress.

*There may be a delay for larger CGNS meshes that can last several minutes.

### Edit #

Undo will revert reverts the last change. (not yet implemented)

Redo recovers the last change. (not yet implemented)

Insert Group inserts a group, such as a Cell Family, into the Tree, based on the currently selected branch. Some family groups have multiple types, which are displayed in a submenu.

Copy Group copies the currently selected group. The name of the new group is the same as the selected group with a suffix of ‘_copy’.

Delete Group deletes the currently selected group, after a confirmation.

### View #

View Toolbar toggles the toolbar.

Use Large/Small Toolbar Icons toggles the size of the icons in the toolbar between 16 and 24 pixels.

View Coordinate Axes toggles the coordinate axes in the mesh window.

View Mesh Info toggles the information box in the mesh window.

Reorient to +/- X/Y/Z axis rotates the mesh to a particular axial orientation.

Zoom In/Out zooms the mesh in or out by approximately 5%.

Reset Zoom resets the zoom on the mesh to a an auto-scaled value.

### Run #

Update CGNS (advanced mode) updates the project’s CGNS file with the data structures shown in the tree viewer. In basic mode, this process is done automatically when the run command is selected.

Run EXN/Aero launches EXN/Aero on the specified compute resources. Once the run begins, a new option appears beneath the run command showing the current solution directory associated with the run and a % completion indicator.

Show displays solver output in a new tab.

Backup causes EXN/Aero to write a solution file to a new CGNS file having the same name as the original CGNS file, with a BAK suffix and sequential numerical designation. EXN/Aero continues to run after a backup.

Update causes EXN/Aero to re-read select fields in the CGNS file (e.g. time step) at the end of the next time step. No solutions are written and EXN/Aero continues to run.

Stop causes EXN/Aero to complete the current iteration, write a solution file for restart to the CGNS file, and then exit

Kill is the same as issuing the kill command at the command prompt. It terminates the EXN/Aero process and no solution files are written.

Properties opens a small dialog window with information about the EXN/Aero session, including the duration of the current session.

### Settings #

Switch to Advanced/Basic Mode toggles the interface between basic and advanced mode.  Advanced mode gives users additional options and forces some processes to be done manually.

View Alpha/Beta features makes new and in-development engineering models & solver features available to the user.

Alpha features generally have not been thoroughly tested or validated, but are known not to affect the integrity of release models. Alpha features might produce spurious results.

Beta features have at least one validation case in our automated test repository and are known not to affect the integrity of release models. Beta models have been tested internally and usually will be included in the next major release.

### Help #

Knowlege Base & Report a Bug links to the knowledge base section of www.envenio.ca. The knowledge base contains helpful articles and links to our zendesk bug reporting system.

### Tree #

The left hand pane of the EXN/Aero UI enables the user to set up physics, initial conditions, boundary conditions and solver settings.

Practically, this information is stored in the project CVC directory in the form of formatted text files (XML structures).

Plus and minus icons next to the branch names expand and collapse branches of the tree. This can also be done by right clicking and selecting expand or collapse from the context menu.

Node Insertion is available by right clicking the Cell Families and BC Families node and selecting the desired branch type from the context menu. Places where a branch can be inserted are designated by the insert icon .

Copy & Delete are available for nodes that were inserted by the user. Deletion and copy operations are available on the right-click context menu. Branches that can be copied and deleted are indicated by the copy-delete icon.

Formatted Text Links can be inserted anywhere a folder icon  is present to the right of a node value. Formatted text is used primarily for monitor point databases and user-defined fields. See the section on formatted text for more information.

Database Connections are available anywhere the ‘hamburger’ database icon  is present. Clicking the icon brings up a list of available databases. Upon selecting a database, a sub-node will appear beneath the connected node.

### Render Window #

The render view displays a wireframe mesh constructed from the project’s CGNS file, and highlights blocks when they are selected.

Left-clicking and dragging the cursor rotates the mesh. The mesh can also be rotated to a specific axial orientation using options in the View Menu and the mesh’s toolbar.

Right-clicking and dragging the cursir translates the mesh. The mesh can also be moved by using the arrow keys.

Scrolling the mouse wheel zooms the display. The zoom can also be adjusted using the ‘+’ and ‘-‘ keys, as well as through options in the View Menu.

Orientation Axes aligned with the mesh axes are displayed for reference. The axes can be toggled on and off using the option in the View Menu or the mesh’s toolbar.

Mesh Orientation controls are displayed when the user clicks the plus sign + in the lower left corner of the render view. These controls are the same as those shown in the View menu.

Mesh Size information is displayed at bottom right in the form of max & min coordinates for reference. These coordinates are multiplied by the ‘mesh scale’ variable (default mesh scale = 1.0). This information can be toggled on and off using the option in the View Menu or the mesh’s toolbar.

### Toolbar #

Located below the menu, this option bar holds icons for the more commonly used features, such as Create New Project, Open Existing Project, and Save Project.

The toolbar can be toggled on and off in the View menu

### Tabs #

Tabs are located above the render window. They allow the user to switch between render view, solver text output and chart views.

Tabs are closed by clicking the small ‘x’ next to the tab name.

Tabs are opened (or reopened) by:

• Clicking “show output” or “show errors” for running jobs in the run menu
• Creating a chart
• right-clicking and selecting “show mesh” if the mesh view is closed
• right-clicking in the tab area and opening the output-stream or error stream for either running or finished jobs
• right-clicking in the tab area and opening any text file a file dialogue opens when this option is selected)

### Status Bar #

The status bar is located beneath the tree view. It performs several functions

• confirmation of actions the user has taken, such as ‘save’ or ‘update cgns’
• progress when importing a mesh
• warnings in yellow, or errors in red, encountered by EXN/Aero

### Setting Up Your Run #

For a successful launch, a user must:

1. Define all mesh cells, including physical models, properties and initial conditions.
2. Define all boundaries.
3. Choose appropriate solver settings based on the problem being studied.
4. Indicate the quantity and frequency of output from the solver.

### Cell Families #

Cell Families contain settings for cell families and zones in the mesh; for new projects this branch is empty.

To add a new family, right-click on the Cell Families branch and move to ‘Insert’. This produces a sub-menu which contains all of the available Cell Families, such as FluidDomains and SolidDomains. Once a Cell Family has been added, options are available to copy and delete it.

Clicking on the cell family branch highlights the regions of the mesh associated with this cell family in the render view. Right-clicking on the cell family produces context menu with a ‘Keep Visible’ option, which can be toggled on and off if the user wishes the highlighting to persist in the render view.

The Cell Blocks node opens a drop-down menu listing available mesh blocks. These can be selected in a group using the block family name, or they can be selected individually using shift+click  or ctrl+click.

### Model Parameters #

This section of the tree enables the user to activate physics models for a given cell and to set options for how those models operate.

Activating a model in this section will enable settings such as initial conditions and boundary conditions farther down the tree so models should be selected first.

### Turbulence #

RANS turbulence models

These options are able to operate in both steady and unsteady modes. See the ‘solution control’ seciton for information on switching between modes.

• Wilcox k-$omega$a two-equation turbulence model that uses a transport equation for both turbulent kinetic energy and turbulent frequency.
• Shear Stress Transport (SST): a two-equation variant of the k-$omega$ model that recovers k-$epsilon$ behaviour far away from walls.

Large-Eddy and Detached Eddy models

These options are able to operate in unsteady mode only.

• Smagorinsky-Lily: a zero-equation model that scales SGS turbulent viscosity on the strain rate tensor of the mean flow
• Canopy K-equation: a one-equation sub-grid stress model that uses the transported SGS turbulent kinetic energy ‘k’ and an assumed length scale to compute SGS turbulent viscosity. Extra source terms are active in the k-equation, and in the momentum equations to model the dissipative, momentum absorbing effect of a vegetative canopy
• SST DES: is a detached eddy formulation where the two-equation SST model serves as the universal turbulence model in the boundary layer, and as the SGS model in unconstrained regions of the flow

Turbulent Flow is the master toggle that activates turbulence models.

LES spatial filter enables the user to pick which LES / DES model that they wish to use. Choices are shown in the ‘models’ section below.

LES Length Scale branch controls the method by which subgrid length scale is computed.

Max Edge Length computes the LES length scale as the maximum edge length of a control volume when set to ‘on’. When set to ‘off’ it computes length scale as the cube root of the element volume.

Cube Root Volume Weight allows the user to specify multipliers on the x, y and z dimensions of a mesh element when computing its volume. Usually this is set to [1 1 1] and is only active when the cube root spatial filter is selected.

### Energy #

Energy Equation is the master toggle for the energy transport equation. Switching this on enables the user to set energy properties, boundaries, initial conditions and solver settings related to the energy equation.

### Multifluid #

The Eulerian multifluid models in EXN/Aero are used to track the motion of heavy particles that are transported by the fluid solution and are able to slip relative to the fluid motion. The models assume that particles are dilute and completely mixed within the confines of a control volume.

Multifluid Flow is the main toggle for the multifluid models.

Multifluid Option gives the choice between the direct quadrature method of moments (DQMOM) model, used to model continuous particle spectra, and the Fixed size Droplet Transport (FSDT) model which is used to model any number of individual monodisperse particle classes.

Algebraic Slip Velocity computes a relative velocity for the transported particles based on assumption of a Stokes drag regime.

Number of Droplet Sizes or Quadrature Level (name switches depending on model selection) allows the user to set the number of transport equations used to describe the particle phase. In DQMOM, this sets the number of distribution moments. In FSDT each transport equation is responsible for a single monodisperse particle size.

Background Diameters must be set to a non-zero value for either DQMOM or FSDT to function properly. It is best to set this several orders of magnitude smaller than the smallest expected particle size of interest.

Background Volume Fractions must be set only for the DQMOM model. The model requires a non-zero background concentration in order to function properly. It is best to set this several orders of magnitude smaller than the lowest expected concentration (e.g. less than the volume of 1 particle per cubic metre).

Turbulent Diffusion Coefficient is used for turbulent flows and allows the user to pick either a Schmidt Number diffusion coefficient, or diffusion based on the Ryan model (Ryan et al. 2009)

Laminar Diffusion Coefficient is used for laminar flows and allows the user to pick a variable diffusion coefficient that is calculated based on Brownian motion theory, or else specify a fixed diffusion coefficient.

Dispersed Ballast Properties sets the density for the dispersed particles.

### Cell Statistics #

The statistics calculator computes a cumulative mean & variance of selected variables in a transient solution. The Welford method is exact in the mean and converges for variance on a longer time frame than the mean.

When a vector is selected (e.g. Velocity) the mean field will have 3 components and the variance is computed as a symmetric covariance tensor having 6 unique elements.

Collect Statistics toggles the statistics calculator on and off

Continue Stats from Solution gives the user the option to include the previous solution’s cumulative average and standard in the re-started solution

Collect Statistics Time Delay gives the user the option to set a time-based delay on computation of mean and variance. This can be used to exclude the ‘start up’ phase of a transient solution.

Variables for Statistics allows the user to choose variables that included in the computation. The mean fields are output as separate fields in the VTU output file.

### Initial Conditions #

This branch sets the initial conditions for the various transport equations for all mesh blocks that are part of a cell family. Initial conditions may be set to constant values by filling in the variable fields, or else linking the initial conditions to spatial data sets (see Database section).

Initial conditions fall into several broad categories listed below.

General Initial Conditions – always present:

Iblank sets the boundary carving value for environmental flow applicaitons. Typically this is linked to a database that contains the terrain contour. If no boundary carving is used, then leave Iblank = 1.

Velocity sets the initial velocity for the domain. This is especially useful for external flows to aid with start-up.

Pressure sets the initial pressure for the domain. This is a gauge pressure, relative to the reference pressure set in Solver Control.

Energy Initial Conditions – present when energy models are active:

Temperature sets the initial temperature for the domain. This is a gauge temperature relative to the reference temperature set in Sol1ver Control.

Turbulence Initial Conditions – present when turbulence models are active:

Turbulence Option gives the user the option to set turbulence initial conditions either as k and epsilon, or as an intensity and length scale. Practically, these values are interchangeable and the option is given for convenience.

Turbulent Kinetic Energy sets the TKE in the domain

Turbulent Dissipation is related to the turbulence length scale and sets the dissipation variable in the domain.

Turbulence Intensity sets the magnitude of the varying component (ie kinetic energy) and is parameterized as a fraction of the mean flow velocity.

Turbulence Length Scale sets the size of the energy-containing eddies (which is related to the dissipation).

Scalar Initial Conditions: present when scalar transport models are active

Scalar 1 sets the initial value for that scalar throughout the domain.

Scalar 2…N sets the initial value for any additional scalars. One initial value for each scalar equation inserted in subdomains.

Particle Transport Initial Conditions: present when particle transport models are active

DQMOM Diameters (DQMOM particle transport model only) sets the initial field for particle diameters

Volume Fractions (DQMOM and FSDT) sets the initial value for concentration of each particle size

### Properties #

This branch sets fluid properties and user-editable coefficients for various transport equations.

### Fluid Prop. #

Many of the fluid properties a user needs are available from online sources and reference tables. All units are SI.

Density sets a constant density when solving incompressible flows. When compressible flow option is activated, the density is variable and is calculated by the solver using the ideal gas equation.

Gas Constant is available when the thermodynamic property model is set to Ideal gas. This sets the ideal gas constant for the fluid.

Thermal Conductivity is a constant value for the fluid

Compressible Opion toggles between compressible flow and incompressible flow. Generally, compressible flow is only required when mach numbers exceed 0.5.

Thermodynamic Property Model is available when the energy equation is activated. The drop-down menu gives options for different properties and specific heat models. When compressible flow is activated, the property model is locked to ideal gas and polynomial Cp.

Dynamic Viscosity sets a constant viscosity for the fluid.

Specific Heat is a constant value when the thermodynamic property model is set to constant density and Cp. Otherwise, the specific heat is defined as a 4th order polynomial, as C1, C2, C3, C4, C5, where C1 is a constand ant C5 is the coefficient of the x^4 term.

Specific heat temp range defines the absolute bounds bounds on the temperature used in the ideal gas equation.

### Subdomain Prop. #

General Scalar Difffusion is available when scalar transport models are active. This sets the diffusion coefficient

### Subdomains #

EXN/Aero uses subdomains where a special model or source term is applied to a limited region of the domain.

If a simulation is to use a subdomain, then labeling & mesh subdivision must be applied at the meshing stage to accomodate it.

For example, a pipe mesh with a sponge media near the exit would be generated in 2 blocks — one for the free fluid flow, and one for the porous region. Both blocks can have their properties set by the same Cell Family, but the subdomain relies on there being a distinct mesh block to define the its boundaries.

### Porous Media (alpha) #

Porous media models add a drag source term to the momentum equations that approximates the average effect of the media on the flow.

Porous regions are defined as subdomains. To designate a mesh block as a porous media subdomain, right click on subdomains and select ‘porous media.

Media Property can be set to homogeneous or ansiotropic.

Permeability and Inertial Resistance Factor  are scalars that control the total resistance of the porous media to fluid flow. Permeability is often termed the ‘Darcy’ coefficient. For laminar flows, the inertial component is ignored.

Viscous Resistance Coefficient and Inertial Resistance Coefficient are specified as symmetric tensors when the anisotropic media option is selected.

EXN/Aero requires the 6 unique components of the symmetric tensor, D1, D2, D3, D4, D5, D6, corresponding to the following positions in the tensor:

### General Scalar #

Scalar transport models activate an extra (or several extra) transport equations that track the movement of a passive, dilute, non-reacting species.

General Scalar Transport toggles the general scalar transport model on and off.

Schmidt Number is the ratio between momentum diffusivity and mass diffusivity.

Scalar Variables becomes a parent node when the user right clicks and inserts scalar variables. Once inserted, a corresponding initial condition and diffusion coefficient will appear for each scalar.

### BC Families #

BC Families contains settings for boundary conditions and surfaces in the mesh; for new projects this branch is empty

To add a new family, right-click on the ‘BC_Families’ branch and move to ‘Insert’. This will produce a sub-menu which contains all of the available BC Families, including:

• BC Connectivity
• Walls
• Inlet
• Outlet
• Symmetry
• Freeslip
• Moving Walls
• Multifluid Inlet

Once a family has been added, it is possible to copy or delete it.

Clicking on the BC family branch highlights the regions of the mesh associated with this BC family in the render view. Right-clicking on the BC family produces context menu with a ‘Keep Visible’ option, which can be toggled on and off if the user wishes the highlighting to persist in the render view.

### Inlet #

Inlets give users a means of fully defining a flow as it enters the computational domain. This is often the most critical boundary condition and numerical stability is very sensitive to the settings here.

BC_Location opens a drop-down menu listing available boundary families and patches. Select a family, a group of families or a group of patches to include them as part of the boundary definition. Families & patches do not need to be contiguous or adjacent.

Momentum Data gives options for defining the driving force behind the inlet. There are several options:

• Specified Velocity gives a fixed mass flow across the inlet plane, moving at the velocity provided in u,v,w components in the data entry field below
• Specified Normal Velocity gives a fixed mass flow across the inlet plane that is always normal to the plane, moving at the velocity provided in the entry data entry field below.
• Specified Static Pressure defines pressure at the inlet plane using the data entry field below. Mass flow rate will change depending on flow resistance within the domain.
• Total Pressure is used to set pressure at the inlet when using the equation of state model with compressible flow.
• Total Pressure Supersonic is used to set pressure at the inlet when using the equation of state model with compressible flow, where the inlet velocity is expected to exceed mach 1.
• Rotating Frame of Reference inlets are used when the domain will be in a rotating frame. Users must specify both the angular rotation vector and the location of the centre of rotation.

Energy Data allows the user to set the inlet temperature using the data entry field below.

Turbulence Option gives the user the option to set turbulence initial conditions either as k and epsilon, or as an intensity and length scale. Practically, these values are interchangeable and the option is given for convenience.

Turbulent Kinetic Energy sets the TKE in the domain

Turbulent Dissipation is related to the turbulence length scale and sets the dissipation variable in the domain.

Turbulence Intensity sets the magnitude of the varying component (ie kinetic energy) and is parameterized as a fraction of the mean flow velocity.

Turbulence Length Scale sets the size of the energy-containing eddies (which is related to the dissipation).

Synthetic Turbulence is used primarily for LES. ST adds velocity perturbations at the inlet plane in order to hasten development of resolved-scale turbulence within the domain.

Synthetic Boundary Turbulence turns the model on and off.

Synthetic Turbulence Option gives the option between synthetic generated turbulence (Smirnov 2001) and the synthetic eddy method (Jarrin 2006).

• For the synthetic eddy method, the inlet must be a plane normal to one of the mesh axes. Users must supply a reynolds stress tensor [r11, r21, r22, r31, r32, r33], a turbulent length scale [lx, ly, lz], a minimum and maximum XYZ location which define the corners of the inlet plane [x,y,z], and a mean convection velocity for eddies through the inlet plane [v] which can usually be set equal to the mean inlet velocity.
• For the synthetic generated turbulence method, the inlet must be a plane normal to one of the mehs axes. Users must supply a reynolds stress tensor [r11, r21, r22, r31, r32, r33], a band pass wavenumber filter [min_x, min_y, min_z, max_x, max_y, max_z], a band pass frequency filter [min, max] and a turbulent length scale vector [lx, ly, lz].

The synthetic eddy models are very complex to set up and often are best used with input databases. Please consult the theory manual or support for assistance setting the models up.

Scalar Data contains input fields for the passive scalar transport equations. There is a scalar data tree node for each scalar equation defined in Cell Families above.

Inlet Scalar Option switches between two inlet options:

• Specified value simply sets the value at the inlet plane. This can be linked to a database if the input of scalar values is not uniform across the plane.
• Zero Flux sets the inlet value to zero.

### RANS / LES / DES BCs #

Guidelines for setting boundary conditions for turbulence models are found here, although some experimentation may be required to find a value that works for a user’s particular case.

Turbulence Option gives the user the option to set turbulence initial conditions either as k and epsilon, or as an intensity and length scale. Practically, these values are interchangeable and the option is given for convenience.

Turbulent Kinetic Energy sets the TKE in the domain

Turbulent Dissipation is related to the turbulence length scale and sets the dissipation variable in the domain.

Turbulence Intensity sets the magnitude of the varying component (ie kinetic energy) and is parameterized as a fraction of the mean flow velocity.

Turbulence Length Scale sets the size of the energy-containing eddies (which is related to the dissipation).

### BC Statistics #

Collect Statistics currently gives the option to calculate only total body force and moment, both instantaneous and cumulative mean, about the mesh origin for a wall boundary group.

BC statistics are written to the SQL database rather than the VTU file at every simulation time step.

### Subdomain BCs #

Scalar equations can be specified at inlets & walls, or else set to zero-flux.

Inlet Scalar Option, or  Wall_Scalar Option can be set to ‘specified value’, which is a constant, or ‘zero flux’ which blocks all movement of the scalar through the boundary.

Scalar Variable indicates which scalar is associated with a particular boundary condition

Specified Value if selected above sets a boundary value. This node is not available when the ‘zero flux’ option is selected.

### Solver Control #

Solver Control contains options related to the EXN/Aero solver including time step, total run time, compute resource allocation, output options and coefficient loop limiters (among other things). These settings are applied globally when the EXN/Aero run is launched.

In Solver_Control, you can:

• Modify global solver settings
• Change simulation timing
• Toggle restart, and restart arrays
• Set relaxation parameters
• Set simulation output frequency and manage output arrays
• Assign compute resources
• Set Iteration limiters and multigrid parameters

### Simulation #

Branches under simulation affect the solution as a whole, and the way that the solver outputs data.

### Simulation Type #

Steady State solution methods converge quickly at the expense of temporal resolution for processes that are statistically stationary in time. These solutions satisfy the unsteady partial differential equations in the limit of very large time steps. EXN/Aero employs a false transient solution method combined with implicit (explicit for pressure only) relaxation schemes of the underlying correction equations.

Steady state mode is controlled primarily by the time step and the relaxation parameter. Time step is found in the main Solver Control  branch. Relaxation parameters and other convergence settings are found in the Convergence Criteria branch.

Unsteady solution methods converge the entire flow field at each real time step, and typically run much more slowly than steady state solutions. Unsteady methods enable temporal resolution of the flow, and are useful for situations where the process is not statistically stationary in time, or where a steady solution is inaccurate (e.g. vortex shedding from a cylinder).

Unsteady mode employs both inner and outer coefficient loops controlled by a series of convergence criteria at both the global and the individual equation level. Settings for unsteady convergence are found in the Convergence Criteria branch

### Misc Solver Settings #

Task Scheduling changes whether physics set up computations are done in serial or parallel. Typically, this should remain on the default ‘concurrent’ setting.

Element Quality Threshold is used on unstructured grids. Element quality is measured by ‘flatness’, ‘skewness’ and ‘aspect ratio’ by the EXN/Aero solver on start up. Any elements that exceed the threshold (big numbers are bad) will have 2nd order treatments turned off. This can create solution inaccuracy, but can also help stabilize solutions on grids with poor element quality. Setting this below 0.8 is not advised.

Transient Scheme Order toggles between 1st and 2nd order transient scheme.

Reference Pressure Level sets the reference for all fluctuating pressures (e.g. BCs and initial conditions) in the simulation. Pressure at any point in the solution (absolute) is the local pressure plus this reference pressure.

Reference Temperature sets the reference for all fluctuating temperatures (e.g. BCs and initial conditions) in the simulation. Temperature at any point in the solution (absolute) is the local temperature plus this reference temperature.

Point Specified Pressure is used when the user wishes a location within the solution to be at a specific pressure. Usually the automatic setting is used and location is specified in XYZ coordinates.

### Solution Timing #

Total Run Time sets the internal time-based stop condition for the solver. When the simulation time reaches this value, EXN/Aero writes output files per transient output settings, writes a restart-able solution to the CGNS file and then exits.

Unsteady Time Step is the amount of simulated time that elapses during the outer coefficient loop.

Steady State Time Step should be comparable in magnitude to the characteristic time scales of the problem. It is good practice to use a small time step at the beginning of the simulation, and to increase it gradually. Increasing the time step rapidly can trigger numerical instability, and so relaxation of the transport equations is needed to aid convergence in many cases.

### Restarting #

When a simulation completes, a single restart-able solution is written to the main CGNS file for the project. Likewise, when ‘backup’ commands (automated or manual) are issued, a single restart-able solution is written to a backup CGNS file.

To restart a completed simulation or to restart from a backup, ensure that the solution is in the main project CGNS file. For completed solutions, it will already be there. For backup solutions, it is necessary to import the mesh from the backup location using the Mesh menu.

Initial time from solution has two options. Upon restart, setting this variable to ‘on’ causes EXN/Aero to use the final simulation time from the previous run as the start time of the current run. If set to ‘off’ then the initial time is set to zero.

Initial Values from Solution enables the user to pick which data arrays are included in a restart. If an array is required but not included in this list on restart, EXN/Aero defaults to the initial value settings in Cell_Families.

Use random seed from solution is used in the synthetic turbulence models. If a user wishes to reuse the random number generator seed from a previous solution in the current solution, this should be set to ‘on’.

### Solution Output #

Solution files for post-processing are written in VTK format. Practically, the VTK for a single solution consists of an index (.vtm) file, and a series of associated solution files (.vtu) in a directory of the same name. There is one VTU file for each BC family and each Cell family set in the tree viewer.

Transient Output Frequency controls the interval in time steps at which solution output files (VTK format) are written to the solution directory. A setting of ‘1’ outputs every time step, while a setting of ‘0’ disables VTK output entirely. Keep in mind that writing VTK files to the hard drive is very time consuming and can slow a simulation significantly if interval is small.

Solution Backup Frequency controls the interval in time steps at which EXN/Aero writes a restart-able solution to a backup CGNS file in the solution directory. A setting of ‘1’ outputs every time step, while a setting of ‘0’ disables backups entirely. Keep in mind that writing solution arrays to the CGNS file is very time consuming and can slow a simulation significantly if interval is small.

Transient Output Variables controls which solution variables are included in VTK output for fluid volumes. Some output variables are multi-component vectors (e.g. Velocity) while others are scalars or individual components of vectors (e.g. Temperature, Velocity_X). Picking a vector in this field is sufficient to include its components — for example, picking Velocity means the user does not also need to select ‘Velocity_X’, ‘Velocity_Y’ and ‘Velocity Z’ also.

Transient Output Variables BC controls which solution variables are included in VTK output, simultaneously with the VTK output for fluid volumes. Vector & scalar picking rules are the same as for Transient Ouput Variables above.

Inner Loop Equations sets which transport equations are computed as part of the main coefficient loop. Any equations not included in this list are by default calculated outside the coefficient loop.

Deactivate Equations lets users selectively deactivate transport equations. For runs, this is usually set to ‘none’ but might be modified as part of diagnostic checks. Deactivated fields are frozen from the point of view of the solver.

### Convergence Criteria #

Unsteady Convergence Criteria tell the solver how hard to work to achieve convergence of a solution.

• Global Settings control the outer coefficient loop and set residual targets. These settings can be modified at the global level, which affects all transport equations, or at the local level on a per-equation basis.
• Individual Equation Settings (e.g. pressure, momentum, etc….) control the inner loop for each equation and can override global residual targets.

Some global settings have a different meaning for steady-state solutions; these are noted below.

All relaxation parameters can be adjusted mid-solution by changing the value in EXN/View and issuing an ‘update’ command from the Run Menu.

Outer loop Exit Condition: determines the exit condition for the outer loop.

• Static: every outer coefficient loop runs to the maximum number of iterations. Mass imbalance and residual targets are not checked
• Dynamic: the outer loop will perform iterations between the minimum and maximum number settings. Mass imbalance and residual targets checked, and the coefficient loop stops if these thresholds are met.

Minimum number of iterations: sets the minimum iteration number for Dynamic outer loops. This is not used for static outer loops.

Maximum number of iterations: sets the maximum iteration number for dynamic and static outer loops outer loops, in all conditions.

Residual Option: sets the normalization equation for residuals.

• Default: residuals are un-normalized and have units according to the equation they are associated with.
• Normalize Globally: The residual for each equation is divided by its a_P coefficient and by the range of that variable in the domain. Globally-normalized residuals are dimensionless.
• Normalize Locally: The residual for each equation is divided by its a_P coefficient and by the local variable in a control volume. Locally-normalized residuals are dimensionless

Mass Imbalance Target: Sets a threshold for mass imbalance below which a dynamic coefficient loop will exit. This target is ignored for static loop exit conditions.

Residual RMS Target Default: Sets the RMS residual threshold below which the dynamic coefficient loop will exit. If an individual equation has a lower threshold, then the solver will use that as an exit criterion instead. This target is ignored for static loop exit conditions.

Residual MAX Target Default: Sets the maximum residual threshold below which the dynamic coefficient loop will exit. If the RMS threshold is met, but a local maximum residual exceeds this threshold, the loop will not exit. This target is ignored for static loop exit conditions.

Outer loop Exit Condition: No outer loops are performed; each false time step is iterated upon a single time.

• Static: The simulation runs to the maximum number of iterations and no convergence targets are checked.
• Dynamic: the simulation iterates a number of times between the minimum and maximum number settings. Mass imbalance and residual targets checked, and the run stops if these thresholds are met.

Minimum number of iterations: the minimum number of false transient time steps taken to reach a steady-state solution.

Maximum number of iterations: the maximum number of false transient time steps taken to reach a steady-state solution.

Residual RMS Target: Same concept as the global setting, except it applies only for a single equation and it overrides the default target in global convergence settings if  is given a lower value. This target is ignored for static loop exit conditions.

Residual MAX Target: Same concept as the global setting, except it applies only for a single equation and it overrides the default target in global convergence settings if it is given a lower value. This target is ignored for static loop exit conditions.

Residual Reduction Level: Sets the target residual reduction threshold for an equation, defined as the ratio of the initial residual to the final residual in any inner loop. If the target is met, then the inner coefficient loop exits. This threshold is dynamically increased to drive a solution toward convergence, pursuant to the Iteration Threshold setting.

MG Residual Reduction Rate: Sets the target residual reduction threshold for an equation, defined as the ratio of the initial residual to the final residual at any multigrid level. If the target is met, then the multigrid solver will not initiate additional multigrid levels.

Max Iterations: The maximum number of iterations at the finest multigrid level in the linear solver for an equation.

Forced Iterations: Forces a user-defined number of iterations at the finest multigrid level in the linear solver for an equation. This setting overrides all residual target thresholds at the individual equation and global level.

MG Forced Iterations: Forces a minimum number of multigrid levels in the linear solver for an equation.

Iteration Threshold: Below this iteration threshold, the Residual Reduction Level is not dynamically modified. Above this number of iterations, the Residual Reduction level is dynamically modeified. If set to a number greater than Max Iterations, this feature is disabled.

Relaxation Factors: must have values between 0 and 1. A typical pressure relaxation is 0.3 and a typical momentum relaxation is 0.7. Other model equations use relaxation factors similar to that of momentum. A general rule of thumb for optimal convergence is to choose pressure and momentum relaxation factors such that their sum equals 1.0.

At early time steps, the pressure correction loop is working harder than at later time steps. It is often necessary to use a smaller pressure relaxation early on and to then increase it as the simulation progresses. To drive final convergence, relaxations should be increased to a value close to 1 as the simulation nears completion.

### Exporting Boundaries Only #

The export boundaries branch controls which solution variables are included in VTK output for selected boundaries. This capability is useful where the user desires to generate a boundary VTK file at a different frequency from the regular transient output controlled above in the main solver control branch. To generate a boundary export group, right click Export Boundaries. Multiple groups can be created, each with unique settings.

BC Locations controls which boundaries or boundary families inherit these output settings.

Variable_List enables the user to pick which variables are included in the VTK output for this group. Picking rules are the same as for solution variables above.

Output_Frequency controls the interval in time steps at which EXN/Aero writes a boundary VTK file to the hard drive. This interval can be a different from the solution output frequency.

### Compute Resources #

CPU Threads sets the number of  CPUs allocated to the EXN/Aero launch. This number cannot be greater than the available cores on the user’s system. A good rule of thumb is to have at least one CPU allocated for every GPU device plus one. More CPUs should be allocated if the multi-block mesh has a large number of internal block-to-block interfaces.

GPU Devices sets the number of GPU cards allocated to the EXN/Aero launch. This number cannot be greater than the available GPU cards installed on the user’s system. Keep in mind that GPUs do not exhibit strong scaling in all cases. Small runs are often faster on a single GPU rather than multiple GPUs.

Memory Size is a multiplier on the default memory allocation for data arrays. Very large runs sometimes require that this limit be increased. If it is too large, the system will report that it has insufficient memory.

### Monitor Points #

Monitor points, otherwise known as probes, are points in the simulation domain which are defined by the user. Monitor points gather data from the solution in progress output them to a SQL database. Monitor points are a low-overhead means of writing transient data at points of interest for the complete time series.

Cartesian Coordinate Locations are the XYZ coordinates, relative to the domain origin, of monitor point positions. These are entered in 2 ways:

1. manually as position vectors separated by commas
2. as a formatted text file

Variable List controls which variables are reported at each monitor point. The variables in the list are the same variables available in the Solution Output Variables node above.

Format for monitor points:

• Comma separated.
• X coordinate, Y coordinate, Z coordinate, in that order, in metres
• New line for each monitor point location.

Multiple monitor point sets can be loaded with different names, and each point will receive its own table in the SQL database.

### Database #

The Database branch holds custom user-defined databases, such as scripted changes to time steps, user-defined boundary fields and user-defined initial conditions. These databases are read from formatted text files and are interpolated to the CFD mesh. The Database branh is divided into two areas: Boundary and Initial Conditions, and User Tables.

Linking to databases and tables is done using the database symbol ==> . Anywhere this is icon is displayed in the cell families, BC families, or solver control branch, that variable can be linked to a database or table. To create the link, the database must first exist in the Database branch below.

### Boundary and Initial Conditions #

Boundary and initial condition databases use a distance-weighted neigbour interpolation to initialize boundaries and mesh volumes from a user-supplied spatial data set. Spatial data sets are 2D for boundaries, and 3D for mesh volumes.

Users do not need to supply a complete data set for the entire mesh volume if it is not specifically required. Where the pre-solver fails to find a nearby neighbour, it will default to the initial value specified in either cell families or BC families.

Database Name allows the user to name the database – no spaces.

Variable Name allows the user to select the variables that are read in from the database. In a spreadsheet, these would be the column headers. Variable names always appear in alphabetic order and must correspond 1:1 to the columns in the input database

Data links to the text file and will display the file name of any loaded file. To load a database into a project, click the folder icon to the right of the data field and navigate to the file location.

Bounding Box Extension controls the search area that is used by the pre-solver. By default (values of zero) the bounding box equals the boundary or mesh block where the interpolation takes place. The extension pads this by the amount specified. Two delta vectors are required. One positive, and one negative to control padding in each direction.

For 2D interfaces  the bounding box must be padded slightly. One control volume in either direction normal to the plane is sufficient. For example, an x-y plane with adjacent control volume thickness of dz should be padded as: [0 0 dz, 0 0 -dz].

DB Grid controls the size of the search area for each point on the grid as the pre-solver cycles through and performs the interpolation. The default value of [0 0 0] means that every boundary element will search every single point in the database to find nearest neighbours. Setting to [1 1 1] places once grid subdivision in each coordinate direction, and so on.

To minimize time loading databases, it is strongly advised that users consider the density of points in their input database relative to the number of neighbours each boundary control volume must find. Ideally, DB Grid is as large as possible while each search area still contains enough neighbours to perform the interpolation

Number Weighted Neighbours controls the number of neighbours used in the interpolation. Setting this value to 1 maps the nearest DB point to a control volume

Weight Power Parameter enables the user to place additional weight on nearby DB points in the interpolation. A value of zero means that distances have face value. Greater powers place greater weight on nearby DB points.

Format for database points:

• Comma separated, one variable in each column
• New line for each database point (XYZ location).
• Each line must contain X, Y, and Z coordinate
• 2D datasets must still contain the 3rd coordinate even if it is equal throughout the set.
• Order columns alphabetically, left to right
• Save as .txt

### User Tables #

User Tables give users the ability to provide custom lookup tables for some variables. Where values are required between table points, they are interpolated.

User Table allows the user to name the database – no spaces.

Variable Name allows the user to select the variables that are read in from the table. In a spreadsheet, these would be the column headers. Variable names are separated by dashes. Abscissa value(s) are prefixed with a period “.”

Tables can have at most 2 abscissa with an ordinate value, therefore the maximum number of variable names is 3.

Data links to the text file and will display the file name of any loaded file. To load a database into a project, click the folder icon to the right of the data field and navigate to the file location. Data can also be entered manually if the table is small. Values in a pair or triplet (as the case may be from the variable name field) are separated by spaces, individual pairs & triplets are separated by commas. An example is provided for time step whenever a user creates a new user table.

Format for user table points in externally linked files:

• Comma separated, one variable in each column, by axis.
• New line for each table point
• one-axis tables should have contain a list of abscissa values in the left column and ordinate values in the right column.
• two-axis tables should be entered as a series of appended one-axis tables using the format above. The number of appended tables is equal to the number of values on the second abscissa. The two left columns should contain the abscissa values and the right column contains the ordinate value.
• Save as .txt

### Variables #

This branch lists all of the variables available for output in EXN/Aero. The two main categories are Cell (mesh block) and Interface (boundary) variables.

Variable visibility depends on model selections made in BC_Families and in Cell_Families — variables associated with inactive models are not shown by default.

### Monitor #

The graphical monitoring tool enables users to view certain solver outputs in a 2D graphical format. These outputs include:

• Convergence information (residuals).
• Monitor point data.
• Mean and variance for boundaries, as selected in the statistics calculator.

Data displayed by the monitor tool is contained in the solution directory within a SQL database file — see the results directory section for more information.

### Graphing #

After about 10 time steps, data begins writing from the solution buffer to the SQL database and the monitor branch appears. Clicking on this branch causes the render view to display available solution files in a project.

To select a run, drag and drop the name of the results file (on the right) over to the Results branch of the tree. This opens a new branch with the name of the results file beneath.

Clicking on the results file name in the tree (at left) shows tables in that SQL database (at right). In this example, ‘Convergence’, is the only choice. Monitor points and boundary statistics are shown if these options are activated for a run.

Dragging and dropping the ‘Convergence’ table under the results file name in the tree, selects that table; multiple tables may be selected if they are available.

Clicking the table in the tree shows the available variables at right.

Right-clicking Charts gives the option to create a new chart.

Once a new chart has been created, variables are added by dragging and dropping them onto the X and Y axis.

Multi-plot (Y-axis only) is available by dragging and dropping multiple variables onto the Y_Variables branch. Multi-selection of variables is available using CTRL+click.

Clicking on the chart causes the render view to display a 2D plot of the selected variables. The legend displays the variables by name and color, and displays the value of that variable based upon the position of the mouse.

Variables can be removed by clicking the small red X icon, or hidden, by un-checking the box next to that variable.

Left clicking and dragging the mouse enables the user to zoom in on sub-regions of the chart. Chart range can be edited directly on the axes in variables outlined by a rectangular border.

Right-clicking gives several additional options for the chart.

Disable Mouse Tracking turns on the live display of variables based on the mouse position.

View Legend toggles the legend on and off.

Save Chart as PNG opens a file dialogue where the user can save an screen capture of the plot.

Save Chart as CSV opens a file dialogue where the user can save the chart data as a comma separated text file of columnar data, containing variable names in the header.

Reset Zoom Sets both X and Y axis ranges to that of the visible variable having the largest range.

Dynamic Updating activates an automated read of the SQL database at the update interval. Automated updates re-scale the x-axis to include new data appended to the database.

Update Interval is the interval in real-world seconds at which the SQL database is re-read. When the SQL file is very large (e.g. long unsteady runs, or runs with many monitor points) it is best to set a long update interval, as re-reading the tables can take awhile.

### Running EXN/Aero #

Launching a run is an important time, but can also be a frustrating process. This section aims to help you overcome any difficulties you may encounter.

Review the “pre-flight checks” section prior to launching your run as it will avoid the most common causes of run failure.

### Pre-Flight Checks #

The following are some checks users can make to avoid frustration:

1. Ensure that your mesh has only positive mesh volumes. This can be checked in Pointwise or in the compatible meshing tool used to create the mesh in the first place
2. Check that all boundaries are assigned and all mesh blocks are assigned. Using block & boundary family names before exporting from the meshing software is a good way to ensure that nothing is missed.
3. Check that your target simulation time (unsteady) and min/max number of iterations (steady-state) are high enough to do at least one time step. This is especially important when restarting an old solution where ‘time from solution’ is set to ‘on’.
4. Check that there are 1:1 connections for control volumes across all mesh block interfaces, unless many-to-many or conjugate heat transfer is being used (see 5.)
5. Check that all BC connectivity information is supplied where many-to-many or conjugate heat transfer is in use.
6. Check that resource allocation does not exceed available resources. Sometimes this means lowering the number of CPU/GPU resources assigned in solver control, and sometimes it means that a larger GPU resource is required (e.g. moving from a single to a dual K80 session)

### Getting a Run to Start #

Time step Decisions:

EXN/Aero does not use automated time stepping for steady-state runs. Use this procedure to get a good guess.

1. Find the largest and smallest length scales of relevance in the domain.
2. Divide each length scale by the average free stream speed, or by the most relevant speed scale (e.g. inlet velocity) to obtain the short and long time scale.
3. Average these two time scales and divide by 10.

Preventing Divergence:

The pressure equation in a steady-state solution should normally work hard.

If the simulation is diverging and the pressure equation iterations are low (as seen in output text) then force pressure solver iterations in Solver Control >> Convergence Criteria >> Steady >> Pressure

Time Step Decisions

EXN/Aero does not use automated time stepping for unsteady runs. Use this procedure to get a good guess.

1. Find the smallest length scales of relevance in the domain.
2. Divide each length scale by the average free stream speed, or by the most relevant speed scale (e.g. inlet velocity) to obtain time scale.
3. Divide this scale by 100 to obtain the delta Time setting.

Speeding up Time Steps:

The pressure equation should not work nearly as hard as in transient solutions owing to the high level of diagonal dominance, provided time step is chosen appropriately.

If the pressure equation is working very hard beyond the start-up stage (e.g. greater than 20 iterations per outer coefficient loop iteration) limit the number of fine and multi-grid (MG) and mg solver iterations are applied.

If the flow is compressible the 20 iteration criteria above should be reduced to 10 iterations, since in this case the pressure equation now also has a transient term making it easier to solve

### Solver Errors #

Solver errors are usually the result of failing to make pre-flight checks, or file corruption. Please review the pre-flight checks section and you will avoid most of these errors.

#### If your run is failing and the error is not shown here, please submit a support ticket and Envenio engineers will respond as soon as possible.

cg_goto error

This solver error occurs when EXN/Aero tries to access a setting in the CGNS file, but fails to find it.

This error is most often triggered when a boundary has not been properly defined, or else, when a XML file is corrupted.

To resolve the error, ensure that XML files are updated to the latest version (automatic in basic mode), and also, that all boundaries are defined. If this does not help, try importing XML files from a previous solution. Envenio technical support can assist in finding the exact source of the error.

infinite loop NLNLNLN…

This solver error means that the CGNS file is corrupted. This can occur occasionally during file transfer, or by miss handling the file (e.g. copying or moving) when it is being written to by EXN/Aero.

To resolve this issue, it is best to re-import a CGNS file from another source. These sources can include:

• The original CGNS file exported from the user’s meshing tool, or a fresh export from the mesher.
• A backup file, either kept manually by the user, or else backed up automatically using settings in ‘solver control >> simulation’.

NOTE: please keep a copy of the corrupted file. Corruption in CGNS files tends to be localized to one or a few nodes. Even though EXN/Aero solver cannot launch a corrupted file, other generic and 3rd party CGNS editing tools can open the file directly and recover solution directories and settings. Please contact Envenio technical support for assistance.

Error: EXN-Aero cvc version( 1.380) and input CGNS file cvc version( 1.370) are incompatible!

This solver error should only occur in advanced mode if the user neglects to update XML structure files to the latest version.

Periodically, XML structure templates are updated to include new features. The solver expects a certain CGNS Version Control (CVC) number and will not attempt to launch a CGNS file that has not been updated

Updating happens can be triggered by the user, or when in basic mode will happen automatically. Updating causes no loss of data

Device real memory exceeded.../coef/mip0 185018483 > 183187236

In the example above, the solver indicates real memory, but may also indicate integer memory, double precision, or character.

This can be resolved in advanced mode by increasing the default memory allocation in the tree view under solver control >> resources.

This error can also be resolved by increasing the number of GPU devices or CPU core.

Error in creation of element neighbour list ... 16005, 15000

The CGNS format is an open source standard but different meshing tools do not apply the standard in an identical manner.

Envenio supports compatibility with CGNS export from Pointwise, ICEM, AnsysMesh.

Insufficient host memory. 18321465 bytes requested, 1655200 bytes free

This solver error occurs when the memory required for the current run is greater than the available memory. Install additional host RAM, reduce the size of the mesh or deactivate memory-intensive settings (e.g. double precision & second order transient)

Cannot have different quadrature levels in a multifluid DQMOM simulation....

The number of quadrature levels is not consistently defined across all Cell Families. Set quadrature levels equal to clear this error

Cannot have mixed buoyant flow cells in simulation....
Cannot have mixed compressible/incompressible flow cells in simulation....
Cannot have mixed turbulent/laminar cells in simulation....
Cannot have mixed non-isothermal/isothermal cells in simulation....

The models referred to in the error messages above must be activated consistently across all cell families.

To resolve this solver error, activate models consistently to clear this error.

Stop, ill posed fine mesh size

EXNAero only allows maximum of 1024 points in each IJK direction for a structured block. To avoid this solver error, subdivide a mesh into a multi-block mesh so that number of points in any one block is less than 1024.

Not found for cell coordinate in calc_database_bc...

When reading inlet conditions from a 2D database, this solver error means some elements in this boundary cannot find the closest point in the database ‘inletDB’.

To clear this error, check that the database points (e.g. Mesh_X,Mesh_Y and Mesh_Z), are large enough to cover all elements at that boundary, or else increase the Bounding_Box_Extension to cover all elements.

### Error Output #

From time to time, during your run, you might receive a warning message at the bottom of the EXNview window. This will indicate which run has triggered the warning and indicate that you should check it out. In general, warnings do not stop a solution, but errors do.

If a warning is triggered, you can view additional information in 2 ways.

1. For running jobs where the warning has not stopped the solver, simply go to the ‘Run’ menu, drop down to the job in question, and click show errors. This will load the error output text file as a new tab over top of the mesh render view.
2. For stopped/failed jobs where the error has stopped the solver, right click over top of the mesh render pane, select ‘show errors’ and then select the solution file containing the failed job.

EXN/Aero jobs are able to continue running in the background even if they are not opened as the current project in EXNView. These are detected by EXNView even though the tree and mesh render is not visible. In this case:

1. For running backgound jobs where the warning has not stopped the solver, use procedure 1. above.
2. For warnings about jobs that have failed in the background, right click above the mesh render pane, click “open text file” and navigate to the solution directory of the job in question.

### File Management #

The project directory contains all data related to a simulation in EXN/Aero, including setup, solver control and results files.  This section walks through the project directory, explaining the purpose of each component. The file structure shown here simulation has already been set-up and several simulations have been performed; some files may not be present in the user’s directory at early stages of the set up process.

When the user creates a new project in EXN/View, the project’s parent folder is created. The image below shows the contents of the Example.proj folder.

### Network Storage #

The permanent storage area is under the /data directory in the Nimbix cloud environment. Nimbix provides users with 1 terabyte (TB) of free persistent data storage for compute jobs.

Data can be uploaded and downloaded at any time whether jobs are running or not. The best tools to use are Secure FTP protocol (SFTP) or the Nimbix File Manager over HTTPS.

Please note, it is best to use SFTP for large file transfers, i.e., any file over 100 GB.

### Zipping CGNS & VTU files #

File compression utilities are unusually effective at shrinking the file size for both CGNS and VTU files — 50% or greater compression is the norm.

Nimbix does not provide a zip or unzip desktop utility, but these commands can easily be issued from the command line.

As an example:

zip at the command line:

zip .zip /nimbix/data//.zip

unzip at the command line:

unzip .zip

Whether storing files for later use, or compressing before transferring a file, zip is an effective way to reduce your space usage on the cloud.

### Transfer files by SFTP #

SFTP is a good choice for moving large files reliably.  Access to cloud storage requires a Nimbix username and the account’s API key.

For users that are behind a corporate firewall, it might be necessary to request OUTBOUND access to port 22 (TCP) from IT management.

1. Download an SFTP client, such as FileZilla or SCP and install.
2. Enter server hostname drop.jarvice.com.
3. Port 22.

For help finding the API Key, see related article Where do I find my JARVICE API Key?

Once connected, transfer files to and from the Nimbix storage drive. It is not necessary to have an active compute session to do this.

In the next Nimbix compute session, files will be available in the /data directory.

### Nimbix File Manager #

The Nimbix file manager is a browser-based utility offered by Nimbix that makes file transfer simple.

1. Login to the Nimbix dashboard.
2. Click “Compute” in the top navigation bar and select Nimbix File Manager. Launch a session using default settings.
3. The browser now returns to the dashboard and shows a running file manager job.
4. Click the blue file manager link in the ‘active jobs’ area to show the session details. Connect to this using the green ‘connect’ tab that appears about 30 seconds after the job is launched.
5. A new browser tab appears and it shows the contents of the account’s /data folder. Use the links to download and upload files.

When data transfer is complete, return to the dashboard and shut down the file manager task to avoid unnecessary charges.

### Parent Directory #

CGNS files (.cgns) contain mesh information and are created by EXN/View when the user imports a mesh. Upon import, the CGNS file contains only mesh and block connectivity information, but eventually will contain custom folders after the problem is set up and ready to run on EXN/Aero.

JSON files (.json) are a reduced version of the mesh information in the CGNS file. These are used by the render view in EXN/View to visualize the mesh outline.

### CVC Directories #

XML structure files are housed in the CVC directories, where #.## represents the version number. CVC stands for ‘CGNS Version Control”.

An example of a CVC directory contents is shown below:

XML structure files contain all project settings and information displayed in the tree viewer. These are created by EXN/View when the user saves a new project. They can also be imported by other projects that share similar settings. The formatted text contained in XML files drives the appearance of the tree viewer in EXN/View, and can be edited manually in any text editor if the user so desires.

CVC.log is a record of version changes.

Info.txt provides some basic information for manual editing of XML files and cvc.log. This might be of interest for advanced users wishing to automate XML structures using scripts for running large parameter sets

### XML Structure Files #

When a user makes changes to the default layout of the tree, they are modifying a default XML file. These custom files are specific to a project, and they are saved in the CVC directory. At launch time, the XML structures are used to write the necessary information to the CGNS file which is then read by the solver.

The physical model settings, solver control configuration settings and the position of these settings in the tree view change incrementally from time to time as EXN/Aero is updated.

Every update to EXN/Aero contains a template set XML structure files that are compatible with that release.

When an older project is loaded in a new update of EXN/Aero, the structures are automatically updated to the newest template, preserving any data and settings from the older files.

Upon updates, older XML structure files are preserved in their original CVC directory and a new CVC directory is created with the up-to-date versions.

XML structures can be edited by the user in any text editor, or edited automatically using a 3rd party script or programming language (e.g. matlab). This can be useful if users are generating a set of permutations on a single mesh.

Formatting of XML structure files is specific to EXN/Aero and must be correct in order to run without errors. Envenio engineers can help explain the formatting if a user wishes to make edits directly.

### Results Directory #

Simulation output data files are contained in the results directories. These are created each time EXN/Aero is launched, and are given a unique name, where ### is the sequential number of the run.

An example of the results directory contents is shown below

output_stream_prop-0.txt contains condensed simulation health data such as residuals and global mass imbalance. This is the file read by the output stream in EXN/View.

SQLite Output (-it_1-ts_1.sqldb) is a SQLite database file that contains an array of solver outputs. By default it contains equation residuals, CPU timing information. Monitor points and integrated quantities also end up in the SQLite file as additional tables.

The SQLite database can be appended and queried in real time, and it is the primary input to the simulation monitoring function within EXN/View. There are several open-source tools available for viewing, editing, appending and exporting data from SQL databases. Some of these (e.g. CSV export of chart data) are available in EXN/View

XML directory contains an exact copy of the structures used at runtime. These are useful if a user wishes to re-create the exact conditions of a run, and can be imported directly into EXN/View.

Memory Map  directory contains memory usage summaries. This data is used mostly for debugging and will not be of interest to the average user.

Parareal Profile Status (parareal_profile_win-1.txt) contains diagnostic information from Envenio’s parallel-in-time solver. This capabilty is in alpha and the contents of this directory will not be of interest to the average user.

# VTK subdirectory

This contains all field data outputs from the simulation, saved in VTK database format. This format is native to Paraview but is supported by several other visualization tools.

Drilling down through this file structure to the Iteration-1 directory, and then a step further into the Example_1 directory shows the following:

VTK Multiblock file (.vtm) contains information on how the various data blocks in the results folder are connected. Its name will always be identical to its corresponding results folder

VTK blocks folder (-###) contains the data blocks that are connected to the multiblock file. In the above image there are four blocks. One each for wall boundaries, outlet plane, inlet plane, and the mesh block.

### Post Processing With Paraview #

Paraview is a general purpose open source post-processing software. It can be used:

1. On a user’s local machine.
2. On a internal server via remote connection.
3. On cloud resources such as Nimbix, which enable users to access their cloud storage drive directly.

Paraview installers & source code can be accessed at:

http://www.paraview.org/

The complete user manual is available at:

http://www.paraview.org/paraview-guide/

We encourage users to learn more about the software from the official Paraview guide. This section is intended to give some basic information to get users up and run quickly Paraview for post-processing of simulation results from EXN/Aero.

### Launching Paraview on the Cloud #

Running Paraview on cloud resources is done in the exact same way as EXN/Aero. Simply find Paraview on the Compute tab of the Nimbix dashboard. There is only one selection available in the drop-down list for the machine type (the charge is \$2.50/h).

Connect to the Paraview desktop either through the browser remote connection or through a VNC connection.

When using Paraview on the cloud, users have access to the same /data directory as EXN/Aero sessions. This is advantageous for situations where results files are too large to download efficiently.

### Paraview Anatomy #

Paraview uses hierarchical data filtration to produce user defined data sets.

• At the top of the hierarchy is the root data file(s). For EXN/Aero output, these output files are found in the VTK sub-directory (see the file management section of this manual). Either .vtu or .vtm output files can be opened in Paraview.
• ‘Filters’ are applied to reduce the complete data set into useful displays, contours and plots.
• Filters can be applied cell-wide, or on interfaces, using any combination of variables included in solver transient output (see solver control for output settings).
• Filters, among other things, can project variables, perform calculations, plots over lines, create contours, produce streamlines, integrate in 2D and 3D, and analyze time-series data sets.

The default Paraview display has 3 main areas.

1. File and filter list displays the filter hierarchy and provides some controls for modifying its branches.
2. File and filter controls are displayed for the filter or data set selected in the list above.
3. The render view displays data from filters in a variety of formats. 3D render is shown in the image above, but 2D charts, spreadsheets and other views are available.

### Common Operations #

Examining a cylinder in cross-flow, we added 2 filters to the root data set cylinder0M_6001.vtm

1. Cell data to point data converts the raw ‘cell’ data output by EXN/Aero to Paraview’s  ‘point’ data format and removes jaggedness from the data in subsequent filters.
2.  Clip slices the data set along a plane and keeps data on one side.

Notice that the render view automatically turns off visibility for higher filters and keeps visibility for the lowest filter. Visibility in the render view can be changed anytime.

Next, we plotted streamwise velocity along a line using a new filter

1. Plot over Line gives the user control over two endpoints of the line and displays results in a split render view (2D), and allows the user to choose which plots are displayed.

Here, rather than applying the filter to the clipped data set, we drew off the same root set (cell data to point data) as the clip filter. The plot over line could easily be applied to the clipped set, but we would need to ensure that the line endpoints lie within the clipped set.

To compute the vortex shedding frequency, we computed lift and drag on the upper surface of the cylinder. To accomplish this we used 3 additional filters:

1. Extract Block enables the user to turn off certain .vtu files from the root data set. In this case, we turned off all but the cylinder surface.
2. Clip was explained above; it is used here to remove the lower half of the cylinder surface.
3. Integrate Variables performs a surface integral for all variables in the data set. We are concerned primarily with pressure in this case, since its surface integral equals force.

In the figure below, we apply extract block to the cell data to point data branch. If we applied this filter to the first clipped block (described above) we would see a truncated cylinder.

An additional filter can be used here for an unsteady simulation. Plot over time gives the time series of the set to which it is applied. When using any ‘over time’ filter in Paraview, it must open each individual .vtu file and perform the exact same computation. When the data set is large or the file size is large (or both) this can be a slow process.

We plan to grow this section of the manual with feedback from our user base. If you have comments or suggestions for Paraview tutorials (in an EXN/Aero context) please send us a message!

Suggest Edit

### See it for yourself!

Book a hands on demo with an engineer