SDF driver for Windows version of Visit

[DanRRRR]

Hi,
Visit or Paraview can load hell amount (~200) of different file formats but not the EPOCH's SDF one. Hence EPOCH provides additional driver for specific version of Visit. Unfortunately there is no version of Visit for my version of Linux, compilation of Visit from GetHub took hours and failed, Visit folks are helpless with this version of Linux too for the last two years, and for Windows version of Visit it is probably harder to create the driver because all MAKE files there are different. But may be anyone has succeeded to make SDF driver for Visit in Windows?

Alternatively can anyone help to get the particles and fields saved to harddrive directly from EPOCH avoiding these FOUR (!!!) intermediate "black box" stages:

saving to SDF --> SDF DRIVER converts SDF file for Visit --> Visit loads data --> Visit dumps the data in ASCII/binary format.

Not even mentioning that we do not need these Visit, Paraview or any other graphics software nice pics, we just need the raw data.
I have made the beta version for this saving but it messes output somewhere. It works only if no MPI parallelization is used or essentially only if one core is used. When say two cores are used the final data does not represent the sum of two parts to make the whole entire simulation box, the pieces of data get distorted somewhere like in a wrong assembled puzzle. Does each core work not with real x,y,z coordinates but with the additionally shifted ones ?

It is completely understandable that the decision to use SDF, HDF etc file formats were taken years ago when there was no compatible way to save and load binary data and saving just the text ASCII is slow and inefficient. But with latest additions to Standard ACCESS = "STREAM" all binary formats now became compatible along all existing compilers and OSes. Plus this way to read/write is 10-100x faster, infinitely simpler and files are ~2x smaller than with SDF

[Status-Mirror]

Hi Dan,

Working directly with the SDF binary is complicated, although I believe some FORTRAN and C++ code is present in the SDF directory.

I also struggle to get Visit working on Linux, and I find the least-effort approach is to use the MATLAB or Python functions for extracting and manipulating SDF data. These are also present in the SDF directory (Python is labelled as utils).

My personal preference is to use the MATLAB's GetDataSDF("0000.sdf") function to get the data contained in the 0000.sdf file. Alternatively, Python users may wish to follow the instructions documented here:
https://epochpic.github.io/documentation/visualising_output/python.html

We are about to start a large rewrite of EPOCH to scale with next generation super-computers, and the I/O will likely be updated here. We appreciate the suggestion of using STREAM - this is worth looking into. I wouldn't expect any big changes any time soon though, the new EPOCH project is a 3 year grant.

Cheers,
Stuart

[TomGoffrey]

A few things to consider:

1. How does your proposed use of raw Fortran output scale to 100k+ processes? Assuming you stick to the current output strategy that'll result in billions of files for a typical simulation? How do you propose to handle that in post-processing? This sort of output strategy may well be problematic for your filesystem too.
2. You seem to imply that 3-7 GB/s is going to out-perform parallel I/O. I don't think this is true. Taking ARCHER (last generation UK HPC) as an example, they achieved something around 16,000MiB/s during benchmarking (around 50% of theoretical peak performance of the file system). This was 6 years ago, and ARCHER has since been decommissioned.
3. If you want to ignore the first 2 points above, as Stuart says EPOCH is open source under a permissive license. It wouldn't be too hard to write your own output routines.
4. As I mentioned above, tools are shipped with EPOCH to translate SDF into plain text, should you want to use that.

[DanRRRR]

Tom,
I though to use your way of conversion but i know it is slow (1000x slower than unformatted Fortran), we used it as a last resort sometimes with HDF5, where there exist the analog of sdffilter called h5dump or our own Python HDF converter. By the way do you or anybody have these utilities compiled for Windows? At least they will be compatible with all version of Windows. I'd try them, may be somehow miraculously they are fast.

I do not get how billions files appeared here? You have one SDF files per intermediate output, in Fortran approach there will be better to separate particles and fields and there will be 2 files. Some other PIC codes have 5-10 files per intermediate output, and at least for myself i find that optimal (electrons and each type of atoms saved also separately).

As to parallel vs serial speed - some codes may need parallel I/O of course. But tasks with up to 50 TB of total output per run this Fortran approach will fit perfectly. Specifically if at some point you plan to have your personal supercomputer at home :)

[TomGoffrey]

The billions of files would occur because the code you've written would be executed by every single process, each writing their own file. You have 100,000 MPI processes running, and each output 10,000 files (extreme, but not unheard of). That gives you a 1e9 files.

One final thing to consider - it's not the total size of the data that is necessarily going to limit if you need to run in parallel. More likely the amount of compute you have to do. Even for 2D simulations we typically require >100,000 processes to carry out our calculations in a reasonable time frame.

[DanRRRR]

I understand. If SDF or any format is used for intermediate scratch data let the code use it as usually. I am talking only about data for final saving to keep the data.

[DanRRRR]

By the way the new standard PCIe 5 also has 16 GB/s bandwidth. So soon every laptop will get the bandwidth of that superduper ARCHER.

[DanRRRR]

I just contacted the developer's forums of Visit and Paraview. They quickly responded that they could be interested to work on developing the ability of their software to read SDF file format. I am surprised why no one contacted them before. They ask where to find the description of SDF ? What one can read in EPOCH description is the following "A detailed specification of this format is available elsewhere, although this is only of interest to developers wishing to write new libraries". Is Self Describing Format the same as SDFormat which also exists in the world wild net confusing things even more?

[Status-Mirror]

Hi Dan,

I'm not an SDF developer, so I can't give you a complete answer - but you may find this helpful. The SDF documentation in .tex form can be found here:
https://github.com/Warwick-Plasma/SDF_documentation/blob/main/sdf_format.tex

Hope this helps,
Stuart

[DanRRRR]

The maintainer of Visit Kathleen Biagas asking : "Since EPOCH has a VisIt plugin for SDF already, I wonder if the EPOCH folks would be amenable to contributing their plugin to VisIt's repo? I don't know if there would be licensing issues or whatnot"

Probably EPOCH developers should be very interested to give the users the ability to visualize the output of the code straight off the run and not through the several more intermediate stages of different software you need to learn to install and use and which may or may not work anyway.