DOLFIN

Saving and retrieving fields to/from xdmf files

Asked by Maximilian Albert

Hi,

I'd like to try out the new support for xdmf files, but unfortunately there don't seem to be any examples or demos available. All I could find was a an email which suggested to do this:

   file = File("foo.xdmf")
   file << u

(Here I presume that u is a dolfin.Function.) I tried to play around with this but am having a couple of questions:

1) What is the correct way to associate timestamps with the data (for example for later inspection in Paraview) - in particular when I'd like to save multiple fields simultaneously?

After digging a bit into the C++ classes and their docstrings, I tried this (in Python):

   file = File("foo.xdmf")
   file << (u, 0.0)
   file << (v, 0.0)
   file << (u, 1.0)
   file << (v, 2.0)

I'd expect this to save both fields u, v at time t=0, whereas at time t=1 only the field u should be saved, and similarly at t=2 only v should be saved. When I open the resulting file in Paraview, it seems to recognise that there are four fields present in the .xdmf file (these are displayed in the 'Properties' pane, both in the 'Blocks' and 'Hierarchy' tabs). However, the 'Display' pane doesn't give me any option to visualise either of the fields. Only when I advance to t=1.0 or t=2.0 (at each of which time steps there is just a single field present) can I visualise that data.

Of course it's possible that I'm just unable to use Paraview correctly, but my hunch is that I'm doing something wrong in saving the data. Any suggestions?

2) How can I read the data back in from a .xdmf file?

More precisely, assume I previously saved multiple fields at different time steps as above. How do I, say, retrieve a field with name 'v' which was stored at time t=2.0 so that I can store it in a df.Function object?

Generally, would it be possible to make some examples (or even just short code snippets) available for the new functionality that was introduced in dolfin 1.1? There seems to be a lot of really cool stuff, but it's of very limited use if it's unclear how to use it in one's own code. :) A case in point is the re-ordering of dofs, where I'm still confused as to what the correct way is to reliably set the values of a df.Function, other than using vertex_to_dof_map() (which is unreleased as yet, I believe?).

Many thanks!
Max

Question information

Language:
English Edit question
Status:
Solved
For:
DOLFIN Edit question
Assignee:
No assignee Edit question
Solved by:
Garth Wells
Solved:
Last query:
Last reply:
Revision history for this message
Best Garth Wells (garth-wells) said : 		#1

On 18 February 2013 23:31, Maximilian Albert
<email address hidden> wrote:
> New question #222230 on DOLFIN:
> https://answers.launchpad.net/dolfin/+question/222230
>
> Hi,
>
> I'd like to try out the new support for xdmf files, but unfortunately there don't seem to be any examples or demos available. All I could find was a an email which suggested to do this:
>
> file = File("foo.xdmf")
> file << u
>
> (Here I presume that u is a dolfin.Function.) I tried to play around with this but am having a couple of questions:
>

That works.

> 1) What is the correct way to associate timestamps with the data (for example for later inspection in Paraview) - in particular when I'd like to save multiple fields simultaneously?
>
> After digging a bit into the C++ classes and their docstrings, I tried this (in Python):
>
> file = File("foo.xdmf")
> file << (u, 0.0)
> file << (v, 0.0)
> file << (u, 1.0)
> file << (v, 2.0)
>
> I'd expect this to save both fields u, v at time t=0, whereas at time t=1 only the field u should be saved, and similarly at t=2 only v should be saved. When I open the resulting file in Paraview, it seems to recognise that there are four fields present in the .xdmf file (these are displayed in the 'Properties' pane, both in the 'Blocks' and 'Hierarchy' tabs). However, the 'Display' pane doesn't give me any option to visualise either of the fields. Only when I advance to t=1.0 or t=2.0 (at each of which time steps there is just a single field present) can I visualise that data.
>

It's under active development, which means features appear and can
disappear. Time series were supported, but I'm not sure if it's there
now. It hasn't yet been added to demos yet because HDF5 is an optional
dependency.

> Of course it's possible that I'm just unable to use Paraview correctly, but my hunch is that I'm doing something wrong in saving the data. Any suggestions?
>

It should work (at least for one function).

> 2) How can I read the data back in from a .xdmf file?
>
> More precisely, assume I previously saved multiple fields at different time steps as above. How do I, say, retrieve a field with name 'v' which was stored at time t=2.0 so that I can store it in a df.Function object?
>

Not yet, and it might nor be supported via XDMF. If will definitely be
supported via HDF5.

>
> Generally, would it be possible to make some examples (or even just short code snippets) available for the new functionality that was introduced in dolfin 1.1? There seems to be a lot of really cool stuff, but it's of very limited use if it's unclear how to use it in one's own code. :) A case in point is the re-ordering of dofs, where I'm still confused as to what the correct way is to reliably set the values of a df.Function, other than using vertex_to_dof_map() (which is unreleased as yet, I believe?).
>

HDF5/XDMF functionality has not been heavily advertised because it's
being developed actively and can change (and break) at any time. It
will stabilise over the next few months.

For single functions or meshes, it is very usable. I've used to for
problems with ~50M cells in parallel. It's very fast.

Garth

> Many thanks!
> Max
>
> --
> You received this question notification because you are a member of
> DOLFIN Team, which is an answer contact for DOLFIN.

Revision history for this message
Maximilian Albert (cilix) said : 		#2

Thank you very much for the detailed reply, Garth! I wasn't aware that XDMF support is still so heavily developed. In this case I'll just remain patient and wait a bit for it to mature. Thanks for the great work!

I also wasn't aware that there is a separate HDF5 writer. What are the differences between the two? (Isn't XDMF also based on the hdf5 file format?) In particular, which one would you recommend using?

Regarding my example with two functions at a single time step, if you say it should work then I'll try to play around and investigate a bit further. Do you happen to have any offhand suggestions how I can test whether what I'm seeing might be a Paraview or dolfin issue?

Many thanks again and best regards,
Max

Revision history for this message
Maximilian Albert (cilix) said : 		#3

Thanks Garth Wells, that solved my question.

Revision history for this message
Garth Wells (garth-wells) said : 		#4

On 19 February 2013 10:41, Maximilian Albert
<email address hidden> wrote:
> Question #222230 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/222230
>
> Status: Answered => Solved
>
> Maximilian Albert confirmed that the question is solved:
> Thank you very much for the detailed reply, Garth! I wasn't aware that
> XDMF support is still so heavily developed. In this case I'll just
> remain patient and wait a bit for it to mature. Thanks for the great
> work!
>
> I also wasn't aware that there is a separate HDF5 writer. What are the
> differences between the two? (Isn't XDMF also based on the hdf5 file
> format?) In particular, which one would you recommend using?
>

XDMF is a very light XML format that describes the data, with the
actual data stored in HDF5. The XDMF files points to the HD5 file.

XDMF is supported by ParaView (and other programs). I suggest using
XDMF for visualisation, and HDF5 to store other data.

> Regarding my example with two functions at a single time step, if you
> say it should work then I'll try to play around and investigate a bit
> further. Do you happen to have any offhand suggestions how I can test
> whether what I'm seeing might be a Paraview or dolfin issue?
>

Look inside the XMDF file (it's small). You can inspect the HD5 file
using h5dump.

Garth

> Many thanks again and best regards,
> Max
>
> --
> You received this question notification because you are a member of
> DOLFIN Team, which is an answer contact for DOLFIN.

Revision history for this message
Maximilian Albert (cilix) said : 		#5

Thanks again, Garth! I was aware of the XML wrapping, but I thought that this means that XMDF is a "superset" (forgive my naive terminology) of HDF5 and thus more powerful/descriptive in some sense. So I was wondering why to use HDF5 directly for accessing data instead of sticking to a single format for both visualisation and data storage/retrieval.

This will probably become obvious once I start playing around, but would it be possibe to save data in XDMF format and then read it back from the .h5 file that was created alongside the .xdmf file? I seem to gather from your previous reply that this isn't natively supported in dolfin yet. For the time being, would you recommend using something like pytables to do this?

Many thanks and best wishes,
Max

Revision history for this message
Chris Richardson (chris-bpi) said : 		#6

At present, XDMF output of time series simply advances a counter each time a Function is output
to the file. As a result, saving multiple functions to the same file will not work properly, even if they have the same
timestamp, as they will appear as separate time steps.
 Whether this will be resolved in the future, I am not sure.

The recommended solution would be to save each field in a separate file.

When saving an XDMF file, the H5 file which is generated is not compatible for reading back in. At present,
for efficiency reasons, when running in parallel, the mesh will contain some duplicated points (with the same values).
It makes it difficult to read back in the H5 file.

To save to HDF5 natively, you can do:

F = HDF5File("filename.h5","w")
F.write(mesh, "mesh_name")

To read the mesh back in, you would use:

F = HDF5File("filename.h5","r")
F.read(mesh, "mesh_name")

This should work (for the mesh at least) - and is much faster than the XML reader in parallel.

Revision history for this message
Maximilian Albert (cilix) said : 		#7

Thanks, Chris! This looks really useful. Unfortunately, HDF5File doesn't seem to be defined, even after saying "from dolfin import *". What's the magic command for accessing it? Also, your example shows how to save a mesh. Is saving functions (ideally, multiple ones) supported?

Revision history for this message
Johannes Ring (johannr) said : 		#8

HDF5File should be available if you are using DOLFIN 1.1 or from trunk and HDF5 support was enabled when building DOLFIN.

Revision history for this message
Chris Richardson (chris-bpi) said : 		#9

At the moment, natively in HDF5 you can only save and read Mesh, and solution Vectors.

If you save these, you should be able to read them back in again, and restore the state of a Function - probably.
In parallel, the Mesh will be partitioned when read in - there is no sure way to know it will always be partitioned the same way.

I am assuming you want to save your Functions for checkpointing, so you can restart a run from that point. This is definitely
something that we are looking at.

Another tool you might want to look at is h5py - which allows you to access the HDF5 data directly from python.

As for XDMF, when saving a Function to XDMF, for visualisation, some data may be lost - e.g. if saving a P2 field, with quadratic
elements, the values at the midpoint nodes of the edges will be lost. XDMF does support quadratic elements, so improving this might be a future development. Other elements (e.g. RT, BDM) involve similar compromises. So really, you should just
use the XDMF interface for visualisation only.

Revision history for this message
Maximilian Albert (cilix) said : 		#10

@Johannes: I'm sure I tried with dolfin 1.1 and HDF5File wasn't available, but I just double-checked and I must have done something wrong as it's certainly there.

@Chris: Many thanks for the explanations. It's extremely helpful to get an idea of the current status and plans. Since you mentioned h5py, I had indeed considered implementing my own saving mechanism (either using h5py or pytables). However, I thought that there is no point in doing that if dolfin supports it natively, and I'm sure there are lots of pitfalls which I'm not aware of (such as how to store the mesh and how to deal with re-ordering of dofs). Thinking about this option I'm having a few questions but I'll open a separate thread for that as they are getting a bit off-topic.

One last comment regarding xdmf support: if it makes any difference at all, imho being able to save multiple functions to the same xdmf file (and associate them with the same time steps) would be super-useful. This must actually be a rather common use case. For example, I'd like to use it for storing multiple fields in a magnetic simulation as they evolve over time. I may be thinking far too naively here, but would it not be possible to write the timestamp instead of the integer counter to the xdmf file in case the user provides it (and fall back to the counter otherwise)? Of course the user would then be responsible for not mixing saving operations with and without time stamps, but at least it would work in both cases if the user is careful.

Revision history for this message
Chris Richardson (chris-bpi) said : 		#11

Re: XDMF with multiple fields, it may be possible, but it is not too obvious how to do it. The problem is that
each time you feed a Function into the XMDFFile object, it would have to understand which one it is.
I think you would have to create a separate TimeSeries for each function - and on the basis of the Function::name()
redirect the data to the correct one.

It probably will not be implemented any time soon, as it is fairly easy to save each field in a separate file.