VTK interface unused?

Plotting in Dolfin depends on VTK (take a look in dolfin/plot/ ).

I think the VTK library is intentionally not used when generating VTK files to be able to produce such files in environments where VTK is not available.

It makes sense to have this as a backup I suppose, but I see this substantial VTK section in the main CMakeLists.txt which sets all kinds of things, e.g., DOLFIN_ENABLE_VTK.
Also, the same logic could be applied to HDF5 which does implement a proper interface.

Sorry, I missed your comment on dolfin/plot/ which apparently is the reason for the CMake VTK code.

On Sat, Oct 6, 2012 at 1:01 PM, Nico Schlömer
<email address hidden> wrote:
> New question #210513 on DOLFIN:
> https://answers.launchpad.net/dolfin/+question/210513
>
> As indicated in ./dolfin/CMakeLists.txt, Dolfin optionally depends on VTK.
> However, in the actual interface at
>
> ./dolfin/io/VTK*
>
> doesn't make use of this interface at all but instead manually creates all VTK files.

VTK is a recent dependency, and is used for light-weight runtime
plotting. I considered VTK for formatting files in the past, but
decided against such a heavy dependency to generate visualisation
output to a file, otherwise without VTK a user can't visualise
anything.

> Not only is this inefficient and less flexible,

It a difference between inefficient and inefficient - the VTK XML
format will always be inefficient. This why we've added XMDF support
with parallel output to HDF5 to the development version.

>but it also introduces bugs such as
>
> https://bugs.launchpad.net/dolfin/+bug/1048221
>

Easy fix.

> It would seem natural to use the VTK interface to harness the full power of the toolkit (e.g., compressing files).

DOLFIN supports compression inside VTK XML files.

> Is something like this being worked on, or is there any help needed?
>

Testing out the XDMF output in the dev version would be good (use the
file extension .xdm ). Let me know how you get on. No doubt some
tweaks will be needed, and we're still adding functionality.

It's possible that we might remove VTK output in the future and just
support XDMF (with support for both ascii data in the XML file and
HDF5 storage).

Garth

> --
> You received this question notification because you are a member of
> DOLFIN Team, which is an answer contact for DOLFIN.

> VTK is a recent dependency.
> otherwise without VTK a user can't visualise anything.

That's exactly why I'm using it as well.

> the VTK XML format will always be inefficient.
> It's possible that we might remove VTK output in the future.

Is that the case?
I'm not an expert in scientific file formats, but I do have experience with VTK and ExodusII. In comparison, I always found that for a single state, the VTK files would be smaller than their ExodusII counterparts. As for XDMF: I was always under the impression that it isn't quite as widely used as the previous two which may be because it is newer (?), but I guess this is still a good reason to use the established standards. For this reason, I would also advice against entirely removing VTK I/O even if XDMF is in place.

> Testing out the XDMF output in the dev version would be good.

I think I will need to get going with the dev version anyways since I'm still ambitious enough to contribute here and there. :)

Is there a guide of how to set up all of FEniCS's/Dolfin's dependencies in the dev version? I'm a little afraid of having to "git clone" and build ten separate packages before getting down to business.

Anyways, getting back to the original issue: What would your opinion be on using "#ifdef HAVE_VTK" in ./dolfin/io/VTKWriter.cpp to add an alternative implementation that makes use of the actual VTK interface? I figure that'd be a good starting point for me to get started with Dolfin development of patch creation and such, but obviously I wouldn't want to spend time on this if it's going to get tossed anyways.

On Sat, Oct 6, 2012 at 2:31 PM, Nico Schlömer
<email address hidden> wrote:
> Question #210513 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/210513
>
> Nico Schlömer posted a new comment:
>> VTK is a recent dependency.
>> otherwise without VTK a user can't visualise anything.
>
> That's exactly why I'm using it as well.
>
>> the VTK XML format will always be inefficient.
>> It's possible that we might remove VTK output in the future.
>
> Is that the case?
> I'm not an expert in scientific file formats, but I do have experience with VTK and ExodusII. In comparison, I always found that for a single state, the VTK files would be smaller than their ExodusII counterparts.

XDMF + HDF5 is smaller than VTK XML (ascii), and much faster.

> As for XDMF: I was always under the impression that it isn't quite as widely used as the previous two which may be because it is newer (?), but I guess this is still a good reason to use the established standards. For this reason, I would also advice against entirely removing VTK I/O even if XDMF is in place.
>

It won't be removed any time soon, but in the future if it becomes a
question of fixing the VTK output code or removing it, it will be in
danger.

XDMF has the advantage of being a standard, having sensible parallel
output via HDF5 (single file with io via MPI-IO) and it doesn't
require storing the entire mesh at each time step.

>> Testing out the XDMF output in the dev version would be good.
>
> I think I will need to get going with the dev version anyways since I'm
> still ambitious enough to contribute here and there. :)
>
> Is there a guide of how to set up all of FEniCS's/Dolfin's dependencies
> in the dev version? I'm a little afraid of having to "git clone" and
> build ten separate packages before getting down to business.
>

There are a few packages to get, but the Python ones are very easy to
install. There are instructions at

    http://fenicsproject.org/download/installation_from_source.html#installation-from-source

> Anyways, getting back to the original issue: What would your opinion be
> on using "#ifdef HAVE_VTK" in ./dolfin/io/VTKWriter.cpp to add an
> alternative implementation that makes use of the actual VTK interface? I
> figure that'd be a good starting point for me to get started with Dolfin
> development of patch creation and such, but obviously I wouldn't want to
> spend time on this if it's going to get tossed anyways.
>

My concern is that it's more code with no obvious advantage. If you
want to get started, my suggestion would be to add text-only XDMF
output. This would permit output without HDF5 being installed. The
code would be similar to the VTK code that we have already.

Garth

> --
> You received this question notification because you are a member of
> DOLFIN Team, which is an answer contact for DOLFIN.

> If you want to get started, my suggestion would be to add text-only XDMF output. This would permit output without HDF5 being
> installed. The code would be similar to the VTK code that we have already.

I don't think I'll be doing exactly this. Reasons:

 * This feels pretty much like bundling code, which is already bad, except that instead of literally including the tried-and-true 3rd party library code you're writing it all from scratch by yourself, probably introducing a whole bunch of bugs on top.
 * If HDF5 isn't available for a certain architecture it probably means XDMF shouldn't be used there either. Anyways, what architecture is there that doesn't support HDF5?
 * When contributing code, my motivation is to make something work for me that didn't work before and have others enjoy the same.

Anyways, I don't wanna turn this into a software engineering discussion board. :)
I'll first go ahead and try to install Dolfin from dev and get back to you if I have any more issues. Those may be about VTK. ;)

For the latest dolfin dev, do I need to install the development version of all of the above packages as well??
I'm asking because with the latest release of all of the dependencies, I get

============================== *snip* ==============================
Traceback (most recent call last):
  File "poisson.py", line 21, in <module>
    T = project(Constant([0,0,1]), FunctionSpace(mesh, 'Nedelec 1st kind H(curl)', 1))
  File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/projection.py", line 104, in project
    form_compiler_parameters=form_compiler_parameters)
  File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/assembling.py", line 259, in assemble_system
    subdomains, form_compiler_parameters)
  File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/form.py", line 74, in __init__
    function_spaces)
  File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/form.py", line 108, in _extract_function_spaces
    r = compiled_form.rank()
  File "/usr/local/lib/python2.7/dist-packages/ufc/ufc.py", line 262, in rank
    def rank(self): return _ufc.form_rank(self)
TypeError: in method 'form_rank', argument 1 of type 'ufc::form const *'
============================== *snap* ==============================

for a reasonably simple test file. (The dolfin release works fine here.)

For a consistent installation you need dev versions of all FEniCS packages.

Johan

On 10/06/2012 09:01 PM, Nico Schlömer wrote:
> Question #210513 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/210513
>
> Status: Solved => Open
>
> Nico Schlömer is still having a problem:
> For the latest dolfin dev, do I need to install the development version of all of the above packages as well??
> I'm asking because with the latest release of all of the dependencies, I get
>
> ============================== *snip* ==============================
> Traceback (most recent call last):
> File "poisson.py", line 21, in <module>
> T = project(Constant([0,0,1]), FunctionSpace(mesh, 'Nedelec 1st kind H(curl)', 1))
> File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/projection.py", line 104, in project
> form_compiler_parameters=form_compiler_parameters)
> File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/assembling.py", line 259, in assemble_system
> subdomains, form_compiler_parameters)
> File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/form.py", line 74, in __init__
> function_spaces)
> File "/usr/local/lib/python2.7/dist-packages/dolfin/fem/form.py", line 108, in _extract_function_spaces
> r = compiled_form.rank()
> File "/usr/local/lib/python2.7/dist-packages/ufc/ufc.py", line 262, in rank
> def rank(self): return _ufc.form_rank(self)
> TypeError: in method 'form_rank', argument 1 of type 'ufc::form const *'
> ============================== *snap* ==============================
>
> for a reasonably simple test file. (The dolfin release works fine here.)
>

That's too bad, but I guess I'll get around it. This doesn't make developing for FEniCS easier of course.
What's your workflow here when you update your packages? Navigate into each and every single one of them and do a bzr pull?

On Sun, Oct 07, 2012 at 11:11:13PM -0000, Nico Schlömer wrote:
> Question #210513 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/210513
>
> Nico Schlömer posted a new comment:
> That's too bad, but I guess I'll get around it. This doesn't make
> developing for FEniCS easier of course. What's your workflow here
> when you update your packages? Navigate into each and every single
> one of them and do a bzr pull?

My workflow is:

1. Download dev version of Dorsal and let it build everything except
DOLFIN (set stable build to false). Install in some suitable location.

2. Build DOLFIN against that install using ./cmake.local.

Whenever some packages that DOLFIN depends on need updating (not very
often), I revisit the Dorsal directory and rebuild, but this time with
all the heavy stuff (PETSc etc) commented out. The reinstall then
typically takes something like 30s.

--
Anders

On 10/08/2012 01:11 AM, Nico Schlömer wrote:
> Question #210513 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/210513
>
> Nico Schlömer posted a new comment: That's too bad, but I guess I'll
> get around it. This doesn't make developing for FEniCS easier of
> course.

You can say that, but this is not different from other large projects.
Even though we have experienced a (compared to pre 1.0) quite stable
period wrt interface changes we have done some changes which requires
upgrade of most FEniCS packages.

Also note that because the FEniCS book relies on the 1.0 release we will
keep this branch for a long time, and do point releases with bug fixes.
We therefor have two development branches for each of the core FEniCS
packages residing at launchpad, one for bug fixes into 1.0.x and trunk.

> What's your workflow here when you update your packages?

I guess this depends on what you like. I bet some have their own
dev.conf and stable.conf script which sets the environment variables for
the particular installation. The conf files are then run for each
terminal session. I also think many depends on dorsal
(https://launchpad.net/dorsal) doing the heavy lifting wrt downloading
and installing the correct versions of, not only all dependencies, but
also all the FEniCS packages.

I have chosen to install everything locally and then just have a remove
script which erase a present FEniCS installation. A bit intrusive, but
it works. The bzr pull and build stage I use a simple bash script for.
Here I could have used dorsal, but I use that mostly for installing
dependencies.

> Navigate into each and every single one of them and do a bzr pull?

Something like that, but automated with scripts.

Johan