Memory consumption of DofMap::build?

On Tue, Dec 18, 2012 at 11:05 PM, Felix Ospald
<email address hidden> wrote:
> New question #217126 on DOLFIN:
> https://answers.launchpad.net/dolfin/+question/217126
>
> Hi,
> Im currently solving a linear elasticity problem with P2 elements in R^3. The function space has about 606.900 dofs.
> Now I'm tring to refine the problem uniformly, which should give a function space with about 4.800.000 dofs.
> The problem is that, ubuntu kills the process, because it takes too much memory (at least more than 6GB of 8GB).
> I tried to figure out what takes so much memory and it seems like it is the DofMap::DofMap() / DofMap::build method, which is called inside adapt(FunctionSpace).
> Can anybody explain why this takes so much memory (1235 byte /dof)?
>

Are you using the latest dev version?

My experience is that that dof re-numbering uses less memory than the
solver. If you want to solve a P2 elasticity problem in 3D with 5M
dofs, I think you'll need more than 8GB for the solver to work.

> I guess it is the Cuthill McKee renumbering algorithm, which takes so much memory?
> Is this necessary and is there a way to change this?

The re-ordering makes the total runtime faster than without. You can
turn is off (in serial) using

   parameters["reorder_dofs_serial"] = False

but I think your basic problem is that you don't have enough memory
for the size problem you want to solve.

If you run in parallel, it will use less memory because the
re-ordering is local to each process.

Garth

> Could this be a problem of the topology of the domain? I'm starting with a cube which has topologically 6 holes in it.
>
> like this:
> http://www.zwirnerandwirth.com/exhibitions/2003/052003FJLS/images/open-cube-corner-piece.jpg
>
> class ShapeDom : public SubDomain
> {
> public:
> bool inside(const Array<double>& x, bool on_boundary) const
> {
> double r1 = 1.0/5.0+DOLFIN_EPS;
> double r2 = 4.0/5.0-DOLFIN_EPS;
> return ((x[0] < r1) || (x[0] > r2) || (x[1] < r1) || (x[1] > r2)) &&
> ((x[0] < r1) || (x[0] > r2) || (x[2] < r1) || (x[2] > r2)) &&
> ((x[1] < r1) || (x[1] > r2) || (x[2] < r1) || (x[2] > r2));
> }
> };
>
> SubMesh mesh(UnitCube(5,5,5), ShapeDom());
>
>
> Thanks,
>
> Felix
>
> --
> You received this question notification because you are a member of
> DOLFIN Team, which is an answer contact for DOLFIN.

> Are you using the latest dev version?
Not the latest, but the dev from 2 months ago.

> but I think your basic problem is that you don't have enough memory
> for the size problem you want to solve.

That's a true statement, thats why ubuntu kills my process. But the problem occurs before I assemble the matrix. It is definitely in the DofMap::build() somewhere. And this sould not need so much space!?
I'll try to find it out.

Thanks

Felix

On Wed, Dec 19, 2012 at 3:41 PM, Felix Ospald
<email address hidden> wrote:
> Question #217126 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/217126
>
> Status: Answered => Open
>
> Felix Ospald is still having a problem:
>
>> Are you using the latest dev version?
> Not the latest, but the dev from 2 months ago.
>

You should update. Changes were made to reduce the memory usage.

Garth

>> but I think your basic problem is that you don't have enough memory
>> for the size problem you want to solve.
>
> That's a true statement, thats why ubuntu kills my process. But the problem occurs before I assemble the matrix. It is definitely in the DofMap::build() somewhere. And this sould not need so much space!?
> I'll try to find it out.
>
> Thanks
>
> Felix
>
> --
> You received this question notification because you are a member of
> DOLFIN Team, which is an answer contact for DOLFIN.

OMG seems like you are doing quite some structural changes in the code.
No dolfin:uint anymore? la_index stuff, tons of new abstract functions to implement...
I get a lot of compile errors for my code now. I think I'll wait until changes are complete... ;-)

> The re-ordering makes the total runtime faster than without. You can
> turn is off (in serial) using
>
> parameters["reorder_dofs_serial"] = False

Hi,

I performed some tests.

Disabling dof reordering did
- increasedsolve time about 25%, when using cg (I assume because of
better inode blocking), for PETSc LU almost no change
- reduced initialization time about 20-40%

In my case the solve time was increased by 0.35s and initialization time
reduced by 4s, total 3.65s less - about 15% of the total runtime.

So it depends on the number of iterations, required for the solution of
the problem,
if enablingdof reordering pays off.

On Sun, Jan 13, 2013 at 10:35:52PM -0000, Felix Ospald wrote:
> Question #217126 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/217126
>
> Status: Answered => Open
>
> Felix Ospald is still having a problem:
>
> > The re-ordering makes the total runtime faster than without. You can
> > turn is off (in serial) using
> >
> > parameters["reorder_dofs_serial"] = False
>
> Hi,
>
> I performed some tests.
>
> Disabling dof reordering did
> - increasedsolve time about 25%, when using cg (I assume because of
> better inode blocking), for PETSc LU almost no change
> - reduced initialization time about 20-40%
>
> In my case the solve time was increased by 0.35s and initialization time
> reduced by 4s, total 3.65s less - about 15% of the total runtime.
>
> So it depends on the number of iterations, required for the solution of
> the problem,
> if enablingdof reordering pays off.

It also affects memory usage, which you can check by calling

  monitor_memory_usage()

in your program.

--
Anders