finer meshes caused the "out of memory" error

On Tue, Sep 28, 2010 at 09:12:47PM -0000, cutejeff wrote:
> New question #127219 on DOLFIN:
> https://answers.launchpad.net/dolfin/+question/127219
>
> I have been doing the mesh refinement convergence study. The simulations worked very well for meshes Rectangle(0,0,5,5,50,50) and Rectangle(0,0,5,5,50,100). But if I use a finer mesh Rectangle(0,0,5,5,100,100), it caused the "out of memory" error.
>
> I have 10G memory on my computer and I looked at the hardinfo, I still have about 5-6g memory left when Fenics was running. So I guess there are some limitations on memory allocation in Fenics. Is there a way to allocate more memory for Fenics?
>
> What should I do in my case? Because I have a high order equation(4th), and it's kind of complicated, I split it into 4 mixed equations. Therefore, it requires V*V*V*V function spaces and makes the linear system size (161604 x 161604) much bigger than a single equation.
>
> I am considering to make less mixed equations. But I don't know how to do it. Isn't mixed-equations the only way for high order PDE?
> For example in 1D case, du/dt+ d/dx(u+u''')=0, what all I know to do this is that, split it into 3 equations
> du/dt+dT=0
> T=u+dK/dx
> K=u''
>
> Then I can make weak form for equation3, and it works. But I have to have 3 mixed equations. Is this the only way to do it? I couldn't find any examples in demos.
>
> Any suggestions would help a lot. Thank you.

Do you have a 64bit operating system? Otherwise, I expect you will not
be able to allocate more than 4GB with any single program.

--
Anders

I do have the 64 bit Ubuntu.

On Wed, Sep 29, 2010 at 12:01:12AM -0000, cutejeff wrote:
> Question #127219 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/127219
>
> Status: Answered => Open
>
> cutejeff is still having a problem:
> I do have the 64 bit Ubuntu.

Strange. If you can provide us with a very simple example that runs
out of memory, I can try it on my machine.

--
Anders

Thank you, Anders.

Can I email you my python script?

On Thu, Sep 30, 2010 at 03:03:32PM -0000, cutejeff wrote:
> Question #127219 on DOLFIN changed:
> https://answers.launchpad.net/dolfin/+question/127219
>
> Status: Answered => Open
>
> cutejeff is still having a problem:
> Thank you, Anders.
>
> Can I email you my python script?

Just post it to the mailing list, but make sure it is a small script
(like 10 lines). We don't want to dig through your whole application
code.

--
Anders

hi Anders, same problem on 64bit Ubuntu LTS machine, after couple of minutes:

Solving linear system of size 397953 x 397953 (PETSc LU solver, umfpack).

UMFPACK V5.4.0 (May 20, 2009): ERROR: out of memory

[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: umfpack_UMF_numeric failed!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 10, Tue Nov 24 16:38:09 CST 2009
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: xxx on a linux-gnu named xxx by emek Fri Nov 26 16:41:18 2010
[0]PETSC ERROR: Libraries linked from /build/buildd/petsc-3.0.0.dfsg/linux-gnu-c-opt/lib
[0]PETSC ERROR: Configure run at Thu Dec 31 09:53:16 2009
[0]PETSC ERROR: Configure options --with-shared --with-debugging=0 --useThreads 0 --with-fortran-interfaces=1 --with-mpi-dir=/usr/lib/openmpi --with-mpi-shared=1 --with-blas-lib=-lblas-3gf --with-lapack-lib=-llapackgf-3 --with-umfpack=1 --with-umfpack-include=/usr/include/suitesparse --with-umfpack-lib="[/usr/lib/libumfpack.so,/usr/lib/libamd.so]" --with-superlu=1 --with-superlu-include=/usr/include/superlu --with-superlu-lib=/usr/lib/libsuperlu.so --with-spooles=1 --with-spooles-include=/usr/include/spooles --with-spooles-lib=/usr/lib/libspooles.so --with-hypre=1 --with-hypre-dir=/usr --with-scotch=1 --with-scotch-include=/usr/include/scotch --with-scotch-lib=/usr/lib/libscotch.so
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatLUFactorNumeric_UMFPACK() line 176 in src/mat/impls/aij/seq/umfpack/umfpack.c
[0]PETSC ERROR: MatLUFactorNumeric() line 2396 in src/mat/interface/matrix.c
[0]PETSC ERROR: PCSetUp_LU() line 222 in src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
  *** Warning: Using LU solver, ignoring preconditioner "default".

----------------------------------------------------------------------------------------------------------------------

wit this code (changing to mesh = Box(0, 0, 0, 3.0, 3.0, 1.0, 30, 30, 30) would yield to solve it, probably direct elimination instead of umfpack, if yes than call the sandia lab. :))

--------------------------------------------
from dolfin import *
import numpy, sys

parameters["form_compiler"]["cpp_optimize"] = True
parameters["form_compiler"]["optimize"] = True

mesh = Box(0, 0, 0, 3.0, 3.0, 1.0, 50, 50, 50)
mesh.order()
V = VectorFunctionSpace(mesh, 'CG', 1)

subdomains = MeshFunction("uint", mesh, mesh.topology().dim()-1)
left, right, back, front, top, bottom = compile_subdomains(["(fabs(x[0]) < DOLFIN_EPS) && on_boundary", "(fabs(x[0] - 3.0) < DOLFIN_EPS) && on_boundary", "(fabs(x[1] - 3.0) < DOLFIN_EPS) && on_boundary", "(fabs(x[1]) < DOLFIN_EPS) && on_boundary", "(fabs(x[2] - 1.0) < DOLFIN_EPS) && on_boundary", "(fabs(x[2]) < DOLFIN_EPS) && on_boundary"])

subdomains.set_all(6)
top.mark(subdomains, 4)

Q = Function(V)
Q = Expression('-1.0*sin(pi*1.0*x[0]/3.0)*sin(pi*1.0*x[1]/3.0)')
u0 = Function(V)
u0 = Expression('0.0')

bc = [DirichletBC(V.sub(1), u0, left), DirichletBC(V.sub(2), u0, left), DirichletBC(V.sub(1), u0, right), DirichletBC(V.sub(2), u0, right), DirichletBC(V.sub(0), u0, back), DirichletBC(V.sub(2), u0, back), DirichletBC(V.sub(0), u0, front), DirichletBC(V.sub(2), u0, front)]

u = TrialFunction(V)
v = TestFunction(V)

nu = 0.3
G = Expression('10.0')
lmbd = 2.0*G*nu / (1.0 - 2.0*nu)

def sigma(u):
 return lmbd*tr(0.5*(grad(u)+grad(u).T))*Identity(u.cell().d) + 2*G*0.5*(grad(u)+grad(u).T)

n = V.cell().n
a = inner(sigma(u), grad(v))*dx
L = dot(Q*n, v)*ds(4)

problem = VariationalProblem(a, L, bcs=bc, exterior_facet_domains=subdomains)
u = problem.solve()
----------------------------------------