petsc-options.rst (3603B)
1 .. include:: ../../global.txt 2 3 .. _sec-petscoptions: 4 5 PETSc options for PISM users 6 ---------------------------- 7 8 All PETSc programs including PISM accept command line options which control how PETSc 9 distributes jobs among parallel processors, how it solves linear systems, what additional 10 information it provides, and so on. The PETSc manual :cite:`petsc-user-ref` is the complete 11 reference on these options. We list some here that are useful to PISM users. They can be 12 mixed in any order with PISM options. 13 14 Both for PISM and PETSc options, there are ways of avoiding the inconvenience of long 15 commands with many runtime options. Obviously, and as illustrated by examples in the 16 previous sections, shell scripts can be set up to run PISM. But PETSc also provides two 17 mechanisms to give runtime options without retyping at each run command. 18 19 First, the environment variable ``PETSC_OPTIONS`` can be set. For example, a sequence of 20 runs might need the same refined grid, and you might want to know if other options are 21 read, ignored, or misspelled. Set (in Bash): 22 23 .. code-block:: none 24 25 export PETSC_OPTIONS="-Mx 101 -My 101 -Mz 51 -options_left" 26 27 The runs 28 29 .. code-block:: none 30 31 pismv -test F -y 100 32 pismv -test G -y 100 33 34 then have the same refined grid in each run, and the runs report on which options were 35 read. 36 37 Alternatively, the file ``.petscrc`` is always read, if present, from the directory where 38 PISM (i.e. the PETSc program) is started. It can have a list of options, one per line. In 39 theory, these two PETSc mechanisms (``PETSC_OPTIONS`` and ``.petscrc``) can be used 40 together. 41 42 Now we address controls on how PETSc solves systems of linear equations, which uses the 43 PETSc "KSP" component (Krylov methods). Such linear solves are needed each time the 44 nonlinear SSA stress balance equations are used (e.g. with the option ``-stress_balance 45 ssa -ssa_method fd``). 46 47 Especially for solving the SSA equations with high resolution on multiple processors, it 48 is recommended that the option :opt:`-ssafd_ksp_rtol` be set lower than its default value 49 of :math:`10^{-5}`. For example, 50 51 52 .. code-block:: none 53 54 mpiexec -n 8 ssa_testi -Mx 3 -My 769 -ssa_method fd 55 56 may fail to converge on a certain machine, but adding "``-ssafd_ksp_rtol 1e-10``" works 57 fine. 58 59 There is also the question of solver *type*, using option :opt:`-ssafd_ksp_type`. Based on 60 one processor evidence from ``ssa_testi``, the following are possible choices in the sense 61 that they work and allow convergence at some reasonable rate: ``cg``, ``bicg``, ``gmres``, 62 ``bcgs``, ``cgs``, ``tfqmr``, ``tcqmr``, and ``cr``. It appears ``bicg``, ``gmres``, 63 ``bcgs``, and ``tfqmr``, at least, are all among the best. The default is ``gmres``. 64 65 Actually the KSP uses preconditioning. This aspect of the solve is critical for parallel 66 scalability, but it gives results which are dependent on the number of processors. The 67 preconditioner type can be chosen with :opt:`-ssafd_pc_type`. Several choices are 68 possible, but for solving the ice stream and shelf equations we recommend only 69 ``bjacobi``, ``ilu``, and ``asm``. Of these it is not currently clear which is fastest; 70 they are all about the same for ``ssa_testi`` with high tolerances (e.g. ``-ssafd_picard_rtol 71 1e-7`` ``-ssafd_ksp_rtol 1e-12``). The default (as set by PISM) is ``bjacobi``. To force 72 no preconditioning, which removes processor-number-dependence of results but may make the 73 solves fail, use ``-ssafd_pc_type none``. 74 75 For the full list of PETSc options controlling the SSAFD solver, run 76 77 .. code-block:: none 78 79 ssa_testi -ssa_method fd -help | grep ssafd_ | less