General improvement on viewing wide and long slog2 files. This added features needed...
[mpe.git] / INSTALL.cross
1                       MPE (Multi-Processing Environment)
2                       ----------------------------------
3
4                           Version 2.5.0. Oct, 2009
5
6                     Mathematics and Computer Science Division
7                            Argonne National Laboratory
8
9 II. CONFIGURATION
10 -----------------
11
12 Please read the CONFIGURATION section II. a) and II. b) in INSTALL to get
13 yourself familar with the Configuration Model of MPE.
14
15 III. INSTALLATION INSTRUCTIONS
16 ------------------------------
17
18 This file describes the Cross-Compilation build instructions for MPE.
19 If you don't need cross-compilation support or you don't know what
20 cross-compilation is, you probably don't need to read this.  INSTALL
21 is all you need to know.
22
23 **** The following is NO longer true in this MPE release ****
24 This release of MPE has better support of cross-compilation and does
25 not require a 2-step process, i.e. no cross-spec file is needed.
26 Cross compilation is similar to native compilation except the addition
27 of --host and --build as required by autoconf(i.e. configure).
28 In short, cross-compile requires specification of CC/F77/MPI_CC/MPI_F77,
29 --host and --build as part of configure command.
30 More documentation will be added later.
31 **************************************************************
32
33 For those who need cross-compilation support, 7 to 10 variables are
34 needed.  These variables are determined through some runtime tests on
35 cross-compiled host (i.e. the host where the libraries/executable are
36 being built for).  The exact number of variables depends on if the
37 MPE's internal Fortran to C MPI profiling wrapper library needs to be
38 built.  7 of the 10 environmental variables are related to the C compiler
39 of the cross-compiled platform.
40
41 CROSS_SIZEOF_CHAR      : sizeof(char), e.g. 1
42 CROSS_SIZEOF_SHORT     : sizeof(short), e.g. 2
43 CROSS_SIZEOF_INT       : sizeof(int), e.g. 4
44 CROSS_SIZEOF_LONG      : sizeof(long), e.g. 8
45 CROSS_SIZEOF_LONG_LONG : sizeof(long long), e.g. 8
46 CROSS_SIZEOF_VOID_P    : sizeof(void *), e.g. 8
47 CROSS_BIGENDIAN        : true if it is a big endian machine, false otherwise.
48
49 If your MPI implementation does NOT provide any Fortran to C MPI profiling
50 wrapper library, then you need to specify 3 more environmental variables.
51 The following environmental variables specify properties of MPI
52 implementation if F2CMPI_LIBS is defined, i.e MPE's internal Fortran
53 to C MPI profiling library is used.  These variables are related to
54 how Fortran primitive type looks like in C.
55
56 CROSS_MPI_STATUS_SIZE  : value of MPI_STATUS_SIZE defined in mpif.h, i.e.
57                          sizeof(MPI_STATUS_SIZE)/sizeof(MPI_Fint), e.g. 4
58 CROSS_FORTRAN2C_TRUE   : value of fortran's .TRUE. in C program, e.g. 1
59 CROSS_FORTRAN2C_FALSE  : value of fortran's .FALSE. in C program, e.g. 0
60
61
62
63 III. a) Cross-Compile Specification
64 -----------------------------------
65
66 In order to simplify and avoid any ambiguity in the determination of these
67 values, a small subpackage called "cross_spec" within MPE can be used.
68 This goal of this subpackage is to generate a cross-compile specification
69 file which will contain all the relevant CROSS_xxx variables listed above.
70 The subapackage has its own independent configure and accepts a similar set
71 of configure variables and options as MPE's configure.  If you are not
72 familar with MPE's configure variables and options, please see INSTALL file's
73 section II. a) and II. b).  Or you can do "mpe2/cross_spec/configure --help"
74 for a list of the cross_spec's options and environment variables.
75 The default cross-compile specification file generated by cross_spec
76 is called cross_spec.txt which can be overriden by option --with-spec.
77
78 Let say we are cross-compiling MPE for x86_64 (64bit linux) backend nodes
79 on a 32bit linux frontend box which has the following cross-compilers:
80
81     the serial C cross-compiler :        /usr/local/bin/lans64_cc,
82     the serial Fortran cross-compiler :  /usr/local/bin/lans64_fc,
83     the MPI C cross-compiler :           /usr/local/bin/lans64_mpicc,
84     the MPI Fortran cross-compiler :     /usr/local/bin/lans64_mpif90
85
86 Usually the frontend will have a command to run executable in the backend,
87 let say this is called "/usr/local/bin/lans64_run -np 1", where "-np 1"
88 means the executable will be run on 1 node.
89
90 Now pick a build directory <build_spec_dir> to build cross_spec, be sure
91 not set <build_spec_dir> as <mpe2_src_dir>.  Doing so will cause conflict
92 between cross_spec's Makefile and mpe2's main Makefile.
93
94 One can configure cross_spec as follows:
95
96     mkdir <build_spec_dir>
97     cd <build_spec_dir>
98     <mpe2_src_dir>/cross_spec/configure --host=x86_64-linux                   \
99                                         AR=ar RANLIB=ranlib                   \
100                                         CC=/usr/local/bin/lans64_cc           \
101                                         F77=/usr/local/bin/lans64_fc          \
102                                         MPI_CC=/usr/local/bin/lans64_mpicc    \
103                                         MPI_F77=/usr/local/bin/lans64_mpif90  \
104                                         MPERUN="/usr/local/bin/lans64_run -np 1"
105     make
106     make run
107
108 Where --host=x86_64_linux informs configure we are building for 64bit linux.
109 Notice there isn't a --prefix option in the configure command, as it isn't
110 necessary to install this subpackage anywhere.  Here AR and RANLIB are used
111 in the configure command to highlight these variables can be added for
112 platform which has specific AR and RANLIB for the backend nodes.  The MPERUN
113 allows users to run the tests on the backend through Makefile, i.e. through
114 the "make run" commad.  Without MPERUN or there isn't a command to launch
115 executable to run on the backend, the "make run" will execute the tests
116 locally on the frontend.  In that case, one has to launch the tests separately
117 by hand.
118
119 There are 2 executables and 1 script created by cross_spec.  They are
120 cross_cc, cross_f2c and cross_env.
121
122     cross_cc:  an executable that specifies various C compiler features
123                in the backend.  (Has to run on the backend)
124     cross_f2c: an executable that specifies various Fortran to C features
125                in the backend. (Has to run on the backend)
126     cross_env: a script that specifies the configure options used in
127                cross_spec and these options will be used in cross-compiling
128                MPE again.  (run on the frontend)
129
130 If running of the cross_cc, cross_f2c and cross_env returns successfully,
131 a cross_spec.txt file will be generated, an example of file is like
132
133 # C compiler runtime characteristics...
134 CROSS_SIZEOF_CHAR=1
135 CROSS_SIZEOF_SHORT=2
136 CROSS_SIZEOF_INT=4
137 CROSS_SIZEOF_LONG=8
138 CROSS_SIZEOF_LONG_LONG=8
139 CROSS_SIZEOF_VOID_P=8
140 CROSS_BIGENDIAN=false
141 # Fortran to C runtime characteristics...
142 CROSS_MPI_STATUS_SIZE=5
143 CROSS_FORTRAN2C_TRUE=1
144 CROSS_FORTRAN2C_FALSE=0
145 # C and Fortran development tool variables...
146 MPI_CC=/usr/local/bin/lans64_mpicc
147 CC=/usr/local/bin/lans64_cc
148 MPI_F77=/usr/local/bin/lans64_mpif90
149 F77=/usr/local/bin/lans64_fc
150 AR=ar
151 RANLIB=ranlib
152
153 Any other configure variables, e.g. CFLAGS, for MPE configure
154 can be appended in the cross_spec.txt, as they will be proprogangated
155 down to other subpackages within MPE configure.
156
157
158
159 III. b) Cross-Compilation With "cross_spec.txt"
160 -----------------------------------------------
161
162 With cross_spec.txt generated by cross_spec subpackage, cross_compile
163 MPE becomes easy.  One can do it as follows:
164
165     mkdir <build_mpe2_for_backend>
166     cd <build_mpe2_for_backend>
167     <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
168                              --host=x86_64_linux                               \
169                              --with-cross-spec=<build_spec_dir>/cross_spec.txt \
170                              MPERUN="/usr/local/bin/lans64_run -np 4"
171     make
172     make install
173     make installcheck-all
174
175 Here AR and RANLIB are added for illustration purposes.
176
177 Where --host is used to inform configure what type of host to be built for
178 the backend.  Again, MPERUN is optional here.  With MPERUN, the 
179 "make installcheck-all" command will launch runtime tests to the backend
180 to check if every selected components are built correctly.  Notice that
181 the MPERUN command here is different from the one used in cross_spec, since
182 we want to run multiple processes job in the "installcheck-all".
183
184
185     instructions for BlueGeneLight (BG/L)
186     -------------------------------------
187
188     **** Cross-Spec build ****
189     mkdir <build_spec_dir>
190     cd <build_spec_dir>
191     <mpe2_src_dir>/cross_spec/configure  CC=blrts_xlc F77=blrts_xlf         \
192                                          MPI_CC=mpixlc MPI_F77=mpixlf77     \
193                                          --host=powerpc-unknown-linux-gnu   \
194                                          --build=powerpc-pc-linux-gnu
195     make
196     cqsub -n 1 -t 5 cross_cc
197     cqsub -n 1 -t 5 cross_f2c
198     cross_env
199
200     Here --host is set to powerpc-unknown-linux-gnu even BG/L is not linux
201     Since we don't know what is the correct host-type for BG/L, --host
202     is arbitary set to  powerpc-unknown-linux-gnu and should be different
203     from --build value so the configure knows we are doing cross-compilation.
204     Also make sure that "cqsub ... cross_cc" is finished __before__
205     "cqsub ... cross_f2c", otherwise the cross_spec.txt may have overlapped
206     output from both cross_cc and cross_f2c at the same time.
207
208     **** MPE2 build ****
209     mkdir <build_mpe2_for_backend>
210     cd <build_mpe2_for_backend>
211     <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
212                              --host=powerpc-unknown-linux-gnu                  \
213                              --build=powerpc-pc-linux-gnu                      \
214                              --with-cross-spec=<build_spec_dir>/cross_spec.txt \
215                              --with-java=/usr/lib/IBMJava2/jre                 \
216                              --disable-misc --disable-graphics                 \
217                              --disable-sample --disable-rlog
218     make
219     make install
220     make installcheck
221
222     instructions for BlueGene P (BG/P)
223     -------------------------------------
224
225     **** Cross-Spec build ****
226     mkdir <build_spec_dir>
227     cd <build_spec_dir>
228     <mpe2_src_dir>/cross_spec/configure  CC=bgxlc_r F77=bgxlf_r             \
229                                          MPI_CC=mpixlc_r MPI_F77=mpixlf77_r \
230                                          --host=powerpc-unknown-linux-gnu   \
231                                          --build=powerpc-pc-linux-gnu
232
233     ** the "_r" is used to enable thread support in the compiler **
234     ** mpixlc_r/mpixlf_r are in /bgsys/drivers/ppcfloor/comm/bin **
235
236     make
237     cqsub -n 1 -t 5 cross_cc
238     cqsub -n 1 -t 5 cross_f2c
239     cross_env
240
241     Here --host is set to powerpc-unknown-linux-gnu even BG/P is not linux
242     Since we don't know what is the correct host-type for BG/P, --host
243     is arbitary set to  powerpc-unknown-linux-gnu and should be different
244     from --build value so the configure knows we are doing cross-compilation.
245     Also make sure that "cqsub ... cross_cc" is finished __before__
246     "cqsub ... cross_f2c", otherwise the cross_spec.txt may have overlapped
247     output from both cross_cc and cross_f2c at the same time.
248
249     **** MPE2 build ****
250     mkdir <build_mpe2_for_backend>
251     cd <build_mpe2_for_backend>
252     <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                      \ 
253                              --host=powerpc-unknown-linux-gnu                 \
254                              --build=powerpc-pc-linux-gnu                     \
255                              --with-cross-spec=<build_spec_dir>/cross_spec.txt\
256                              --disable-safePMPI                               \
257                              --disable-misc --disable-graphics                \
258                              --disable-sample --disable-rlog
259     make
260     make install
261     make installcheck
262
263
264     instructions for SiCortex (SC5812)
265     -------------------------------------
266
267     On the i686 linux cross-compiling frontend (bblogin)
268
269     **** Cross-Spec build ****
270     mkdir <build_spec_dir>
271     cd <build_spec_dir>
272     <mpe2_src_dir>/cross_spec/configure                                      \ 
273                               CC="scpathcc -n32"                             \
274                               F77="scpathf95 -fno-second-underscore -n32"    \
275                               MPI_CC="scmpicc --pathscale -n32"              \
276                               MPI_F77="scmpif90 -fno-second-underscore -n32" \
277                               LIBS=-lpthread                                 \
278                               --host=mips64el-gentoo-linux-gnu               \
279                               --build=x86_64_linux        
280     make
281
282     The above build is for n32 binary.  For 64-bit binary, replace -n32 by -64.
283     Now log onto sicortex's login node (fd-login)
284     to launch programs with srun to get the cross_spec.txt
285
286     srun -n 1 cross_cc
287     srun -n 1 cross_f2c
288     srun -n 1 cross_env
289
290     **** MPE2 build ****
291     mkdir <build_mpe2_for_backend>
292     cd <build_mpe2_for_backend>
293     <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                      \
294                              --host=mips64el-gentoo-linux-gnu                 \
295                              --build=x86-linux-gnu_linux                      \
296                              --with-cross-spec=<build_spec_dir>/cross_spec.txt\
297                              --disable-safePMPI
298     make
299     make install
300     make installcheck
301
302
303     instructions for Cray X1E (Phoenix)
304     ----------------------------------
305
306     On the i686 linux cross-compiling frontend (robin)
307
308     **** Cross-Spec build ****
309     mkdir <build_spec_dir>
310     cd <build_spec_dir>
311     <mpe2_src_dir>/cross_spec/configure CC=cc F77="ftn -s real64 -dp"         \
312                                         MPI_CC=cc MPI_F77="ftn -s real64 -dp" \
313                                         --host=craynv-cray-unicosmp3.0.X      \
314                                         --build=i686-pc-linux-gnu
315     make
316
317     # Now create a cross_tests.cmd as follows:
318     cat > cross_tests.cmd <<EOF
319 #!/bin/sh
320 #PBS -l walltime=5:00,mppe=1
321 #PBS -j oe
322 #PBS -m abe
323 #PBS -A xxxxx
324
325 # job starts in home directory, cd to the submission directory
326 cd ${PBS_O_WORKDIR}
327
328 echo "running cross_cc"
329 aprun -n 1 ./cross_cc
330
331 echo "running cross_f2c"
332 aprun -n 1 ./cross_f2c
333
334 EOF
335
336     qsub cross_tests.cmd
337     # After the job is done, then run cross_env on the front-end
338     ./cross_env
339
340
341     Now make sure that java (SUN or IBM's jdk) is in your path.
342
343     **** MPE2 build ****
344     mkdir <build_mpe2_for_backend>
345     cd <build_mpe2_for_backend>
346     <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
347                              --host=craynv-cray-unicosmp3.0.X                  \
348                              --build=i686-pc-linux-gnu                         \
349                              --with-cross-spec=<build_spec_dir>/cross_spec.txt \
350                              --disable-misc --disable-graphics                 \
351                              --disable-threadlogging
352
353     --disable-threadlogging disables the pthread logging support since Cray
354     backend does not seem to support pthread calls.
355
356     make
357     make install
358     make installcheck
359
360
361     instructions for Cray XT4 (jaguar)
362     ----------------------------------
363
364     Compiling for the catamount(no thread support) on the frontend
365
366     **** Cross-Spec build ****
367     mkdir <build_spec_dir>
368     cd <build_spec_dir>
369     <mpe2_src_dir>/cross_spec/configure CC="cc --target=catamount"            \
370                                         F77="ftn --target=catamount"          \
371                                         MPI_CC="cc --target=catamount"        \
372                                         MPI_F77="ftn --target=catamount"      \
373                                         --host=x86_64-catamount-linux         \
374                                         --build=x86_64-linux
375     make
376
377 # allocate an interactive PBS job slot to run cross_cc and cross_f2c
378     qsub -A XXXXX -V -I -lwalltime=00:15:00,size=2
379
380 # cd to where cross_cc and cross_f2c re
381     cd <mpe2_src_dir>/cross_spec
382
383 # run cross_cc and cross_f2c
384     yod -sz 1 cross_cc
385     yod -sz 1 cross_f2c
386
387 # Now make sure that java (SUN or IBM's jdk) is in your path if it is available
388
389     **** MPE2 build ****
390     mkdir <build_mpe2_for_backend>
391     cd <build_mpe2_for_backend>
392     <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
393                              --host=x86_64-catamount-linux                     \
394                              --build=x86_64-linux                              \
395                              --with-cross-spec=<build_spec_dir>/cross_spec.txt \
396                              CC="cc --target=catamount"                        \
397                              F77="ftn --target=catamount"                      \
398                              MPI_CC="cc --target=catamount"                    \
399                              MPI_F77="ftn --target=catamount"
400     make
401     make install
402                       
403
404
405
406
407 III. c) Cross-Compilation Without "cross_spec.txt"
408 --------------------------------------------------
409
410 The section contains some examples of cross-compiling MPE on older platforms:
411
412 For cross-compilation with an existing version of MPICH(or LAM). e.g. ASCI-Red.
413
414 *** If the Fortran to C MPI profiling wrapper library exists (it is called
415     libfmpich.a in MPICH and liblamf77mpi.a in LAM), do
416
417     setenv CC cicc
418     setenv F77 cif77
419     setenv CROSS_SIZEOF_CHAR 1
420     setenv CROSS_SIZEOF_SHORT 2
421     setenv CROSS_SIZEOF_INT 4
422     setenv CROSS_SIZEOF_LONG_LONG 8
423     setenv CROSS_BIGENDIAN false
424     ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
425                              --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
426                              --with-f2cmpilibs=-lfmpich \
427                              --disable-graphics
428     make
429     make install PREFIX=${MPE_INSTALL_DIR}
430
431 Here assume mpicc/mpif77 scripts exist.  If not, mpicc should include lines:
432
433 #! /bin/sh
434 cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>
435
436
437 mpif77 can be composed by including the following line
438
439 #! /bin/sh
440 cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>
441
442
443 <-lmpi> refers to various MPI libraries included in your MPI implementation.
444
445
446 *** If the Fortran to C MPI profiling wrapper library does NOT exist,
447     then the MPE's own Fortran to C MPI profiling wrapper library needs
448     to be built.  3 additional environment variables are needed.  They
449     are CROSS_MPI_STATUS_SIZE, CROSS_FORTRAN2C_TRUE & CROSS_FORTRAN2C_FALSE
450
451     setenv CC cicc
452     setenv F77 cif77
453     setenv CROSS_SIZEOF_CHAR 1
454     setenv CROSS_SIZEOF_SHORT 2
455     setenv CROSS_SIZEOF_INT 4
456     setenv CROSS_SIZEOF_LONG_LONG 8
457     setenv CROSS_BIGENDIAN false
458     setenv CROSS_MPI_STATUS_SIZE 4
459     setenv CROSS_FORTRAN2C_TRUE  1
460     setenv CROSS_FORTRAN2C_FALSE 0
461     ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
462                              --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
463                              --disable-graphics
464     make
465     make install PREFIX=${MPE_INSTALL_DIR}
466
467
468 For Cray X1E MPI implementation: cross-compilation at a linux front-end(robin)
469 (e.g. phoenix at Oak Ridge)
470
471     ${MPE_SRC_DIR}/configure                                              \
472     MPI_F77="ftn -s real64 -dp"                                           \
473     F77="ftn -s real64 -dp"                                               \
474     MPI_CC=cc                                                             \
475     CC=$HOME/phoenix/bin/gcc-robin                                        \
476     --host=craynv-cray-unicosmp3.0.X                                      \
477     --build=i686-pc-linux-gnu                                             \
478     CROSS_SIZEOF_CHAR=1                                                   \
479     CROSS_SIZEOF_SHORT=2                                                  \
480     CROSS_SIZEOF_INT=4                                                    \
481     CROSS_SIZEOF_LONG_LONG=8                                              \
482     CROSS_BIGENDIAN=true                                                  \
483     CROSS_MPI_STATUS_SIZE=6                                               \
484     CROSS_FORTRAN2C_TRUE=1                                                \
485     CROSS_FORTRAN2C_FALSE=0                                               \
486     --disable-misc --disable-graphics --disable-collchk                   \
487     --disable-sample --disable-rlog