To all InterCall users, ------------------------------------------ The second InterCall Newsletter. May 1993. ------------------------------------------ I n t e r C a l l A Mathematica package that allows interactive calling of external code ----------- First, welcome to all the new InterCall users. Like the first Newsletter, this letter is just to keep you up to date with all the changes and other developments relating to the InterCall package. For those of you new to InterCall, and who missed out on the earlier newsletter, then you should be able to obtain a copy from MathSource. See section 3 below if you are unfamiliar with how to access MathSource. In addition to keeping you up to date, this newsletter also includes some additional documentation for a new InterCall pre-compiler called icc, together with a shell script implementation. ========================================== 0. CONTENTS OF THIS NEWSLETTER ========================================== 1. WHAT'S NEW 2. EXTERNAL COMPILER 3. USING MATHSOURCE 4. BUG FIXES 5. DOCUMENTATION A.1 APPENDIX: Erroneous warning message. A.2 APPENDIX: Missing _MAIN_ module. A.3 APPENDIX: Absoft Fortran77 on the NeXT. A.4 APPENDIX: Source code for icc. Terry Robb Email: tdr@vaxc.cc.monash.edu.au Mathematics Department Phone: +61 3 565-5666 Monash University Fax : +61 3 565-4403 Clayton 3168, Melbourne, Australia. For ordering and site license information etc, contact: Analytica Email: analytic@earwax.pd.uwa.oz.au PO Box 522 Phone: +61 9 386 5666 Nedlands, WA 6009 Fax : +61 9 386 5666 Australia. Or email to: intercall_forward@wri.com ========================================== 1. WHAT'S NEW ========================================== By now most of you should have realized that InterCall is not just a Mathematica link to the IMSL and NAG libraries. That is just one application area, but in general you can Import in your own code and that opens up many useful possibilities. One current major application involves importing a large computational fluid dynamics program into Mathematica. Another example, albeit on a smaller scale, involves the Schroedinger equation solver which some of you may have already seen in Notebook form at MathSource. Those two applications represent an important use of InterCall: they are true hybrid symbolic+numeric applications. In the Schroedinger equation case, one symbolically defines a potential function within Mathematica, and gets an external numeric routine to interact with that function. Evolution of the wave-function is then computed by the external routine, and this evolution is animated back in Mathematica. This type of functionality is not easy to achieve directly with MathLink, but with InterCall it is a relatively easy matter and requires virtually no external programming on the users part. With these new applications one finds that a common set of external routines needs to be Imported into Mathematica every time the application is used. To save time used by the normal dynamic Import mechanism, a new external InterCall compiler, named icc, has been developed to make loading much faster. Sections 2 and 5 deal in more detail with icc and it's usefulness. So what else is new? There are some new InterCall Notebooks at MathSource, including the above mentioned Schroedinger equation solver. And soon to appear will be a set of routines useful for Computational Fluid Dynamics that can be Imported into Mathematica with InterCall. There is also a new soon to appear Cray Y-MP remote driver, which allows your Mathematica Kernel to connect to a Cray using InterCall. As with the other remote drivers available for various machines, you can request this for free. Drivers are stored in directories with names of the form driver_*, which means you can have many drivers for different machines all stored on the same file system if necessary. There has also been some progress in extending the datatypes supported by InterCall to include some parallel datatypes. At the same time a datatype is being added to allow support for the 'struct' datatype in C, and this will also allow fortran common blocks to be accessed without having to write an external routine. In the meantime I should mention that Heino Falcke has developed a big InterCall pre-processor called ICPP which can be used to help automate some of the tasks involved in accessing fortran common blocks and writing AddDefault settings. Contact him via email at for details about his ICPP. The internal InterCall compiler is now being written in C (rather than as a set of Mathematica functions) to speed up compiling of very large routines. When this is finished an even larger number of Mathematica constructs will also be compilable, such as Reverse[] and Transpose[] etc. At present most of the constructs that you would ever need to use when interacting with normal external numeric routines are supported, but now some users are starting to use the internal InterCall compiler in its own right without accessing external routines. The InterCall pseudo-code interpreter is already very fast so there are speed gains to be made in compiling Mathematica functions using the InterCall compiler. The Mandelbrot Notebook at MathSource gives an example. While we are on the subject of producing fast code from Mathematica, I would like to briefly raise the topic of a proposed new command that could be used to convert a user-written Mathematica function into a C or fortran routine. The proposed form of the new command would have a syntax of the form: WriteCode[ name_ = mathematicaFunction_, intercallDatatype_ ] and would take options like Language->C or FORTRAN or Cstar (for parallel C code) etc, and OutputFile->"stdout", Optimize->True or False etc. A trivial examples would be: In[2]:= WriteCode[ fred = Function[x,1/(1-x^3)], RF[R], Language->FORTRAN ] Out[2]= real*8 function fred(x) real*8 x fred = 1/(1-x**3) end But in general one would want to be able to define an arbitrary Mathematica statement or function such as: In[3]:= f = Function[{x,y,z,t}, y = Append[ Drop[x,1], 0 ]; z = Log[y]^2 + Reverse[x]; t = z[[1]] + z[[3]]^3 - Last[y]^First[y] ]; and then invoke a command to convert it to C or fortran. The command would use InterCall datatypes to control the conversion, for example In[4]:= WriteCode[ fred = f, S[R[10],R[10],R[10],R], Language->FORTRAN ] Out[4]= subroutine fred(x,y,z,t) integer i,N parameter (N=10) real*8 x(N),y(N),z(N),t,log intrinsic log do i=1,N-1 y(i) = x(i+1) end do y(N) = 0 do i=1,N z(i) = log(y(i))**2 + x(N-i+1) end do t = z(1) + z(3)**3 - y(10)**y(1) end Now this is a very hard problem, but is something that would work very nicely with InterCall once written. If anyone has any ideas about what they would like to see in such a command, then please email to me. I doubt whether anything will happen with this before the end of the year, but eventually I hope something could be done. And one final piece of news to mention is that at some point a book is to be written containing many applications and other topics of interests using InterCall and Mathematica. Again if you have ideas for what you would like to see... ------------------------------------------ ========================================== 2. EXTERNAL COMPILER ========================================== InterCall has a built-in numeric compiler which was discussed in the first newsletter. The latest version of InterCall now features a separate external compiler named icc. This new external compiler offers many features and advantages, but the main one is that it allows the Import command to work very much faster. Due to the internal design of InterCall it has proved very easy to implement this special external compiler, and it is not even necessary for you to get an updated version of InterCall to benefit from this new feature. All you need is a special unix shell script to implement icc, and a version is given in appendix A.4 for you to copy and use. Note however that if you received InterCall in the last few months then you will have already received a copy of icc, and what's more it will have been automatically installed for you by the installer script. For documentation about the purpose and use of icc, see the last section of this newsletter. A lot of you will find that using icc is much more convenient than using dynamic importing. For your well debugged applications you should now use icc with InterCall. ------------------------------------------ ========================================== 3. USING MATHSOURCE ========================================== This section is just intended for new users who haven't seen the first InterCall newsletter and who don't know how to access MathSource or get a Notebook-to-TeX converter. Ignore this section otherwise. To get some help on how to use MathSource, email the line 'help intro' to 'mathsource@wri.com'. For example if you are using a unix machine, then type % mail mathsource@wri.com help intro % MathSource can also be accessed via anonymous ftp. To do this type % ftp mathsource.wri.com But if your name-server doesn't know about mathsource.wri.com, then type instead % ftp 140.177.10.5 Next type anonymous as the username, and for a password just type anything -- although the standard convention is to type your email address. This should give you access to all the MathSource material. You can then use 'cd' to move around, and 'ls' to do a listing, and 'get' to retrieve a file. Type 'help' for other commands. The first InterCall newsletter can be obtained by typing to an ftp prompt the following lines cd /pub/Programming/MathLink/InterCall get IC-News1.txt quit This will copy the first newsletter to your current directory, where you can then print or preview it etc. For example: % more IC-News1.txt Other InterCall related items are kept in the files: /pub/AppliedMath/Physics/fluids.* /pub/AppliedMath/Physics/schroed.* /pub/AppliedMath/Physics/schroed2D.* /pub/PureMath/Calculus/EllipticPDE.* /pub/Programming/MathLink/InterCall/Info.* /pub/Programming/MathLink/InterCall/IC-News1.* /pub/Programming/MathLink/InterCall/Defaults.* /pub/Programming/MathLink/InterCall/Examples/InterCall-Examples.* /pub/Programming/MathLink/InterCall/Examples/ICversion2demo.* /pub/Programming/MathLink/InterCall/Examples/Traveling.* /pub/Programming/MathLink/InterCall/Examples/mandel.* Most of the InterCall related material is written in Mathematica Notebook format because of graphics and animations they contain. If you don't have a Mathematica front-end that supports Notebooks, then there are several options to allow you to look at the Notebooks. First, if you have a Macintosh available, then you can use the public domain MathReader program. This allows you to view a Mathematica Notebook, and a copy of MathReader is available at MathSource in the file: /pub/Utilities/Macintosh/MathReader2.1.sit.hqx Second, you can turn a Notebook into a TeX file using a Notebook to TeX converter. The TeX file can then be printed as usual if you have TeX on your computer. Notebook to TeX converters are available at MathSource in the files: /pub/Utilities/NB2TeX/DECnb2tex.tar.Z /pub/Utilities/NB2TeX/HPnb2tex.tar.Z /pub/Utilities/NB2TeX/Sun4nb2tex.tar.Z ------------------------------------------ ========================================== 4. BUG FIXES ========================================== InterCall has been very heavily debugged and tested and this process has, I think, eliminated virtually every major bug before any of you ever received a copy of InterCall. Very few of you have therefore encountered bugs, but naturally some bugs have slipped through the net whether you have noticed them or not. To date all known bugs have been corrected, and for your reference a list of these bug fixes is included below. None of the bugs are very serious, but if you have an older version of InterCall which contains these bugs then email to me direct and I'll send you the latest version with the corrections. When more features are added to InterCall you will receive a free update anyway, so most of you may like to wait for that instead. Also all versions shipped in the last few months have the fixes. Bugs that have been fixes: -- S[] and RF[] datatypes (i.e no arguments) failed in some versions. -- empty characters strings failed in some versions. -- buffering of output was not switched on -- it's line buffered now. -- slight CleanSlate incompatibility -- a trivial fix, see appendix A.1. -- some minor problems in the SPARC installation script have been fixed. In addition to all known bugs being fixed, there still remain the cases where some strange behaviour has been reported. These reports are not entirely due to problems with InterCall (or even Mathematica), but they do present themselves when using InterCall. Strange behaviour: -- There is a rare bug in some versions of the SPARC loader: sometimes you get an erroneous message about a missing _MAIN_ module. See hints below in appendix A.2 on how to make this message go away. -- Sometimes an InterCall[Off] command doesn't close properly and an or process is left in the background. This is not a major problem, as the process consumes no resources and always disappears when Mathematica exits. So don't worry too much if it happens with your setup. Also it should be fixed properly by the next release. -- An Import command should be reasonably fast, but sometimes it can be slow. Note that under Mathematica 2.0 a command like OpenWrite["!ls"] (and similar examples) was fast, but under Mathematica 2.1 it could take a long time if memory is low, because a copy of Mathematica gets temporarily spawned. This is fixed with Mathematica 2.2, but in any case the new icc compiler effectively eliminates the problem if it actually does happen with your particular setup under Mathematica 2.1. -- The Absoft Fortran77 compiler on the NeXT is slightly awkward to use in conjunction with C object code. See hints in appendix A.3 on how to get around the difficulties. -- Some versions of the SGI family (for example Iris, crimson and indigo) appear to have a slight binary incompatibility between some versions of the operating system. If you upgrade your SGI then you may need to get a new (and free) ilink.SGI executable. ------------------------------------------ ========================================== 5. DOCUMENTATION ========================================== ------------------------------- The InterCall icc compiler ------------------------------- There is some good news that will make importing very much faster. Most new InterCall users will already have received a shell script called icc in the directory named driver_*. Other users will not have a copy of icc, so a DECstation version is appended at the end of this newsletter in appendix A.4 which you can copy and modify to work with your particular system. Whether you are a new or longer term user however, you will not have received any documentation or other information about icc. So what is icc? Well, if you know what routines you are going to Import in advance of running Mathematica, then you can pre-configure a special InterCall executable using icc, and then use that executable over and over again. The pre-configured executable can be imported virtually instantaneously into Mathematica, and this is the big advantage of using icc, but it does require some planning ahead. In some respects the InterCall icc compiler is similar to the MathLink mcc compiler, but with it you get all the ease and simplicity of use of a normal InterCall import. An example of such a pre-configured executable, created with icc, is the one used in conjunction with the schroed2d.ma Notebook available from MathSource. The executable, called schroedinger, will be available for various system (including DEC, HP9000S700, IBMRS6000, NeXT, SGI and SPARC) from MathSource eventually, but you can email to me direct in the meantime to get a free copy. Anyway here is what the icc program does: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ icc is InterCall's cc compiler -- its options are similar to those of unix's cc compiler or MathLink's mcc compiler, but instead of taking a file name like *.c (C source) or *.tm (MathLink template) it takes a file name like *.ic which just contains names of routines that you may later wish to Import into Mathematica using InterCall. Here is a simple example to illustrate its use. Suppose you want to always use the routines MYFUNC, NEWFUN, ERSET, and DAI, and that the code for those routines are in the object files named mynewf.o, mycode.o and also the libimsl.a (IMSL) archive library. Write those routine names in a file with extension *.ic, eg: % cat myImports.ic MYFUNC NEWFUN ERSET DAI Now invoke the icc compiler, which in this instance will output a special pre-configured executable called myImports, by typing % icc -o myImports myImports.ic mynewf.o mycode.o -limsl Then anytime later you can use that executable with InterCall. E.g % math In[1]:= <&1 to $CC -O -o $TMPICALLE ...etc... $LIBRS $LOPTS 2> /dev/null The above change should throw all loader messages to /dev/null and thus not echo them at Mathematica's end. This will hide the bug for you. In general you want to see compiler messages if/when they occur though because sometimes they are actually informative -- but if the loader persists in giving erroneous warnings then it is probably better to suppress all those messages. ------------------------------------------ ========================================== A.3 APPENDIX: Absoft Fortran77 on the NeXT. ========================================== The Absoft Fortran77 compiler on the NeXT produces object code that is not easy to call from C code. This makes it inconvenient (although not impossible) to call using InterCall. Most other fortran compilers, such as f77 for most unix systems, and fort for an IBMRS6000, produce object code that is easy to call from C. If you have fortran code on a NeXT then there are two main strategies to fix the Absoft Fortran77 difficulties. ******** Try using a fortran-to-C converter. The public domain fortran-source to C-source converter (f2c) is an excellent tool. If you have access to the fortran source code, then your problems are solved, as you can run it through f2c, and then use the standard cc compiler to produce the object code that InterCall needs. Your NeXT computer will probably have f2c as a standard utility, but if not then email to me or check out NETLIB for a public domain copy. ******** Try a jacket-function. If you don't have access to the fortran-source (for example someone has given you a file with Absoft fortran object code, without the source), then you can use the 'jacket-function' technique. This technique can also be used in conjunction with parallel or vector code, where some datatype structures are not directly supported by InterCall. Absoft fortran requires that double precision functions (and also integer and complex etc functions) be converted to C's datatype structure using a special DoubleConvert() routine. What you could do is write a small C function (called a jacket-function), that calls the fortran function you really want. In the C function you put all the DoubleConvert(..) things that the Absoft fortran manual says you should do. Then from Mathematica's end, you mention the object file that has the small jacket C functions that interface to the real Absoft fortran routines you want. For example, AddDefault[ ...etc... ...etc..., "mycode.o jackets.o" ] where jacket.o was written in C by you, and defines C functions to call any or all of the fortran routines in the Absoft fortran object mycode.o. In driver*/icall you may also have to mention CConvert.o in LIBRS and mention any other code that the Absoft manual suggests. ------------------------------------------ ========================================== A.4 APPENDIX: Source code for icc. ========================================== Here is the driver_DEC/icc shell script for a DECstation. Modify as indicated for other systems (see the hints section). #<----------------------------- CUT HERE ------------------------------------> #! /bin/sh # I am the InterCall C compiler icc, and I need the object file icall.o . # You may modify the indicated section below. Don't touch anything else. # Edit the LOPTS and ICDRVDIR variables -- see hints further down. # Also make me executable: chmod uo+x icc # and place me in the directory driver_*/ #--------------------------------- EXAMPLE USAGE ----------------------------- #**** % cat myImports.ic #**** MYFUNC #**** NEWFUN #**** ERSET #**** DAI #**** % icc -o myImports myImports.ic mynewf.o mycode.o -limsl #**** % math #**** In[1]:= < $TMPIBINDH /bin/cat /dev/null > $TMPIBINDC /bin/cat >> $TMPIBINDC << EOF #include /* gets fflush stdout */ #include /* strcmp strcpy */ char* malloc(); typedef char* addr; static void Insert1(); #define NBUF 80 EOF N=0; COMMENT=NULL echo '/* This line purposely here. */' >> $TMPIBINDH echo '#include "'$TMPIBINDH'"' >> $TMPIBINDC echo 'static void InsertRoutines() {' >> $TMPIBINDC for ROUTINE in `/bin/cat $PREICFILE | sed 's/*/#/g'`; do if [ "$ROUTINE" = "/#" ]; then COMMENT=TRUE; continue; fi if [ "$ROUTINE" = "#/" ]; then COMMENT=NULL; continue; fi if [ "$COMMENT" = TRUE ]; then continue; fi N=`expr $N + 1`; ROUTINEX=`echo $ROUTINE$UNDSC | /usr/bin/tr '[A-Z]' '[a-z]'` echo 'void '$ROUTINEX'(); /* id='$N' */' >> $TMPIBINDH echo 'Insert1("'$ROUTINE'",(addr)'$ROUTINEX');' >> $TMPIBINDC done echo '}' >> $TMPIBINDC echo '#define N '$N >> $TMPIBINDC echo '#define LIBRARY "'$PREICALLE'"' >> $TMPIBINDC /bin/cat >> $TMPIBINDC << EOF static char* r[N+1]; static addr a[N+1]; static int kk=0; static void Insert1(routine,address) char *routine; addr address; { r[kk] = (char *) malloc(NBUF); strcpy(r[kk],routine); a[kk] = address; kk++; } static void UndefinedRoutine() {printf("Not installed in %s.\n",LIBRARY);} extern void Insert(); extern int Icallmain(); int main(argc, argv) int argc; char* argv[]; { int k; char routine[NBUF]; addr address; char objfile[NBUF]; int argc1=2; char* argv1[2]; argv1[0]=(char*)malloc(8); argv1[1]=(char*)malloc(8); if (isatty(0) && argc==1) exit(1); InsertRoutines(); printf("#:\n(%s)\n#.\n",LIBRARY); fflush(stdout); while (*gets(routine)) { address=0; for (k=0;k