MovieDigitizer
MathLink for Automatic Digitizing of QuickTime Movie
Chikara MIYAJI

Institute of Health and Sport Sciences, University of Tsukuba,
Tsukuba city, Ibaraki, JAPAN

E-mail:miyaji@taiiku.tsukuba.ac.jp

MovieDigitizer is a MathLink program that runs on the Macintosh. It reads QuickTime movie files, and communicates the graphical information to Mathematica.

Biomechanics of Sports

Biomechanics analyses human movement as mechanics. How do humans move and how can we improve  movement more efficiency.

When analyzing body movements, it is necessary to track a given point on a body using video processing. Modeling of body movement is built upon from the motions of these points.

Why I wrote MovieDigitizer

Flexible Automatic Digitizing is necessary for biomechanical Analysis

Computer have the ability to directly capture the Video information to memory.

QuickTime: Standard Movie format make it easy to process Video.

Program Design

For automatic Digitizing System, there are many different level of processing.

[1]Low Level Pixel Access

This is best done using the C language to access toolbox routines

[2]Numerical Processing on the Pixel Data, such as Filtering, in order to to find the object

Needs flexible Programing environment.

[3]Prediction of the subsequent object positions

Prediction needs the model of object,
This needs flexible Programing environment.

MovieDigitizer's Design

Let's try MovieDigitizer

Invoke MovieDigitizer

Thomas Chin's Dialog will appear

Open A QuickTime File

Connect from Mathematica

lnk = Install["5000",LinkHost->"130.158.112.209",
LinkMode->Connect]

lnk = Install["MovieDigitizer",
	LinkMode->Connect]

The Commands of MovieDigitizer

QTGet[]

QTGet[] will return the coordinate of the point where you click 
the mouse.

The origin of coordinate system is top left corner of the window.

QTGet[]

QTStep[n]

You can move around movie frame. QTStep[n] moves movie n frame.
If n is negative, movie will move backward. If n is zero, movie will move to start.

Do[QTStep[1],{i,5}]

QTPix[{x,y},{x,y}]

QTPix[{x,y},{x,y}]will return the Raster Array format of the area {x,y}
and {x,y}. The format is RasterArray.

Show[QTPix[{280,40},{300,50}]]

Show[QTPix[QTGet[],QTGet[]]]

QTImage[{x,y},{x,y}]

QTImage is similar to QTPix but it return only RGB number as list of list.
{{{r,g,b},{r,g,b},....{r,b,g}},
{{r,g,b},{r,g,b},....{r,b,g}},
....
{{r,g,b},{r,g,b},....{r,b,g}}}

area = QTImage[{0,0},{10,10}]

Image2Pix[image]

You can convert Image format to RasterArray format.

Show[Image2Pix[area]]

Programming

Client or Server?

MovieDigitizer act as stand alone program, also Mathemtica runs as normal session. The relation between both is equal.

Mathematica is client

MathLink for Excel
IN&OUT(sample)
Link Tutor

.tm generates client program

.tm generates standard Macintosh program

This is the normal main event loop of tm.c

int handle_user_event( MLINK mlp, MLYieldParameters yp) {
#pragma unused( mlp)
	struct _p { short v; unsigned long s;}* _yp = (struct _p*)yp;
	unsigned long ticks = _yp && _yp->v == 1 ? _yp->s : 0;
	EventRecord event;

	if( WaitNextEvent(everyEvent, &event, ticks, nil)){
		long      menuResult = 0;
		short     menuID, menuItem;
		WindowPtr window;
		Str255    daName;

		switch ( event.what ) {
		case mouseDown:
			if( FindWindow(event.where, &window) == inMenuBar)
				menuResult = MenuSelect(event.where);
			break;
		case keyDown:
			if( event.modifiers & cmdKey )
				menuResult = MenuKey(event.message & charCodeMask);
			break;
		case kHighLevelEvent:
			AEProcessAppleEvent(&event);
			break;

This is the MovieDigitizer.tm.c.

int handle_user_event( MLINK mlp, MLYieldParameters yp) {
#pragma unused( mlp)
	struct _p { short v; unsigned long s;}* _yp = (struct _p*)yp;
	unsigned long ticks = _yp && _yp->v == 1 ? _yp->s : 0;
	EventRecord event;

/* Local start */
	Boolean isEvent;
	int id;
	
	isEvent = WaitNextEvent(everyEvent, &event, ticks, nil);
	maintainCursor(&event, MAINLOOP);
	chkMovieEvent(&event); 
	if(isEvent){
/* Local end */
		long      menuResult = 0;
		short     menuID, menuItem;
		WindowPtr window;
		Str255    daName;

		switch ( event.what ) {
		case mouseDown:
/* Local start */
			id = FindWindow(event.where, &window);
			switch(id){
				case inMenuBar:
					menuResult = MenuSelect(event.where);
					break;
				case inDrag:
					doDrag(&event,window);
					break;
				case inGoAway:
					doGoAway(window, event.where);
					break;
		}
		break;
		case updateEvt:
			doUpdate(&event);
/* Local end */
			break;
		case keyDown:
			if( event.modifiers & cmdKey )

Steal instead of Wrting

MLDlg.c 222 lines.(0 lines of my code)

int MLMain( argc, argv)
	int argc; char* argv[];
{
	MLINK mlp;
	/* Local Start */
	void*			env;

	if( !init_macintosh() ) return 3;
	if( !(env = MLInitialize(NULL)) ) return 3;
	MLSetDialogFunction (env, MLRequestArgvFunction, 
		(MLDialogFunctionType)DoMLRequestArgv);
/* Local End */

MovieDigitizer.c 400 lines.( 250 lines of my code)

MovieDigitizer.tm.c 222 lines.( 21 lines of my code)

:Begin:
:Function:	get_mouse
:Pattern:	QTGet[]
:Arguments:	{}
:ArgumentTypes:	{}
:ReturnType:	Manual
:End:

:Begin:
:Function: get_pixrect
:Pattern: GetPix[{x_,y_}, h_, v_]
:Arguments: {x, y, h, v}
:ArgumentTypes: {Integer,Integer,Integer,Integer}
:ReturnType: Manual
:End:

:Begin:
:Function: dostep
:Pattern: QTStep[n_]
:Arguments: {n}
:ArgumentTypes: {Integer}
:ReturnType: Integer
:End:

C or Mathematica

I must decide which procedures run on C side or Mathematica side

I try to put on Mathematica side as many part as possible.

Mathematica Side

QTPix[{x1_,y1_},{x2_,y2_}] := Module[
{h=Abs[x1-x2]+1,
 v=Abs[y1-y2]+1},
	Graphics[RasterArray[
		Partition[
			Map[(#/.List->RGBColor)&,
				Partition[
					Map[(If[#<0,#+65537,#]/65536.0)&,
						GetPix[{Min[x1,x2],Max[y1,y2]},h, v]],
			3]
		],
	h]
],AspectRatio->v/h]]

C Side

for(i=0; i < v; ++i)
		for(j=0; j < h; ++j){
			GetCPixel(x+j, y-i, &cpix);
			(*buf)[3*(i*h+j)] = cpix.red;
			(*buf)[3*(i*h+j)+1] = cpix.green;
			(*buf)[3*(i*h+j)+2] = cpix.blue;
		}
	MLPutIntegerList(stdlink, *buf, 3 * h * v);

Automatic Digitizing

Distance of two RGB color value

distance gets distance of two RGB color.

distance[px1_,px2_]:=
	N[Sqrt[Apply[Plus,(px1-px2)^2]]]

Convert image to Grayscale level distance

color2gray[image_,col_] :=
	Map[
		Map[(1-distance[#,col]/Sqrt[3.])&, #]&,
	image]

Convert Grayscale to Raster format

gray2raster[gim_]:=
	Module[{d=Dimensions[gim]},
		Graphics[Raster[gim],
		AspectRatio->d[[1]]/d[[2]]]]

Image Filtering

Filtering the Grayscale image.

t2[l_] :=
	Take[l,{2,Length[l]-1}]

tt2[l_] :=
	Transpose[t2[Transpose[l]]]

tt3[l_,n_] :=
	Transpose[Drop[Transpose[t2[l]],n]]

filter[gim_] :=
	Module[{i1,i2,i3,i4,i5},
		i1 = tt2[Drop[gim,2]];
		i2 = tt2[t2[gim]];
		i3 = tt2[Drop[gim,-2]];
		i4 = tt3[gim,2];
		i5 = tt3[gim,-2];
		0.6 i3 + 0.1 (i1 + i2 + i4 + i5)]

filter2[gim_] :=
	Module[{i1,i2,i3,i4,i5},
		i1 = tt2[Drop[gim,2]];
		i2 = tt2[t2[gim]];
		i3 = tt2[Drop[gim,-2]];
		i4 = tt3[gim,2];
		i5 = tt3[gim,-2];
		0.2 (i1 + i2 + i3 + i4 + i5)]

Sample

Raw Image

g1 = QTImage[QTGet[],QTGet[]];

Show[Image2Pix[g1]]

Convert image to grayscale from {1,1,1} distance

g2=color2gray[g1,{1,1,1}];

Show[gray2raster[g2]]

Filtering the Image

Show[gray2raster[filter[g2]]]

Show[gray2raster[filter2[g2]]]

Cutoff some level

Cut GrayScale level at some level.

cutoff[rim_,lev_] :=
	Map[Map[If[#>lev,#,0]&,#]&,rim]

Show[gray2raster[cutoff[filter2[g2],0.5]]]

Show[gray2raster[cutoff[g2,0.8]]]

Compute Geometric Center

weight[rim_] :=
	Module[{t1},
		t1 = Position[rim,x_/;x!=0];
		Apply[Plus,Map[rim[[#[[1]],#[[2]]]]#&, t1]]/
	Apply[Plus,Flatten[rim]]-0.5]

Automatic Digitizing

Get small image area pointed by p, 
Filter it, and cut some level.
Return the geometric center of it.

getcg[p_,level_,col_] :=
	Module[{g,z,xy,n=8},
	g =color2gray[
		QTImage[Floor[p-n],Floor[p+n]],col];
	z = cutoff[filter2[g],level];
	Show[gray2raster[z]];
	xy = weight[z];
	{xy[[2]]+p[[1]]-n+1,p[[2]]+n-xy[[1]]-1}
]

getcg[QTGet[],0.65,{1,1,1}]

Apply getcg to Automatic Tracking.

Try[lev_,col_,n_] :=
	Module[{p,p1,p2,p3,op1,op2,op3},
	p = QTGet[];
	op1 = p1 = getcg[p,lev,col];
	QTStep[1];
	p = p1;
	op2 = p2 = getcg[p,lev,col];
	QTStep[1];
	p = 2 p2 - p1;
	op3 = p3 = getcg[p,lev,col];
	Join[{op1,op2,op3},
		Table[
			(QTStep[1];
			p = 3 p3 - 3 p2 + p1;
			p1 = p2;
			p2 = p3;
			p3 = getcg[p,lev,col]),{i,n-3}]
		]
	]

d = Try[0.65,{1,1,1},17]

Thank you.