Wheelchair guidance by facial movements

L M Bergasa, M Mazo, A Gardel and R Barea at the Electronics Department, Polytechnic School, Alcala University, Madrid, describe work on a wheelchair with a vision-based guidance system, to be controlled by the handicapped via their facial movements. This experimental, non-intrusive system, which suits all skin colours, learns the user's facial characteristics, in an unsupervised set-up. The system is adaptive and sensitive to indoor light and background changes.

Visual interfaces facilitate natural and simple interfaces for human-robot interaction. Nowadays there are many applications using these, such as teleconferencing with improved visual sensation, virtual reality systems, lip readers, assistance for mobility assistance for the disabled, etc. The use of head movements and gestures offers a natural way for severely disabled people, who cannot use a joystick, to control an electric wheelchair.

Photo: Wheelchair guidance by facial movements

System architecture ( Fig.1)
Facial images are acquired via a CCD colour micro camera, placed in front of the user. These are digitised by a frame-grabber and loaded onto a Pentium memory PC. To locate the head in the image, an original skin colour segmentation algorithm - the Unsupervised and Adaptive Gaussian Skin-Colour Model (UAGM) - is used. This segments any colour skin, under changing light conditions and random backgrounds, in an unsupervised and adaptive manner. To achieve this, a stochastic adaptive model of the skin colour in a normalised red, green (RG) colour space is adjusted for the user, using a clustering process. Then the parameters of the model are adapted by a linear combination of those known and using the maximum likelihood criterion. A 2D facial tracking is applied to the skin blob and, depending on its state vectors, a fuzzy detector of head movements activates the transitions of a high control state machine, which generates the wheelchair’s linear and angular speed (Vcmd,Wcmd). Applying the Kinematic model, linear and angular speed become angular speeds for each wheel (wr, cmd wl,cmd) and are sent to the low level control. At this level a PI controller has been designed to control the velocity of each wheel. A visual feedback loop can be clearly seen as the human user reacts to changing circumstances.

Fuzzy commands generation
Applying the criteria of simplicity and robustness, and taking into account the disabilities of an intended user, the estimated minimum number of commands necessary to guide a wheelchair are: on/off, forward/backward and speed commands (turn left/right and increase/decrease speed).
Head, eyes and mouth movements, e.g. hiding the lips and winking an eye at certain intervals (the latter not covered in this report), have been chosen to activate speed commands, direction and on/off commands. Thus, if the user turns his head to the right or left the wheelchair will turn in that direction. Head rising and lowering controls the increase and decrease speed of the wheelchair, commands shown in Fig.2(a); the 2D face tracker calculates the following parameters: centre of gravity (x,y), horizontal and vertical size of the skin blob (h,v), to obtain its position and orientation. A zero-th order Kalman filter is used to estimate two independent state vectors, one of them for the horizontal variation (xh =(x,h)) and the other one for the vertical variation (xv =(y,v)). Thus reducing computation time. Derivatives of the estimated state vectors are the fuzzy inputs for a Sugeno detector with an output. It controls the transitions of the machine which is responsible for generating commands. The last transition (trans(n-1)) is an extra input that controls the rules used by the detector each time. The knowledge-base of the fuzzy system is made up of the initial calibration process that normalises all the input variables within a range (-1,1). Each input variable has three membership functions (Neg, Zero, Pos). These functions are adjusted in an experimental way and fuzzy rules are chosen by direct observation. Fig.2(b) shows a detection sequence of several head movements, as well as the temporal evolutions of the state variable derivatives used in the fuzzy detector. The state machine generates linear and angular speed (Vcmd, Wcmd) as a function of time and the command activated. Turning commands modify angular speed in fixed quantities each 100 m/s, and depending on the direction and on/off states. Acceleration and braking commands work in a similar way but with the linear speed. Speeds are saturated to a pre-arranged limit, to improve the system’s security.

Experimental results and conclusions
The vision system process up to 25 images per second, with a resolution of 128x128 pixels. Ten commands per second are issued to the low level controller. The maximum wheelchair linear velocity was set at 1 m/s and the angular at two rad/s.

To increase safety during navigation, the prototype wheelchair is fitted with an ultrasonic ring and bumpers.

After some training on a simulator, eight users tested the system in the labs and corridors of the Electronics Department. The run performed by one user, and the evolution of the wheelchair’s linear and angular velocity, can be seen in Fig.3. The test lasted 100 seconds, taking five samples per second.
Contact: bergasa@depeca.uah.es