Continuous Uncertain Interaction


High quality, original versions of the figures from the thesis are available from this page. All original images are copyright John Williamson 2006. You may use these in your own publications, talks etc. as long as appropriate citation is made. You may also modify these figures. If you do so, please make it clear in the caption (or equivalent) that the figure is a modification of the original.

Figures in the thesis which are reproduced from other sources are listed, but are not available from this page -- a reference to the original paper is instead provided.

Images are in png format for bitmap originals, and in eps format for vectors. Most vector images are also available in svg or ai (Adobe Illustrator) versions. The svg images can be opened and edited with Inkscape. A few images are originally in dia format. A few others are in sxd format, which can be opened by OpenOffice. You can convert the eps format images to other vector formats using pstoedit along with Ghostview on most platforms.

Jump to:
Chapter I: Introduction
Chapter II: Theory and Definitions
Chapter III: Dynamic Probabilistic Feedback
Chapter IV: Predictive Uncertain Display
Chapter V: Augmented Dynamics
Chapter VI: Active Selection

Back to the main thesis page.

Chapter I: Introduction

1.1 The evolution of the human-computer interface. From the rigid, time delayed interactions of the punch card interface, to the conventional GUI, and beyond to tightly-coupled extensions of the mind. SVG | EPS

1.2 A Morse key; an example of a highly refined physical interaction device. This device has evolved over a period of a century, and is extremely effective at transforming intentions into sequences of Morse code; the weighting, bearings, spring tensions and shape of the device have all been carefully adapted to provide the optimal dynamics to maximise the transfer of information. Building interfaces which have this quality of interaction, but also the flexibility of software control, is a major challenge. PNG

1.3 The effect of delays on control. (a) Very short delay: simple comparison of current state and desired state is sufficient. (b) Medium delay: a model of the system dynamics must be introduced to compensate. (c) Long delay: capacity of the delay exceeds complexity of model and control becomes bursty. All real world systems have some delay in both directions, but the lengths of the delays may be significantly different. SVG | EPS

1.4 An overview of the structure of the thesis. Chapters III and IV discuss feedback from the goals; Chapter V explores feedback from the inference process to the interface dynamics; and Chapter VI describes general techniques for building selection systems. SVG | EPS

Chapter II: Theory and Definitions

2.1 Bit-rate curves for different interfaces. Many interface design decisions involve trading off ease-of-learning against ultimate interaction rates. SVG | EPS

2.2 A basic negative feedback control loop. The error signal is fed back to maintain control. SVG | EPS

2.3 The human-computer interaction as a closed-loop control process. The “imaginary path” between the intention of the user and the actions of the system is shown as a dashed line. Real communication must take place through the interface, via the control loop. The circles indicate comparator units. AI | EPS

2.4 The interface divided into the evidence space, the goal space and the state machine. Only the input side of the interface is illustrated, the feedback paths being omitted. SVG | EPS

2.5 The changing entropy H(x) across regions of the goal space, shown here for a three goal system. Entropy is shown as the dotted surface above the simplex – distance from the surface indicates entropy at that point. It reaches its maximum of 1.584 bits at <1/3, 1/3, 1/3> and drops to zero at the vertices. PNG

2.6 The plots at the top show four example densities on a two dimensional space. The contours show the total probability at each point (i.e. sum of all three goal probabilities). The red line is the path of a test trajectory, beginning at the lower right.
(a) shows a Gaussian/squared exponential kernel $e^{\frac{-x^2}{\sigma^2}}$,
(b) a Cauchy kernel $\frac{1} {\pi \left( 1+ \frac{x}{\sigma} \right) }$,
(c) a Laplace/double exponential kernel $\frac{1}{ e^{\sqrt{\frac{x}{\sigma}}} }$,
and (d) a raised cosine kernel $1+\cos\left( \pi \frac{\sqrt{x}}{\sigma}\right)$.
The lower plots show the goal space trajectories corresponding to the example path on the 2-simplex for each of the p.d.f's. These densities transform spatial (i.e. sensed) measurements into positions in the goal space (distributions over goals).

2.7 Time-dependent utility of information and the arrival of evidence. Each curve shows the utility of information as a function of time (solid green, fixed for all three cases) and the arrival of evidence for that piece of information (dotted blue). In (a) evidence arrives at nearly the same time as the maximal utility of the information; effective control is possible in this case. (b) shows a situation where the system under control is very predictable; evidence for the state of the system is accumulated well in advance. In (c), information is not reliable until long after the event has occurred, and is no longer useful at that point. Such a system cannot reliably be controlled. SVG | EPS

2.8 Some roughly estimated delays in an interaction, for an audio display. The upper bound time is 2145ms, the lower is 57ms. For a visual display, the transmission time would drop to almost zero and the visual processing delay in the brain would increase by »20ms. SVG | EPS

2.9 The interface as a series of nested control loops. Layers move outward from physiological loops to complex learning behaviours. SVG | EPS

2.10 Some potential sources of uncertainty in the control process. AI | EPS

2.11 Terminal events in conventional interaction devices. In each case, a single button style motion is used to trigger the action. Evidence accumulates throughout the interaction, but is not used until this action occurs, and when it does occur, it is treated as if it were absolutely certain. PNG

2.12 The transfer functions used for mouse control in Windows XP. Taken from the Microsoft website ( input/pointer-bal.mspx , retrieved 10th of June 2006). The mouse control panel has four “speed” settings which select one of these transfer functions. This function is applied after any on-mouse filtering.

2.13 Oscilloscope traces for two different buttons being pushed. Note that the behaviour is quite complex, with noise and ringing around the transient. Debouncing algorithms are needed to clean up these signals for button controls. The top trace in both cases is the raw signal values; the bottom indicates the corresponding logic states. These two images are taken from Jack Ganssle’s debouncing guide ( debouncing.pdf , retrieved 15th April 2006).

Chapter III: Dynamic Probabilistic Feedback

3.1 Exposed mechanics. Good feedback should reveal the workings of a system in a way users can model. PNG

3.2 Some postulated visual uncertain displays in a map navigation example. Left to right, top to bottom: Single point, no uncertainty; point cloud (no alpha); point cloud (alpha blended, anti-aliased); density alpha overlay; outline of two standard deviation bound; world convolved with position density. PNG

3.3 The granulation process (shown here for three grains). A number slices are taken from a set of original sounds. Sample positions are chosen by some process, here by sampling from a p.d.f. These slices are then enveloped (to eliminate clicking artifacts), pitch adjusted and spatialized according to the densities for each parameter. All of the slices are summed together to produce an output sound stream. In practice many hundreds of grains are active simultaneously. AI | EPS

3.4 Some possible enveloping windows for granular synthesis. These are applied to waveform sections to eliminate clicking artifacts. The effect of windowing a sine wave is illustrated. EPS

3.5 Multiple models in a recognition task are directly sonified via granular synthesis. AI | EPS

3.5 Image from the mixture of Gaussians demo. A number of Gaussian densities are arranged in a 2d state space, each with its associated waveform. The probability of each source given the current cursor position is sonified. In this image, isoentropy contours are shown. The colours indicate the mixture of probabilities at that source. PNG

3.5 (a) Audio densities are sparse and non-overlapping, leading to a jumpy sound. (b) Densities are adjusted so as to cover the entire inter-interval region. Sound is smooth and clean AI | EPS

3.5 An image from the arc length mapping demo. The dark path indicates the mean positions, while the outer circles indicate the one standard deviation bound. The current position along the path is given by the triangle extending from the cursor; as the mouse moves closer and further the potential positions contract and expand respectively. Audio is produced corresponding to the section of the curve highlighted by the two endpoints. PNG

Chapter IV: Predictive Uncertain Display

4.1 Quickened display for helicopter hover manoeuvres. Adapted from Hess and Gorder [1990]. The acceleration, velocity and current position of the aircraft are displayed on a map to aid in low speed movements. The dashed black line is not part of the display. It indicates the path an object would take given the velocity and acceleration vectors shown here (with 1 acceleration unit = .07 velocity units) AI | EPS

4.2 Kelley’s postulated predictor model for spacecraft manoeuvres. This display is intended to be based on a fast-time model of the process; a simulation approach to predictive control, based upon Ziebolz’s (Ziebolz and Paynter [1954]) work. Reproduced from C. R. Kelley. Manual and Automatic Control: A Theory of Manual Control and Its Applications to Manual and to Automatic Systems. Academic Press, 1968. p.147

4.3 Monte Carlo propagation example. Shown are ten particles being propagated through a dynamic system (vector field shown in blue), starting at the points marked with black circles. At each step the deterministic dynamics are applied, followed by the noise diffusion process. The right hand side shows a close up of the process. Particle positions at each step are marked in red. The dynamics step is shown as a green line, the noise step as a magenta line. The initial condition is centred on a saddle point, leading to a strongly bimodal terminal distribution. SVG | EPS

4.4 Artillery fire as an an analogy to adjustable time horizon Monte Carlo prediction horizons. Higher angles (up to 45o) have greater range (in time) but increased diffusion. SVG | EPS

4.5 Simulated surface with goals distributed across it. The dynamics of the system are defined by the landscape. In this case, each goal has a smooth depression around it (leading to an attractor at that point). The solid blue spheres mark the centres of the goal distributions. Control is effected by virtually “tilting” the surface (i.e. by adjusting the effective gravitational vector) PNG

4.6 The ball-bearing simulation model. The ball-bearing can freely move over the surface. The mouse alters the orientation of the surface, and thus the effect of gravity on the motion of the bearing. Predictions are shown in red. The predictions at the time horizon are highlighted in green. There are a number of waveforms distributed on the surface; the centres of the density associated with them are shown as blue solid circles (at the centre of each depression). PNG

4.7 InterTrax head tracker used for the manipulation of the time horizon in the rolling object demo. Tilting up and down extends and retracts the prediction horizon. PNG

4.8 bearing_timeseriesPredicted and actual states in the rolling ball simulation. Actual and predicted positions for the bearing are shown. This shows prediction at approximately two seconds ahead (frame rate is slightly variable but is around 40Hz. 80 step ahead prediction is used) EPS

4.9 Predicted and actual states in the rolling ball simulation (2d projection of the trajectory). EPS

4.10 Visualization of the Monte Carlo sampled predictions in the X-Plane simulator. The “strands” extending from the helicopter body represent future possible paths for the aircraft PNG

4.11 Dependencies and densities in the helicopter goal model. Arrows indicate dependencies. Densities are represented as filled semicircles. The oval objects are goals which are sonified. This graph is simple to evaluate as dependencies flow from left to right, and the left variables are known (they are samples from the Monte Carlo process). SVG | EPS

4.12 GPS position variation while the unit is at standstill in a two minute period. Units are of degrees (the area shown corresponds to approximately 27 x 39 meters). This data is recorded in a clear area away from obstructions, at around 55 degrees north with the MESH GPS device (using a standard GPS chipset) and an external antenna. Both jitter and long-term drift are apparent. Noise would be even more pronounced in heavily built up areas. EPS

4.13 A shadow map computed for a region of the Maynooth campus in Ireland. This is estimated via a raytracing technique given the height of the surrounding area, and the angle of GPS satellites. PNG

4.14 A likelihood map computed for a region of the Maynooth campus in Ireland. This is based on building positions. Entrance into buildings is not modelled in this example. PNG

4.15 PocketPC augmented with GPS sensor pack. The sensor pack include GPS, 3-axis accelerometers, 3-axis gyroscopes, 3-axis magnetometers, capacitive touch sensing and vibrotactile display. PNG

4.16 GPS Monte Carlo navigation application. This image is from a desktop mockup with simulated GPS input. The GPS enabled version runs on the PocketPC but does not have such visualisations. The first shows predictions with a likelihood map designed for walking behaviour. The second figure shows the effect of changing to a likelihood map suitable for bike navigation, where movement is more tightly constrained to pathways. The lower two figures illustrate the effect the shadow map has on prediction. The first of these two is in a relatively non-occluded area, and predictions begin, and remain concentrated. The second has poor satellite visibility; predictions are initially diffuse, but coalesce as obstacles reduce the likely movement paths. They subsequently re-diverge in open space. (4 separate images) PNG 1 PNG 2 PNG 3 PNG 4

4.17 True GPS position and artificially disturbed GPS position during the navigation experiment. The true position is shown in blue; the readings after the introduction of noise are shown as red points. Hollow green circles indicate the targets the participant had to locate. EPS

4.17 Position and heading of a typical user during the navigation task. Positions are shown as blue points; the arrows indicate the heading sensed by the inertial sensors. EPS

4.18 Time to complete task in mean and uncertain display cases. Time is reduced for all participants except the first in the uncertain display condition. EPS

4.19 Heading signal energy in mean and uncertain display cases (energy is computed as sum_{t=0}^T \frac{dk^2}{dt} \frac{1}{T}) . Energy is reduced in the uncertain display condition. EPS

4.20 Energy in the 0.1–2Hz band of the heading signal for the period after a 63% reduction in error (i.e. the fine-tuning phase). The blue box plots show the energy for the mean condition, while the green show the uncertain condition. EPS

4.21 Boxplots of acquisition times for orientation acquisition. Again, blue represents the mean condition, green the uncertain. EPS

4.22 Variance of error for orientation acquisition. Blue shows the variance of the error (true angle-sensed angle) for the mean condition, green for the uncertain. EPS

4.23 Time series for a typical experimental run in the mean noisy case. The start and end of the acquisitions are marked, along with the point at which 63% of the error was removed (this was the criterion used to determine the end of the gross movement phase). EPS

4.24 Time series for a typical experimental run in the uncertain noisy case. The start and end of the acquisitions are marked, along with the point at which 63% of the error was removed. Compared with Figure IV.24, there is less wild overshooting in this case, although participants occasionally “hover” just outside the capture zone. EPS

4.25 Histogram of error for one experimental trial in the mean noisy case. EPS

4.26 Histogram of error for one experimental trial in the uncertain noisy case. The distribution is significantly narrower than that show in Figure IV.26 for the mean case. EPS

4.27 (a) Illustration of the basic elements of a one-dimensional particle filter. Each time step involves deterministic dynamics, diffusion and importance sampling. The similarity to the Monte Carlo prediction methods is apparent. The major difference is the introduction of an importance sampling step. SVG | EPS

4.27 (b) Illustration of the basic elements of a one-dimensional particle filter. Each time step involves deterministic dynamics, diffusion and importance sampling. The similarity to the Monte Carlo prediction methods is apparent. The major difference is the introduction of an importance sampling step. SVG | EPS

4.28 Sequence of screenshots from the particle filter gesture recogniser during the performance of an “r”-shaped gesture. The states of the particles are drawn as partially transparent circles, with lines extending showing the future state of the particle. The result is nebulous display of the distribution of particles. PNG

Chapter V: Augmented Dynamics

5.1 Feeding the results of the inference back into the control loop. DIA | EPS

5.2 The visual space (a) and motor space image (b) of a dialog box with variable controldisplay ratios. The “save” button, which has higher utility to the user, is enlarged in motor space. From R. Blanch, Y. Guiard, and M. Beaudouin-Lafon. Semantic pointing: improving target acquisition with control-display ratio adaptation. In CHI ’04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 519–526, New York, NY, USA, 2004. ACM Press.

5.3 An example of how introducing “helping” dynamics into the system can improve control performance. This figure shows a simulated acquisition task in which an operator – modelled as PID controller controlling a second order system with saturation, strong drifts and Gaussian noise – attempts to acquire a target (the circular region to the right). In the unaided case (left), many of the acquisitions miss the mark (only »25% acquire the target). When the simulated surface shown at the far right is introduced (a smooth “dimple” around a path heading towards the target), the acquisition is significantly cleaner (centre figure), with more than 80% of the acquisitions being succesful. This is equivalent to an increase of target size of » 360%. (three separate images) EPS 1 EPS 2 EPS 3

5.4 The Quikwriting text entry system. Text is entered by moving back and forth from the octagonal sections. Taken from K. Perlin. Quikwriting: continuous stylus-based text entry. In UIST ’98: Proceedings of the 11th annual ACM symposium on User interface software and technology, pages 215–216, New York, NY, USA, 1998. ACM Press.

5.5 TCube text entry. Entry is based around pie menus, which the user can pop up at any position on screen. From D. Venolia and F. Neiberg. T-cube: a fast, self-disclosing pen-based alphabet. In CHI ’94: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 265–270, New York, NY, USA, 1994. ACM Press.

5.6 Dasher continuous text entry system. Text is entered as a continuous one-dimensional gesture. The sizes of the targets are adapted according to a probability model. Taken from D. J. Ward, A. F. Blackwell, and D. J. C. MacKay. Dasher - a data entry interface using continuous gestures and language models. In UIST 2000. ACM Press, 2000.

5.7 Shark gestural text entry system. Words are entered as continuous gesture on the virtual keyboard display. The system then uses a language model to deform the path to the path for the most likely word given the gesture. Taken from P.-O. Kristensson and S. Zhai. Shark2: A large vocabulary shorthand writing system for pen-based computers. In 17th Annual ACM Symposium on User Interface Software and Technology (UIST 2004), pages 43–52. ACM Press, 2004.

5.8 SonicText. The joystick can move in eight directions; each letter requires either a single directional movement, or a directional movement followed by a “roll”. Audio feedback is presented during the interaction to reduce the need for visual attention. From M. Rinott. Audio-tactile interactions with mobile devices. Master’s thesis, Interaction Design Institute, IVREA, 2004

5.9 The Cirrin text entry interface. Text is entered as a continuous movement, entering and leaving one segment of the circle to produce a character. The layout is chosen to reduce user effort for likely words. From J. Mankoff and G. D. Abowd. Cirrin: A word-level unistroke keyboard for pen input. In UIST ’98, pages 213–214. ACM Press, 1998.

5.10 TiltText Tilting text entry with accelerometers. From K. Partridge, S. Chatterjee, V. Sazawal, G. Borriello, and R. Want. Tilttype: accelerometer-supported text entry for very small devices. In UIST ’02: Proceedings of the 15th annual ACM symposium on User interface software and technology, pages 201–204, New York, NY, USA, 2002. ACM Press

5.11 Unigesture tilt text entry. Orientation is divided into seven zones plus one for space. Users tilt the device into a zone for each character; the system disambiguates the letters at the end of a word. From V. Sazawal, R. Want, and G. Borriello. The unigesture approach. In Mobile HCI 2002, pages 256–270, London, UK, 2002. Springer-Verlag

5.12 The TUP touchwheel text entry system. The user can type by tapping on the circular pad. The system estimates the most likely letters given the position of the tap and a language model. From M. Proschowsky, N. Schultz, and N. E. Jacobsen. An intuitive text input method for touch wheels. In CHI 2006, pages 467–470, 2006.

5.13 Keypad text entry. Letters are coded as sequences of digits. In the T9 system, each letter is coded as a single digit, and the entire word decoded at the end. PNG

5.14 PocketPC augmented with XSens miniature linear accelerometer. PNG

5.15 The hexagonal grid is formed by taking the Voronoi tessellation of the hexagonallyarranged points (heavy black points). Transitions detection is simply a matter of observing changes in the nearest hexagon centre point. EPS

5.16 The default layout of letters in Hex. AI | EPS

5.17 A series of images from Hex in action. “O” is being entered. The cursor history is shown as a series of black dots. PNG

5.18 The trajectories corresponding to words using the layout given in Figure V.16. These are created by fitting a cubic spline through the medians of the transition edges which would create the word. ZIP (8 zipped EPS files)

5.19 The transfer function applied to the control input in Hex AI | EPS

5.20 A real physical model showing the design of Hex; rolling a marble (the cursor) across the undulating surface defined by the language model. PNG

5.21 A vector field, illustrating the virtual forces applied to the cursor in Hex EPS

5.22 Input and output trajectories for the word “the”. Overlaid spatial plots are shown in the top two images; the lower quadrants show the time series. ZIP (3 zipped EPS files)

5.23 Input and output trajectories for the word “squeak”. Overlaid spatial plots are shown in the top two images; the lower quadrants show the time series. ZIP (3 zipped EPS files)

5.24 Input and output trajectories for the word “take”. Overlaid spatial plots are shown in the top two images; the lower quadrants show the time series. ZIP (3 zipped EPS files)

5.25 Transitioning through a vertex is ambiguous. Very slight changes initial position or heading lead to distinct jumps in state. Therefore, the physical model prohibits transitions in this region. AI | EPS

5.26 A section of the Hex language model (probabilities are from a short test training set, not from the true corpus). The probability of jumping to a new character given a prefix is shown along the edges. Black dots indicate pruned regions of the tree. DIA | EPS
5.27 Some example virtual landscapes in Hex, showing how the probability affects the virtual surface. These are snapshots of the landscape during an interaction. ZIP (9 zipped EPS files)

5.28 The simple shake detection algorithm used in Accelerometer-Hex. The z-axis signal is shown at the right. The red line indicates the positive shake threshold, the green the negative. If a the series crosses the positive threshold going upwards, the negative going downwards and then the positive going upwards again within 150ms the movement is considered a shake. Crossing the first threshold locks the state of Hex until 150ms has passed, minimising the effect of unrecognised shakes on disturbing the state. SVG | EPS

5.29 Monte Carlo sampling display on the desktop version of Hex. Sampling is performed over the language model, given the current prefix. Spline paths for each sample word are drawn on the surface; the thickness of the line displays the probability of that path. For clarity, this example shows only two predictions, and the top-ranked predicted word is shown at the top centre. PNG

5.30 Cubic spline word trajectory generation. The knots are at the centres of the hexagons and at the medians of the edges the path goes through. SVG | EPS

5.31 An optimised layout of letters in Hex. This minimises the jerk of word trajectories for the training corpus. SVG | EPS

5.32 Hex in the form of hierarchical control loops. The world is sensed via accelerometers, which control the landscape dynamic system. This is modulated by the language model, which infers users goals. Hex has no adaptation, but if it did, the adaptation process would modulate the inferential mechanism. SVG | EPS

5.33 Cloud plots of trajectories in Hex, to produce the word “the”. (a) shows ten trajectories with the normal physical model, (b) shows the effect of removing the dynamics model (variance is increased here), (c) shows the effect of doubling the forces strength, creating a tighter but jumpier result, and (d) shows the effect of inverting the language model probabilities, significantly impairing the ability of the user to control the system. AI | EPS

5.34 Hex used with EEG brain computer interface control. In collaboration with the BCI group at Fraunhofer First IDA. PNG

Chapter VI: Active Selection

6.1 Some traditional selection mechanisms. SVG | EPS

6.2 Asymmetry in the communication channels between an interactor and a computer. Human sensory channels have much higher bandwidth than the actuation channels. SVG | EPS

6.3 The agents as independent interactors. Each user interface goal (a vertex in the goal space) is now an active agent. AI | EPS

6.4 The structure of an agent loop. SVG | EPS

6.5 One postulated user model, from R. J. Jagacinski and J. M. Flach. Control theory for humans : quantitative approaches to modeling performance. L. Erlbaum Associates, Mahwah, N.J., 2003. (p.205), in turn adapted from T. B. Sheridan and W. R. Ferrell. Man-machine Systems: Information, Control, and Decision Models of Human Performance. M.I.T. Press, Cambridge, USA, 1974. (p.254)

6.6 Another postulated user model, with a general predictive controller. Adapted from C. R. Kelley. Manual and Automatic Control: A Theory of Manual Control and Its Applications to Manual and to Automatic Systems. Academic Press, 1968. (p.140). EPS

6.7 Costello’s (Costello [1968]) surge controller. The error phase plane is shown (error against derivative of error). When the current state lies outside the diamond, bang-bang (full on or full off) control is applied, until the signal enters the central region, where proportional linear control takes over. SVG | EPS

6.8 (a) Step responses for first and second order lags with various parameters. Such responses (combined with delays) are often good fits to human tracking behaviour. EPS

6.8 (b) Step responses for first and second order lags with various parameters. Such responses (combined with delays) are often good fits to human tracking behaviour. EPS

6.9 Compensation with first-order and second-order lags, for a single dimensional mouse tracking task. The first plot shows a generated disturbance signal (red), and the observed control input (blue). The sum-of-squared error between these signals is 3.19x10^4. The middle plot shows the effect of compensation with a 100ms delay and a first-order lag (with an optimised lag parameter). The fit is much closer with an SSE of 1.58x10^4. The bottom figure shows the application of a second-order lag and delay. The fit is extremely close with an SSE of 1.00x10^4. The second-order system (with an appropriate perceptual delay) models the dynamics of the operator well. ZIP (3 zipped EPS files)

6.10 Some frequency components in human motion. Several bands are dominated by signals which do not relate to intentional control activity (with respect to the system). SVG | EPS
6.11 Controlled and uncontrolled agents and a computed Gaussian fit, for a single dimensional tracking task using mouse motion and a Brownian noise disturbance. Top-left shows the histogram (blue) and Gaussian fit (green) of an agent (the one the user is trying to select) without applying the control inputs. The effect of applying the control inputs to the agent is shown to the right, where the variance is markedly reduced. The lower pair of figures show the same process for another agent present during the selection task. Exactly the same control inputs were applied to this agent; however since the disturbances are independent there is relatively little change in the distribution of offset. ZIP (4 zipped EPS files)

6.12 Probability and entropy time series for a selection task. The top panel shows the probability of each agent, and the entropy (óni =0pi log pi) is shown below in blue. The gradually changing likelihoods of the hypotheses are visible. Selection events are indicated by the rapid increases in entropy (the vertical jumps in the lower diagram). EPS

6.13 Simple computation of the mutual information between a onedimensional display (position on screen) and control variable (mouse input) during a tracking task using simple histograms. The joint probability p(x,y) is shown as colour level. The marginals p(x) and p(y) are shown in green. The mutual information is ÓxÓy p(x,y) p(x)p(y) . The left figure uses shows a controlled agent; the right shows a second agent which was not being controlled. The mutual information between the disturbance and sensed signals in the controlled case is 2.17 bits, while it is 1.11 bits in the uncontrolled case (although the bit measure is strongly dependent on the estimation procedure). This technique becomes less effective in higher dimensions (longer time windows, or high-DOF inputs). EPS 1 EPS 2

6.14 A sequence of images from Brownian motion selection example, from the initial state, through partial selection, to the completion of the selection process. Here, there are a number of objects (labelled with names from the phonetic alphabet) with smooth random disturbances. Correlating the mouse motion with the object motion results in selection. PNG

6.15 Probability and entropy time series for the Brownian motion selection example. The probability of each agent is shown, with the overall entropy shown below in blue. The lower figure shows a close up of the time series, where the continuous change of probabilities is visible. Green ’*’ symbols mark succesful selection events. The sharp drops in entropy happen when selection has occured and the state is reset. Each selection was separated by approximately 0.5–1 second; the visible onset of the increase in probability occurs about half a second after selection behaviour begins. SXD EPS

6.16 Probability time series for the 100 agent Brownian motion selection example. The lower figure shows a close up of the time series. Green ’*’ symbols mark succesful selection events. SXD EPS

6.17 Probability time series for the 1000 agent Brownian motion selection example. The lower figure shows a close up of the time series, highlighting one single selection where many hypotheses become likely before fading away as new evidence arrives. Green ’*’ symbols mark succesful selection events. SXD EPS

6.18 Time series from the “saccadic” disturbance function, for one dimension in orientation space. The function features plateaux with smooth, rapid transitions between the levels EPS

6.19 Here, there are again a number of objects (the “eggheads”), with relatively rapid but infrequent perturbations to their state. The orientation offset of all heads can be controlled with the input device. Maintaining a head in the “looking straightforward” position causes selection. PNG

6.20 Probability time series for the “eggheads” orientation selection example. The probability of each agent is shown. The lower figure shows a close up of the time series, where the continuous change of probabilities is visible. Green ’*’ symbols mark succesful selection events. SXD | EPS

6.21 Probability time series for the “eggheads” orientation selection example. with 200 objects. The lower figure shows a close up of the time series. Green ’*’ symbols mark succesful selection events. SXD | EPS

6.21 Probability time series for the “eggheads” orientation selection example. with 200 objects. The lower figure shows a close up of the time series. Green ’*’ symbols mark succesful selection events. SXD | EPS

6.22 Probability time series for the “eggheads” orientation selection example. with 200 objects using a joystick for input . The lower figure shows a close up of the time series. Green ’*’ symbols mark succesful selection events. EPS

6.23 An image of the continuous selection interface. The red line shows the current state of the agent function. The black gridded surface extending backwards is a prediction of the future state of the selection line. Vertical mouse movement moves the entire surface up and down. Stabilising one part of the display increases the likelihood of interest in that neighbourhood; this is displayed via the circles visible at the top of the image, which represent the log likelihood at each sample point. PNG

6.24 Perlin noise examples, showing the flexibility of the function. Left to right, top to bottom: 2 octave function with low persistance; 2 octave with high persistance; 4 octave, low persistance; 4 octave high persistance; 6 octave low persistance; 6 octave high persistance; 4 octave, medium persistance with an “octave” size of 1.6 instead of 2 (i.e. with dense frequency bands); 4 octave, medium persistance, octave size of 3.9 (widely spaced frequency bands). ZIP (6 zipped EPS figures)

6.25 Probability time series for the continuous selection mechanism with a long length-scale. The rising ridge near the centre indicates the increasing probability of interest in that region. In this example, the interactor is attempting to select a region near x = 40. Probabilities are smooth in both space and time. EPS

6.26 Probability time series for the continuous selection mechanism with a short length-scale. Here, the interactor is selecting a region near x = 50. The localisation is tighter than that of Figure VI.25. EPS

6.27 A physical resonant system. Weights are suspended from a bar by cord of varying length. It is easy to excite modes of the system such that one weight vigorously oscillates while the others stay relatively steady. PNG

6.28 A resonant interface setup for selection. Input signals are passed through a bank of high-Q bandpass filters whose centre frequency drifts randomly, at a rate determined by their current energy. This type of interface exhibits time-varying dependence on feedback. SVG | EPS

6.29 Probability time series from the one-dimensional resonant interface, where the fourth agent (light blue) is being selected. In this case selection would have been declared around t = 30, however the process was allowed to continue for the purposes of illustration. EPS

6.30 Position time series from the example one-dimensional resonant interface during the selection of the fourth agent from ten (light blue). The sinusoidal resonance of the agents is apparent. EPS

6.31 Probability time series for the footpedal selection example. As in the Brownian motion time series, each sharp drop indicates a selection event. Bit-rates of approximately 0.35 bits per second. are obtained. EPS

6.31 footpedals used in this example. From CH Products. PNG

6.32 Some example two dimensional salience maps, for a three class synthetic data example. Each class is drawn from a Gaussian distribution (with different covariance). Data points are shown as ’+’, ’*’, ’.’ symbols, one for each class; contours give the salience density. Labelled data is used for illustrative purposes; it is not a requirement. ZIP (3 zipped EPS files)

6.33 Proxy agents and intention agents in exploration. The intention agents identify aspects of the data the user is interested in; this is used to weight the combination of salience maps into a single map, which the proxy agents explore and visualise/sonify. SVG | EPS

6.34 Markov Chain Monte Carlo exploration example. This is a 2- dimensional example for ease of display. (a) shows data points in a space, with a Parzen window kernel density estimate overlaid as contours. (b) shows a (Gaussian) density around the isocontour p = 0.2 of the density in (a). (c) shows the MCMC process (with Cauchy proposal density) where the estimated goal is the density shown in (b), overlaid on the original data. The process walks the boundaries of the data density. (d) shows the MCMC process with increased proposal width (this would correspond to higher goal entropy). (e) shows the MCMC process relaxing back to a exploration of the data point density ZIP (5 zipped eps files)