This work describes a novel perspective on the theoretical foundation of human-computer interfaces, framing the problem as a continuous control process. In this view, the system continuously infers a distribution over potential user goals, and provides continuous feedback about its beliefs as it does so. The proper representation and manipulation of uncertainties in interaction -- via probability theory -- and the explicit inclusion of temporal characteristics -- in the form of dynamic systems -- are inherent to this framework.

The framework is used to derive a novel approach to interaction design, particularly in situations where rich or unusual sensing and display modalities are present. A number of key tools for describing and implementing systems which are consistent with this perspective are presented. The role of system dynamics as a mediating element between sensed state and decision making is described. The work sets out a paradigm for interaction which brings probabilistic models -- and thus many of the techniques of modern machine learning -- into the interface in a clean and principled manner.

The three major techniques for supporting the paradigm outlined in the thesis are: the display of changing probabilistic beliefs; dynamically adjusting system handling qualities according to an inference model; and a general probabilistic selection technique based on the detection of control.

Methods are presented for displaying the state of a system with appropriate representation of uncertainty, via Monte Carlo sampling techniques. Specifically, the use of granular synthesis for auditory display of the distributions involved in a period of interaction is described, and it is shown how predictive elements can be introduced into goal directed displays, mitigating delays present in the interaction loop. The use of these techniques in displaying particle filtering processes is illustrated.

The process by which the results of goal inference can be fed back into the dynamics that the user directly controlled is presented in general form, and applied specifically to the problem of text entry. It is shown that this inference feedback mechanism unifies a range of conventional techniques, including semantic pointing and bubble pointing. A novel text entry system for mobile devices augmented with inertial sensors is developed to illuminate the inference feedback technique.

By viewing human behaviour as a control process, a general, dynamic selection for many possible sensors is developed. This is fully probabilistic and widely applicable in many domains. It provides a sound method for designing the dynamics of an interactive selection system. This is extended to general exploration in high-dimensional spaces via controlled Markov Chain Monte Carlo processes.

Main thesis page