The Integration Manager and Natural Interaction Processor (IMNIP) merges multiple streams of input data and derives the intent represented by that data. It can be implemented in conventional desktop and notebook computers or embedded within small microprocessor-based control devices. While specifically designed for the Archimedes intelligent Total Access System (ITAS), the IMNIP can be used in any situation requiring a person to command and control a computer-based device. The Integration Manager (IM) is able to integrate inputs from multiple sources such as keyboards, speech recognizers, touch panels, Morse code, gestures and special sensors. The Natural Interaction Processor (NIP) functions operate independently of the input modalities. The strategies developed for the IMNIP provide significant advantages over earlier agent based NIP designs.
The IMNIP performs all of the message merging and decoding functions for the new Intelligent Total Access System Modules (iTASK).
One way of explaining the operation of the IMNIP is to think of it as a word filter, as depicted in the following diagram.
As depicted in this diagram, the IMNIPs distribute intelligence throughout a system.User IMNIPs use a corpus matched to the types of user input expected from its input data. User IMNIPs know about users but not about devices, and device IMNIPs know about the commands for the devices to which they are connected but nothing about users. Each IMNIP extracts predefined keywords and macros from messages that are received at its inputs. This type of word filter is necessary in practical systems since a device designer has no control over the words that may be chosen by a user to express an intended action. The designer does have full control, however, over the choice of words used to specify each function a target device is able to perform. The traditional solution to this problem is to force the user to learn scripts containing the exact words and phrases that have been predefined for each command on each device. Fundamental flaws with this traditional solution include:
The IMNIP overcomes these problems by allowing a user to describe an intended outcome in his or her own words and gestures as if asking another person to perform the action. Whenever an IMNIP recognizes an intended action, it instructs the system to perform all of the necessary steps without requiring the user to participate in their execution or to even know what the steps are.
While a single IMNIP can perform all of the necessary merging and decoding operations for a particular combination of user inputs and output devices this results in very specialized and inflexible system. It is much more versatile when the IMNIP functions are distributed between input devices that handle the human centered functions and output devices that handle all of the the machine-dependent functions. A serial, or cascaded configuration of the IMNIPs enables systems to handle ad hoc combinations of users and target devices with minimum overhead. The first IMNIP handles user dependent details independently of the device requirements and the second IMNIP handles device specific details independently of the human factors. Parallel operation of IMNIPs as depicted by the four parallel input IMNIPs in the preceding figure, adds another level of flexibility by allowing different processing and different constraints to be independently applied to each of the input and output channels.
The ability for the IMNIP in each device controller to combine multiple inputs messages allows accessors to be used in collaborative modes without the need for channeling all of the information through a single central processor. The first IMNIP translates intuitive user messages into a compact, well-defined intermediate format that can be efficiently and unambiguously decoded by a second IMNIP to generate the control codes required by a particular device.
As depicted in the figure, a user can generate any input word or phrase either directly, such as by typing or talking to a speech recognizer, or indirectly such as by touching the image of an object on a touch screen tablet that generates a text message. After passing each input though an input IMNIP, only words or synonyms that are included in the ITAS corpus will survive. Filtered messages from multiple accessors can be directed to a single iTASK Module in which the IMNIP will integrate the messages into a single stream, extract command words, parameters, and macros, and generate the stream of output control codes necessary to achieve the intended outcome.