Smart space unites the physical world and the digital world, and changes the lifestyle of its users. Enhanced sensors and digital devices interact with the user and provide enhanced services. However, new and different types of devices and communication are essential for users to configure the smart space for a natural user interface or natural user experience (NUI/NUX).
Communicating with context-aware devices by designing an NUI that interacts with the user encounters some limitations. First, digital devices use different languages, such as Windows, iOS, HTML, and Android, which restrict what developers can do. Second, the user interface differs from machine to machine, so developers must maintain these interfaces separately. Third, NUI developers have to write specifications about each device interface for the equipment’s users. To ease these burdens, this paper proposes a standardized context schema, called Human Interaction Markup Language (HIML), and a middleware architecture supporting schema to manage the structural conditions efficiently and provide a successful NUI model. The practicality of the proposed middleware will be demonstrated by implementing and experimenting with an NUI module that accepts electroencephalogram (EEG) input. The HIML middleware structure proposed in this paper consists of Authentication Manager, Appliance Manager Server, and NUI Analyzer. Authentication Manager identifies users who have access permissions for the device, Appliance Manager Server provides a service that required by each device, and NUI Analyzer creates commands that correspond to each NUI/NUX environment. In this paper, we test HIML middleware supporting NUI/NUX using Emotiv’s EEG measuring device to change the channel on a television. Users’ EEG signals map the left and right sensorimotor through the FC5 and FC6 channels to issue TV channel change commands. NUI Analyzer separates the EEG signal input into 8–13 Hz bands via fast Fourier transform. If event-related desynchronization appears, the command for up and down channel is categorized and an HIML document is created using the Linear discriminant analysis (LDA) algorithm. The HIML Converter converts an inputted HIML document into commands based on the tree structure in HIMLDocParser. Command Tree maps key values such as device model, serial number, transmission command, and interface tag of the corresponding device. Many developers who have studied the implementation of smart space see context-aware computing as playing a pivotal role in providing intelligent services. In this paper, a middleware structure that focuses on user interaction and supports NUI is proposed. HIML is designed to create make messages among context agents. The structure design and HIML proposed in this paper are applicable to almost every service in the smart space environment.
For further development, additional adoption of NUI/NUX models is required to provide more natural interaction for users in the smart space. Moreover, it would be possible to create even more intelligent services in the future if complicated situations can be handled using data-mining techniques.

*ITRC 기여율 = 1