Improvising through the senses: a performance approach with the indirect use of technology

This article explores and proposes new ways of performing in a technology-mediated environment. We present a case study that examines feedback loop relationships between a dancer and a pianist. Rather than using data from sensor technologies to directly control and affect musical parameters, we captured data from a dancer’s arm movements and mapped them onto a bespoke device that stimulates the pianist’s tactile sense through vibrations. The pianist identifies and interprets the tactile sensory experience, with his improvised performance responding to the changes in haptic information received. Our system presents a new way of technology-mediated performer interaction through tactile feedback channels, enabling the user to establish new creative pathways. We present a classification of vibrotactile interaction as means of communication, and we conclude how users experience multi-point vibrotactile feedback as one holistic experience rather than a collection of discrete feedback points.     

KAT II: Tactile Display Mouse for Providing Tactile and Thermal Feedback

This paper proposes a tactile display mouse providing both pin-array-type tactile feedback and thermal feedback. The pin-array-type tactile display is composed of a 6 × 5 pin-array that is actuated by 30 piezo-electric bimorphs. Micro shape and vibrotactile feedback can be generated by the device, and various planar distributed patterns can be displayed as can Braille cell patterns. The thermal feedback device is composed of a thin-film resistance temperature detector, a Peltier thermoelectric heat pump and a water cooling jacket. Users can discriminate among different materials by considering the temperature variation that can be sensed as they touch an object's surface. This paper also includes an experimental evaluation of the tactile display mouse to prove the effectiveness of displaying textures. Evaluation of the ability to identify material properties was conducted using the thermal feedback part that displays a simulated temperature profile. To investigate thermo–tactile interaction, an experiment determining perceived magnitude of vibrotactile stimulus according to different temperature conditions was conducted.     

Bidirectional deformable matching with application to handwritten character extraction

To achieve integrated segmentation and recognition in complex scenes, the model-based approach has widely been accepted as a promising paradigm. However, the performance is still far from satisfactory when the target object is highly deformed and the level of outlier contamination is high. In this paper, we first describe two Bayesian frameworks, one for classifying input patterns and another for detecting target patterns in complex scenes using deformable models. Then, we show that the two frameworks are similar to the forward-reverse setting of Hausdorff matching and that their matching and discriminating properties are complementary to each other. By properly combining the two frameworks, we propose a new matching scheme called bidirectional matching. This combined approach inherits the advantages of the two Bayesian frameworks. In particular, we have obtained encouraging empirical results on shape-based pattern extraction, using a subset of the CEDAR handwriting database containing handwritten words of highly varying shape.     

Vibrotactile and vibroacoustic communications: pairs in interaction and play—an interactive structure and bodies in an urban environment

We designed a vibrotactile vest and The Humming Wall, a vibroacoustic interactive furniture set in an urban environment to interact with each other. We developed the vibrotactile patterns in the vest as a form of vibrotactile language to convey information to the wearer. In addition, we designed a set of interactive movements on The Humming Wall that would trigger patterns on the vest and elicit sensations and encourage body movements onto the wearer’s body. We invited people to interact in pairs at The Humming Wall, with one at the wall and one wearing the vest (they later swapped roles). Actions by the one at the wall, such as swiping or knocking on the wall were repeated on the vest wearer’s body. In addition, participants could ‘feel’ (vibroacoustically) and hear their own heartbeats and breath rates at the wall. We conducted a field trial with 39 participants over a 5-week period. Participants wearing the vest (and their pair) completed a set of tasks. We logged use and responses, recorded all activities on video and conducted post-experiment interviews and questionnaires. The results depicted the participants’ experience, communication and connection while wearing the vibrotactile vest and interacting with the wall. The findings show convincing, strong and positive responses to novel interactions between the responsive vibroacoustic environment and the vibrotactile vest. This work constitutes the first field trial with people ‘working’ in pairs with a vibrotactile wearable responding to and driving vibroacoustic displays with an interactive vibroacoustic environment.     

Spatial planning for musical output

This article describes an expert system called ENGRAVE that produces musical notation. This task is a resource allocation problem involving the layout of musical objects in space. to do this, we use a spatial planning technique called constraint guidance to plan the layout of the music. an example is given to illustrate the technique. an additional example is given in another domain to show that constraint guidance has general applicability for spatial planning problems.     

Assisting people with visual impairments in aiming at a target on a large wall-mounted display

Large interactive displays have become ubiquitous in our everyday lives, but these displays are designed for the needs of sighted people. In this paper, we specifically address assisting people with visual impairments to aim at a target on a large wall-mounted display. We introduce a novel haptic device, which explores the use of vibrotactile feedback in blind user search strategies on a large wall-mounted display. Using mid-air gestures aided by vibrotactile feedback, we compared three target-aiming techniques: Random (baseline) and two novel techniques – Cruciform and Radial. The results of our two experiments show that visually impaired participants can find a target significantly faster with the Cruciform and Radial techniques than with the Random technique. In addition, they can retrieve information on a large display about twice as fast by augmenting speech feedback with haptic feedback in using the Radial technique. Although a large number of studies have been done on assistive interfaces for people who have visual impairments, very few studies have been done on large vertical display applications for them. In a broader sense, this work will be a stepping-stone for further research on interactive large public display technologies for users who are visually impaired.     

Synthesizing the note-specific atoms based on their fundamental frequency,used for single-channel musical source separation

The musical source separation deals with extracting the musical signals from a mixture. To attain this goal, one of the efficient methods is to decompose the mixture into a dictionary of some basic functions that inherently describe the instruments. Usually, a unique function is synthesized for each of the notes of each instrument, called the note-specific atom. In this paper, a sine-harmonic model is utilized to synthesize note-specific atoms and the note’s fundamental frequency is used as a prior information to determine the model parameters. To calculate these parameters, the training signal spectrum is processed only around the main note harmonics. Experimental results demonstrated that the proposed method is much faster in note-specific atoms synthesis, without decreasing the source separation quality and can also eliminate the single-frequency noise from training signals.

     

Proposal of a SkilMate Hand and its component technologies for extravehicular activity gloves

In this study, we propose a SkilMate Hand for space extravehicular activity gloves which is equipped with devices of both a power assist and a tactile media. The paper focuses on development of component technologies for constructing a SkilMate Hand, which is proposed for recovering deteriorated haptic sensation in human hands. First, we manufacture a power assist device which compensates the bending moment exerted at a human finger joint utilizing a standing-wave-type ultrasonic motor. We plot the examined characteristics, propose a control policy of the actuators and show some control perforce in Bode plots. Second, we produce a tactile media device which is composed of a vibrotactile sensor element on the outer side and a vibrotactile display element on the inner side at the fingertips of the SkilMate Hand. Piezo-rubber is chosen to be used as a vibrotactile sensing transducer whose sensing performance is examined in the extremely high/low-temperature regions. We locate SkilMate in a wider framework of wearable intelligent machines which assist in affording such working surroundings that they can exhibit their skills in spite of their necessity for wearing special suits typical in hazardous environments. A proposal of a wearable intelligent machine such as that of a SkilMate with its concept has not been made before.     

Stream segregation algorithm for pattern matching in polyphonic music databases

As music can be represented symbolically, most of the existing methods extend some string matching algorithms to retrieve musical patterns in a music database. However, not all retrieved patterns are perceptually significant because some of them are, in fact, inaudible. Music is perceived in groupings of musical notes called streams. The process of grouping musical notes into streams is called stream segregation. Stream-crossing musical patterns are perceptually insignificant and should be pruned from the retrieval results. This can be done if all musical notes in a music database are segregated into streams and musical patterns are retrieved from the streams. Findings in auditory psychology are utilized in this paper, in which stream segregation is modelled as a clustering process and an adapted single-link clustering algorithm is proposed. Supported by experiments on real music data, streams are identified by the proposed algorithm with considerable accuracy.

基于查询子主题分类的多样性搜索评价方法

多样化检索结果的评测通常假设一个查询词包含多个权重各不相同的用户子意图,并在此假设的基础上对检索结果进行评测.虽然大多数已经存在的多样化检索评测方法利用了这些特性对检索结果进行评测,但在评测过程中,它们都忽略了查询子意图的类型信息;而不同类型的查询子意图对信息需求具有不同的特点.首先,通过引入衰减函数对这种特点进行描述,进而对用户子意图的分类方法进行抽象;在此基础上,提出了利用查询子意图类型信息进行多样化检索结果评测的框架,该框架定义了利用查询子意图类型信息进行多样化检索评测的方法应该具有的结构;然后,讨论了在用信息类和导航类作为子意图分类方法的前提下,其对应的衰减函数的形式;最后,在TREC与NTCIR测试集上的实验结果表明了所提出方法的有效性.     

Effects of amplitude modulation on vibrotactile flow displays on piezo-actuated thin touch screen

This paper investigates effects of different amplitude modulation methods of frequency-swept excitation patterns for creating vibrotactile flow displays on a spatiotemporally actuated thin touch screen. Four piezoelectric actuators (bimorph type) located at the midpoints of each side of a screen can be used to make eight directional (horizontal, vertical, diagonal) vibrotactile flows by spatiotemporally actuating two or more piezo-actuators. To determine which amplitude modulation method is the most effective in generating directional vibrotactile flows, an excitation pattern having sweeping frequencies in the range of 50 Hz to 250 Hz with four different time-varying amplitudes are investigated: 1) constant amplitude modulation, 2) linear amplitude modulation (linearly increasing and decreasing amplitude), 3) exponential amplitude modulation (exponentially increasing and decreasing amplitude), and 4) logarithmic amplitude modulation (logarithmically increasing and decreasing amplitude). A user study shows that the exponentially increasing and decreasing the amplitude pattern is the worst in terms of human perception (accuracy, validity, and refinement), while other amplitude modulation methods give similar results. Another user study shows that the eight directional vibrotactile flows can easily be distinguished on the users?? palms. As such, it is expected that these vibrotactile flows can be applied to the user interface design of a mobile device, for example, for identifying callers or giving directions to the elderly and the blind.     

On the Shared Control of an EMG-Controlled Prosthetic Hand: Analysis of User–Prosthesis Interaction

An anthropomorphic underactuated prosthetic hand, endowed with position and force sensors and controlled by means of myoelectric commands, is used to perform experiments of hierarchical shared control. Three different hierarchical control strategies combined with a vibrotactile feedback system have been developed and tested by able-bodied subjects through grasping tasks used in activities of daily living (ADLs). The first goal is to find a good tradeoff between good grasping capabilities and low attention required by the user to complete grasping tasks, without addressing advanced algorithm for electromyographic processing. The second goal is to understand whether a vibrotactile feedback system is subjectively or objectively useful and how it changes users' performance. Experiments showed that users were able to successfully operate the device in the three control strategies, and that the grasp success increased with more interactive control. Practice has proven that when too much effort is required, subjects do not do their best, preferring, instead, a less-interactive control strategy. Moreover, the experiments showed that when grasping tasks are performed under visual control, the enhanced proprioception offered by a vibrotactile system is practically not exploited. Nevertheless, in subjective opinion, feedback seems to be quite important.     

Automatic music genre classification based on musical instrument track separation

The aim of this article is to investigate whether separating music tracks at the pre-processing phase and extending feature vector by parameters related to the specific musical instruments that are characteristic for the given musical genre allow for efficient automatic musical genre classification in case of database containing thousands of music excerpts and a dozen of genres. Results of extensive experiments show that the approach proposed for music genre classification is promising. Overall, conglomerating parameters derived from both an original audio and a mixture of separated tracks improve classification effectiveness measures, demonstrating that the proposed feature vector and the Support Vector Machine (SVM) with Co-training mechanism are applicable to a large dataset.     

Tactile display and vibrotactile pattern recognition on the torso

A wirelessly controlled tactile display has been designed, fabricated and tested for use as a navigation aid. The display comprises a 4 × 4 array of vibrating motors that is mounted on a waist band and stimulates the skin across the lower back. Three types of electromechanical actuators were evaluated for use in the display; based on their mechanical performance and power requirements, two of these motors were then used to fabricate tactile displays. The performance of the displays and the wireless tactile control units was assessed experimentally by having subjects identify which of eight possible vibrotactile patterns was presented to the lower back. The results indicated that subjects could recognize the vibrotactile patterns with almost perfect accuracy and that there was no difference between the two types of motor used for the displays. Moreover, the ability to recognize the pattern of vibrotactile stimulation was superior on the back as compared to the forearm. A further experiment confirmed that the tactile display can be used as a navigation aid outdoors and that the vibrotactile patterns presented can be interpreted as directional or instructional cues with almost perfect accuracy.     

基于视觉缩略图的高维音乐信息可视化的关键技术研究

对基于视觉缩略图的高维音乐信息可视化技术作了深入探讨。对代表性的音乐可视化、特别是视觉缩略图的各种技术进行了广泛调研,通过一系列的用户调查分析了音乐内容的视觉缩略图应具备的基本特征。在此基础之上,提出了一种新颖的视觉缩略图ThumbnailDJ,并对其进行了一系列的用户测试。在对实验结果进行分析后,讨论了音乐信息可视化的应用前景、发展方向及相关研究重点。高维音乐内容的视觉描述有助于提高音乐库的浏览和检索效率,本研究将有助于缩小音乐视觉描述与用户音乐感知之间的语义鸿沟,提高音乐库的浏览和检索效率。同时,研究成果也将对高维数据的信息可视化研究起到重要的借鉴作用。     

HMM-based writer identification in music score documents without staff-line removal

Writer identification from musical score documents is a challenging task due to its inherent problem of overlapping of musical symbols with staff-lines. Most of the existing works in the literature of writer identification in musical score documents were performed after a pre-processing stage of staff-lines removal. In this paper we propose a novel writer identification framework in musical score documents without removing staff-lines from the documents. In our approach, Hidden Markov Model (HMM) has been used to model the writing style of the writers without removing staff-lines. The sliding window features are extracted from musical score-lines and they are used to build writer specific HMM models. Given a query musical sheet, writer specific confidence for each musical line is returned by each writer specific model using a log-likelihood score. Next, a log-likelihood score in page level is computed by weighted combination of these scores from the corresponding line images of the page. A novel Factor Analysis-based feature selection technique is applied in sliding window features to reduce the noise appearing from staff-lines which proves efficiency in writer identification performance. In our framework we have also proposed a novel score-line detection approach in musical sheet using HMM. The experiment has been performed in CVC-MUSCIMA data set and the results obtained show that the proposed approach is efficient for score-line detection and writer identification without removing staff-lines. To get the idea of computation time of our method, detail analysis of execution time is also provided.     

AR-based serious game framework for post-stroke rehabilitation

Stroke is considered one of the main causes of death around the world. Survivors often suffer different kinds of disabilities in terms of their cognitive and motor capabilities, and are therefore unable to perform their day-to-day activities. To regain some of their cognitive as well as motor abilities, they require rehabilitation. To this end, we present a serious game framework based on augmented reality technology that may motivate the patients’ involvement in the rehabilitation exercise. Additionally, we analyze the requirements for such a framework and describe the concept and implementation of the proposed approach. Furthermore, we designed a wireless vibrotactile output device that is attached to a tangible object. The tangible object that is connected to the framework can give haptic as well as audio-visual feedback to the patient in a more motivating and entertaining environment for rehabilitation exercises. The suitability and utility of the proposed framework was evaluated with real stroke patients and compared against the performance of a healthy control group, thus facilitating occupational therapists in assessing a patient’s progress. Our evaluations show that the serious games with vibrotactile feedback are well accepted by patients.     

基于均匀混合模型的运动检测

文章提出一种新的实时运动检测方法,即均匀混合模型运动检测法。该方法对每个象素用均匀混合分布建模,然后利用自适应的学习率在线更新模型,使背景模型更加准确。均匀混合模型运动检测方法显著的特点是,首先模型简单实用,适于运动检测的实时处理;其次利用帧间信息对学习率进行自适应调整,使算法在复杂情况下也能进行正确的运动检测。实验表明该方法较目前常用的高斯混合模型运动检测法有更好的实时性和可靠性。     

Monaural Musical Sound Separation Based on Pitch and Common Amplitude Modulation

Monaural musical sound separation has been extensively studied recently. An important problem in separation of pitched musical sounds is the estimation of time-frequency regions where harmonics overlap. In this paper, we propose a sinusoidal modeling-based separation system that can effectively resolve overlapping harmonics. Our strategy is based on the observations that harmonics of the same source have correlated amplitude envelopes and that the change in phase of a harmonic is related to the instrument's pitch. We use these two observations in a least squares estimation framework for separation of overlapping harmonics. The system directly distributes mixture energy for harmonics that are unobstructed by other sources. Quantitative evaluation of the proposed system is shown when ground truth pitch information is available, when rough pitch estimates are provided in the form of a MIDI score, and finally, when a multi pitch tracking algorithm is used. We also introduce a technique to improve the accuracy of rough pitch estimates. Results show that the proposed system significantly outperforms related monaural musical sound separation systems.     

Frameworks for multivariate m-mediods based modeling and classification in Euclidean and general feature spaces

This paper presents an extension of m-mediods based modeling technique to cater for multimodal distributions of sample within a pattern. The classification of new samples and anomaly detection is performed using a novel classification algorithm which can handle patterns with underlying multivariate probability distributions. We have proposed two frameworks, namely MMC-ES and MMC-GFS, to enable our proposed multivarite m-mediods based modeling and classification approach workable for any feature space with a computable distance metric. MMC-ES framework is specialized for finite dimensional features in Euclidean space whereas MMC-GFS works on any feature space with a computable distance metric. Experimental results using simulated and complex real life dataset show that multivariate m-mediods based frameworks are effective and give superior performance than competitive modeling and classification techniques especially when the patterns exhibit multivariate probability density functions.