A qualitative human-centric evaluation of flexibility in middleware implementations

Today middleware is much more powerful, more reliable and faster than it used to be. Nevertheless, for the application developer, the complexity of using middleware platforms has increased accordingly. The volume and variety of application contexts that current middleware technologies have to support require that developers be able to anticipate the widest possible range of execution environments, desired and undesired effects of different programming strategies, handling procedures for runtime errors, and so on. This paper shows how a generic framework designed to evaluate the usability of notations (the Cognitive Dimensions of Notations Framework, or CDN) has been instantiated and used to analyze the cognitive challenges involved in adapting middleware platforms. This human-centric perspective allowed us to achieve novel results compared to existing middleware evaluation research, typically centered around system performance metrics. The focus of our study is on the process of adapting middleware implementations, rather than in the end product of this activity. Our main contributions are twofold. First, we describe a qualitative CDN-based method to analyze the cognitive effort made by programmers while adapting middleware implementations. And second, we show how two platforms designed for flexibility have been compared, suggesting that certain programming language design features might be particularly helpful for developers.