Editorial: Special issue on approximate dynamic programming and reinforcement learning

We are extremely pleased to present this special issue of the Journal of Control Theory and Applications.Approximate dynamic programming (ADP) is a general and effective approach for solving optimal control and estimation problems by adapting to uncertain environments over time.ADP optimizes the sensing objectives accrued over a future time interval with respect to an adaptive control law,conditioned on prior knowledge of the system,its state,and uncertainties.A numerical search over the present value of the control minimizes a Hamilton-Jacobi-Bellman (HJB) equation providing a basis for real-time,approximate optimal control.     

A model-based approximate λ-policy iteration approach to online evasive path planning and the video game Ms. Pac-Man

This paper presents a model-based approximate λ-policy iteration approach using temporal differences for optimizing paths online for a pursuit-evasion problem, where an agent must visit several target positions within a region of interest while simultaneously avoiding one or more actively pursuing adversaries. This method is relevant to applications, such as robotic path planning, mobile-sensor applications, and path exposure. The methodology described utilizes cell decomposition to construct a decision tree and implements a temporal difference-based approximate λ-policy iteration to combine online learning with prior knowledge through modeling to achieve the objectives of minimizing the risk of being caught by an adversary and maximizing a reward associated with visiting target locations. Online learning and frequent decision tree updates allow the algorithm to quickly adapt to unexpected movements by the adversaries or dynamic environments. The approach is illustrated through a modified version of the video game Ms. Pac-Man, which is shown to be a benchmark example of the pursuit-evasion problem. The results show that the approach presented in this paper outperforms several other methods as well as most human players.     

Usability evaluation methods for the web: A systematic mapping study

Context

In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers’ usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring.

Objective

The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14 years.

Method

A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner.

Results

The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners.

Conclusions

From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.     

To Score or Not to Score? Tripling Insights for Participatory Design

For evaluating visual-analytics tools, many studies confine to scoring user insights into data. For participatory design of those tools, we propose a three-level methodology to make more out of users' insights. The Relational Insight Organizer (RIO) helps to understand how insights emerge and build on one each other.     

Visual methods for analyzing time-oriented data

Providing appropriate methods to facilitate the analysis of time-oriented data is a key issue in many application domains. In this paper, we focus on the unique role of the parameter time in the context of visually driven data analysis. We will discuss three major aspects - visualization, analysis, and the user. It will be illustrated that it is necessary to consider the characteristics of time when generating visual representations. For that purpose we take a look at different types of time and present visual examples. Integrating visual and analytical methods has become an increasingly important issue. Therefore, we present our experiences in temporal data abstraction, principal component analysis, and clustering of larger volumes of time-oriented data. The third main aspect we discuss is supporting user-centered visual analysis. We describe event-based visualization as a promising means to adapt the visualization pipeline to needs and tasks of users.     

Human cytidine deaminase: a three-dimensional homology model of a tetrameric metallo-enzyme inferred from the crystal structure of a distantly related dimeric homologue

Cytidine deaminase (CDA) is a cytosolic metalloprotein whose functional unit can be either a homotetramer (T-CDA) or a homodimer (D-CDA), depending on the species. In 1994, the first crystal structure of the dimeric Escherichia coli CDA has been published. However, a crystal structure of a tetrameric CDA was not determined until 2002. Prior to the disclosure of the experimentally elucidated structure of a tetrameric CDA, we derived a homology model of the human T-CDA employing the crystal structure of the dimeric E. coli CDA as a template. The comparison of our theoretical model with the crystal structure of the human T-CDA, subsequently published in 2004, validates our prediction: not only of the structural features of the monomer and the details of the binding site, but also the multimeric arrangement of the subunits were determined with high accuracy in our model. By means of a phylogenetic analysis conducted on CDAs from various organisms, we demonstrate that the E. coli CDA is one of the furthest known homologues of the human enzyme. Nonetheless, despite the evolutionary distance and, more importantly, the different multimeric arrangement of their functional units, the E. coli CDA proved to have all the necessary information to accurately infer the structure of its human homologue.     

Identification of a Maturation Plasma Cell Index through a Highly Sensitive Droplet Digital PCR Assay Gene Expression Signature Validation in Newly Diagnosed Multiple Myeloma Patients

DNA microarrays and RNA-based sequencing approaches are considered important discovery tools in clinical medicine. However, cross-platform reproducibility studies undertaken so far have highlighted that microarrays are not able to accurately measure gene expression, particularly when they are expressed at low levels. Here, we consider the employment of a digital PCR assay (ddPCR) to validate a gene signature previously identified by gene expression profile. This signature included ten Hedgehog (HH) pathways’ genes able to stratify multiple myeloma (MM) patients according to their self-renewal status. Results show that the designed assay is able to validate gene expression data, both in a retrospective as well as in a prospective cohort. In addition, the plasma cells’ differentiation status determined by ddPCR was further confirmed by other techniques, such as flow cytometry, allowing the identification of patients with immature plasma cells’ phenotype (i.e., expressing CD19+/CD81+ markers) upregulating HH genes, as compared to others, whose plasma cells lose the expression of these markers and were more differentiated. To our knowledge, this is the first technical report of gene expression data validation by ddPCR instead of classical qPCR. This approach permitted the identification of a Maturation Index through the integration of molecular and phenotypic data, able to possibly define upfront the differentiation status of MM patients that would be clinically relevant in the future.     

Fatty Acids and Lipid Paradox-Neuroprotective Biomarkers in Ischemic Stroke

Stroke is the primary cause of death and disability worldwide, with ischemic stroke up to 80% of the total cases. Lipid profile was established as a major risk factor for stroke, but recent studies show a paradoxical relationship between serum values and the outcome of stroke patients. Our study aims to analyze the impact of the classic extended lipid profile, including fatty acids as potential neuroprotective biomarkers for the outcome of ischemic stroke patients. We included 298 patients and collected clinical, paraclinical, and outcome parameters. We used a method consisting of high-performance liquid chromatography coupled with mass spectrometry to quantify serum fatty acids. We observed a negative correlation between National Institutes of Health Stroke Scale (NIHSS) at admission and total cholesterol (p = 0.040; r = −0.120), respectively triglycerides (p = 0.041; r = −0.122). The eicosapentaenoic to arachidonic acid ratio has a negative correlation, while the docosahexaenoic to eicosapentaenoic acid ratio positively correlates with all the prognostic parameters, showing a potential neuroprotective role for eicosapentaenoic acid in preventing severe ischemic stroke. The impact of the lipid profile paradox and the dependency relationship with the fatty acids represent a significant predictive factor for the functional and disability prognostic of ischemic stroke patients.     

Discovery of Mitophagy Inhibitors with Therapeutic Potential in Different Familial Amyotrophic Lateral Sclerosis Mutations

Mitophagy is the selective degradation of mitochondria by autophagy. It promotes the turnover of mitochondria and prevents the accumulation of dysfunctional mitochondria, which can lead to cellular degeneration. Mitophagy is known to be altered in several pathological conditions, especially in neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS). We recently demonstrated an increase in autophagy flux in lymphoblasts from ALS patients bearing a mutation in SOD1. Thus, the identification of mitophagy inhibitors may be a therapeutic option to recover mitochondrial homeostasis. Here, using a phenotypic mitophagy assay, we identified a new mitophagy inhibitor, the small molecule named IGS2.7 from the MBC library. Interestingly, the treatment of different cellular and in vivo models of ALS with mutations on SOD1 and TARDBP with this inhibitor restores autophagy to control levels. These results point mitophagy inhibitors, especially IGS2.7, to a new therapeutic approach for familial ALS patients.