Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews

Mobile app reviews by users contain a wealth of information on the issues that users are experiencing. For example, a review might contain a feature request, a bug report, and/or a privacy complaint. Developers, users and app store owners (e.g. Apple, Blackberry, Google, Microsoft) can benefit from a better understanding of these issues – developers can better understand users’ concerns, app store owners can spot anomalous apps, and users can compare similar apps to decide which ones to download or purchase. However, user reviews are not labelled, e.g. we do not know which types of issues are raised in a review. Hence, one must sift through potentially thousands of reviews with slang and abbreviations to understand the various types of issues. Moreover, the unstructured and informal nature of reviews complicates the automated labelling of such reviews. In this paper, we study the multi-labelled nature of reviews from 20 mobile apps in the Google Play Store and Apple App Store. We find that up to 30 % of the reviews raise various types of issues in a single review (e.g. a review might contain a feature request and a bug report). We then propose an approach that can automatically assign multiple labels to reviews based on the raised issues with a precision of 66 % and recall of 65 %. Finally, we apply our approach to address three proof-of-concept analytics use case scenarios: (i) we compare competing apps to assist developers and users, (ii) we provide an overview of 601,221 reviews from 12,000 apps in the Google Play Store to assist app store owners and developers and (iii) we detect anomalous apps in the Google Play Store to assist app store owners and users.     

CSLabel: An Approach for Labelling Mobile App Reviews

Mobile apps (applications) have become a popular form of software, and the app reviews by users have become an important feedback resource. Users may raise some issues in their reviews when they use apps, such as a functional bug, a network lag, or a request for a feature. Understanding these issues can help developers to focus on users’ concerns, and help users to evaluate similar apps for download or purchase. However, we do not know which types of issues are raised in a review. Moreover, the amount of user reviews is huge and the nature of the reviews’ text is unstructured and informal. In this paper, we analyze 3 902 user reviews from 11 mobile apps in a Chinese app store — 360 Mobile Assistant, and uncover 17 issue types. Then, we propose an approach CSLabel that can label user reviews based on the raised issue types. CSLabel uses a cost-sensitive learning method to mitigate the effects of the imbalanced data, and optimizes the setting of the support vector machine (SVM) classifier’s kernel function. Results show that CSLabel can correctly label reviews with the precision of 66.5%, the recall of 69.8%, and the F₁ measure of 69.8%. In comparison with the state-of-the-art approach, CSLabel improves the precision by 14%, the recall by 30%, the F₁ measure by 22%. Finally, we apply our approach to two real scenarios: 1) we provide an overview of 1 076 786 user reviews from 1 100 apps in the 360 Mobile Assistant and 2) we find that some issue types have a negative correlation with users’ evaluation of apps.     

On the automatic classification of app reviews

App stores like Google Play and Apple AppStore have over 3 million apps covering nearly every kind of software and service. Billions of users regularly download, use, and review these apps. Recent studies have shown that reviews written by the users represent a rich source of information for the app vendors and the developers, as they include information about bugs, ideas for new features, or documentation of released features. The majority of the reviews, however, is rather non-informative just praising the app and repeating to the star ratings in words. This paper introduces several probabilistic techniques to classify app reviews into four types: bug reports, feature requests, user experiences, and text ratings. For this, we use review metadata such as the star rating and the tense, as well as, text classification, natural language processing, and sentiment analysis techniques. We conducted a series of experiments to compare the accuracy of the techniques and compared them with simple string matching. We found that metadata alone results in a poor classification accuracy. When combined with simple text classification and natural language preprocessing of the text—particularly with bigrams and lemmatization—the classification precision for all review types got up to 88–92 % and the recall up to 90–99 %. Multiple binary classifiers outperformed single multiclass classifiers. Our results inspired the design of a review analytics tool, which should help app vendors and developers deal with the large amount of reviews, filter critical reviews, and assign them to the appropriate stakeholders. We describe the tool main features and summarize nine interviews with practitioners on how review analytics tools including ours could be used in practice.     

Studying the consistency of star ratings and the complaints in 1 & 2-star user reviews for top free cross-platform Android and iOS apps

How users rate a mobile app via star ratings and user reviews is of utmost importance for the success of an app. Recent studies and surveys show that users rely heavily on star ratings and user reviews that are provided by other users, for deciding which app to download. However, understanding star ratings and user reviews is a complicated matter, since they are influenced by many factors such as the actual quality of the app and how the user perceives such quality relative to their expectations, which are in turn influenced by their prior experiences and expectations relative to other apps on the platform (e.g., iOS versus Android). Nevertheless, star ratings and user reviews provide developers with valuable information for improving the overall impression of their app. In an effort to expand their revenue and reach more users, app developers commonly build cross-platform apps, i.e., apps that are available on multiple platforms. As star ratings and user reviews are of such importance in the mobile app industry, it is essential for developers of cross-platform apps to maintain a consistent level of star ratings and user reviews for their apps across the various platforms on which they are available. In this paper, we investigate whether cross-platform apps achieve a consistent level of star ratings and user reviews. We manually identify 19 cross-platform apps and conduct an empirical study on their star ratings and user reviews. By manually tagging 9,902 1 & 2-star reviews of the studied cross-platform apps, we discover that the distribution of the frequency of complaint types varies across platforms. Finally, we study the negative impact ratio of complaint types and find that for some apps, users have higher expectations on one platform. All our proposed techniques and our methodologies are generic and can be used for any app. Our findings show that at least 79% of the studied cross-platform apps do not have consistent star ratings, which suggests that different quality assurance efforts need to be considered by developers for the different platforms that they wish to support.     

Fresh apps: an empirical study of frequently-updated mobile apps in the Google play store

Mobile app stores provide a unique platform for developers to rapidly deploy new updates of their apps. We studied the frequency of updates of 10,713 mobile apps (the top free 400 apps at the start of 2014 in each of the 30 categories in the Google Play store). We find that a small subset of these apps (98 apps representing ?1 % of the studied apps) are updated at a very frequent rate — more than one update per week and 14 % of the studied apps are updated on a bi-weekly basis (or more frequently). We observed that 45 % of the frequently-updated apps do not provide the users with any information about the rationale for the new updates and updates exhibit a median growth in size of 6 %. This paper provides information regarding the update strategies employed by the top mobile apps. The results of our study show that 1) developers should not shy away from updating their apps very frequently, however the frequency varies across store categories. 2) Developers do not need to be too concerned about detailing the content of new updates. It appears that users are not too concerned about such information. 3) Users highly rank frequently-updated apps instead of being annoyed about the high update frequency.     

一种基于支持向量机和主题模型的评论分析方法

在移动应用软件中,用户评论是一种重要的用户反馈途径.用户可能提到一些移动应用使用中的问题,比如系统兼容性问题、应用崩溃等.随着移动应用软件的广泛流行,用户提供大量无结构化的反馈评论.为了从用户抱怨评论中提取有效信息,提出一种基于支持向量机和主题模型的评论分析方法RASL（review analysis method based on SVM and LDA）以帮助开发人员更好、更快地了解用户反馈.首先对移动应用的中、差评提取特征,然后使用支持向量机对评论进行多标签分类.随后使用LDA主题模型（latent dirichlet allocation）对各问题类型下的评论进行主题提取与代表句提取.从两个移动应用中爬取5 141条用户原始评论,并对这些评论分别用RASL方法和ASUM方法进行处理,得到两个新的文本.与经典方法ASUM相比,RASL方法的困惑度更低、可理解性更佳,包含更完整的原始评论信息,冗余信息也更少.     

Using frame semantics for classifying and summarizing application store reviews

Text mining techniques have been recently employed to classify and summarize user reviews on mobile application stores. However, due to the inherently diverse and unstructured nature of user-generated online textual data, text-based review mining techniques often produce excessively complicated models that are prone to overfitting. In this paper, we propose a novel approach, based on frame semantics, for app review mining. Semantic frames help to generalize from raw text (individual words) to more abstract scenarios (contexts). This lower-dimensional representation of text is expected to enhance the predictive capabilities of review mining techniques and reduce the chances of overfitting. Specifically, our analysis in this paper is two-fold. First, we investigate the performance of semantic frames in classifying informative user reviews into various categories of actionable software maintenance requests. Second, we propose and evaluate the performance of multiple summarization algorithms in generating concise and representative summaries of informative reviews. Three different datasets of app store reviews, sampled from a broad range of application domains, are used to conduct our experimental analysis. The results show that semantic frames can enable an efficient and accurate review classification process. However, in review summarization tasks, our results show that text-based summarization generates more comprehensive summaries than frame-based summarization. Finally, we introduces MARC 2.0, a review classification and summarization suite that implements the algorithms investigated in our analysis.     

数据驱动的移动应用用户接受度建模与预测

应用市场（app market）已经成为互联网环境下软件应用开发和交付的一种主流模式.相对于传统模式,应用市场模式下,软件的交付周期更短,用户的反馈更快,最终用户和开发者之间的联系更加紧密和直接.为应对激烈的竞争和动态演变的用户需求,移动应用开发者必须以快速迭代的方式不断更新应用,修复错误缺陷,完善应用质量,提升用户体验.因此,如何正确和综合理解用户对软件的接受程度（简称用户接受度）,是应用市场模式下软件开发需考量的重要因素.近年来兴起的软件解析学（software analytics）关注大数据分析技术在软件行业中的具体应用,对软件生命周期中大规模、多种类的相关数据进行挖掘和分析,被认为是帮助开发者提取有效信息、作出正确决策的有效途径.从软件解析学的角度,首先论证了为移动应用构建综合的用户接受度指标模型的必要性和可行性,并从用户评价数据、操作数据、交互行为数据这3个维度给出基本的用户接受度指标.在此基础上,使用大规模真实数据集,在目标用户群体预测、用户规模预测和更新效果预测等典型的用户接受度指标预测问题中,结合具体指标,提取移动应用生命周期不同阶段的重要特征,以协同过滤、回归融合、概率模型等方法验证用户接受度的可预测性,并讨论了预测结果与特征在移动应用开发过程中可能提供的指导.     

Effects of the aesthetic design of icons on app downloads: evidence from an android market

With the rapid development of the mobile app market, understanding the determinants of mobile app success has become vital to researchers and mobile app developers. Extant research on mobile applications primarily focused on the numerical and textual attributes of apps. Minimal attention has been provided to how the visual attributes of apps affect the download behavior of users. Among the features of app “appearance”, this study focuses on the effects of app icon on demand. With aesthetic product and interface design theories, we analyze icons from three aspects, namely, color, complexity, and symmetry, through image processing. Using a dataset collected from one of the largest Chinese Android websites, we find that icon appearance influences the download behavior of users. Particularly, apps with icons featuring higher colorfulness, proper complexity, and slight asymmetry lead to more downloads. These findings can help developers design their apps.     

SNAF: Observation filtering and location inference for event monitoring on twitter

Twitter has recently emerged as a popular microblogging service that has 284 million monthly active users around the world. A part of the 500 million tweets posted on Twitter everyday are personal observations of immediate environment. If provided with time and location information, these observations can be seen as sensory readings for monitoring and localizing objects and events of interests. Location information on Twitter, however, is scarce, with less than 1% of tweets have associated GPS coordinates. Current researches on Twitter location inference mostly focus on city-level or coarser inference, and cannot provide accurate results for fine-grained locations. We propose an event monitoring system for Twitter that emphasizes local events, called SNAF (Sense and Focus). The system filters personal observations posted on Twitter and infers location of each report. Our extensive experiments with real Twitter data show that, the proposed observation filtering approach can have about 22% improvement over existing filtering techniques, and our location inference approach can increase the location accuracy by up to 36% within the 3km error range. By aggregating the observation reports with location information, our prototype event monitoring system can detect real world events, in many case earlier than news reports.     

Understanding the impact of personality traits on mobile app adoption – Insights from a large-scale field study

The sheer amount of available apps allows users to customize smartphones to match their personality and interests. As one of the first large-scale studies, the impact of personality traits on mobile app adoption was examined through an empirical study involving 2043 Android users. A mobile app was developed to assess each smartphone user's personality traits based on a state-of-the-art Big Five questionnaire and to collect information about her installed apps. The contributions of this work are two-fold. First, it confirms that personality traits have significant impact on the adoption of different types of mobile apps. Second, a machine-learning model is developed to automatically determine a user's personality based on her installed apps. The predictive model is implemented in a prototype app and shows a 65% higher precision than a random guess. Additionally, the model can be deployed in a non-intrusive, low privacy-concern, and highly scalable manner as part of any mobile app.     

Twitter: An Underutilized Potential during Sudden Crises?

The widespread use of Twitter by citizens during sudden crises has convinced communications experts that governments should also actively use Twitter during crises. However, this position seems insufficiently empirically validated. In this article, we want to provide empirical building blocks for an informed approach to the use of Twitter by the government. To this end, we analyze the tweets posted by citizens and governments about the large‐scale fire in Moerdijk (2011), the Netherlands. The results show that by far, most tweets do not contain any new and relevant information for governments and that the tweets posted by governments got buried under an avalanche of citizen tweets. We may conclude that the Moerdijk case does not give rise to advocate a (more) active role of governments on Twitter during sudden crises.     

Multi-objective mobile app recommendation: A system-level collaboration approach

The tremendous increase of mobile apps has given rise to the significant challenge of app discovery. To alleviate such a challenge, recommender systems are employed. However, the development of recommender systems for mobile apps is at a slow pace. One main reason is that a general framework for efficient development is still missing. Meanwhile, most existing systems mainly focus on single objective recommendations, which only reflect monotonous app needs of users. For such reasons, we initially present a general framework for developing mobile app recommender systems, which leverages the multi-objective approach and the system-level collaboration strategy. Our framework thus can satisfy ranges of app needs of users by integrating the strengths of various recommender systems. To implement the framework, we originally introduce the method of swarm intelligence to the recommendation of mobile apps. To be detailed, we firstly present a new set based optimization problem which is originated from the collaborative app recommendation. We then propose a novel set based Particle Swarm Optimization (PSO) algorithm, namely, the Cylinder Filling Set based PSO, to address such a problem. Furthermore, we implement the algorithm based on three popular mobile app recommender systems and conduct evaluations. Results verify that our framework and algorithm are with promising performance from both the effectiveness and efficiency.     

Time,place, technology: Twitter as an information source in the Wisconsin labor protests

Recently, Twitter has become a prominent part of social protest movement communication. This study examines Twitter as a new kind of citizen journalism platform emerging at the aggregate in the context of such “crisis” situations by undertaking a case study of the use of Twitter in the 2011 Wisconsin labor protests. A corpus of more than 775,000 tweets tagged with #wiunion during the first 3 weeks of the protests provides the source of the analyses. Findings suggest that significant differences exist between users who tweet via mobile devices, and thus may be present at protests, and those who tweet from computers. Mobile users post fewer URLs overall; however, when they do, they are more likely to link to traditional news sources and to provide additional hashtags for context. Over time, all link-posting declines, as users become better able to convey first-hand information. Notably, results for most analyses significantly change when restricted to original tweets only, rather than including retweets.     

Field experience with obfuscating million-user iOS apps in large enterprise mobile development

In recent years, mobile apps have become the infrastructure of many popular Internet services. It is now common that a mobile app serves millions of users across the globe. By examining the code of these apps, reverse engineers can learn various knowledge about the design and implementation of the apps. Real-world cases have shown that the disclosed critical information allows malicious parties to abuse or exploit the app-provided services for unrightful profits, leading to significant financial losses. One of the most viable mitigations against malicious reverse engineering is to obfuscate the apps. Despite that security by obscurity is typically considered to be an unsound protection methodology, software obfuscation can indeed increase the cost of reverse engineering, thus delivering practical merits for protecting mobile apps. In this paper, we share our experience of applying obfuscation to multiple commercial iOS apps, each of which has millions of users. We discuss the necessity of adopting obfuscation for protecting modern mobile business, the challenges of software obfuscation on the iOS platform, and our efforts in overcoming these obstacles. We especially focus on factors that are unique to mobile software development that may affect the design and deployment of obfuscation techniques. We report the outcome of our obfuscation with empirical experiments. We additionally elaborate on the follow-up case studies about how our obfuscation affected the app publication process and how we responded to the negative impacts. This experience report can benefit mobile developers, security service providers, and Apple as the administrator of the iOS ecosystem.     

移动应用安全解析学:成果与挑战

随着移动应用(App)的广泛使用,移动应用的安全事件也频频发生。从数以亿计的移动应用中准确地识别出潜在的安全隐患成为了信息安全领域重要的难题之一。移动应用数量级增长的同时,也产生了海量的应用安全数据。这些数据使得移动应用的安全解析成为了可能。本文分别从用户界面解析、重打包应用检测、应用功能与安全行为一致性检测、基于上下文的恶意行为检测、终端用户应用管理和使用行为分析这五个方面介绍了移动应用安全解析学目前的成果。同时,基于以上的研究成果,对未来的研究方向进行了展望,并讨论了这些研究方向面临的挑战。     

Tweet categorization by combining content and structural knowledge

Twitter is a worldwide social media platform where millions of people frequently express ideas and opinions about any topic. This widespread success makes the analysis of tweets an interesting and possibly lucrative task, being those tweets rarely objective and becoming the targeting for large-scale analysis. In this paper, we explore the idea of integrating two fundamental aspects of a tweet, the proper textual content and its underlying structural information, when addressing the tweet categorization task. Thus, not only we analyze textual content of tweets but also analyze the structural information provided by the relationship between tweets and users, and we propose different methods for effectively combining both kinds of feature models extracted from the different knowledge sources. In order to test our approach, we address the specific task of determining the political opinion of Twitter users within their political context, observing that our most refined knowledge integration approach performs remarkably better (about 5 points above) than the textual-based classic model.     

Decision-making in the aisles: informing, overwhelming or nudging supermarket shoppers?

Recent findings suggest that while shopping people apply ‘fast and frugal’ heuristics: short-cut strategies where they ignore most product information and instead focus on a few key cues. But rather than supporting this practice, mobile phone shopping apps and recommender systems overwhelm shoppers with information. This paper examines the amount and structure of product information that is most appropriate for supermarket shoppers, finding that in supermarkets, people rapidly make decisions based on one or two product factors for routine purchases, often trading-off between price and health. For one-off purchases, shoppers can be influenced by reading customer star ratings and reviews on a mobile phone app. In order to inform decision-making or nudge shoppers in supermarkets, we propose using embedded technologies that provide appropriate feedback and make key information salient. We conclude that rather than overwhelming shoppers, future shopping technology design needs to focus on information frugality and simplicity.     

Detecting plagiarized mobile apps using API birthmarks

This paper addresses the problem of detecting plagiarized mobile apps. Plagiarism is the practice of building mobile apps by reusing code from other apps without the consent of the corresponding app developers. Recent studies on third-party app markets have suggested that plagiarized apps are an important vehicle for malware delivery on mobile phones. Malware authors repackage official versions of apps with malicious functionality, and distribute them for free via these third-party app markets. An effective technique to detect app plagiarism can therefore help identify malicious apps. Code plagiarism has long been a problem and a number of code similarity detectors have been developed over the years to detect plagiarism. In this paper we show that obfuscation techniques can be used to easily defeat similarity detectors that rely solely on statically scanning the code of an app. We propose a dynamic technique to detect plagiarized apps that works by observing the interaction of an app with the underlying mobile platform via its API invocations. We propose API birthmarks to characterize unique app behaviors, and develop a robust plagiarism detection tool using API birthmarks.