Learning curve models and applications: Literature review and research directions

Learning curves (LCs) are deemed effective tools for monitoring the performance of workers exposed to a new task. LCs provide a mathematical representation of the learning process that takes place as task repetition occurs. These curves were originally proposed by Wright in 1936 upon observing cost reduction due to repetitive procedures in production plants. Since then, LCs have been used to estimate the time required to complete production runs and the reduction in production costs as learning takes place, as well as to assign workers to tasks based on their performance profile. Further, effects of task interruption on workers’ performance have also being modeled by modifications on the LCs. This wide variety of applications justifies the relevance of LCs in industrial applications. This paper presents the state of the art in the literature on learning and forgetting curves, describing the existing models, their limitations, and reported applications. Directions for future research on the subject are eventually proposed.

Relevance to industry

The Learning Curve (LC) models described here can be used in a wide variety of industrial applications where workers endeavor new tasks. LC modeling enables better assignment of tasks to workers and more efficient production planning, and reduces production costs.     

非平稳数据流下的持续学习灾难性遗忘问题求解策略综述

持续学习作为一种在非平稳数据流中不断学习新任务并能保持旧任务性能的特殊机器学习范例,是视觉计算、自主机器人等领域的研究热点,但现阶段灾难性遗忘问题仍然是持续学习的一个巨大挑战。围绕持续学习灾难性遗忘问题展开综述研究,分析了灾难性遗忘问题缓解机理,并从模型参数、训练数据和网络架构三个层面探讨了灾难性遗忘问题求解策略,包括正则化策略、重放策略、动态架构策略和联合策略;根据现有文献凝练了灾难性遗忘方法的评估指标,并对比了不同灾难性遗忘问题的求解策略性能。最后对持续学习相关研究指出了未来的研究方向,以期为研究持续学习灾难性遗忘问题提供借鉴和参考。     

The effects of task complexity and experience on learning and forgetting: a field study

This paper examines the effects of task complexity and experience on parameters of individual learning and forgetting. Three attributes of task complexity and experience are addressed: the method, machine, and material employed. The task involved a high-manual-dexterity skill taken from an operating textile assembly plant; there were 2853 individual participant learning/forgetting episodes. A parametric model of individual learning and forgetting that allows the evaluation of worker response to the attributes of task complexity and experience is discussed. Results indicate that both task complexity and experience significantly affect learning and forgetting rates. Potential applications of this research include the allocation of workers to tasks based on individual learning/forgetting characteristics.     

The Impacts of Visual Display Units Used for Highly Cognitive Tasks on Learning Curve Models

This paper compares progress (learning) curves for a task that requires high manual dexterity where information is obtained from a video display unit (VDU) to a manual assembly line task. Sixteen volunteer subjects either assembled peg-boards or performed a task on a VDU. Subjects were randomly assigned to a task or control group for each task. Each subject performed their assigned task for approximately two hours. The peg-board assembly simulated a very low cognitive but highly manual task, and the VDU task simulated a high technological industrial operation. There were no statistical differences in learning between the control and test group for either task. The progress curve rates were 95 percent for the pegboard and 91 percent for the VDU. Steady states for the peg-board and VDU tasks were reached at the 25th and 30th iterations respectively. Subjects then were tested four weeks later. During this 28 day period the test group refrained from participating in any activity that would approximate their assigned task. The control group maintained proficiency by practicing 45 minutes per week. There was no difference between the control and test groups for the peg-board task. Thus, there was no significant forgetting. However, there was a significant difference between the control and test groups for the video task. For the control group (VDU) the cycle time for the first iteration was reduced by 68.0 percent. The group's progress curve rate was 97.4 percent and steady state was reached at the 23rd iteration. Though the test group did improve their cycle time for the first iteration, it was only 33 percent of the control group's improvement. The test group's progress curve rate was 94.5 percent and steady state was reached at the 33rd iteration. This significant difference between the two curves is explained by forgetting. The forgetting rate decrease in learning for the VDU was 3.2 percent. This rate gradually decreased until the 45th iteration where there was no significant differences in cycle time between the two groups. Thus, forgetting occurs in the earlier iterations. To maintain its competitive edge in the world marketplace, industry has begun to implement many processes that rely heavily on VDUs. Inherent in these processes are shorter production runs and multiple set-ups. Traditional curves developed from low cognitive tasks do not reflect the forgetting that occurs in more typical modern industrial tasks. This impacts on industrial engineers and production managers whose operations require information obtained from VDUs. To set standard times for these type tasks, more consideration should be given to forgetting rather than relying totally on the more traditional models.     

LFKT:学习与遗忘融合的深度知识追踪模型

知识追踪任务旨在根据学生历史学习行为实时追踪学生知识水平变化,并且预测学生在未来学习表现.在学生学习过程中,学习行为与遗忘行为相互交织,学生的遗忘行为对知识追踪影响很大.为了准确建模知识追踪中学习与遗忘行为,本文提出了一个兼顾学习与遗忘行为的深度知识追踪模型LFKT.LFKT模型综合考虑了四个影响知识遗忘因素,包括学生重复学习知识点的间隔时间、重复学习知识点的次数、顺序学习间隔时间以及学生对于知识点的掌握程度.结合遗忘因素,LFKT采用深度神经网络,利用学生答题结果作为知识追踪过程中知识掌握程度的间接反馈,建模融合学习与遗忘行为的知识追踪模型.通过在真实在线教育数据集上的实验,与当前知识追踪模型相比,LFKT可以更好地追踪学生知识掌握状态,并具有较好的预测性能.     

A two-stage heuristic for multi-activity and task assignment to work shifts

The multi-activity assignment problem consists of assigning interruptible activities to given work shifts so as to match as much as possible for each activity a demand curve in function of time. In this paper we consider an extension to this problem, called the multi-activity and task assignment problem, that additionally considers the assignment of uninterruptible pieces of work, called tasks. These possess properties such as worker qualifications, time windows for completion, fixed lengths and precedence relationships. We propose a mixed-integer programming formulation and a two-stage method to solve this problem. The first stage consists of an approximation mixed-integer programming model to assign tasks approximately taking into account the activities and the second involves a column generation heuristic for assigning activities and reassigning tasks at the same time. We suggest four different strategies for reassigning tasks. We conducted extensive computational tests on randomly generated instances in order to validate our method and to compare the various strategies. One strategy proved universally best when compared to the other three policies.     

A two-stage heuristic for multi-activity and task assignment to work shifts

The multi-activity assignment problem consists of assigning interruptible activities to given work shifts so as to match as much as possible for each activity a demand curve in function of time. In this paper we consider an extension to this problem, called the multi-activity and task assignment problem, that additionally considers the assignment of uninterruptible pieces of work, called tasks. These possess properties such as worker qualifications, time windows for completion, fixed lengths and precedence relationships. We propose a mixed-integer programming formulation and a two-stage method to solve this problem. The first stage consists of an approximation mixed-integer programming model to assign tasks approximately taking into account the activities and the second involves a column generation heuristic for assigning activities and reassigning tasks at the same time. We suggest four different strategies for reassigning tasks. We conducted extensive computational tests on randomly generated instances in order to validate our method and to compare the various strategies. One strategy proved universally best when compared to the other three policies.     

Visual Analytics on Network Forgetting for Task-Incremental Learning

Task-incremental learning (Task-IL) aims to enable an intelligent agent to continuously accumulate knowledge from new learning tasks without catastrophically forgetting what it has learned in the past. It has drawn increasing attention in recent years, with many algorithms being proposed to mitigate neural network forgetting. However, none of the existing strategies is able to completely eliminate the issues. Moreover, explaining and fully understanding what knowledge and how it is being forgotten during the incremental learning process still remains under-explored. In this paper, we propose KnowledgeDrift, a visual analytics framework, to interpret the network forgetting with three objectives: (1) to identify when the network fails to memorize the past knowledge, (2) to visualize what information has been forgotten, and (3) to diagnose how knowledge attained in the new model interferes with the one learned in the past. Our analytical framework first identifies the occurrence of forgetting by tracking the task performance under the incremental learning process and then provides in-depth inspections of drifted information via various levels of data granularity. KnowledgeDrift allows analysts and model developers to enhance their understanding of network forgetting and compare the performance of different incremental learning algorithms. Three case studies are conducted in the paper to further provide insights and guidance for users to effectively diagnose catastrophic forgetting over time.     

A Review of: “The Dictionary of Developmental and Educational Psychology”. Edited by ROM HARRE and ROGER LAMB (Oxford: Black well, 1986.) [Pp. xi + 271.] Price unknown,ISBN 0-631-14604-0.

Preventive measures aimed at minimizing the occurrence of work-related musculoskeletal disorders of the upper limbs (WMSDs) associated with repetitive tasks can be divided into three categories: structural, organizational and educational. Whenever specific risk and injury assessments have shown the need for preventive action, this is most often implemented within the framework of a range of assorted measures. In particular, structural measures involve optimizing the layout of the work area and furnishings, and the 'ergonomic' properties of work tools and equipment. Such measures serve to alleviate the problems caused by the use of excessive force and awkward postures. The authors refer to the principles guiding such structural measures, in the light of the extensive literature that has been published on the subject. Organizational (or reorganizational) measures essentially relate to job design (i.e. distribution of tasks, speeds and pauses). They serve to alleviate problems connected with highly repetitive and frequent actions, excessively lengthy tasks and inadequate recovery periods. Very few relevant findings are available: the authors therefore illustrate in some detail a practical trial conducted in a major engineering firm. The objective was to lower to acceptable limits the frequency of certain repetitive tasks performed using the upper limbs. The trial made it possible to identify a suitable plan and schedule of measures taking into due consideration the impact of the plan on production levels (and costs). The fundamental principles guiding the adoption of specific educational and training programmes for the workers and their supervisors are presented and discussed.     

Grid Scheduling with Makespan and Energy-Based Goals

The need for better energy efficiency in grid computing is significant given the massive amount of energy dissipated by large grids. We approximate the optimal allocation of compute nodes to a job stream, with each job consisting of multiple tasks, and while considering both the computing requirements and a desired balance of shorter makespans and lower energy consumption. The approach is widely applicable to many grid scenarios and does not require the scheduler to have administrative rights to change the workers’ DVFS or hibernation state. A discrete particle swarm optimisation (PSO) determines the worker assignments based on estimations of the tasks’ service times and energy consumption using an online learning process, and taking into account pending task executions from prior jobs. The performance of the proposed system is then evaluated through extensive Monte Carlo simulations using traces of real multi-threaded program executions on representative computer hardware. The results demonstrate the latent energy savings that are possible in grid computing through an energy-aware task scheduling.     

Variant versus invariant time to total forgetting: the learn–forget curve model revisited

Understanding and quantifying the learning–forgetting process helps predict the performance of an individual (or a group of individuals), estimate labor costs, bid on new and repeated orders, estimate costs of strikes, schedule production, develop training programs, set time standards, and improve work methods [IIE Trans. 29 (1997) 759]. Although there is agreement that the form of the learning curve is as presented by [J. Aeronaut. Sci. 3 (1936) 122], scientists and practitioners have not yet developed a full understanding of the behavior and factors affecting the forgetting process. The paucity of research on forgetting curves has been attributed to the practical difficulties involved in obtaining data concerning the level of forgetting as a function of time [IIE Transactions 21 (1989) 376]. The learn–forget curve model (LFCM) was shown to have many advantages over other theoretical models that capture the learning–forgetting relationship. However, the deficiency of the LFCM is in the assumption that the time for total forgetting is invariant of the experience gained prior to interruption. This paper attempts to correct this deficiency by incorporating the findings of [Int. J. Ind. Ergon. 10 (1992) 217] into the LFCM. Numerical examples are used to illustrate the behavior of the modified LFCM (MLFCM) and compare results to those of the LFCM.     

A new repetitive controller for mechanical manipulators

A new repetitive learning controller for motion control of mechanical manipulators undergoing periodic tasks is developed. This controller does not require exact knowledge of the manipulator dynamic structure or its parameters, and is computationally efficient. In addition, no actual joint accelerations or any matrix inversions are needed in the control law. The global asymptotic stability of the ideal and the robust stability of the nonideal control system is proven, taking into account the full nonlinear dynamics of the manipulator. Simulation results of this algorithm applied to a realistic Scara type manipulator, which includes dry friction, pay-load inertia variations, actuator/sensor noise, and unmodelled dynamics are also presented.     

F-TCKT：融合遗忘因素的深度时序卷积知识追踪模型

智慧教育中，对学生的知识水平进行追踪是很重要的技术之一。传统的深度知识追踪方法的主要关注点集中在循环神经网络(recurrent neural network, RNN)上，但RNN存在梯度消失或者梯度爆炸的问题，并且很多知识追踪方法没有考虑到学习过程中遗忘行为对结果的影响。针对以上问题，为了准确地预测学生的知识水平，提出了一种融合遗忘因素的深度时序卷积知识追踪模型(temporal convolutional knowledge tracking with forgetting, F-TCKT)。该模型引入了三个影响学生遗忘行为的因素，包括学习相同知识点的时间间隔、学习的时间间隔和同一知识点的学习次数。首先利用全连接网络计算得到表示学生遗忘程度的向量并与学生的答题记录进行拼接，然后使用梯度稳定的时间卷积网络(temporal convolutional network, TCN)和注意力机制预测学生下一次答题正误的概率。经实验验证，与传统方法相比，F-TCKT具有更好的预测性能。     

多任务学习在不良言论与个体特征检测中的应用

多任务学习在自然语言处理领域有广泛应用, 但多任务模型往往对任务间的相关性比较敏感. 如果任务相关性较低或信息传递不合理, 可能会严重影响任务性能. 本文提出了一种新的共享-私有结构的多任务学习模型BB-MTL (BERT-BiLSTM multi-task learning model), 并借助元学习的思想为其设计了一种特殊的参数优化方式MLL-TM (meta-learning-like train methods). 进一步引入一个新的信息融合门SoWLG (Softmax weighted linear gate), 用于选择性地融合每项任务的共享特征与私有特征. 实验验证所提出的多任务学习方法, 考虑到用户在网络上的行为与其个体特征密切相关, 文中结合了不良言论检测、人格检测和情绪检测任务进行了一系列实验. 实验结果表明, BB-MTL能够有效学习相关任务中的特征信息, 在3项任务上的准确率分别达到了81.56%、77.09%和70.82%.     

Exploring augmented reality for worker assistance versus training

This paper aims at advancing the fundamental understanding of the affordances of Augmented Reality (AR) as a workplace-based learning and training technology in supporting manual or semi-automated manufacturing tasks that involve both complex manipulation and reasoning. Between-subject laboratory experiments involving 20 participants are conducted on a real-life electro-mechanical assembly task to investigate the impacts of various modes of information delivery through AR compared to traditional training methods on task efficiency, number of errors, learning, independence, and cognitive load. The AR application is developed in Unity and deployed on HoloLens 2 headsets. Interviews with experts from industry and academia are also conducted to create new insights into the affordances of AR as a training versus assistive tool for manufacturing workers, as well as the need for intelligent mechanisms that enable adaptive and personalized interactions between workers and AR. The findings indicate that despite comparable performance between the AR and control groups in terms of task completion time, learning curve, and independence from instructions, AR dramatically decreases the number of errors compared to traditional instruction, which is sustained after the AR support is removed. Several insights drawn from the experiments and expert interviews are discussed to inform the design of future AR technologies for both training and assisting incumbent and future manufacturing workers on complex manipulation and reasoning tasks.     

A learning curve for tasks with cognitive and motor elements

This paper develops a new learning curve model that has cognitive and motor components. The developed model is fitted to experimental data of a repetitive manual assembly-and-disassembly task. The fits are compared to those of two other known models from the literature, which are the renowned power form learning curve and its aggregated version. The model developed in this paper performed the best. The fits of the models are evaluated using the mean squared error method. Furthermore, the developed learning curve model is investigated by incorporating it into the economic production quantity model, a topic which has been frequently studied by researchers. The results show that assuming an inappropriate learning curve may produce biased inventory policies by over- or underestimating production rates and consequently inventory levels.     

基于潜层向量对齐的持续零样本学习算法

在持续学习多任务过程中,持续零样本学习旨在积累已见类知识,并用于识别未见类样本.然而,在连续学习过程中容易产生灾难性遗忘,因此,文中提出基于潜层向量对齐的持续零样本学习算法.基于交叉分布对齐变分自编码器网络框架,将当前任务与已学任务的视觉潜层向量对齐,增大不同任务潜层空间的相似性.同时,结合选择性再训练方法,提高当前任务模型对已学任务判别能力.针对不同任务,采用已见类视觉-隐向量和未见类语义-隐向量训练独立的分类器,实现零样本图像分类.在4个标准数据集上的实验表明文中算法能有效实现持续零样本识别任务,缓解算法的灾难性遗忘.     

Neural reactive path planning with Riemannian motion policies for robotic silicone sealing

Due to its excellent chemical and mechanical properties, silicone sealing has been widely used in many industries. Currently, the majority of these sealing tasks are performed by human workers. Hence, they are susceptible to labor shortage problems. The use of vision-guided robotic systems is a feasible alternative to automate these types of repetitive and tedious manipulation tasks. In this paper, we present the development of a new method to automate silicone sealing with robotic manipulators. To this end, we propose a novel neural path planning framework that leverages fractional-order differentiation for robust seam detection with vision and a Riemannian motion policy for effectively learning the manipulation of a sealing gun. Optimal control commands can be computed analytically by designing a deep neural network that predicts the acceleration and associated Riemannian metric of the sealing gun from feedback signals. The performance of our new methodology is experimentally validated with a robotic platform conducting multiple silicone sealing tasks in unstructured situations. The reported results demonstrate that compared with directly predicting the control commands, our neural path planner achieves a more generalizable performance on unseen workpieces and is more robust to human/environment disturbances.     

Tolerating Concept and Sampling Shift in Lazy Learning Using Prediction Error Context Switching

In their unmodified form, lazy-learning algorithms may have difficulty learning and tracking time-varying input/output function maps such as those that occur in concept shift. Extensions of these algorithms, such as Time-Windowed forgetting (TWF), can permit learning of time-varying mappings by deleting older exemplars, but have decreased classification accuracy when the input-space sampling distribution of the learning set is time-varying. Additionally, TWF suffers from lower asymptotic classification accuracy than equivalent non-forgetting algorithms when the input sampling distributions are stationary. Other shift-sensitive algorithms, such as Locally-Weighted forgetting (LWF) avoid the negative effects of time-varying sampling distributions, but still have lower asymptotic classification in non-varying cases. We introduce Prediction Error Context Switching (PECS) which allows lazy-learning algorithms to have good classification accuracy in conditions having a time-varying function mapping and input sampling distributions, while still maintaining their asymptotic classification accuracy in static tasks. PECS works by selecting and re-activating previously stored instances based on their most recent consistency record. The classification accuracy and active learning set sizes for the above algorithms are compared in a set of learning tasks that illustrate the differing time-varying conditions described above. The results show that the PECS algorithm has the best overall classification accuracy over these differing time-varying conditions, while still having asymptotic classification accuracy competitive with unmodified lazy-learners intended for static environments.     

Applying incremental Deep Neural Networks-based posture recognition model for ergonomics risk assessment in construction

Monitoring and assessing awkward postures is a proactive approach for Musculoskeletal Disorders (MSDs) prevention in construction. Machine Learning models have shown promising results when used in recognition of workers’ posture from Wearable Sensors. However, there is a need to further investigate: i) how to enable Incremental Learning, where trained recognition models continuously learn new postures from incoming subjects while controlling the forgetting of learned postures; ii) the validity of ergonomics risk assessment with recognized postures. The research discussed in this paper seeks to address this need through an adaptive posture recognition model– the incremental Convolutional Long Short-Term Memory (CLN) model. The paper discusses the methodology used to develop and validate this model’s use as an effective Incremental Learning strategy. The evaluation was based on real construction workers’ natural postures during their daily tasks. The CLN model with “shallow” (up to two) convolutional layers achieved high recognition performance (Macro F1 Score) under personalized (0.87) and generalized (0.84) modeling. Generalized CLN model, with one convolutional layer, using the “Many-to-One” Incremental Learning scheme can potentially balance the performance of adaptation and controlling forgetting. Applying the ergonomics rules on recognized and ground truth postures yielded comparable risk assessment results. These findings support that the proposed incremental Deep Neural Networks model has a high potential for adaptive posture recognition. They can be deployed alongside ergonomics rules for effective MSDs risk assessment.