A study of two penalty-parameterless constraint handling techniques in the framework of MOEA/D

Penalty functions are frequently employed for handling constraints in constrained optimization problems (COPs). In penalty function methods, penalty coefficients balance objective and penalty functions. However, finding appropriate penalty coefficients to strike the right balance is often very hard. They are problems dependent. Stochastic ranking (SR) and constraint-domination principle (CDP) are two promising penalty functions based constraint handling techniques that avoid penalty coefficients. In this paper, the extended/modified versions of SR and CDP are implemented for the first time in the multiobjective evolutionary algorithm based on decomposition (MOEA/D) framework. This led to two new algorithms, CMOEA/D-DE-SR and CMOEA/D-DE-CDP. The performance of these new algorithms is tested on CTP-series and CF-series test instances in terms of the HV-metric, IGD-metric, and SC-metric. The experimental results are compared with NSGA-II, IDEA, and the three best performers of CEC 2009 MOEA competition, which showed better and competitive performance of the proposed algorithms on most test instances of the two test suits. The sensitivity of the performance of proposed algorithms to parameters is also investigated. The experimental results reveal that CDP works better than SR in the MOEA/D framework.     

Penalty resolution of the babuska circle paradox

If the solution for a simply-supported circular plate is approximated by a sequence of finite element solutions on successively refined polygonal domains the finite element approximations converge to the solution of a different problem. In the present study we present a finite element formulation with boundary penalty that produces valid approximations to the simply-supported plate. The penalty term is shown to require the use of reduced integration. The dependence of the penalty parameter on mesh size h is also examined. Numerical experiments confirm the validity of the method and determine rates of convergence. A second approach involving a modified corner condition is also considered and error estimates determined. This scheme is implemented also using a discrete penalty technique. The results and rates are compared with the boundary penalty method and their relative merits discussed.     

Online deadline scheduling with preemption penalties

This paper presents a study of the problem of online deadline scheduling under the preemption penalty model of Zheng, Xu, and Zhang (2007). In that model, each preemption incurs a penalty of ρ times the weight of the preempted job, where ρ ? 0 is the preemption penalty parameter. The objective is to maximise the total weight of jobs completed on time minus the total penalty.     

Evolution strategies based adaptive Lp LS-SVM

Not only different databases but two classes of data within a database can also have different data structures. SVM and LS-SVM typically minimize the empirical ?-risk; regularized versions subject to fixed penalty (L₂ or L₁ penalty) are non-adaptive since their penalty forms are pre-determined. They often perform well only for certain types of situations. For example, LS-SVM with L₂ penalty is not preferred if the underlying model is sparse. This paper proposes an adaptive penalty learning procedure called evolution strategies (ES) based adaptive L_p least squares support vector machine (ES-based L_p LS-SVM) to address the above issue. By introducing multiple kernels, a L_p penalty based nonlinear objective function is derived. The iterative re-weighted minimal solver (IRMS) algorithm is used to solve the nonlinear function. Then evolution strategies (ES) is used to solve the multi-parameters optimization problem. Penalty parameterp, kernel and regularized parameters are adaptively selected by the proposed ES-based algorithm in the process of training the data, which makes it easier to achieve the optimal solution. Numerical experiments are conducted on two artificial data sets and six real world data sets. The experiment results show that the proposed procedure offer better generalization performance than the standard SVM, the LS-SVM and other improved algorithms.     

Boundary penalty techniques

We examine techniques for enforcing Dirichlet boundary data in finite element solution of boundary-value problems. The common programming strategy of adding a large number to certain diagonal entries and scaling the right-side is compared with penalty methods. By selectively underintegrating the continuous penalty formulation and selecting the penalty parameter ε^?1 optimally as a function of mesh size h we deduce the ‘large number’ strategy and hence show that the optimal rate is preserved. Numerical experiments to study the rates for continuous penalty, underintegrated penalty and ‘large number’ strategies corroborate the argument.     

Face recognition by sparse discriminant analysis via joint L2,1-norm minimization

Recently, joint feature selection and subspace learning, which can perform feature selection and subspace learning simultaneously, is proposed and has encouraging ability on face recognition. In the literature, a framework of utilizing L_2,1-norm penalty term has also been presented, but some important algorithms cannot be covered, such as Fisher Linear Discriminant Analysis and Sparse Discriminant Analysis. Therefore, in this paper, we add L_2,1-norm penalty term on FLDA and propose a feasible solution by transforming its nonlinear model into linear regression type. In addition, we modify the optimization model of SDA by replacing elastic net with L_2,1-norm penalty term and present its optimization method. Experiments on three standard face databases illustrate FLDA and SDA via L_2,1-norm penalty term can significantly improve their recognition performance, and obtain inspiring results with low computation cost and for low-dimension feature.     

A new constraint handling method based on the modified Alopex-based evolutionary algorithm

In this paper, a new constraint handling method based on a modified AEA (Alopex-based evolutionary algorithm) is proposed. Combined with a new proposed ranking and selecting strategy, the algorithm gradually converges to a feasible region from a relatively feasible region. By introduction of an adaptive relaxation parameter μ, the algorithm fully takes into account different functions corresponding to different sizes of feasible region. In addition, an adaptive penalty function method is employed, which adaptively adjust the penalty coefficient so as to guarantee a moderate penalty. By solving 11 benchmark test functions and two engineering problems, experiment results indicate that the proposed method is reliable and efficient for solving constrained optimization problems. Also, it has great potential in handling many engineering problems with constraints, even with equations.     

A novel L1/2 regularization shooting method for Cox’s proportional hazards model

Nowadays, a series of methods are based on a L ₁ penalty to solve the variable selection problem for a Cox’s proportional hazards model. In 2010, Xu et al. have proposed a L _1/2 regularization and proved that the L _1/2 penalty is sparser than the L ₁ penalty in linear regression models. In this paper, we propose a novel shooting method for the L _1/2 regularization and apply it on the Cox model for variable selection. The experimental results based on comprehensive simulation studies, real Primary Biliary Cirrhosis and diffuse large B cell lymphoma datasets show that the L _1/2 regularization shooting method performs competitively.     

Sparse regularization for semi-supervised classification

Manifold regularization (MR) is a promising regularization framework for semi-supervised learning, which introduces an additional penalty term to regularize the smoothness of functions on data manifolds and has been shown very effective in exploiting the underlying geometric structure of data for classification. It has been shown that the performance of the MR algorithms depends highly on the design of the additional penalty term on manifolds. In this paper, we propose a new approach to define the penalty term on manifolds by the sparse representations instead of the adjacency graphs of data. The process to build this novel penalty term has two steps. First, the best sparse linear reconstruction coefficients for each data point are computed by the l¹-norm minimization. Secondly, the learner is subject to a cost function which aims to preserve the sparse coefficients. The cost function is utilized as the new penalty term for regularization algorithms. Compared with previous semi-supervised learning algorithms, the new penalty term needs less input parameters and has strong discriminative power for classification. The least square classifier using our novel penalty term is proposed in this paper, which is called the Sparse Regularized Least Square Classification (S-RLSC) algorithm. Experiments on real-world data sets show that our algorithm is very effective.     

Isoparametric C 0 interior penalty methods for plate bending problems on smooth domains

In this paper we develop isoparametric C ⁰ interior penalty methods for plate bending problems on smooth domains. The orders of convergence of these methods are shown to be optimal in the energy norm. We also consider the convergence of these methods in lower order Sobolev norms and discuss subparametric C ⁰ interior penalty methods. Numerical results that illustrate the performance of these methods are presented.     

Convergence Analysis of a Symmetric Dual-Wind Discontinuous Galerkin Method

A new symmetric discontinuous Galerkin method for second order elliptic problems is analyzed. We show that the numerical method is stable for any positive penalty parameter and converges with optimal order provided the exact solution is sufficiently regular. These results are also shown to hold for some non-positive penalty parameters. Numerical experiments are presented that support the theoretical results.     

Analysis of a Penalty Method

In this article we study the simplest one-dimensional transport equation u _t +au _x =f and study the implementation of the boundary condition using a penalty method combined with a P1 finite element discretization. We discuss the convergence of the method when both the penalty parameter ? and the mesh size h go to zero, in sequence or simultaneously. Some numerical simulations are reported also showing the efficiency of the method. Numerical simulations are also made for the similar problem in space dimension?2.     

TROP-ELM: A double-regularized ELM using LARS and Tikhonov regularization

In this paper an improvement of the optimally pruned extreme learning machine (OP-ELM) in the form of a L₂ regularization penalty applied within the OP-ELM is proposed. The OP-ELM originally proposes a wrapper methodology around the extreme learning machine (ELM) meant to reduce the sensitivity of the ELM to irrelevant variables and obtain more parsimonious models thanks to neuron pruning. The proposed modification of the OP-ELM uses a cascade of two regularization penalties: first a L₁ penalty to rank the neurons of the hidden layer, followed by a L₂ penalty on the regression weights (regression between hidden layer and output layer) for numerical stability and efficient pruning of the neurons. The new methodology is tested against state of the art methods such as support vector machines or Gaussian processes and the original ELM and OP-ELM, on 11 different data sets; it systematically outperforms the OP-ELM (average of 27% better mean square error) and provides more reliable results - in terms of standard deviation of the results - while remaining always less than one order of magnitude slower than the OP-ELM.     

Variable selection via combined penalization for high-dimensional data analysis

We propose a new penalized least squares approach to handling high-dimensional statistical analysis problems. Our proposed procedure can outperform the SCAD penalty technique (Fan and Li, 2001) when the number of predictors p is much larger than the number of observations n, and/or when the correlation among predictors is high. The proposed procedure has some of the properties of the smoothly clipped absolute deviation (SCAD) penalty method, including sparsity and continuity, and is asymptotically equivalent to an oracle estimator. We show how the approach can be used to analyze high-dimensional data, e.g., microarray data, to construct a classification rule and at the same time automatically select significant genes. A simulation study and real data examples demonstrate the practical aspects of the new method.     

Special boundary approximation methods for laplace equation problems with boundary singularities— Applications to the motz problem

We investigate the convergence of special boundary approximation methods (BAMs) used for the solution of Laplace problems with a boundary singularity. In these methods, the solution is approximated in terms of the leading terms of the asymptotic solution around the singularity. Since the approximation of the solution satisfies identically the governing equation and the boundary conditions along the segments causing the singularity, only the boundary conditions along the rest of the boundary need to be enforced. Four methods of imposing the essential boundary conditions are considered: the penalty, hybrid, and penalty/hybrid BAMs and the BAM with Lagrange multipliers. A priori error analyses and numerical experiments are carried out for the case of the Motz problem, and comparisons between all methods are made.     

Nonconforming Maxwell Eigensolvers

Three Maxwell eigensolvers are discussed in this paper. Two of them use classical nonconforming finite element approximations, and the other is an interior penalty type discontinuous Galerkin method. A main feature of these solvers is that they are based on the formulation of the Maxwell eigenproblem on the space H ₀(curl;Ω)∩H(div⁰;Ω). These solvers are free of spurious eigenmodes and they do not require choosing penalty parameters. Furthermore, they satisfy optimal order error estimates on properly graded meshes, and their analysis is greatly simplified by the underlying compact embedding of H ₀(curl;Ω)∩H(div⁰;Ω) in L ₂(Ω). The performance and the relative merits of these eigensolvers are demonstrated through numerical experiments.     

A Posteriori Error Control for a Weakly Over-Penalized Symmetric Interior Penalty Method

A reliable and efficient residual based a posteriori error estimator is constructed for a weakly over-penalized symmetric interior penalty method for second order elliptic problems. Numerical results that demonstrate the performance of the error estimator are presented.     

Multiobjective Markov chains optimization problem with strong Pareto frontier: Principles of decision making

In this paper, we present a novel approach for computing the Pareto frontier in Multi-Objective Markov Chains Problems (MOMCPs) that integrates a regularized penalty method for poly-linear functions. In addition, we present a method that make the Pareto frontier more useful as decision support system: it selects the ideal multi-objective option given certain bounds. We restrict our problem to a class of finite, ergodic and controllable Markov chains. The regularized penalty approach is based on the Tikhonov’s regularization method and it employs a projection-gradient approach to find the strong Pareto policies along the Pareto frontier. Different from previous regularized methods, where the regularizator parameter needs to be large enough and modify (some times significantly) the initial functional, our approach balanced the value of the functional using a penalization term (μ) and the regularizator parameter (δ) at the same time improving the computation of the strong Pareto policies. The idea is to optimize the parameters μ and δ such that the functional conserves the original shape. We set the initial value and then decrease it until each policy approximate to the strong Pareto policy. In this sense, we define exactly how the parameters μ and δ tend to zero and we prove the convergence of the gradient regularized penalty algorithm. On the other hand, our policy-gradient multi-objective algorithms exploit a gradient-based approach so that the corresponding image in the objective space gets a Pareto frontier of just strong Pareto policies. We experimentally validate the method presenting a numerical example of a real alternative solution of the vehicle routing planning problem to increase security in transportation of cash and valuables. The decision-making process explored in this work correspond to the most frequent computational intelligent models applied in practice within the Artificial Intelligence research area.     

The element-free Galerkin method for the nonlinear p-Laplacian equation

The element-free Galerkin (EFG) method is developed in this paper for solving the nonlinear p-Laplacian equation. The moving least squares approximation is used to generate meshless shape functions, the penalty approach is adopted to enforce the Dirichlet boundary condition, the Galerkin weak form is employed to obtain the system of discrete equations, and two iterative procedures are developed to deal with the strong nonlinearity. Then, the computational formulas of the EFG method for the p-Laplacian equation are established. Numerical results are finally given to verify the convergence and high computational precision of the method.     

A new kind of combinations between the Ritz-Galerkin and finite element methods for singularity problems

The coupling techniques of simplified hybrid plus penalty functions are first presented for matching the Ritz-Galerkin method and thek(k>-1)-order Lagrange finite element methods to solve complicated problems of elliptic equations, homogeneous or nonhomogeneous, in particular with singularities or unbounded domains. Optimal convergence rates of numerical solutions have been proved in the Sobolev norms. Moreover, the theoretical results obtained in this paper have been verified by numerical experiments for the singular Motz problem.