Nonconvex Vector Optimization and Optimality Conditions for Proper Efficiency

In this paper, we consider, a new nonlinear scalarization function in vector spaces which is a generalization of the oriented distance function. Using the algebraic type of closure, which is called vector closure, we introduce the algebraic boundary of a set, without assuming any topology, in our context. Furthermore, some properties of this algebraic boundary set are given and present the concept of the oriented distance function via this set in the concept of vector optimization. We further investigate Q-proper efficiency in a real vector space, where Q is some nonempty (not necessarily convex) set. The necessary and sufficient conditions for Q-proper efficient solutions of nonconvex optimization problems are obtained via the scalarization technique. The scalarization technique relies on the use of two different scalarization functions, the oriented distance function and nonconvex separation function, which allow us to characterize the Q-proper efficiency in vector optimization with and without constraints.