Friction is a nonlinear phenomenon which has destructive effects on performance of control systems. To obviate these effects, friction compensation is an effectual solution. In this paper, an adaptive technique is proposed in order to eliminate limit cycles as one of the undesired behaviors due to presence of friction in control systems which happen frequently. The proposed approach works for nonlinear dynamic and static friction models and is applicable to a wide range of different mechanical systems. It is also applied to a simple inverted pendulum on a cart as a highly nonlinear under-actuated system. A nonlinear optimal controller based on the approximate solution of Hamilton-Jacobi-Bellman partial differential equation is designed to fulfill our control objectives and achieve preferable performance compared to those of the linear optimal controllers. It causes to have more accuracy in system's response and positioning in the presence of friction. Simulation result approve the effectiveness of both the presented technique and controller.