A Vision-Based Data-Driven Online Learning Approach for Autonomous Manipulation of Deformable Objects
Visual servoing manipulation without having prior knowledge of the object and environment
Objective: In this reserach, we are trying to address the challenges associated with deformable object manipulation originating from uncertain deformation behavior due to highly nonlinear dynamics, uniform or nonuniform physical properties - including geometry, stiffness, mass distribution, viscoelasticity, friction, ect.
Research Pipeline: (1) Developed a data-driven combined deformation-projection Jacobian formulation which encodes the object’s deformation behavior and imaging projection and uses Quasi-Newton numerical optimization algorithm (Broyden algorithm), allowing online estimation of the parameters recursively and in real time. (2) Developed a computer vision object tracking algorithm in Python using Lucas-Kanade optical flow method and OpenCV library to track feature points in the streamed video to provide visual feedback. (3) Implemented optimization-based control in Da Vinci Robotic system using constrained linear-least-square MATLAB lsqlin function to update control action at each time step, allowing manipulating a deformable object with unknown physical properties in an environment with potential disturbances.
