Image moments have been widely used for designing robust shape descriptors that are invariant to rigid transformations. In this work, we address the problem of estimating non-rigid deformation fields based on image moment variations. By using a single family of polynomials to both parameterize the deformation field and to define image moments, we can represent image moments variation as a system of quadratic functions, and solve for the deformation parameters. As a result, we can recover the deformation field between two images without solving the correspondence problem. Additionally, our method is highly robust to image noise. The method was tested on both synthetically deformed MPEG-7 shapes and cardiac MRI sequences.