In this paper, we approach the problem of determining the basic components from repetitive textured surfaces undergoing free-form deformations. Traditional methods for texture modeling are usually based on measurements performed on fronto-parallel planar surfaces. Recently, affine invariant descriptors have been proposed as an effective way to extract local information from non-planar texture surfaces. However, affine transformations are unable to model local image distortions caused by changes in surface curvature. Here, we propose a method for selecting the most representative candidates for the basic texture elements of a texture field while preserving the descriptors’ affine invariance requirement. Our contribution in this paper is twofold. First, we investigate the distribution of extracted affine invariant descriptors on a nonlinear manifold embedding. Secondly, we describe a learning procedure that allows us to group repetitive texture elements while removing candidates presenting high levels of curvature-induced distortion. We demonstrate the effectiveness of our method on a set of images obtained from man-made texture surfaces undergoing a range of non-rigid deformations.