Asteroids: No little pebbles allowed?