Just stumbled across this piece of info in the technet2 doco.
You can set the peoplepicker-onlysearchwithinsitecollection property to yes to only show users that are a member of that site. This is great for extranet type site collections that you want to stop them from browsing all of you AD to find details on your users.
The example syntax given on the site is:-
stsadm -o setproperty –url http://server –pn peoplepicker-onlysearchwithinsitecollection –pv yes
Note that if a user knows the fully qualified login name they can still select those names.
But not limit at site level, only at SITE COLLECTION level…
i tried but gives command line error let me know what i should write in url is web application url or central administration url