Do you know robots.txt?
百度百科