Build a robots.txt file by defining crawler rules, allow/disallow paths, and sitemap location.
User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/
