The X-Robots-Tag can be used as an element of the HTTP header response for a given URL of the web page. If you do not want to use robots.txt or robots meta tag, or need to block non-HTML content, then use X-Robots-Tag. .