That moment when you realise only 2 lines of code prevent everyone

the only reason id ever respect a faggy robots.txt is if im releasing something public on github and even then if your site doesn't have a public api in 2017, to get the information i need im going to crawl it with different common user agents and varying proxies until you stop being a fucking idiot

>>59378219
I don't have robots.txt, only X-Robots

robots.txt is actually deprecated. If you have it, Google will still index your page, but replacing all the pages with "This page can't be displayed due to robots.txt" etc. which completely defeats the point.

It's good to not have robots.txt

>>59378280
You can see this with chiru.no. you can't google it and it has no robots.txt

File: frog.png (378KB, 600x760px) Image search: [Google]

378KB, 600x760px

>>59378219
>write website entirely in javascript
>crawlers can't index anything because they dont support running js
>as an added bonus, freetards also won't visit your website

>>59378336
>He still thinks crawlers can't index javascript-based website :^)

>>59378336
m8, crawler generates a websites loaded state in its VM or something and then interacts with it.

>>59378336
Javascript is rendered before Google archives it. Otherwise everything would say "Enable Javascript" in the description. It's a lot of bullshit to do somethig incorrectly.

Also, the page still gets indexed even if the description won't load.

>>59378219
what did he mean when he said this?

>>59378219
The primary purpose of robots.txt is to inform crawling bots not to index something that's bad to index, like if a page is generated automatically and thus pointless to index, or if the page links back to itself using randomly generated URLs so the bot doesn't get stuck in an infinite loop.

robots.txt is a tool for websites to help crawling bots, not hurt them or block them. Crawling bots are not obligated to respect it.

If you're trying to block crawlers, keep in mind that there is literally no way for you to detect a crawler that perfectly mimics your regular users. It's like bailing out a flooding aircraft carrier with a teaspoon.

>>59378446
Don't assume my gender ever again, faggot nigger.

>>59378567
Don't assume my race and sexuality. I'm a non binary gender, pansexual jewish male lesbian