LogFAQs > #940300968

LurkerFAQs, Active DB, DB1, DB2, DB3, DB4, DB5, Database 6 ( 01.01.2020-07.18.2020 ), DB7, DB8, DB9, DB10, DB11, DB12, Clear

Topic List	Page List: 1
Topic	What is the worst thing that could happen if somebody web-scraped a site without
OniRonin 06/07/20 2:34:19 AM #4:	if you want to be respectful and follow a site's guidelines for scraping, you can look at their robots.txt file - for example for the fire emblem site you mentioned https://serenesforest.net/robots.txt this is essentially like asking the owner what parts of the site you can scrape this one means anybody can scrape any part of the website, other than the wp-admin directory. although for some reason it does allow scraping some random php file in that directory. --- god is dumb #NotAllGamers #YesAllLandlods <cite>OniRonin posted [p:940300968] in What is th... [t:78752218]...</cite> <quote>if you want to be respectful and follow a site's guidelines for scraping, you can look at their robots.txt file - for example for the fire emblem site you mentioned https://serenesforest.net/robots.txt this is essentially like asking the owner what parts of the site you can scrape this one means anybody can scrape any part of the website, other than the wp-admin directory. although for some reason it does allow scraping some random php file in that directory. --- god is dumb #NotAllGamers #YesAllLandlods</quote> ... Copied to Clipboard!
Topic List	Page List: 1