Not exactly sure what is going on here… My only guess is that the basic auth is up to stop bots, but was it such a problem that robots.txt alone wasn’t enough?
Not exactly sure what is going on here… My only guess is that the basic auth is up to stop bots, but was it such a problem that robots.txt alone wasn’t enough?