diff options
| author | Frankie B <git@diskfloppy.me> | 2024-06-11 18:02:01 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-06-11 18:02:01 +0100 |
| commit | 0f52d80ca67a49258b235f5831163dd72fbd54cf (patch) | |
| tree | 9c5cd36b6e0a233e09ac88a4409fb68c63e4781a /public/robots.txt | |
| parent | a64bcc2c4639d5804b6dada23151bfcb8b198121 (diff) | |
Merge MVC rewrite into master (#21)
* Just commit it all
* Require auth
* crap
* Update homepage
* Block AI scrapers
* Update cache update script
* Add dummy file
* Remove unnecessary lastfm config var
* Use withQueryParameters for LastFM API
* Fix embeds
* Update example env
* Smard
Diffstat (limited to 'public/robots.txt')
| -rw-r--r-- | public/robots.txt | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/public/robots.txt b/public/robots.txt index e65f07c..582552d 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -3,3 +3,19 @@ Disallow: /admin Disallow: /login Disallow: /js Disallow: /css + +User-Agent: GPTBot +Disallow: / + +User-Agent: ChatGPT-User +Disallow: / + +User-Agent: Google-Extended +Disallow: / + +User-Agent: CCBot +Disallow: / + +User-Agent: PerplexityBot +Disallow: / + |