Типа - пример для примера:
Код: Выделить всё
https://www.phpbbguru.net/community/1
https://www.phpbbguru.net/community/%F
https://www.phpbbguru.net/community/m
https://www.phpbbguru.net/community/xz
Код: Выделить всё
https://www.phpbbguru.net/community/1
https://www.phpbbguru.net/community/%F
https://www.phpbbguru.net/community/m
https://www.phpbbguru.net/community/xz
Код: Выделить всё
/thankslist/givens/185/true
/thankslist/givens/211/false
/thankslist/givens/58/true
/thankslist/givens/65/false
Код: Выделить всё
User-agent: *
Disallow: /forum/*?sid=*
Disallow: /forum/adm/*
Disallow: /forum/download/*
Disallow: /forum/faq.php
Disallow: /forum/mcp.php
Disallow: /forum/memberlist.php
Disallow: /forum/posting.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/ucp.php
Disallow: /forum/viewtopic.php?p=*
Disallow: /forum/viewtopic.php?f=*&t=*&start=0$
Disallow: /forum/viewtopic.php?f=*&t=*&view=next
Disallow: /forum/viewtopic.php?f=*&t=*&view=previous
Sitemap: http://nashauk.ru/forum/sitemap.xml
Clean-param: sid /forum/index.php
Clean-param: sid /forum/viewforum.php
Clean-param: sid /forum/viewtopic.php
User-agent: Yandex
Disallow: /forum/*?sid=*
Disallow: /forum/adm/*
Disallow: /forum/download/*
Disallow: /forum/faq.php
Disallow: /forum/mcp.php
Disallow: /forum/memberlist.php
Disallow: /forum/posting.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/ucp.php
Disallow: /forum/viewtopic.php?p=*
Disallow: /forum/viewtopic.php?f=*&t=*&start=0$
Disallow: /forum/viewtopic.php?f=*&t=*&view=next
Disallow: /forum/viewtopic.php?f=*&t=*&view=previous
Host: nashauk.ru
Sitemap: http://nashauk.ru/forum/sitemap.xml
Clean-param: sid /forum/index.php
Clean-param: sid /forum/viewforum.php
Clean-param: sid /forum/viewtopic.php