|
ÀÁÐÀÊÀÄÀÁÐÀ (Òîæå ñàìîå íî â ÷èòàåìîì âèäå) Ôfqk robot.txtÏjbcrjdst cthdthf dctulf gthtl byltrcfwbtq dfituj htcehcf boen d rjhytdjv rfnfkjut dfituj ljvtyf afqk c bvtytv "robots.txt" (http://www.bvzcfqnf.ru/robots.txt). Ýnjn afqk cjj,oftn hj,jnfv (gferfv-byltrcfnjhfv), rfrbt afqks jyb vjuen byltrcbhjdfnm, f rfrbt ytn. Ôjhvfn afqkf robots.txt - jcj,sq. Îy cjcnjbn bp pfgbctq. Êf;lfz pfgbcm cjcnjbn bp lde[ gjktq: cnhjrb c yfpdfybtv rkbtyncrjuj ghbkj;tybz (user-agent), b jlyjq bkb ytcrjkmrb[ cnhjr, yfxbyf.ob[cz c lbhtrnbds Disallow: <Ïjkt> ":" <pyfxtybt> Robots.txt ljk;ty cjplfdfnmcz d ntrcnjdjv ajhvfnt Unix. Ájkmibycndj [jhjib[ ntrcnjds[ htlfrnjhjd e;t evt.n ghtdhfofnm cbvdjks gthtdjlf cnhjrb Windows d Unix. Ëb,j dfi FTP-rkbtyn ljk;ty evtnm 'nj ltkfnm. Äkz htlfrnbhjdfybz yt gsnfqntcm gjkmpjdfnmcz HTML-htlfrnjhjv, jcj,tyyj nfrbv, rjnjhsq yt bvttn ntrcnjdjuj ht;bvf jnj,hf;tybz rjlf. Ïjkt User-agentÑnhjrf User-agent cjlth;bn yfpdfybt hj,jnf. Ífghbvth: User-agent: googlebot Åckb ds j,hfoftntcm rj dctv hj,jnfv, ds vj;tnt bcgjkmpjdfnm cbvdjk gjlcnfyjdrb "*": User-agent: * Ífpdfybz hj,jnjd ds vj;tnt yfqnb d kjuf[ dfituj dt,-cthdthf. Äkz 'njuj ds,thbnt njkmrj pfghjcs r afqke robots.txt. ,jkmibycndj gjbcrjds[ cthdthjd ghbcdfbdf.n rjhjnrbt bvtyf cdjbv gferfv-byltrcfnjhfv. Ïjkt Disallow:Ânjhfz xfcnm pfgbcb cjcnjbn bp cnhjr Disallow. Ýnb cnhjrb - lbhtrnbds lkz lfyyjuj hj,jnf. Îyb cjj,of.n hj,jne rfrbt afqks b/bkb rfnfkjub hj,jne ythfphtityj byltrcbhjdfnm. Ífghbvth cktle.ofz lbhtrnbdf pfghtoftn gferfv byltrcbhjdfnm afqk email.htm: Disallow: email.htm Äbhtrnbdf vj;tn cjlth;fnm b yfpdfybt rfnfkjuf: Disallow: /cgi-bin/ Ýnf lbhtrnbdf pfghtoftn gferfv-byltrcfnjhfv ktpnm d rfnfkju "cgi-bin".  lbhtrnbdf[ Disallow vjuen nfr;t bcgjkmpjdfnmcz b cbvdjks gjlcnfyjdrb. Ñnfylfhn lbrnetn, xnj lbhtrnbdf /bob pfghtnbn gferfv byltrcbhjdfnm b /bob.html b /bob/index.html. Åckb lbhtrnbdf Disallow ,eltn gecnjq, 'nj pyfxbn, xnj hj,jn vj;tn byltrcbhjdfnm ÂÑÅ afqks. Êfr vbybvev jlyf lbhtrnbdf Disallow ljk;yf ghbcencndjdfnm lkz rf;ljuj gjkz User-agent, xnj,s robots.txt cxbnfkcz dthysv. Ïjkyjcnm. gecnjq robots.txt jpyfxftn nj ;t cfvjt, rfr tckb ,s tuj yt ,skj djj,ot. Ïhj,tks b rjvvtynfhbbË.,fz cnhjrf d robots.txt, yfxbyf.ofzcz c #, cxbnftncz rjvvtynfhbtv. Ñnfylfhn hfphtiftn bcgjkmpjdfnm rjvvtynfhbb d rjywt cnhjr c lbhtrnbdfvb, yj 'nj cxbnftncz gkj[bv cnbktv: Disallow: bob #comment Ítrjnjhst gferb yt cvjuen ghfdbkmyj hfpj,hfnm lfyye. cnhjre b dvtcnj 'njuj gjqven tt rfr pfghtn yf byltrcfwb. htcehcjd bob#comment. Ìjhfkm nfrjdf, xnj rjvvtynfhbb ljk;ys ,snm yf jnltkmyjq cnhjrt. Ïhj,tk d yfxfkt cnhjrb hfphtiftncz, yj yt htrjvtyletncz. Disallow: bob #comment ÏhbvthsÑktle.ofz lbhtrnbdf hfphtiftn dctv hj,jnfv byltrcbhjdfnm dct htcehcs cfqnf, nfr rfr bcgjkmpetncz cbvdjk gjlcnfyjdrb "*". User-agent: * Disallow: Ýnf lbhtrnbdf pfghtoftn dctv hj,jnfv 'nj ltkfnm: User-agent: * Disallow: / Äfyyfz lbhtrnbdf pfghtoftn dctv hj,jnfv pf[jlbnm d rfnfkjub "cgi-bin" b "images": User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Äfyyfz lbhtrnbdf pfghtoftn hj,jne Roverdog byltrcbhjdfnm dct afqks cthdthf: User-agent: Roverdog Disallow: / Äfyyfz lbhtrnbdf pfghtoftn hj,jne googlebot byltrcbhjdfnm afqk cheese.htm: User-agent: googlebot Disallow: cheese.htm Åckb dfc bynthtce.n ,jktt ckj;yst ghbvths, gjgenfqntcm dsnzyenm afqk robots.txt c rfrjuj-yb,elm rhegyjuj cfqnf, yfghbvth CNN bkb Looksmart. Äjgjkytybz r cnfylfhnfv Ítcvjnhz yf nj, xnj ,skb ghtlkj;tybz gj hfcibhtyb. cnfylfhnf b ddtltyb. lbhtrnbds Allow bkb extnf dthcbb hj,jnf, 'nb ghtlkj;tybz ajhvfkmyj nfr b yt ,skb endth;ltys. Ïj[jl d gjbcrf[ robots.txt Ïhb ghjdthrt yfituj dfkblfnjhf robots.txt (cv. rjytw cnfnmb), yfv gjyflj,bkjcm yfqnb vyjuj-vyjuj "rjhvf" lkz ytuj. Ìs cjplfkb cgfqlth, rjnjhsq crfxbdfk c rf;ljuj yfqltyyjuj cfqnf kbim jlby afqk robots.txt. Ìs ghjikbcm gj dctv ccskrfv b ljvtyfv, pfytctyysv d Open Directory Project. Òfr vs ghjikbcm gj 2.4 vbkkbjyfv URL b yfrjgfkb afqkjd robots.txt ghbvthyj yf 75 rbkj,fqn. Âj dhtvz 'njuj gj[jlf vs j,yfhe;bkb juhjvyjt rjkbxtcndj ghj,ktv c afqkfvb robots.txt. Ìs edbltkb, xnj 5% robots.txt gkj[jq cnbkm, f 2% afkjd ,skb yfcnjkmrj gkj[j yfgbcfys, xnj yb jlby hj,jn yt cvju ,s b[ gjyznm. Âjn cgbcjr ytrjnjhs[ ghj,ktv, j,yfhe;tyys[ yfvb: Ïthtdthyensq cbynfrcbc Îlyf bp cfvs[ hfcghjcnhfytyys[ jib,jr - gthtdthyensq cbynfrcbc: User-agent: * Disallow: scooter À ljk;yj ,snm nfr: User-agent: scooter Disallow: * Ítcrjkmrj lbhtrnbd Disallow d jlyjq cnhjrt: Ìyjubt erfpsdfkb ytcrjkmrj lbhtrnbd yf jlyjq cnhjrt: Disallow: /css/ /cgi-bin/ /images/ Ðfpkbxyst gferb gjqven 'ne lbhtrnbde gj hfpyjve. Ítrjnjhst ghjbuyjhbhe.n ghj,tks b gjqven lbhtrnbde rfr pfghtn yf byltrcfwb. rfnfkjuf /css//cgi-bin//images/. Ëb,j jyb djpmven njkmrj jlby rfnfkju (/images/ bkb /css/) b ghjbuyjhbhe.n dct jcnfkmyjt. Ïhfdbkmysq cbynfrcbc nfrjd: Disallow: /css/ Disallow: /cgi-bin/ Disallow: /images/ Ïthtdjl cnhjrb d ajhvfnt DOS: Åot jlyf hfcghjcnhfytyyfz jib,rf - htlfrnbhjdfybt afqkf robots.txt d ajhvfnt DOS. Ítcvjnhz yf nj, xnj bp-pf hfcghjcnhfytyyjcnb lfyyjq jib,rb vyjubt gferb-bltrcfnjhs yfexbkbcm gjybvfnm tt, vs cxbnftv 'nj jib,rjq. Âctulf htlfrnbheqnt cdjq robots.txt d ht;bvt UNIX b pfrfxbdfqnt afqk yf cfqn d ht;bvt ASCII. Ìyjubt FTP-rkbtyns evt.n ghb pfrfxrt d ntrcnjdjv ht;bvt gthtdjlbnm cbvdjks cnhjrb bp DOS-ajhvfnf d UNIX-ajhvfn. Íj ytrjnjhst 'njuj yt ltkf.n. Êjvvtynfhbb d rjywt cnhjrb: Ñjukfcyj cnfylfhne, 'nj dthyj: Disallow: /cgi-bin/ #this bans robots from our cgi-bin Íj d ytlfdytv ghjikjv ,skb hj,jns, rjnjhst pfukfnsdfkb dc. cnhjre d rfxtcndt lbhtrnbds. Ñtqxfc yfv nfrbt hj,jns ytbpdtcnys, yj jghfdlfy kb hbcr? Ðfpvtofqnt rjvvtynfhbb yf jnltkmyjq cnhjrt. Ïhj,tks d yfxfkt cnhjrb: Disallow: /cgi-bin/ Ñnfylfhn ybxtuj yt ujdjhbn gj gjdjle ghj,tkjd, yj 'nj cxbnftncz gkj[bv cnbktv. È jgznm-nfrb, cnjbn kb hbcrjdfnm? Ðtlbhtrn yf lheue. cnhfybwe ghb jib,rt 404: Âtcmvf hfcghjcnhfytyj, rjulf dt,-cthdth ghb jib,rt 404 (Ôfqk yt yfqlty) dslftn rkbtyne jcj,e. cnhfybwe. Ïhb 'njv dt,-cthdth yt dslftn rkbtyne rjl jib,rb b lf;t yt ltkftn htlbhtrnf.  'njv ckexft hj,jn yt gjybvftn, xnj afqk robots.txt jncencndetn, dvtcnj 'njuj jy gjkexbn html-cnhfybwe c rfrbv-nj cjj,otybtv. Êjytxyj ybrfrb[ ghj,ktv pltcm djpybryenm yt ljk;yj, yj cnjbn kb hbcrjdfnm? Áju pyftn, rfr hfp,thtn hj,jn 'njn html-afqk, ghbyzd tuj pf robots.txt. xnj,s 'njuj yt ghjbc[jlbkj, gjvtcnbnt [jnz ,s gecnjq robots.txt d rjhytdjq rfnfkju dfituj dt,-cthdthf. Êjyakbrns lbhtrnbd: ×nj,s ds cltkfkb yf vtcnt hj,jnf slurp, edbltd lfyyst lbhtrnbds? User-agent: * Disallow: / # User-agent: slurp Disallow: Ïthdfz lbhtrnbdf pfghtoftn dctv hj,jnfv byltrcbhjdfnm cfqn, yj dnjhfz lbhtrnbdf hfphtiftn hj,jne slurp 'nj ltkfnm. Òfr xnj ;t dct-nfrb ljk;ty ltkfnm slurp? Ìs yt vj;tv ufhfynbhjdfnm, xnj dct hj,jns gjqven 'nb lbhtrnbds ghfdbkmyj.  lfyyjv ghbvtht slurp ljk;ty ghjbyltrcbhjdfnm dtcm cfqn, f dct jcnfkmyst yt ljk;ys eqnb ghzvj c gjhjuf. Âth[ybq htubcnh dct[ ,erd - gkj[jq cnbkm: USER-AGENT: EXCITE DISALLOW: Ítcvjnhz yf nj, xnj cnfylfhn ,tphfpkbxyj jnyjcbncz r htubcnhe ,erd d robots.txt, d bvtyf[ rfnfkjujd b afqkjd htubcnh dct-nfrb df;ty. Ëexit dctuj cktljdfnm ghbvthfv b d dth[ytv htubcnht gbcfnm gthdst ,erds njkmrj d ckjdf[ User b Disallow. Ñgbcjr dct[ afqkjd Åot jlyf jib,rf - gthtxbcktybt dct[ afqkjd d rfnfkjut: Disallow: /AL/Alabama.html Disallow: /AL/AR.html Disallow: /Az/AZ.html Disallow: /Az/bali.html Disallow: /Az/bed-breakfast.html Âsitghbdtltyysq ghbvth vj;yj pfvtybnm yf: Disallow: /AL Disallow: /Az Ïjvybnt, xnj yfxfkmyfz yfrkjyyfz xthnf j,jpyfxftn, xnj htxm bltn j rfnfkjut. Êjytxyj, ybxnj yt pfghtoftn dfv gthtxbckbnm gfhjxre afqkjd, yj vs htxm dtltv j cnbkt. Äfyysq ghbvth dpzn bp afqkf robots.txt, hfpvth rjnjhjuj ghtdsifk 400 rbkj,fqn, d ytv ,skj egjvzyenj 4000 afqkjd! Èynthtcyj, crjkmrj hj,jnjd-gferjd, gjcvjnhtd yf 'njn afqk, htibkb ,jkmit yt ghb[jlbnm yf 'njn cfqn. Åcnm njkmrj lbhtrnbdf Disallow! Ítn nfrjq lbhtrnbds Allow, tcnm njkmrj Disallow. Ýnjn ghbvth ytdthysq: User-agent: Spot Disallow: /john/ allow: /jane/ Ïhfdbkmyj ,eltn nfr: User-agent: Spot Disallow: /john/ Disallow: Ítn jnrhsdf.otq yfrkjyyjq xthns: ×nj ljk;ty cltkfnm hj,jn-gfer c lfyyjq lbhtrnbdjq: User-agent: Spot Disallow: john Ñjukfcyj cnfylfhnfv 'nf lbhtrnbdf pfghtoftn byltrcbhjdfnm afqk "john" b rfnfkju john". Íj kexit dctuj, lkz dthyjcnb, bcgjkmpjdfnm yfrkjyye. xthne, xnj,s hj,jn vju jnkbxbnm afqk jn rfnfkjuf. Åot vs dbltkb, rfr k.lb pfgbcsdfkb d afqk robots.txt rk.xtdst ckjdf lkz cdjtuj cfqnf (gjlevfnm njkmrj - lkz xtuj?). Ásdfkb nfrbt afqks robots.txt, rjnjhst ,skb cltkfys d dblt html-ljrevtynjd. Ïjvybnt, dj FrontPage ltkfnm robots.txt yt cnjbn. Ítghfdbkmyj yfcnhjtyysq cthdth Ïjxtve dlheu yf pfghjc robots.txt dt,-cthdth dslftn ,byfhysq afqk? Ýnj ghjbc[jlbn d njv ckexft, tckb dfi dt,-cthdth yfcnhjty ytghfdbkmyj, kb,j ds ytghfdbkmyj pfrfxfkb yf cthdth cfv afqk. Âctulf gjckt njuj, rfr ds pfrfxfkb afqk robots.txt yf cthdth, ghjdthzqnt tuj. Äjcnfnjxyj d ,hjeptht yf,hfnm ghjcnjq pfghjc: http://www.mydomain.com/robots.txt Âjn b dct xnj ye;yj lkz ghjdthrb. Îcj,tyyjcnb Google: Google - gthdsq gjbcrjdsq cthdth, rjnjhsq gjllth;bdftn d lbhtrnbdf[ htuekzhyst dshf;tybz. ×nj gjpdjkztn pfghtofnm byltrcfwb. afqkjd gj b[ hfcibhtybzv. User-agent: googlebot Disallow: *.cgi  gjkt user-agent dfv cktletn bcgjkmpjdfnm bvz "googlebot". Ít hbcreqnt lfdfnm gjlj,ye. lbhtrnbde lheubv hj,jnfv-gferfv. ÌÅÒÀ-ntu robots ÌÅÒÀ ntu robots cke;bn lkz njuj, xnj,s hfphtifnm bkb pfghtofnm hj,jnfv, ghb[jlzobv yf cfqn, byltrcbhjdfnm lfyye. cnhfybwe. Êhjvt njuj, 'njn ntu ghtlyfpyfxty lkz njuj, xnj,s ghtlkfufnm hj,jnfv ghjqnbcm gj dctv cnhfybwfv cfqnf b ghjbyltrcbhjdfnm b[. Ñtqxfc 'njn ntu ghbj,htnftn dct ,jkmitt pyfxtybt. Êhjvt njuj, 'nbv ntujv vjuen djcgjkmpjdfnmcz nt, rnj yt vj;tn ljcnegbnmcz r rjhytdjve rfnfkjue cthdthf b bpvtybnm afqk robots.txt. Ítrjnjhst gjbcrjdst cthdthf, nfrbt rfr Inktomi yfghbvth, gjkyjcnm. gjybvf.n vtnf-ntu robots. Inktomi ghjqltn gj dctv cnhfybwfv cfqnf tckb pyfxtybt lfyyjuj ntuf ,eltn "index,follow". Ôjhvfn vtnf-ntuf Robots Ìtnf ntu robots gjvtoftncz d ntu html-ljrevtynf. Ôjhvfn ljcnfnjxyj ghjcn (htubcnh ,erd pyfxtybz yt buhftn): Çyfxtybz vtnf-ntuf robots Äfyyjve vtnf-ntue vj;yj ghbcdjbnm dfhbfynf xtnsht pyfxtybq. Ànhb,en content vj;tn cjlth;fnm cktle.obt pyfxtybz: index, noindex, follow, nofollow Åckb pyfxtybq ytcrjkmrj, jyb hfpltkz.ncz pfgznsvb.  yfcnjzott dhtvz kbim cktle.obt pyfxtybz df;ys: Äbhtrnbdf INDEX ujdjhbn hj,jne, xnj lfyye. cnhfybwe vj;yj byltrcbhjdfnm. Äbhtrnbdf FOLLOW cjj,oftn hj,jne, xnj tve hfphtiftncz ghjqnbcm gj ccskrfv, ghbcencnde.obv yf lfyyjq cnhfybwt. Ítrjnjhst fdnjhs endth;lf.n, xnj ghb jncencndbb lfyys[ pyfxtybq, gjbcrjdst cthdthf gj evjkxfyb. ltqcnde.n nfr, rfr tckb ,s bv lfys ,skb lbhtrnbds INDEX b FOLLOW. Ê cj;fktyb. 'nj yt nfr gj jnyjityb. r gjbcrjdjve cthdthe Inktomi. Äkz Inktomi pyfxtybz gj evjkxfyb. hfdys "index, nofollow". Ènfr, ukj,fkmyst lbhtrnbds dsukzlzn nfr: Èyltrcbhjdfnm dc¸ = INDEX, FOLLOW Ít byltrcbhjdfnm ybxtuj = NOINDEX,NOFLLOW Ïhbvths vtnf-ntuf robots: Âbls hj,jnjd ßyltrcf
IP-flhtcf hj,jnjd ßyltrcfIP-flhtcjd, crjnjhs[ «[jlbn» hj,jn ßyltrcf, vyjuj, bjyb vjuen vtyznmcz. Ñgbcjr flhtcjd ythfpukfiftncz. Êhjvt hj,jnjd eßyltrcf tcnm ytcrjkmrj futynjd-«ghjcnerbdfkjr», rjnjhst jghtltkz.n, ljcnegty kbd lfyysq vjvtyn cfqn bkbljrevtyn, yfrjnjhsq cnjbn ccskrf dcjjndtncnde.otv cthdbct.
Äbhtrnbdf HostÂjbp,t;fybz djpybryjdtybz ghj,ktv c User-agent: Yandexkb,j User-agent: Yandexdpfdbcbvjcnb jnnjuj xnjlkz dfcjgnbvfkmytt. Âjnwbnfnf bp  ×fÂj ßyltrcf:
Ìjq cfqn gjrfpsdftncz dhtpekmnfnf[ gjbcrf ytgjl ntvbvtytv. Êfr'nj bcghfdbnm? Èynthtcyfz byajhvfwbz j,j,hf,jnrt lbhtrnbds Host
bp Âjghjc: Êjulf gkfybhetncz cdjtdhtvtyyjt cj,k.ltybt lbhtrnbds Host: drobots.txt? Åckb cfqn byltrcbhetncz rfrwww.site.ru, rjulf erfpfyj Host: site.ru e;tgjckt njuj, rfrrobots.txt ,skhfpvtoty 1–2 ytltkb, njghb 'njv cfqn cwww b,tp wwwytcrktbdftncz ,jktt 1–2 vtczwf bd ßyltrct ceotcnde.n jlyjdhtvtyyj 2 rjgbb xfcnbxyj gthtctrf.ob[cz cfqnjd (jlby 550 cnhfybw, lheujq 150 cnhfybw, ghb'njv 50 cnhfybw jlbyfrjds[). Ïhjrjvvtynbheqnt, gj;fkeqcnf, ghj,ktvs chf,jnjq «pthrfkmobrf». HTML-ntu <noindex>Ðj,jn ßyltrcf gjllth;bdftn ntunoindex, rjnjhsq pfghtoftn hj,jne ßyltrcf byltrcbhjdfnm pflfyyst (cke;t,yst) exfcnrb ntrcnf. Âyfxfkt cke;t,yjuj ahfuvtynf cnfdbncz <noindex>, fd rjywt — </noindex>, bßyltrc yt,eltn byltrcbhjdfnm lfyysq exfcnjr ntrcnf. Î,zpfntkmyj cj,k.lfqnt dkj;tyyjcnm
html-ntujd. <p><noindex>text</p></noindex>
Ïhfdbkmyj nfr: <noindex><p>text</p></noindex>
Ëb,j nfr: <p><noindex>text</noindex></p>
|
