How do I write a robots.txt rule that blocks a folder whose name is in Arabic?
Write the rule in the folder name's encoded form, not in Arabic letters. Crawlers request an Arabic path percent-encoded, starting with % signs, and the builder compares paths as written without decoding them. Encode the folder name in the URL encoder, paste the result into Disallow, and test it with a full URL.
Why a rule in Arabic letters does not work
Your browser shows /منتجات-قديمة/ in Arabic letters because it is easier to read, but the request sent to the server carries the name encoded, each Arabic letter becoming a group of characters starting with %. A crawler compares your rules against that encoded form.
We tested it in the Robots.txt builder: the rule /منتجات-قديمة/ in Arabic letters did not match the encoded path of a page inside the folder. The encoded rule did. Note that the tool accepted the Arabic rule and wrote it into the file with no warning, so the mistake only shows up in the tester.
Steps
- Open the URL encoder/decoder, encode in 'Full URL' mode, and paste the path /منتجات-قديمة/.
- Copy the output, the encoded line you can see in the file below.
- Paste it into Disallow in the robots.txt builder as it is.
- In 'Test a path', paste a full URL for a page inside the folder, Arabic letters and all, as in your browser, such as https://example.com/منتجات-قديمة/oud-1.
- Check the verdict is 'Blocked from crawling' with your encoded rule as the winner.
User-agent: *
Disallow: /%D9%85%D9%86%D8%AA%D8%AC%D8%A7%D8%AA-%D9%82%D8%AF%D9%8A%D9%85%D8%A9/
A trap in testing: full URL or path?
When you paste a full URL starting with https://, the tool extracts the path and encodes it automatically, giving /%D9%85...%D8%A9/oud-1, the form a crawler really requests. If you type the path alone in Arabic letters, the tool compares it as typed. The Arabic rule would then seem to work in the tester even though it will not match a real crawler's request. So always test with the full URL, or with the encoded path.
Encoding details
- The case of the codes matters to the tool: a rule written with lower-case codes such as %d9%85 did not match a path with upper-case %D9%85. Copy the encoder's output exactly.
- Hyphens, digits and Latin letters in the name stay as they are; only the Arabic is encoded.
- Do not put a space in a rule. A space in a name is encoded as %20, and the tool warns if you type one.
- Blocking stops crawling, not indexing, and it hides the folder from nobody, since the file is public.
Real crawlers may handle encoding differently in the details, and the tool uses a strict text comparison so it never gives you a rosier answer than reality. If your store is on a hosted platform, the file is usually not yours to change, but you can test the platform's file the same way.
Common questions
Should I write both the Arabic and the encoded rule?
The encoded one covers what crawlers request. The Arabic one does no harm but does not match an encoded request.
Does the tool encode the rule for me?
No. It writes the rule as you type it, and only encodes the full URL you paste into the tester.
Does the same apply to the sitemap?
The sitemap generator encodes Arabic URLs itself. For robots.txt, encoding the rule is up to you.