Searching the Quran without diacritics: Arabic text matching in PostgreSQL

Our Quran app stores the complete Uthmani text: 6,236 ayahs, fully marked with tashkeel (short vowels), Quranic annotation signs and pause marks. That text is exactly what readers expect to see on screen. It is also almost impossible to search directly.

Someone typing on a phone keyboard enters plain letters. They will type الرحمن, not ٱلرَّحْمَٰنِ. Byte for byte, those two strings share very little. This post explains how we made search ignore the marks while still showing the fully marked text in the results.

Why full-text search did not work

PostgreSQL's full-text search was the obvious first idea. It works well for our translations (English, Urdu, Roman Urdu):

WHERE t.language = $1
  AND to_tsvector('simple', at.text) @@ plainto_tsquery('simple', $2)
ORDER BY ts_rank(to_tsvector('simple', at.text), plainto_tsquery('simple', $2)) DESC

For the Arabic text it found nothing. A tsvector splits text into tokens and compares tokens exactly. Every Uthmani token carries its diacritics, so a plain-typed query token never equals a stored one. PostgreSQL has no built-in Arabic dictionary that strips tashkeel, and stemming Arabic properly is a research problem of its own.

We did not need stemming. We needed "the same letters, ignoring the marks".

Step 1: define what "the same letters" means

We wrote a normaliser in TypeScript that does three things:

/** Tashkeel, superscript alef, Quranic annotation marks, tatweel, BOM. */
const ARABIC_MARKS =
  /[ؐ-ًؚ-ٰٟۖ-ۜ۟-۪ۨ-ۭـ]/g

/** Strip diacritics and unify alef/ya variants so plain-typed queries match Uthmani text. */
export function normalizeArabic(input: string): string {
  return input
    .normalize('NFC')
    .replace(ARABIC_MARKS, '')
    .replace(/[آأإٱ]/g, 'ا') // آ أ إ ٱ → ا
    .replace(/ى/g, 'ي')                     // ى → ي
    .trim()
}

1. Unicode normalisation (NFC) so that composed and decomposed forms of the same character compare equal.
2. Remove the marks. The ranges cover the harakat (fatha, damma, kasra, sukun, shadda, tanwin), the superscript alef (U+0670), the small Quranic annotation signs used in the Uthmani script, the tatweel stretching character (U+0640), and one more character that we will come back to.
3. Unify letter variants. Alef appears as آ, أ, إ and the wasla form ٱ, which is everywhere in Uthmani text and on almost no keyboard. All of them become a plain ا. Alef maqsura ى becomes ي. Users do not reliably distinguish these when typing, and for search they should not have to.

Step 2: apply the same rule inside SQL

Normalising the query is only half the job. The stored text has to be normalised the same way at comparison time. PostgreSQL can do that with regexp_replace and translate:

/** SQL-side mirror of normalizeArabic; POSIX classes can't express these ranges, so the set is inlined. */
const SQL_NORMALIZE = (column: string) =>
  sql.raw(
    `translate(regexp_replace(${column}, ` +
      `'[\\u0610-\\u061A\\u064B-\\u065F\\u0670\\u06D6-\\u06DC\\u06DF-\\u06E8\\u06EA-\\u06ED\\u0640\\uFEFF]', '', 'g'), ` +
      `'آأإٱى', 'ااااي')`,
  )

regexp_replace removes the marks and translate maps each variant letter to its plain form, one character to one character. The two functions mirror the TypeScript version exactly, and a comment says so. If one side changes without the other, searches start missing in ways that are very hard to notice.

The query itself is deliberately simple:

SELECT a.id, a.surah_number, a.ayah_number, a.text_uthmani
FROM ayahs a
WHERE <normalised a.text_uthmani> ILIKE '%' || $1 || '%'
ORDER BY a.global_number
LIMIT $2 OFFSET $3

The user gets back the original, fully marked text_uthmani, in mushaf order. Normalisation only exists for matching. Nothing is shown to the reader without its tashkeel.

Is a full scan acceptable?

ILIKE '%…%' with an expression on the column cannot use a normal B-tree index, so every search scans the table. For most tables that would be a red flag. Here it is fine, and the code says why: the Quran has 6,236 ayahs and that number will never change. A sequential scan with a regular expression over a few megabytes of text runs in milliseconds, and the response is cacheable.

If it ever became a problem, the next step would be a generated column holding the normalised text, with a trigram (pg_trgm) index on it. We have not needed it. Choosing the boring option because the data is small and fixed is a design decision, not a shortcut.

The invisible character

The last entry in the marks list is , the byte order mark, also called the zero-width no-break space. Text that passes through files, imports and copy-paste can pick it up. It is invisible in every editor and has no width on screen, but a string containing it does not match the same string without it. So we strip it.

The first version of the regular expression contained the BOM as a literal character, pasted straight into the character class. It worked perfectly. Then CI failed:

error  Irregular whitespace not allowed  no-irregular-whitespace

ESLint flags literal invisible whitespace in source, which is exactly the right call: a character you cannot see is a character nobody can review. The failing lint job also blocked the image publish, so a search fix could not ship until it was resolved.

The fix was a one-character change: write the escape  instead of the literal. It matches the same code point, all 42 search tests still pass, and now the next person reading the regex can actually see what it does.

We took a broader rule from this, and our mobile app's Mushaf code follows it too: never put literal Arabic marks or invisible characters in source code. Use Unicode escapes. Copy and paste, fonts and some editors can silently replace one character with a visually identical one that has a different code point, and a regex built from it will quietly stop matching.

What we would recommend for Arabic search

• Decide up front which differences should be ignored (marks, letter variants) and write them down as one function.
• Apply the same normalisation to the query and to the stored text, and keep both implementations side by side with a comment.
• Return the original text. Normalise only for comparison.
• Use full-text search for translations, and normalised matching for the Arabic.
• Use Unicode escapes for every mark in source code.
• Measure before indexing. Small, fixed datasets often do not need an index at all.