Fulltext indexes

Analyzers define how fulltext search reads text: how to split it into terms, and how to normalize each term. They are named definitions a fulltext index points at.

The shape

applied to SurrealDB 3.3.0

schema.surdl

sdl

namespace acme;
database shop;

/// English text search.
analyzer english {
  tokenizers: class;
  filters: snowball(english), lowercase;
};

surform build

sql

DEFINE NAMESPACE OVERWRITE acme;
USE NS acme;
DEFINE DATABASE OVERWRITE shop;
USE NS acme DB shop;
DEFINE ANALYZER OVERWRITE `english`
    TOKENIZERS CLASS
    FILTERS SNOWBALL(ENGLISH), LOWERCASE
    COMMENT 'English text search.';

Three optional clauses — an analyzer with none is legal but inert:

tokenizers: — how text splits. blank (whitespace), camel (camelCase boundaries), class (alphanumeric runs), punct (punctuation).
filters: — term transformations, order matters: ascii, lowercase, uppercase, snowball(<language>) (stemming — 18 languages, bare keywords, short aliases like en accepted), edgengram(min, max) (prefix matching for autocomplete), ngram(min, max) (fuzzy matching).
function: — a preprocessing function of your own (it must accept and return a string), running before tokenization.

Building blocks for search features

Autocomplete and fuzzy search are analyzer combinations:

applied to SurrealDB 3.3.0

schema.surdl

sdl

namespace acme;
database shop;

analyzer autocomplete {
  filters: lowercase, edgengram(2, 10);
};
analyzer fuzzy {
  filters: ascii, ngram(3, 5);
};

surform build

sql

DEFINE NAMESPACE OVERWRITE acme;
USE NS acme;
DEFINE DATABASE OVERWRITE shop;
USE NS acme DB shop;
DEFINE ANALYZER OVERWRITE autocomplete FILTERS LOWERCASE, EDGENGRAM(2,10);
DEFINE ANALYZER OVERWRITE fuzzy FILTERS ASCII, NGRAM(3,5);

Point a fulltext index at one and queries like … WHERE title @@ 'sur' start matching prefixes. The linter checks your ngram windows (SDL-L0004 — a min larger than the max can only degrade silently).

Preprocessing with your own function

applied to SurrealDB 3.3.0

schema.surdl

sdl

namespace acme;
database shop;

fn strip_markup($s: string) -> string {
  string::replace($s, '<em>', '')
};

analyzer preprocessed {
  function: fn::strip_markup;
  tokenizers: blank;
  filters: lowercase;
};

surform build

sql

DEFINE NAMESPACE OVERWRITE acme;
USE NS acme;
DEFINE DATABASE OVERWRITE shop;
USE NS acme DB shop;
DEFINE FUNCTION OVERWRITE fn::strip_markup($s: string) -> string { string::replace($s, '<em>', '') };
DEFINE ANALYZER OVERWRITE preprocessed FUNCTION fn::strip_markup TOKENIZERS BLANK FILTERS LOWERCASE;

The function: clause is a name, not a call — no trailing parentheses.

Lists, repeats, and wrapping

Lists are comma-separated and may wrap across lines; repeating a clause replaces the whole list (last one wins); a duplicate entry inside one list is accepted by the database but flagged by the linter:

applied to SurrealDB 3.3.0

schema.surdl

sdl

namespace acme;
database shop;

analyzer everything {
  tokenizers: blank,
              camel,
              punct;
  filters: ascii, lowercase, snowball(english);
};

surform build

sql

DEFINE NAMESPACE OVERWRITE acme;
USE NS acme;
DEFINE DATABASE OVERWRITE shop;
USE NS acme DB shop;
DEFINE ANALYZER OVERWRITE everything
    TOKENIZERS BLANK,CAMEL,PUNCT
    FILTERS ASCII, LOWERCASE, SNOWBALL(ENGLISH);

Where next

Indexes — attaching an analyzer to a field.
Functions — writing the preprocessing function.