Zimanyar API
    Preparing search index...

    Function tokenize

    • Split a text into word, number and punctuation tokens. A ZWNJ never splits a word; an apostrophe between letters is kept; hyphenated compounds split unless keepHyphen is set.

      Parameters

      • text: string
      • keepHyphen: boolean = false

      Returns Token[]

    Settings
    Member Visibility