Type Alias: TokenizeOptions
TokenizeOptions =
object
Options controlling tokenizeWithOptions. All default off except
camelCase.
distinct- drop duplicate tokens (first occurrence wins).lowerCase- lowercase every token.capitalize- upper-case each token’s first letter (then TOKENIZE_OVERRIDES fix upai->AI,fs->FS, andv2->V2).omitUriScheme- strip a leadingscheme://before tokenizing.omitEmailDomain- keep only the local part of an email.camelCase- split on camelCase boundaries / digit runs / acronyms (defaulttrue); whenfalse, split only on non-alphanumerics.
Properties
Section titled “Properties”camelCase?
Section titled “camelCase?”
optionalcamelCase?:boolean
capitalize?
Section titled “capitalize?”
optionalcapitalize?:boolean
distinct?
Section titled “distinct?”
optionaldistinct?:boolean
lowerCase?
Section titled “lowerCase?”
optionallowerCase?:boolean
omitEmailDomain?
Section titled “omitEmailDomain?”
optionalomitEmailDomain?:boolean
omitUriScheme?
Section titled “omitUriScheme?”
optionalomitUriScheme?:boolean