• clb92
    link
    fedilink
    191 year ago

    Sure it can create new words. It can’t create new tokens, would be more correct, I think. But a token is just a text fragment, and, as far as I know, they can range from being several words to being single characters.

    • @average650
      link
      61 year ago

      I got it it say vilumplox. It doesn’t return any Google search results.

    • Dr Cog
      link
      fedilink
      11 year ago

      Tokens are usually never multiple words. Think of them like “information units”. If you have a plural word “hats”, there are two tokens: the “hat” one and the "s’ that adds more information about it being plural. Combinations of words only really occur for proper names.