Estimating prompt size in tokens

Neural networks work with texts by representing words and sentences as tokens.

AI Studio uses its own tokenizer for text processing. To calculate the token size of a text or prompt to a YandexGPT model, use the Tokenize method of the text generation API or Yandex Cloud ML SDK.

The token count of the same text may vary from one model to the next.

Getting started

To use the examples:

  1. Create a service account and assign it the ai.languageModels.user role.

  2. Get and save the service account's API key with yc.ai.foundationModels.execute for its scope.

    The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.

    Note

    If you are using Windows, we recommend installing the WSL shell first and using it to proceed.

  3. Install Python 3.10 or higher.

  4. Optionally, install Python venv to create isolated virtual environments in Python.

  5. Optionally, create a new Python virtual environment and activate it:

    python3 -m venv new-env
        source new-env/bin/activate

  6. Use the pip package manager to install the ML SDK library:

    pip install yandex-ai-studio-sdk

Get API authentication credentials as described in Authentication with the Yandex Cloud AI Studio API.

To use the examples, install cURL.

Calculating prompt size

The example below estimates the size of a prompt to a YandexGPT model.

  1. Create a file named token.py and paste the following code into it:

    #!/usr/bin/env python3
        
        from __future__ import annotations
        from yandex_ai_studio_sdk import AIStudio
        
        messages = "Generative models are managed using prompts. A good prompt should contain the context of your request to the model (instruction) and the actual task the model should complete based on the provided context. The more specific your prompt is, the more accurate the model's output is going to be."
        
        
        def main():
            sdk = AIStudio(
                folder_id="<folder_ID>",
                auth="<API_key>",
            )
        
            model = sdk.models.completions("yandexgpt")
        
            result = model.tokenize(messages)
        
            for token in result:
                print(token)
        
        
        if __name__ == "__main__":
            main()

    Where:

    Note

    As input data for a request, Yandex Cloud ML SDK can accept a string, a dictionary, an object of the TextMessage class, or an array containing any combination of these data types. For more information, see Yandex Cloud ML SDK usage.

    • messages: Message text.
    • model: Model URI. For more information, see Chat domain.

  2. Run the file you created:

    python3 token.py

    The request will return a list of all received tokens.

    Result
    {"tokens":
            [{"id":"1","text":"\u003cs\u003e","special":true},
            {"id":"6010","text":"▁Gener","special":false},
            {"id":"1748","text":"ative","special":false},
            {"id":"7789","text":"▁models","special":false},
            {"id":"642","text":"▁are","special":false},
            {"id":"15994","text":"▁managed","special":false},
            {"id":"1772","text":"▁using","special":false},
            {"id":"80536","text":"▁prompts","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"379","text":"▁A","special":false},
            {"id":"1967","text":"▁good","special":false},
            {"id":"19099","text":"▁prompt","special":false},
            {"id":"1696","text":"▁should","special":false},
            {"id":"11195","text":"▁contain","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"7210","text":"▁context","special":false},
            {"id":"346","text":"▁of","special":false},
            {"id":"736","text":"▁your","special":false},
            {"id":"4104","text":"▁request","special":false},
            {"id":"342","text":"▁to","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"355","text":"▁(","special":false},
            {"id":"105793","text":"instruction","special":false},
            {"id":"125855","text":")","special":false},
            {"id":"353","text":"▁and","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"9944","text":"▁actual","special":false},
            {"id":"7430","text":"▁task","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"1696","text":"▁should","special":false},
            {"id":"7052","text":"▁complete","special":false},
            {"id":"4078","text":"▁based","special":false},
            {"id":"447","text":"▁on","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"6645","text":"▁provided","special":false},
            {"id":"7210","text":"▁context","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"671","text":"▁The","special":false},
            {"id":"1002","text":"▁more","special":false},
            {"id":"4864","text":"▁specific","special":false},
            {"id":"736","text":"▁your","special":false},
            {"id":"19099","text":"▁prompt","special":false},
            {"id":"125827","text":",","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"1002","text":"▁more","special":false},
            {"id":"16452","text":"▁accurate","special":false},
            {"id":"912","text":"▁will","special":false},
            {"id":"460","text":"▁be","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"4168","text":"▁results","special":false},
            {"id":"13462","text":"▁returned","special":false},
            {"id":"711","text":"▁by","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"3","text":"[NL]","special":true},
            {"id":"29083","text":"▁Apart","special":false},
            {"id":"728","text":"▁from","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"19099","text":"▁prompt","special":false},
            {"id":"125827","text":",","special":false},
            {"id":"1303","text":"▁other","special":false},
            {"id":"4104","text":"▁request","special":false},
            {"id":"9513","text":"▁parameters","special":false},
            {"id":"912","text":"▁will","special":false},
            {"id":"8209","text":"▁impact","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"125886","text":"'","special":false},
            {"id":"125811","text":"s","special":false},
            {"id":"5925","text":"▁output","special":false},
            {"id":"2778","text":"▁too","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"7597","text":"▁Use","special":false},
            {"id":"12469","text":"▁Foundation","special":false},
            {"id":"27947","text":"▁Models","special":false},
            {"id":"118637","text":"▁Playground","special":false},
            {"id":"2871","text":"▁available","special":false},
            {"id":"728","text":"▁from","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"7690","text":"▁management","special":false},
            {"id":"15302","text":"▁console","special":false},
            {"id":"342","text":"▁to","special":false},
            {"id":"2217","text":"▁test","special":false},
            {"id":"736","text":"▁your","special":false},
            {"id":"14379","text":"▁requests","special":false},
            {"id":"125820","text":".","special":false}],
        "modelVersion":"23.10.2024"
        }

  1. Create a file named tbody.json with the request parameters:

    {
          "modelUri": "gpt://<folder_ID>/yandexgpt",
          "text": "Generative models are managed using prompts. A good prompt should contain the context of your request to the model (instruction) and the actual task the model should complete based on the provided context. The more specific your prompt is, the more accurate the model's output is going to be."
        }

    Where <folder_ID> is the ID of the Yandex Cloud folder for which your account has the ai.languageModels.user role or higher.

  2. Send a request to the model:

    export IAM_TOKEN=<IAM_token>
        curl --request POST \
          --header "Authorization: Bearer ${IAM_TOKEN}" \
          --data "@tbody.json" \
          "https://ai.api.cloud.yandex.net/foundationModels/v1/tokenize"

    Where:

    • <IAM_token>: Value of the IAM token you got for your account.
    • tbody.json: JSON file with the request parameters.

    The request will return a list of all received tokens.

    Result
    {"tokens":
            [{"id":"1","text":"\u003cs\u003e","special":true},
            {"id":"6010","text":"▁Gener","special":false},
            {"id":"1748","text":"ative","special":false},
            {"id":"7789","text":"▁models","special":false},
            {"id":"642","text":"▁are","special":false},
            {"id":"15994","text":"▁managed","special":false},
            {"id":"1772","text":"▁using","special":false},
            {"id":"80536","text":"▁prompts","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"379","text":"▁A","special":false},
            {"id":"1967","text":"▁good","special":false},
            {"id":"19099","text":"▁prompt","special":false},
            {"id":"1696","text":"▁should","special":false},
            {"id":"11195","text":"▁contain","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"7210","text":"▁context","special":false},
            {"id":"346","text":"▁of","special":false},
            {"id":"736","text":"▁your","special":false},
            {"id":"4104","text":"▁request","special":false},
            {"id":"342","text":"▁to","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"355","text":"▁(","special":false},
            {"id":"105793","text":"instruction","special":false},
            {"id":"125855","text":")","special":false},
            {"id":"353","text":"▁and","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"9944","text":"▁actual","special":false},
            {"id":"7430","text":"▁task","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"1696","text":"▁should","special":false},
            {"id":"7052","text":"▁complete","special":false},
            {"id":"4078","text":"▁based","special":false},
            {"id":"447","text":"▁on","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"6645","text":"▁provided","special":false},
            {"id":"7210","text":"▁context","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"671","text":"▁The","special":false},
            {"id":"1002","text":"▁more","special":false},
            {"id":"4864","text":"▁specific","special":false},
            {"id":"736","text":"▁your","special":false},
            {"id":"19099","text":"▁prompt","special":false},
            {"id":"125827","text":",","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"1002","text":"▁more","special":false},
            {"id":"16452","text":"▁accurate","special":false},
            {"id":"912","text":"▁will","special":false},
            {"id":"460","text":"▁be","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"4168","text":"▁results","special":false},
            {"id":"13462","text":"▁returned","special":false},
            {"id":"711","text":"▁by","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"3","text":"[NL]","special":true},
            {"id":"29083","text":"▁Apart","special":false},
            {"id":"728","text":"▁from","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"19099","text":"▁prompt","special":false},
            {"id":"125827","text":",","special":false},
            {"id":"1303","text":"▁other","special":false},
            {"id":"4104","text":"▁request","special":false},
            {"id":"9513","text":"▁parameters","special":false},
            {"id":"912","text":"▁will","special":false},
            {"id":"8209","text":"▁impact","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"2718","text":"▁model","special":false},
            {"id":"125886","text":"'","special":false},
            {"id":"125811","text":"s","special":false},
            {"id":"5925","text":"▁output","special":false},
            {"id":"2778","text":"▁too","special":false},
            {"id":"125820","text":".","special":false},
            {"id":"7597","text":"▁Use","special":false},
            {"id":"12469","text":"▁Foundation","special":false},
            {"id":"27947","text":"▁Models","special":false},
            {"id":"118637","text":"▁Playground","special":false},
            {"id":"2871","text":"▁available","special":false},
            {"id":"728","text":"▁from","special":false},
            {"id":"292","text":"▁the","special":false},
            {"id":"7690","text":"▁management","special":false},
            {"id":"15302","text":"▁console","special":false},
            {"id":"342","text":"▁to","special":false},
            {"id":"2217","text":"▁test","special":false},
            {"id":"736","text":"▁your","special":false},
            {"id":"14379","text":"▁requests","special":false},
            {"id":"125820","text":".","special":false}],
        "modelVersion":"23.10.2024"
        }

See also

Previous
Batch processing
Next
Implementing knowledge base search