Streaming synthesis

Yandex SpeechKit allows you to send texts for streaming synthesis one by one within a single session using the API v3.

The example uses the following synthesis parameters:

• Synthesized audio file format: LPCM with a sample rate of 22,050 Hz, WAV container (default).
• Volume normalization: LUFS (default).

To convert and record the result, you will need the grpcio-tools and pathlib packages.

Authentication is performed under a service account using an API key or IAM token. To learn more about SpeechKit API authentication, see Authentication with the SpeechKit API.

To synthesize multiple texts one by one:

1. Create a service account to work with the SpeechKit API.

2. Assign the service account the ai.speechkit-tts.user role or higher for the folder where it was created.

3. Clone the Yandex Cloud API repository:

   git clone https://github.com/yandex-cloud/cloudapi
           

4. Install the grpcio-tools and pathlib packages using the pip package manager:

   pip install grpcio-tools && \
           pip install pathlib
           

   You need the grpcio-tools package to generate the interface code for the API v3 synthesis client and the pathlib package to process the audio files you get.

5. In the directory hosting the cloned API repository, create a directory named output, and generate the client interface code there:

   cd <path_to_cloudapi_directory>
           mkdir output
           python3 -m grpc_tools.protoc -I . -I third_party/googleapis \
             --python_out=output \
             --grpc_python_out=output \
             google/api/http.proto \
             google/api/annotations.proto \
             yandex/cloud/api/operation.proto \
             google/rpc/status.proto \
             yandex/cloud/operation/operation.proto \
             yandex/cloud/validation.proto \
             yandex/cloud/ai/tts/v3/tts_service.proto \
             yandex/cloud/ai/tts/v3/tts.proto
           

   This will create the tts_pb2.py, tts_pb2_grpc.py, tts_service_pb2.py, and tts_service_pb2_grpc.py client interface files, as well as dependency files, in the output directory.

6. Create a file (e.g., test.py) in the root of the output directory, and add the following code to it:

   import pathlib
           import grpc
           import yandex.cloud.ai.tts.v3.tts_pb2 as tts_pb2
           import yandex.cloud.ai.tts.v3.tts_service_pb2_grpc as tts_service_pb2_grpc
           
           # Parameters for authentication with the API key
           YANDEX_API_KEY = "<API_key>"
           # Parameters for authentication with an IAM token
           # YANDEX_IAM_TOKEN = "<IAM_token>"
           
           def stream_synthesis():
               cred = grpc.ssl_channel_credentials()
           
               auth_meta = (
                   ('authorization',
                   f'Api-key {YANDEX_API_KEY}'),
                   # or
                   # ('authorization',
                   # f'Bearer {YANDEX_IAM_TOKEN}'),
               )
               with grpc.secure_channel('tts.api.cloud.yandex.net:443', cred) as channel:
                   stub = tts_service_pb2_grpc.SynthesizerStub(channel)
           
                   # Synthesis settings
                   synthesis_options = tts_pb2.SynthesisOptions(
                       voice="masha",
                       role="good",
                       output_audio_spec=tts_pb2.AudioFormatOptions(
                           # Output audio settings. Specify using the RawAudio or ContainerAudiio objects.
                           
                           # raw_audio=tts_pb2.RawAudio(
                           #     audio_encoding=tts_pb2.RawAudio.LINEAR16_PCM,
                           #     sample_rate_hertz=16000
                           # )
                           container_audio=tts_pb2.ContainerAudio(
                               container_audio_type=tts_pb2.ContainerAudio.ContainerAudioType.WAV
                           )
                       )
                   )
           
                   # Request generator
                   def request_generator():
                       # Providing settings in the first message
                       yield tts_pb2.StreamSynthesisRequest(options=synthesis_options)
           
                       # Sending texts for synthesis one by one
                       # Each phrase you send must end with a space 
                       # so they do not turn into a single word during synthesis.
                       # Short text
                       yield tts_pb2.StreamSynthesisRequest(synthesis_input=tts_pb2.SynthesisInput(text="Hello! "))
                       # Very long text to synthesize in chunks
                       yield tts_pb2.StreamSynthesisRequest(synthesis_input=tts_pb2.SynthesisInput(
                           text=(
                               "I’m Masha. I work at Classy Service Ltd. "
                               "I can help you select your order and arrange a delivery. "
                               "I will also find the right person to assist you. I can help with the blank canvas syndrome. I can tell a joke or recite a poem. I also know the weather forecast for tomorrow."
                           )
                       ))
                       # Short text
                       yield tts_pb2.StreamSynthesisRequest(synthesis_input=tts_pb2.SynthesisInput(text="What would you like to ask?"))
           
                   # In finished applications, you will instantly play the audio you get.
                   # For example purposes, let’s create a single file from all audio recordings.
                   audio_chunks = []
                   for response in stub.StreamSynthesis(request_generator(), metadata=auth_meta):
                       if response.audio_chunk.data:
                           print(f"Audio obtained: {len(response.audio_chunk.data)} b")
                           audio_chunks.append(response.audio_chunk.data)
                       if response.text_chunk.text:
                           print("Text to turn into speech:", response.text_chunk.text)
                       print(f"Start: {response.start_ms} ms, length: {response.length_ms} ms")
           
                   synthesized_audio = b"".join(audio_chunks)
                   pathlib.Path("result.wav").write_bytes(synthesized_audio)
                   
           
           if __name__ == '__main__':
               stream_synthesis()
           

   Where:

   • <API_key>: API key with the yc.ai.speechkitTts.execute scope.

     If you are using an IAM token for authentication, modify the Python script.

7. Run the Python script you created:

   python test.py
           

   As a result, a file named result.wav with synthesized speech will be created in the cloudapi directory.