Streaming speech recognition

In streaming mode, you can send audio for recognition and get recognition results within a single connection. You can also get intermediate recognition results while the speaker has not yet finished the utterance. After a pause, SpeechKit will return the final results and start recognizing the next utterance.

Voice assistants and smart speakers work using this recognition mode. When you activate the assistant, it starts transmitting speech to the server for recognition. The server processes the data and returns the intermediate and final recognition results for each utterance. The intermediate results show the recognition progress. After the final results, the assistant performs an action, such as playing music or calling another person.

Streaming recognition restrictionsStreaming recognition restrictions

SpeechKit streaming recognition has a number of restrictions to consider when creating an application. For the full list of SpeechKit restrictions, see Quotas and limits in SpeechKit.

	Streaming recognition
Use cases	Phone assistants and robots Virtual assistants
Input data	Real-time voice
How it works	Exchanging messages with the server within a single connection
Supported APIs	gRPC v2 gRPC v3
Maximum duration of audio data	5 minutes
Maximum amount of transmitted data	10 MB
Number of recognition channels	1

Using the serviceUsing the service

To use the service, create an application that will send audio fragments and process responses with recognition results.

Client application interface codeClient application interface code

SpeechKit has the two streaming recognition API versions: API v3 and API v2. We recommend using the API v3 for new projects.

For your application to access the service, clone the Yandex Cloud API repository and generate the client interface code for your programming language from the API v2 or API v3 specification file.

Client application examples:

• Audio file streaming recognition using the API v3.
• Microphone speech streaming recognition using the API v3.
• Example of using the streaming recognition API v2.

See also the gRPC guides for detailed instructions on how to generate interfaces and implement client apps in various programming languages.

To get the entire response, increase the maximum message size limit:

• For Go, use the MaxCallRecvMsgSize function.
• For C++, in the call method, set the max_receive_message_size value.

Authentication in the serviceAuthentication in the service

In each request, the application must provide an IAM token or API key for authentication in the service, as well as the ID of the folder for which the account has the ai.speechkit-stt.user role or higher. For more on required permissions, see Access management.

The most straightforward way to authenticate an application is to use a service account. When authenticating as a service account, do not indicate the folder ID in your requests: SpeechKit will use the folder where the service account was created.

Learn more about authentication in SpeechKit.

Recognition requestRecognition request

For speech recognition, the application must first send a message with recognition settings:

• For the API v3, the RecognizeStreaming message of the session_options type.
• For the API v2, the StreamingRecognitionRequest message of the RecognitionConfig type.

When the session is set up, the server will wait for messages with audio fragments (chunks). Send the RecognizeStreaming message of the session_options type or the StreamingRecognitionRequest message of the audio_content type to the API v2. Consider the following recommendations when sending messages:

• Do not send audio fragments too often or infrequently. The interval between messages to the service should be approximately the same as the length of the audio fragments you send, but no more than 5 seconds. For example, send 400 ms of audio for recognition every 400 ms.
• Maximum duration of transmitted audio for the entire session: 5 minutes.
• Maximum size of transmitted audio data: 10 MB.

If the service receives no messages within 5 seconds or the data duration or size limit is reached, the session is terminated. To continue speech recognition, establish a new connection and send a new message with the speech recognition settings.

SpeechKit will return intermediate speech recognition results before a message stream with audio fragments has finished.

Recognition resultRecognition result

In each recognition result message (StreamingResponse or StreamingRecognitionResponse), the SpeechKit server returns one or more speech chunks it was able to recognize during this period. A list of recognized text variants is specified for each chunk (alternatives).

The SpeechKit server returns recognition results with their type:

• partial: For intermediate results.

• final: For final results.

• final_refinement: For normalized final results.

  With normalization enabled, you will get the final and final_refinement results.

In the API v2, if recognition is still in progress, the results will contain the final parameter set to False.

The speech recognition completes and delivers final results upon EOU (end-of-utterance). It is a marker of an utterance end. The EOU occurs in the following cases:

• The gRPC session is terminated.

• Silence is recognized in the last speech fragment. Silence can be denoted using one of these two parameters:

  • chunk: Sound recognized as silence.
  • silence_chunk: Silence duration in milliseconds. This parameter allows you to reduce the audio packet size by excluding silence that does not require recognition.

Use casesUse cases

• Audio file streaming recognition using the API v3
• Microphone speech streaming recognition using the API v3
• Streaming speech recognition with auto language detection using the API v3
• Example of using the streaming recognition API v2

Useful linksUseful links

• Supported audio formats
• Supported languages and recognition models
• Authentication with the SpeechKit API
• API v2 for streaming recognition
• API v3 reference

Folder: Space containing the Yandex Cloud resources. To authenticate to AI Studio, you need a folder ID. To get it, in the AI Studio interface, hover over the folder name at the top of the screen and click . The ID will be copied to the clipboard.