Speech Recognition on Office 365 SharePoint with Azure Cognitive Service Speech API

In this post, let us see how speech recognition can be implemented using Microsoft Speech API on SharePoint portals using JavaScript client side libraries/SDKs. This shows how speech to text can be converted on Office 365 SharePoint using Azure Cognitive Services.

In our previous post, we have seen implementing Speech recognition using browser SpeechRecognition objects on SharePoint portals. This post is special for those people who love implementing speech recognition using Microsoft Speech API (Azure Cognitive service speech API).

Note: If you are interested only in the implementation, scroll down to the bottom section. :)

Azure Cognitive Service - Bing Speech API

Microsoft Speech API supports both speech to text and text to speech conversions. In this case, only we are focusing on speech to text conversion. Microsoft Speech API provides two approaches of speech to text conversion. One using the REST API and the other way is using the client libraries. We will be leveraging the client libraries, which provides speech SDK bundles.

Speech Recognition on Office 365 SharePoint

Let us look how the speech recognition can be implemented on Office 365 SharePoint portals. The article contains the introduction to speech recognition service (speech to text conversion), detailed approach for SharePoint, code and snapshots for easier implementation.

Speech Recognition

Speech recognition helps recognizing the real-time audio from the microphone and converts it to the respective text. This kind of interfaces helps in building the voice triggered apps like chat bots, etc.

There are two approaches of implementing speech recognition on SharePoint.

• First using the Speech Recognition interfaces.
• Other way is using the Azure Bing Speech API. It is built on top of WebSockets API. The Speech SDK is available as extensions, which can be leveraged for development.

Let us go with the first approach in this post. We are going to see how the speech recognition interfaces are integrated on to SharePoint applications. In my future articles, I will detail out the integration of Bing Speech API (second approach).

Text Translation on Office 365 SharePoint

Let us look how the text added to Office 365 SharePoint list are translated and saved back to SharePoint using Microsoft Flow and Translator.

Consider a business scenario of having multi-lingual content to be available in SharePoint list and you need write the text in all the required languages. Instead, Microsoft Flow can be leveraged for translating the content on Office 365 SharePoint for the required languages with the help of Translator service.

There are several ways available for implementing the above scenario. The best possible solution is explained in this post.

Note: This article only focuses on translating content present on SharePoint lists. For enabling multilingual feature for whole portal, you can refer to the Microsoft article here.

Building SharePoint Webhook Services with Serverless Architecture

Before looking out at the implementation of webhook models in SharePoint, here let us look at how Serverless web hooks concept can be feasible.

What is Serverless architecture? It allows us to build the required services, without the need of servers. Some Azure services like Azure functions helps building such Serverless models.

The components like webhook service, storage queues, web jobs and SQL database are hosted on Azure. The following picture depicts the Serverless webhooks process, with the help of Serverless components. The components present on the Azure box, depicts serverless architecture model.

Serverless Architecture for SharePoint Webhook Services

Processing Events from SharePoint Webhook Services

Let us theoretically look how the event changes are retrieved from the storage queues, processed and stored on to the logs like SharePoint history lists.

In the previous post, you have seen how to subscribe to the event changes on SharePoint online and how the events are stored on the storage queue. Here, you could see how the events stored can be processed.

Process Data from the Queue - Web Jobs

If we closely look in the previous post, SharePoint service has sent few details to the notification service when the event change had happened on SharePoint. In the details, you can only see what has changed at high level. That is, which list has changes. It doesn’t define, what are the changes.

Event Change Details