Processing Events from SharePoint Webhook Services

Let us theoretically look how the event changes are retrieved from the storage queues, processed and stored on to the logs like SharePoint history lists.

In the previous post, you have seen how to subscribe to the event changes on SharePoint online and how the events are stored on the storage queue. Here, you could see how the events stored can be processed.

Process Data from the Queue - Web Jobs

If we closely look in the previous post, SharePoint service has sent few details to the notification service when the event change had happened on SharePoint. In the details, you can only see what has changed at high level. That is, which list has changes. It doesn’t define, what are the changes.

Event Change Details

Here, you can only get the resource id details but not the exact list item change details. These changes are saved to the storage queues, as depicted in the previous post.

The web jobs can be written for processing the data present in the queue. The web jobs can run continuously or on demand to get the queue data and query back the SharePoint service for the changes. The following snapshot shows the detailed flow.

Web jobs processing Event Changes

In the architecture above, 

• You can see the web jobs pulling the data (SharePoint list details) from the storage queue.
• Here, the web job can build the query for the corresponding SharePoint list looking out for changes in the particular time duration. Change token is built, and passed along the query.
• SharePoint service responds back to the web jobs component with the changes happened in the particular time span for that particular list.
• Then on the web jobs, the mechanism should be written to loop through the changes set and the changes are processed.
• Then the changes are saved on some storage place as logs. The storage place can be anything like SharePoint lists or other suitable storage places.

Efficient Way of Processing Events – Saving the Change Tokens

In the previous step, you could have seen accessing data for particular time duration. Instead of relying on the time duration for building change tokens, the change tokens can be saved on storage like SQL databases or tables. Then the change token values can be retrieved from the storage and sent along the queries. This will help in removing the duplicates data, since the query will always get only the changes from the latest change.

Web Jobs accesses Change Tokens from Databases

1. As you could see, first time change token is retrieved manually using the request (like postman tool) and saved on to the storage like SQL database.
2. Post that, the following flow repeats as and when storage queue is updated with the event change details.
• Web job will retrieve the last change token value from the SQL database.
• Then web job queries for the changes from SharePoint and process the changes as explained in the previous post.
• Once the events are processed, the current change token value will be stored on to SQL database for the next run.
3. The above step (Step 2) will be repeated, whenever there is event update on the subscribed list.

Next, i will hint on going with the serverless components and let us look practically on subscribing to the event changes and processing the events from storage queues.