Friday, 16 March 2018

Building SharePoint Webhook Services with Serverless Architecture

Before looking out at the implementation of webhook models in SharePoint, here let us look at how Serverless web hooks concept can be feasible.

What is Serverless architecture? It allows us to build the required services, without the need of servers. Some Azure services like Azure functions helps building such Serverless models.

The components like webhook service, storage queues, web jobs and SQL database are hosted on Azure. The following picture depicts the Serverless webhooks process, with the help of Serverless components. The components present on the Azure box, depicts serverless architecture model.

Serverless Architecture for SharePoint Webhook Services
Serverless Architecture for SharePoint Webhook Services


Required Components for Building Serverless SharePoint Webhooks:


To implement Serverless webhooks for SharePoint online events, the following are the required components.
  • SharePoint Online Subscription
  • Azure Subscription
    • AD tenant – For enabling permissions
    • Azure Functions – Building the system to receive the notifications
    • Azure Webjobs – To process the data stored on queues
    • Azure Storage – Queues, Databases
  • Postman client – For sending the requests (request test tool on desktops) 

In the previous post, we have seen theoretically how the events can be processed from SharePoint services.

The previous posts might help in understanding the webbook services better.



Process the Notifications sent by SharePoint service


The webhook service requires two components.
  • Create the Azure Function,
  • Create the Azure storage queue.

The Azure function acts as webhook service, which will process the notifications sent by SharePoint service. And the notifications will be pushed to the Azure storage queue.

The following code snippet depicts the above flow.
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;
using System;
using Newtonsoft.Json;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Queue;
using System.Collections.Generic;
namespace spschennai.spwebhooks
{
public static class webhookserviceforstoragequeue
{
[FunctionName("webhookserviceforstoragequeue")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
log.Info($"Webhook was triggered!");
// Grab the validationToken URL parameter
string validationToken = req.GetQueryNameValuePairs()
.FirstOrDefault(q => string.Compare(q.Key, "validationtoken", true) == 0)
.Value;
// If a validation token is present, we need to respond within 5 seconds by
// returning the given validation token. This only happens when a new
// web hook is being added
if (validationToken != null)
{
log.Info($"Validation token {validationToken} received");
var response = req.CreateResponse(HttpStatusCode.OK);
response.Content = new StringContent(validationToken);
return response;
}
log.Info($"SharePoint triggered our webhook...great :-)");
var content = await req.Content.ReadAsStringAsync();
log.Info($"Received following payload: {content}");
var notifications = JsonConvert.DeserializeObject<ResponseModel<NotificationModel>>(content).Value;
log.Info($"Found {notifications.Count} notifications");
if (notifications.Count > 0)
{
log.Info($"Processing notifications...");
foreach (var notification in notifications)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("Azure Storage Account Connection String");
// Get queue... create if does not exist.
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueClient.GetQueueReference("webhooksdata");
queue.CreateIfNotExists();
// add message to the queue
string message = JsonConvert.SerializeObject(notification);
log.Info($"Before adding a message to the queue. Message content: {message}");
queue.AddMessage(new CloudQueueMessage(message));
log.Info($"Message added :-)");
}
}
// if we get here we assume the request was well received
return new HttpResponseMessage(HttpStatusCode.OK);
}
}
// supporting classes
public class ResponseModel<T>
{
[JsonProperty(PropertyName = "value")]
public List<T> Value { get; set; }
}
public class NotificationModel
{
[JsonProperty(PropertyName = "subscriptionId")]
public string SubscriptionId { get; set; }
[JsonProperty(PropertyName = "clientState")]
public string ClientState { get; set; }
[JsonProperty(PropertyName = "expirationDateTime")]
public DateTime ExpirationDateTime { get; set; }
[JsonProperty(PropertyName = "resource")]
public string Resource { get; set; }
[JsonProperty(PropertyName = "tenantId")]
public string TenantId { get; set; }
[JsonProperty(PropertyName = "siteUrl")]
public string SiteUrl { get; set; }
[JsonProperty(PropertyName = "webId")]
public string WebId { get; set; }
}
public class SubscriptionModel
{
[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
public string Id { get; set; }
[JsonProperty(PropertyName = "clientState", NullValueHandling = NullValueHandling.Ignore)]
public string ClientState { get; set; }
[JsonProperty(PropertyName = "expirationDateTime")]
public DateTime ExpirationDateTime { get; set; }
[JsonProperty(PropertyName = "notificationUrl")]
public string NotificationUrl { get; set; }
[JsonProperty(PropertyName = "resource", NullValueHandling = NullValueHandling.Ignore)]
public string Resource { get; set; }
}
}
view raw webhookserviceforstoragequeue.cs hosted with ❤ by GitHub


Subscribing to the Event Notifications


To access the SharePoint list item for subscription or from services related to webhooks, authentication token has to be generated from the Azure AD. 
  • Register the application on Azure AD, with the necessary reply URL and providing read/write permissions on the SharePoint lists.
  • Or you need to hardcode the credentials for accessing the SharePoint list data. 
We will go with the first option. So before hitting the subscription post query, we need to have the access token generated for oAuth authentications.

Postman tool can be used for subscribing to the events. This step will help registering the webhook service (built in the previous step) on the SharePoint list for event changes.

Paste the subscription URL of the list and select the action as POST.
https://nakkeerann.sharepoint.com/_api/web/lists('9AC9CF26-55E2-46DE-96B7-310714417132')/subscriptions

Get the access token, with the following parameters passed.


Get oAuth Token
Generate oAuth Token for Subscriptions

Once you have the oAuth token, use it for subscription. Then using the post request with the following parameters, register the webhook service.
  • Headers – Accept : application/json;odata=nometadata
  • Content-Type : application/json
  • Request Body will contain the 
    • Resource URL
    • Notification URL
    • Expiration date time and 
    • Client state (optional).
           {
  "resource": "https://nakkeerann.sharepoint.com/_api/web/lists('9AC9CF26-55E2-46DE-96B7-310714417132')",
  "notificationUrl": "https://azurefunctionurl/api/webhookservice",
  "expirationDateTime": "2018-04-27T12:11:37+00:00",
  "clientState": "GUID"
              }
Subscribe Webhook Service for Event Changes
Subscribe Webhook Service for Event Changes

Once the request is posted using postman tool, 
  • The SharePoint will send a request with the validation token to webhook service built using Azure functions. 
  • The webhook service should respond back with the valid token within 5 seconds. 
  • If that happens, SharePoint will respond back to postman tool with a success message. 

This flow ensures the registration of webhook service on SharePoint list. The response will contain
  • Subscription ID
  • Expiration date
  • Resource ID
  • Tenant ID
  • Site URL and 
  • Web ID.

Processing the notifications using WebJobs


Next, let us see how to process the notifications present on Azure storage queues and extract the change information from the respective lists. Then the changes will be saved on SharePoint lists as history or logs.

The following code snippet shows the webjob which will process the notifications present on Azure storage queue and save the changes into the logs list present in the SharePoint.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.SharePoint.Client;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Table;
using Newtonsoft.Json;
namespace spschennai.spwebhooks.jobs
{
public class Functions
{
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([QueueTrigger("webhooksdata")] string message, TextWriter log)
{
log.WriteLine("Webjob Starts! ");
log.WriteLine(message);
Console.WriteLine("Webjob start " + message + " ends");
processMessage(message, log);
log.WriteLine("Webjob Ends! ");
}
private static void processMessage(string message, TextWriter log)
{
var notification = JsonConvert.DeserializeObject<NotificationModel>(message);
Console.WriteLine($"Found {notification.Resource} notifications");
#region usercreds
string siteUrl = "https://nakkeerann.sharepoint.com/";
string userName = "abc@nakkeerann.onmicrosoft.com";
string password = "password";
#endregion
OfficeDevPnP.Core.AuthenticationManager authManager = new OfficeDevPnP.Core.AuthenticationManager();
Console.WriteLine($"Processing notifications...");
var clientContext = authManager.GetSharePointOnlineAuthenticatedContextTenant(siteUrl, userName, password);
try
{
ListCollection lists = clientContext.Web.Lists;
Guid listId = new Guid(notification.Resource);
IEnumerable<List> results = clientContext.LoadQuery<List>(lists.Where(lst => lst.Id == listId));
clientContext.ExecuteQueryRetry();
List changeList = results.FirstOrDefault();
// Logs or history list
List historyList = clientContext.Web.GetListByTitle("SPSWebHookQueueLogHistory");
if (historyList == null)
{
// Create if it doesnt exist
historyList = clientContext.Web.CreateList(ListTemplateType.GenericList, "SPSWebHookQueueLogHistory", false);
}
// Query to retrieve the changes
ChangeQuery changeQuery = new ChangeQuery(false, true);
changeQuery.Item = true;
changeQuery.FetchLimit = 1000;
ChangeToken lastChangeToken = null;
Guid id = new Guid(notification.SubscriptionId);
Console.WriteLine("Subscription ID: " + notification.SubscriptionId);
// Get the change token from the Azure storage (in this case Azure Tables)
var lastChangeTokenValue = GetLastChangeToken(id);
if(lastChangeTokenValue != null)
{
lastChangeToken = new ChangeToken();
lastChangeToken.StringValue = lastChangeTokenValue;
}
// Get the changes
bool allChangesRead = false;
do
{
if (lastChangeToken == null)
{
// If change token is not present on the Azure table, build the token for getting last minute changes
lastChangeToken = new ChangeToken();
lastChangeToken.StringValue = string.Format("1;3;{0};{1};-1", notification.Resource, DateTime.Now.AddMinutes(-1).ToUniversalTime().Ticks.ToString());
}
Console.WriteLine(lastChangeToken.StringValue);
// Change token
changeQuery.ChangeTokenStart = lastChangeToken;
// Execute the change query
var changes = changeList.GetChanges(changeQuery);
clientContext.Load(changes);
clientContext.ExecuteQueryRetry();
// Process the changes
if (changes.Count > 0)
{
foreach (Change change in changes)
{
lastChangeToken = change.ChangeToken;
Console.WriteLine(lastChangeToken.StringValue);
if (change is ChangeItem)
{
// Add the changes to history list on SharePoint
ListItemCreationInformation newItem = new ListItemCreationInformation();
ListItem item = historyList.AddItem(newItem);
item["Title"] = string.Format("List {0} had a Change of type {1} on the item with Id {2}.", changeList.Title, change.ChangeType.ToString(), (change as ChangeItem).ItemId);
item.Update();
clientContext.ExecuteQueryRetry();
}
}
if (changes.Count < changeQuery.FetchLimit)
{
allChangesRead = true;
}
}
else
{
allChangesRead = true;
}
// Are we done?
} while (allChangesRead == false);
UpdateLastChangeToken(id, lastChangeToken.StringValue);
}
catch (Exception ex)
{
// Log error
Console.WriteLine(ex.ToString());
}
finally
{
if (clientContext != null)
{
clientContext.Dispose();
}
}
Console.WriteLine("All Done");
}
private static CloudTable GetorCreateTable()
{
// Get or creates Azure Table for storing the change tokens
// Retrieve the storage account from the connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("Azure Storage Account Connection String");
// Create the table client.
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
// Retrieve a reference to the table.
CloudTable table = tableClient.GetTableReference("webhookchangetokens");
// Create the table if it doesn't exist.
table.CreateIfNotExists();
return table;
}
private static void UpdateLastChangeToken(Guid id, string lastChangeTokenValue)
{
// Persist or Save the last change token value into Azure Table
CloudTable tokenTable = GetorCreateTable();
TableOperation retrieveOperation = TableOperation.Retrieve<CustomerEntity>(Convert.ToString(id), "tokenchanges");
TableResult retrievedResult = tokenTable.Execute(retrieveOperation);
CustomerEntity entity = (CustomerEntity)retrievedResult.Result;
if (entity != null)
{
//update
Console.WriteLine(entity.LastChangeToken + " == " + lastChangeTokenValue);
if (!entity.LastChangeToken.Equals(lastChangeTokenValue, StringComparison.InvariantCultureIgnoreCase))
{
entity.LastChangeToken = lastChangeTokenValue;
TableOperation updateOperation = TableOperation.Replace(entity);
tokenTable.Execute(updateOperation);
Console.WriteLine("Table updated with lastChangeTokenValue UPDATE");
}
}
else
{
//add
CustomerEntity newEntity = new CustomerEntity(Convert.ToString(id), "tokenchanges");
newEntity.LastChangeToken = lastChangeTokenValue;
TableOperation newOperation = TableOperation.Insert(newEntity);
tokenTable.Execute(newOperation);
Console.WriteLine("Table updated with lastChangeTokenValue ADD");
}
}
private static string GetLastChangeToken(Guid id)
{
// Get the change token from Azure Table
CloudTable tokenTable = GetorCreateTable();
TableQuery<CustomerEntity> tableQuery =
new TableQuery<CustomerEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, Convert.ToString(id)));
tableQuery.TakeCount = 1;
Console.WriteLine("Before execution");
var entities = tokenTable.ExecuteQuery(tableQuery);
string lastChangeToken = null;
if (entities.Count() > 0)
{
CustomerEntity entity = null;
entity = entities.ElementAt(0);
lastChangeToken = entity.LastChangeToken;
}
Console.WriteLine("After execution");
Console.WriteLine(lastChangeToken);
return lastChangeToken;
}
}
// supporting classes
public class CustomerEntity : TableEntity
{
public CustomerEntity(string lastName, string firstName)
{
this.PartitionKey = lastName;
this.RowKey = firstName;
}
public CustomerEntity() { }
public string LastChangeToken { get; set; }
}
public class ResponseModel<T>
{
[JsonProperty(PropertyName = "value")]
public List<T> Value { get; set; }
}
public class NotificationModel
{
[JsonProperty(PropertyName = "subscriptionId")]
public string SubscriptionId { get; set; }
[JsonProperty(PropertyName = "clientState")]
public string ClientState { get; set; }
[JsonProperty(PropertyName = "expirationDateTime")]
public DateTime ExpirationDateTime { get; set; }
[JsonProperty(PropertyName = "resource")]
public string Resource { get; set; }
[JsonProperty(PropertyName = "tenantId")]
public string TenantId { get; set; }
[JsonProperty(PropertyName = "siteUrl")]
public string SiteUrl { get; set; }
[JsonProperty(PropertyName = "webId")]
public string WebId { get; set; }
}
public class SubscriptionModel
{
[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
public string Id { get; set; }
[JsonProperty(PropertyName = "clientState", NullValueHandling = NullValueHandling.Ignore)]
public string ClientState { get; set; }
[JsonProperty(PropertyName = "expirationDateTime")]
public DateTime ExpirationDateTime { get; set; }
[JsonProperty(PropertyName = "notificationUrl")]
public string NotificationUrl { get; set; }
[JsonProperty(PropertyName = "resource", NullValueHandling = NullValueHandling.Ignore)]
public string Resource { get; set; }
}
}
view raw Function.cs hosted with ❤ by GitHub


The following picture shows the webhook service notification changes being saved on SharePoint.

Processed Webhook Changes stored on SharePoint list
Processed Webhook Changes stored on SharePoint list

By at
Labels: , , , ,