AI driven Knowledge Management

I have been working with a large insurance company to define its knowledge management strategy and solution architecture. As I dug deeper into business scenarios, I realized that ‘Knowledge Management’ is the key area where A.I technologies will have immediate and significant impact on insurer’s bottom line.

Many of the critical insurance processes such as underwriting and claims servicing rely on unstructured documents. For example, when underwriters are evaluating risks and pricing, they need to refer to underwriting guidelines document to understand the eligibility requirements for different industries (SIC codes), classes and exposures. When claims handlers are servicing claims, they need to refer to multiple documents such as special instructions, medical documents, claims damage assessment reports and account specific guidelines etc. Most of these documents are unstructured and do not follow any particular nomenclature or versioning system. Some of the documents are generated by partners or third party providers, so there is not much control over the format as well. In this environment, it is very difficult to search the relevant and most recent documents. As a result, a lot of processing errors and wasted time!

To improve search results, many insurers map documents to pre-defined taxonomies and add metadata to the documents, however, it gets overwhelming to classify and annotate ever increasing number of documents manually.

This is where A.I can help.

Why A.I for Knowledge Management?

1. A.I will enhance the quality of search results with contextual relevance ranking
2. A.I will enhance user’s experience by suggesting google-like type-ahead based on search words that are used together. A.I can also be leveraged for developing digital assistants that can parse ‘natural language’ and understand the intent of the question and search accordingly
3. A.I will add value by recommending other relevant products/documents that typically go together
4. A.I will do the heavy lifting of classifying and annotating millions of documents to improve the relevance of search

Architecture of a Modern, A.I driven Knowledge Management System

Before I talk about the key architecture building blocks and AI algorithms, I would like to introduce a concept called ‘Knowledge graph’ that is at the heart of the solution.

Knowledge Graphs – Providing context to A.I systems

Google talked about knowledge graphs in 2012. It is one of the key reasons behind the effectiveness of Google’s search engine. It helped Google in moving from searching ‘strings’ to searching ‘things’ that are relevant to your context. For example, when you search for ‘Cold Play’, Google understands that you are searching for a British rock band and not the strings such as ‘Cold’ and ‘Play’. There are many graph databases and emerging standards that will allow implementing similar concept for enterprise knowledge management system.

Knowledge graphs capture information related to people, processes, applications, data and things, and the relationships among them. It is easy to store and traverse knowledge graphs so as to understand the ‘context’ for the search. For example, it will tell about historical searches and choices of the user.  In Insurance context, you can model knowledge graphs to indicate which documents are most relevant for the claims handlers at different stages of claims for different LOBs.   This kind of information is very important for A.I driven applications to understand the ‘context’ and query/rank results accordingly.

Building blocks of Knowledge Management Systems

There are multiple ways to architect a modern knowledge management system. The key building blocks will be as shown below:

As discussed before, the heart of the solution will be the knowledge graphs. The knowledge management system will use ‘knowledge graphs’ to refine user’s query by adding the context to the query, and also to process results by filtering/ordering/rescoring for better relevance.  [ Here is some good material that may help further. ]

Leveraging A.I algorithms

The A.I Machine learning works in the background for ensuring effectiveness (i.e. precision and recall) of the search as well as for improving user’s experience by providing type-ahead suggestions and relevant recommendation of other products/documents.

Following algorithms are typically used in the process:

1) Topic Spotting

• This is for automated mapping of documents to user defined Taxonomy categories
• Supervised machine-learning technique such as  Naïve Bayes can be leveraged to assign automatically a document into one or more predefined topics
• The performance of topic detection can be improved by using such algorithm in conjunction with techniques such as Continuous bag of words

2) Automated classification

• Although supervised machine learning algorithms are likely to be more effective, it requires a large training data set and efforts that may not be practical for everyone.
• It is possible to apply unsupervised clustering algorithms (e.g. K-Means, Latent Dirichlet Allocation, Neural Topic model etc) to aggregate documents into clusters such that each cluster represents a topic. These clusters will not necessarily map to user-defined taxonomy, and manual annotation may be required after grouping.

3) Summarization

• When the search results are presented to the users, it will be helpful to show a summary of the document so that the users can make a right selection.
• For summarization, a supervised algorithm can be leveraged. There are two types of algorithms: (1) Extractive: it is based on the extracted keywords. It is much easier, but the summary may not be grammatically correct, (2) Abstractive: It is about constructing grammatically correct sentences to summarize the document. It is much more difficult and requires a deep learning algorithm.

4) Entity Extraction

• Many times there are ‘named entities’ in the document such as persons, organizations, locations, dates, quantities, currencies, monetary values, percentages, etc.
• You may want to extract these named entities and search on it. For example, you may want to search for the agreements where one particular re-insurer and one particular broker are involved.
• To extract such entities a technique called Named Entity Recognition (NER) is leveraged.

5) Recommendation Engine

• While users are searching, it will be useful to provide a type-ahead recommendation for next set of words by observing frequency of words used together. Also, after the search, you can provide recommendations on other Products/Documents that are purchased/viewed together
• For such purpose, a data mining algorithm called APriori can be used. It is a type of unsupervised machine learning that is used to mine frequent data item-sets and relevant association rules.

Technology Choices

1. 'Do It Yourself' using GraphDBs and Machine Learning frameworks – There are many choices to construct your own solution by leveraging Graph DBs (e.g. Neo4j, AWS Neptune), Search Engines ( e.g. Elasticsearch, Solr) and Machine Learning/NLP libraries ( Google Tensorflow, sci-kit learn, NLTK etc)
2. Leverage AI driven KM Products –There are multiple Knowledge Management (KM) focused products that will make it easier to integrate with multiple data sources, search engines and AI algorithms. Few salient examples are - PoolParty, BA Insight, Sinequa etc. 

****

I hope this is useful for those rethinking about their knowledge management systems. Please let me know your thoughts and comments.