Tutorial: Using LIFTI in an MVC 3 web application

Updated 25/02/2012 - it was highlighted that some of the seach phrases used towards the end of this article were not returning the expected results - that was down to me making assumptions about which words would be stemmed - these examples have been updated.

This tutorial will take you through an end-to-end implementation of a simple web site to manage a list of employees. LIFTI will be used to perform full text searching of plain text information associated to the employees.

Whilst the site will be built upon MVC 3, Entity Framework Code First and Ninject, the use of these technologies is largely arbitrary - you should be able to switch them out for any other appropriate frameworks.

Important: This tutorial relies on you having nuget installed - trust me, it makes the process of setting up your project dependencies so much easier.

Getting started

To get started, create a new empty MVC 3 application so you have a basic structure for the web application to be built from.

Now use nuget to add LIFTI - open the Package Manager Console (View/Other Windows/Package Manager Console) and type:

1
2Install-Package LIFTI

The LIFTI assembly will be downloaded and added as a reference to your project.

Install the EntityFramework.SqlServerCompact and Ninject.MVC3 packages using the package manager:

1
2Install-Package EntityFramework.SqlServerCompact
3Install-Package Ninject.MVC3

Adding these packages will automatically pull through all these packages (some of them are the dependencies of the two you added):

• EntityFramework
• WebActivator
• SqlServerCompact
• EntityFramework.SqlServerCompact
• Ninject
• Ninject.MVC3

Both Ninject.MVC3 and EntityFramework.SqlServerCompact packages add code files into a folder called App_Start. The SqlServerCompact class configures the default connection factory for the entity framework library to use the SQL Server Compact connection factory; the NinjectMVC3 class allows you to configure your dependency injection - you’ll come onto that later.

Create a model and data context

Add Employee.cs to the Models folder of the project containing:

 1
 2public class Employee
 3{
 4    [Key]
 5    public int EmployeeId { get; set; }
 6
 7    [StringLength(100), Required]
 8    public string Name { get; set; }
 9
10    [Required]
11    public DateTime DateOfEmployment { get; set; }
12
13    public string Notes { get; set; } 
14}

Add EmployeeDataContext.cs to the Models folder:

1
2public class EmployeeDataContext : DbContext
3{
4    public DbSet<Employee> Employees
5    {
6        get;
7        set;
8    }
9}

Scaffold out the site

Make sure that you have built the project and right-click on the Controllers folder, selecting Add Controller…:

• Name the controller EmployeesController
• Make sure the template “Controller with read/write actions and views…” is selected
• Select Employee as the model
• Select EmployeeDataContext as the data context
• Press Add

The scaffolded actions and views will be created for you (very handy in tutorials like this!).

Before you run the project, make sure you have added the App_Data ASP.NET folder to the project by right clicking on the project and selecting Add > Add ASP.NET Folder > App_Data.

At this point you should be able to run the project to make sure that everything you’ve done so far is correct. You should be able to fire up the project and navigate to http://localhost:xxxx/Employees where xxxx is the port number for your project:

Ok, so it doesn’t look very sexy, but it’s a website with a SQL Server Compact database sitting behind it, all up and running in just a few steps.

Introducing LIFTI

The full text index that you will be using is an updatable full text index that will contain the IDs of employees indexed against the text in their notes. More specifically, this will be an instance of PersistedFullTextIndex - this type of index is backed by a file store, which means that if the web application is stopped and started, the index will not lose all its data and will be able to pick up where it left off. Significantly this means that you will be able to keep the index and database in sync, assuming that whenever the database is updated you also update the index.

Under most circumstances there should only ever be one instance of your index in memory. This means that all of your code, whatever thread it is on, should interact with the same index (don’t worry, LIFTI’s implementation of the full text index is thread safe.). You could implement this is any number of ways:

• Using the singleton pattern
• Storing the index in a static variable somewhere that can be access by the depending code
• Storing the index in Application state
• Using dependency injection to provide one common index to any depending code

Like all good developers these days, I’m sure you’ll want to take the dependency injection route! The first step to this is to add your index into the dependency injection framework.

Open App_Start\NinjectMVC3.cs and change the RegisterServices method so it looks like this:

 1
 2private static void RegisterServices(IKernel kernel)
 3{
 4    string filePath = Path.Combine(
 5        (string)AppDomain.CurrentDomain.GetData("DataDirectory"), 
 6        "Index.dat");
 7
 8    kernel.Bind<IUpdatableFullTextIndex<int>>()
 9        .To<PersistedFullTextIndex<int>>()
10        .InSingletonScope()
11        .WithConstructorArgument("backingFilePath", filePath)
12        .OnActivation((IUpdatableFullTextIndex<int> i) =>
13        {
14            i.WordSplitter = new StemmingWordSplitter();
15            i.QueryParser = new LiftiQueryParser();
16        });
17}

The filePath variable is configured so that the index data file (index.dat) will be stored in the App_Data folder for the application. (that’s where “DataDirectory” points by default in a web application.)

Then the Ninject kernel is instructed that:

• Whenever an instance of IUpdatableFullTextIndex is requested (Bind)
• Map it to an instance of PersistedFullTextIndex (To)
• And re-use it globally, i.e. only ever create one instance (InSingletonScope)
• When it is constructed, pass the path to the index data to the “backingFilePath” parameter (WithConstructorArgument)
• And finally, when it is activated, set the WordSplitter and QueryParser properties to instances of StemmingWordSplitter and LiftiQueryParser, respectively. (OnActivation)

Now you just have to consume the index and use it in the controller.

Updating the index

Open the EmployeesController class and add a constructor:

1
2private IUpdatableFullTextIndex<int> index;
3public EmployeesController(IUpdatableFullTextIndex<int> index)
4{
5    this.index = index;
6}

Ninject (in conjunction with the nice dependency resolution in MVC 3) will take care of providing your controller with the relevant instance of the index whenever it is constructed.

There are 3 places in the controller that you need to interact with the index to keep it in sync with the database:

• Creating an employee - the index should be updated if notes are provided for the employee
• Updating an employee - the index should be updated if the employee has notes, or have any indexed text removed if the employee no longer has any notes
• Deleting an employee - any previously indexed notes for the employee should be removed

Update the HttpPost Create method so it looks like this:

 1
 2[HttpPost]
 3public ActionResult Create(Employee employee)
 4{
 5    if (ModelState.IsValid)
 6    {
 7        db.Employees.Add(employee);
 8        db.SaveChanges();
 9
10     if (!String.IsNullOrEmpty(employee.Notes))
11        {
12            this.index.Index(employee.EmployeeId, employee.Notes);
13        }
14
15        return RedirectToAction("Index");  
16    }
17
18    return View(employee);
19}

Then the HttpPost Edit method:

 1
 2[HttpPost]
 3public ActionResult Edit(Employee employee)
 4{
 5    if (ModelState.IsValid)
 6    {
 7        db.Entry(employee).State = EntityState.Modified;
 8        db.SaveChanges();
 9
10     if (String.IsNullOrEmpty(employee.Notes))
11        {
12            this.index.Remove(employee.EmployeeId);
13        }
14        else
15        {
16            this.index.Index(employee.EmployeeId, employee.Notes);
17        }
18
19        return RedirectToAction("Index");
20    }
21    return View(employee);
22}

And finally, the HttpPost Delete method:

 1
 2[HttpPost, ActionName("Delete")]
 3public ActionResult DeleteConfirmed(int id)
 4{            
 5    Employee employee = db.Employees.Find(id);
 6    db.Employees.Remove(employee);
 7    db.SaveChanges();
 8
 9 this.index.Remove(employee.EmployeeId);
10
11    return RedirectToAction("Index");
12}

Try the application out again, you should be able to create, update and delete employees without any problems - the full text index will be built up in the background.

Searching for employees

The last step is to allow users of your site to search for interesting text within the employee notes.

Update the Views\Employees\index.cshtml file with a search textbox just after the tag:

 1
 2<h2>Index</h2>
 3
 4@using (Html.BeginForm()) {
 5<p>
 6    @Html.Label("searchCriteria", "Search employee notes:")
 7    @Html.TextBox("searchCriteria") 
 8    <input type="submit" value="Search" />
 9</p>
10}

Now add a new method to the EmployeesController to handle the posting of the search text:

 1
 2[HttpPost]
 3public ViewResult Index(string searchCriteria)
 4{
 5    if (String.IsNullOrEmpty(searchCriteria))
 6    {
 7        return Index();
 8    }
 9
10    var matchingIds = this.index.Search(searchCriteria).ToArray();
11    var employees = this.db.Employees
12        .Where(e => matchingIds.Contains(e.EmployeeId))
13        .ToList();
14
15    return View(employees);
16}

The astute amongst you will notice that the data context isn’t injected in the same way that the index is - good spot. This tutorial is long enough, so I’ll leave that as an exercise for you to fix up if it’s bugging you that much!

Testing the application

Build and run the project create these employees:

Try some of these search criteria on the index page:

doubts

Andy is obviously matched - in his notes he has “doubts” specified exactly. However there’s more going on here because Tracy is also matched. This is because you’re using the stemming word splitter process words in the index, and that automatically removes some word suffixes, such as the “s” from “doubts” and “ing” in Tracy’s “doubting” - this means that when you searched for “doubts” you were actually searching for anything that stemmed to “doubt”.

emp

Nothing will come back - not even Tracy. This is because the default behaviour for the LIFTI query parser is to match words exactly.

emp*

Tracy will be returned - she is the only employee with notes containing a word that starts with “emp”.

lunch & break (or lunch break)

Both Ralph and Bob will be returned - both of these contain derivatives of “lunch” and “break” in their notes.

“lunch break”

Only Ralph contains a phrase that contains “lunch” followed immediately by a derivative of “break”.

… your search criteria here

The LIFTI search engine is quite powerful and there are loads of different search permutations you could try out, so try creating a few more employees and searching using some of the other operators, such as or (|) or near (~).