How to make Laravel and Elasticsearch become friends
This article is outdated, you can find the new relevant version here.
Laravel is my favourite framework so far. I like that it offers lots of possibilities out of the box. The Laravel team thinks over every part of their product to make the instrument that solves all the common tasks web-developers bump into during their routine.
They’ve even made a cool tool that helps you create a search engine on your site. It’s called Laravel Scout. The only unpleasant thing here is the driver that comes with the default installation of Laravel. It’s the Algolia driver. Of course, Algolia is a good and quick way to add search capabilities to your models. But I and, I suppose, many other developers would prefer to have Elasticsearch as a default driver.
Fortunately, Laravel team gave us opportunity to create our own drivers. That’s quite easy and you can find all the necessary information regarding that in the Scout docs. In this article, I’m going to use my own solution, that I’ve created being not satisfied with existing ones. I tried to follow the Laravel ideology in order to create a tool that lets you deal with Elasticsearch in a very easy and convenient way.
In this tutorial I will lead you through boring stuff, like setting up and configuring different software, to interesting part of performing search queries. We will learn how to search data in Elasticsearch, how to filter data without specifying a search string and how to perform query using related model fields.
Homestead
I prefer using Homestead for the development, a virtual machine provided by Laravel team. For me it’s a convenient way to separate a working environment from my personal software and data to prevent a clutter on my machine.
In this tutorial I’m going to use Homestead, thus all console instructions are given for Ubuntu.
If you are using VM, connect to your machine by SSH and we will start with configuring software.
Elasticsearch Installation
To perform a search request we have to install Elasticsearch first. It can be easily done with the packet manager:
// download and install the Elasticsearch Signing Key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -// install the apt-transport-https package
sudo apt-get install apt-transport-https// save the repository definition
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list// install the elasticsearch package
sudo apt-get update && sudo apt-get install elasticsearch
But as usual it’s not the only action we have to do in order to make it work.
Open the file /etc/elasticsearch/elasticsearch.yml with any editor, for example vim, and set network.host to localhost:
network.host: localhostNow we need to restart the service:
sudo service elasticsearch restartIt takes some time to restart the service. As soon as it finishes check if it works by simple http query:
curl http://localhost:9200/It has to return an answer similar to the following:
{
"name" : "UhEc17r",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "QoNneSUhTG6EJaA4LMqDlA",
"version" : {
"number" : "5.4.0",
"build_hash" : "780f8c4",
"build_date" : "2017-04-28T17:43:27.229Z",
"build_snapshot" : false,
"lucene_version" : "6.5.0"
},
"tagline" : "You Know, for Search"
}I hope that you had no troubles here and we can move on.
Creating New Laravel Project
Let’s create a fresh installation of the Laravel framework. If you use Homestead, then you have the Laravel installer out of the box, if don’t then you can install it using Composer.
In Homestead all projects are located in the home Code directory. Move to the directory and make a new project, using the Laravel installer:
cd ~/Code/
laravel new search-tutorialIn this tutorial we are going to use sqlite database, just to keep things simple. Go to the .env file end set the DB_CONNECTION option to sqlite and comment other database options out:
DB_CONNECTION=sqlite
#DB_HOST=127.0.0.1
#DB_PORT=3306
#DB_DATABASE=homestead
#DB_USERNAME=homestead
#DB_PASSWORD=secretFinally, to store data we have to create a file:
touch database/database.sqliteScout Driver
We’ve already talked a bit about Laravel Scout. It adds full-text search to your Eloquent models. I’ve also mentioned that we’re going to use a custom Scout engine, because there is no Elasticsearch driver out of the box.
Let’s install the package with Composer:
composer require babenkoivan/scout-elasticsearch-driverAs you could guess, we are going to make some configurations. Firstly, go to the config/app.php and add two strings to the providers section:
'providers' => [
//... // the Scout package itself
Laravel\Scout\ScoutServiceProvider::class,
// the driver for Elasticsearch
ScoutElastic\ScoutElasticServiceProvider::class, //...
]
Secondly, we have to publish the packages settings to configure Scout. Run the following commands in the console:
php artisan vendor:publish --provider="Laravel\Scout\ScoutServiceProvider"php artisan vendor:publish --provider="ScoutElastic\ScoutElasticServiceProvider"
Finally, append SCOUT_DRIVER=elastic to the end of the .env file.
Index Configurator
Elasticsearch Index is like a database in traditional relational model, whilst Elasticsearch Type is like a table. We have to create both of them before we can send any data to Elasticsearch. We will start from creating an index.
At first, we will make a special class — Index Configurator:
php artisan make:index-configurator TutorialIndexConfiguratorThis class describes the Tutorial index. We are free to specify either settings (for example, analyzers) or default mapping in it. But for our purposes we don’t need any special settings, we are good with the default ones.
At last, let’s create an index entity in ElasticSearch according to the index configurator:
php artisan elastic:create-index "App\TutorialIndexConfigurator"Searchable Models
This tutorial is all about adding full-text search to your Eloquent models, but we haven’t created any of them yet. In this section I’m going to fill the gap and make two entities: a book and an author.
Let’s say that each book has a title, a description, a release year and an author. An author has only a name. Let’s also establish the following simplification: an author can write several books, but a book can be written by only one author.
We can create models using the Artisan make command:
php artisan make:searchable-model Book --index-configurator="TutorialIndexConfigurator" --migrationphp artisan make:searchable-model Author --index-configurator="TutorialIndexConfigurator" --migration
Searchable models are extended versions of usual ones, but with some predefined behaviour that allows them to perform search requests in Elasticsearch.
You can notice that we’ve also made migrations and specified the index configurator for each model. This is important, because we have to be explicit regarding what Elasticsearch Index use for Elasticsearch Type.
You can find both ( Book.php and Author.php) created files in the app folder of the project. We’re going to edit them in order to specify mapping for each Ealsticsearch Type. It’s vital to set mapping, because it tells Elasticsearch how to treat fields.
Here is the full code of the Book class after editing:
<?phpnamespace App;use ScoutElastic\SearchableModel;class Book extends SearchableModel
{
// We don't want to use timestamps in this tutorial
public $timestamps = false; protected $indexConfigurator = TutorialIndexConfigurator::class;// We don't analyze numbers, all text is in English
protected $mapping = [
'properties' => [
'id' => [
'type' => 'integer',
'index' => 'not_analyzed'
],
'title' => [
'type' => 'string',
'analyzer' => 'english'
],
'description' => [
'type' => 'string',
'analyzer' => 'english'
],
'year' => [
'type' => 'integer',
'index' => 'not_analyzed'
],
'author_id' => [
'type' => 'integer',
'index' => 'not_analyzed'
]
]
]; // Each book belongs to one author
public function author()
{
return $this->belongsTo(Author::class);
}
}
The code of the Author model is below:
<?phpnamespace App;use ScoutElastic\SearchableModel;class Author extends SearchableModel
{
public $timestamps = false; protected $indexConfigurator = TutorialIndexConfigurator::class; protected $mapping = [
'properties' => [
'id' => [
'type' => 'integer',
'index' => 'not_analyzed'
],
'name' => [
'type' => 'string',
'analyzer' => 'english'
]
]
]; // Each author can write several books
public function books()
{
return $this->hasMany(Book::class);
}
}
Migrations
During model creation we’ve also made two migration files. You can find them in the database/migrations folder. Find the file, that ends with books_table.php and describe table fields as following:
<?phpuse Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;class CreateBooksTable extends Migration
{
public function up()
{
Schema::create('books', function (Blueprint $table) {
$table->increments('id');
$table->string('title');
$table->text('description');
$table->integer('year');
$table->integer('author_id');
});
} public function down()
{
Schema::dropIfExists('books');
}
}
Now find the authors_table.php file and describe authors table fields:
<?phpuse Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;class CreateAuthorsTable extends Migration
{
public function up()
{
Schema::create('authors', function (Blueprint $table) {
$table->increments('id');
$table->string('name');
});
} public function down()
{
Schema::dropIfExists('authors');
}
}
To create tables run the migrate command in the console:
php artisan migrateDummy Data
To play with the search engine we need some data. The easiest way to get it is to create some fake data. Let’s add factories for the Book model and the Author model in the database/factories/ModelFactory.php file:
<?php$factory->define(App\Author::class, function (Faker\Generator $faker) {
return [
'name' => "{$faker->firstName} {$faker->lastName}"
];
});$factory->define(App\Book::class, function (Faker\Generator $faker) {
return [
'title' => ucfirst($faker->realText(15)),
'description' => $faker->realText(200),
'year' => $faker->year,
'author_id' => function () {
// We take the first random author from the table
return App\Author::inRandomOrder()->first()->id;
}
];
});
Laravel has a cool PHP console, called Tinker. You can launch it using artisan:
php artisan tinkerLet’s create 50 authors:
factory(App\Author::class, 50)->create()and 200 books:
factory(App\Book::class, 200)->create()Great! Now we have the data to play with.
Search Queries
All preparations are done, we are ready to make our first search query. Let’s find all books that have the word Alice:
// In this query we say Elasticsearch - give us 10 records, that contains the word Alice in any field
App\Book::search('Alice')->take(20)->get()I’m sure that you will get some results, because the Faker\Generator::realText method returns a random text from the Alice in a Wonderland book.
We didn’t tell to Elasticsearch which fields have more priority, that’s why we can see a mess in the result. It would be much nicer to have records with Alice in a title at the top and records with Alice in a description at the bottom. But how can we achieve this? The answer is a search rule.
For you convenience there is a command for search rule creation:
php artisan make:search-rule BookSearchRuleA search rule is a class, that describes how search query will be performed. You have to implement only one method — buildQueryPayload. This method must return a bool query.
We want to search by title and description for now, so let’s modify the BookSearchRule class according to our needs:
<?phpnamespace App;use ScoutElastic\SearchRule;class BookSearchRule extends SearchRule
{
public function buildQueryPayload()
{
$query = $this->builder->query; return [
'should' => [
[
'match' => [
'title' => [
'query' => $query,
'boost' => 2
]
]
],
[
'match' => [
'description' => [
'query' => $query,
'boost' => 1
]
]
]
]
];
}
}
Now let’s tell the Book model to use new search rule by default:
<?php//...class Book extends SearchableModel
{
protected $searchRules = [
BookSearchRule::class
]; /...
}
Let’s make the same query one more time:
App\Book::search('Alice')->take(20)->get()You can see different result now: the records with Alice in a title are at the top. And this is exactly what we wanted.
Okay, let’s move on to the next example. What if we want to get all the books that have been written after the year of 2010? No problem! We can do that like this:
// We say Elasticsearch - find everything that was released after 2010
App\Book::search('*')->where('year', '>', 2010)->take(10)->get()Let’s do something more difficult. Now I want to search books by an author’s name. How can I do that? There are several options. The first is to find an author by name using the Author model and then pass an author id to the where clause of a search query. The second is to make the Book model find records by author’s name itself.
We can specify the toSearchableArray method to determine which fields must be indexed by Elasticsearch:
<?php//...class Book extends SearchableModel
{
//... public function toSearchableArray()
{
return array_merge(
// By default all model fields will be indexed
parent::toSearchableArray(),
['author_name' => $this->author->name]
);
}
}
Now we have to add author_name to the model mapping:
<?php//...class Book extends SearchableModel
{
//... protected $mapping = [
'properties' => [
'id' => [
'type' => 'integer',
'index' => 'not_analyzed'
],
'title' => [
'type' => 'string',
'analyzer' => 'english'
],
'description' => [
'type' => 'string',
'analyzer' => 'english'
],
'year' => [
'type' => 'integer',
'index' => 'not_analyzed'
],
'author_id' => [
'type' => 'integer',
'index' => 'not_analyzed'
],
'author_name' => [
'type' => 'string',
'analyzer' => 'english'
]
]
]; //...
}
One more thing, we have to modify the search rule:
<?phpnamespace App;use ScoutElastic\SearchRule;class BookSearchRule extends SearchRule
{
public function buildQueryPayload()
{
$query = $this->builder->query; return [
'should' => [
[
'match' => [
'title' => [
'query' => $query,
'boost' => 3
]
]
],
[
'match' => [
'author_name' => [
'query' => $query,
'boost' => 2
]
]
],
[
'match' => [
'description' => [
'query' => $query,
'boost' => 1
]
]
]
]
];
}
}
Now we are ready to search by new field, but it hasn’t been indexed: Elasticsearch didn’t know anything about this field when we were creating fake data. Luckily, we can reindex data with the import command:
php artisan scout:import "App\Book"Take the first author name from the database:
App\Author::first()->nameIn my case it’s Roxanne Boehm. Let’s try to find Roxanne’s books:
App\Book::search('Roxanne Boehm')->get()It works!
Afterwords
If you want to get more information about possibilities of Laravel Scout check the official documentation and the page of Elasticsearch driver on GitHub.
If you have any questions or you need help don’t hesitate to leave a comment below.
Thank everyone for your feedback! I’ve changed the Elasticsearch installation part to provide you with the installation steps of the latest version. I’ve also fixed some inaccuracies in the article and released the new version of the Scout Driver with fixed match_all query.
