Creating a Command to Index Data in Meilisearch

In This Lesson

We'll create an Ace CLI command to index our pre-existing data within our Meilisearch database so it is ready to search against. We'll walk our way through this and point out what to watch out for and how to view task errors.

Created by: @tomgobich
Published: 11 months ago

Join the Discussion 0 comments

Be the first to comment!

Transcript

MeleeSearchIndex command in the last lesson, as well as a search service, which we created to be able to use either an admin or a search authorized instance of
MeleeSearch via their package. In this lesson, we're going to utilize that service to fill out this command here to actually index the data that we have inside of our database.
So we can go ahead and give this command a description that will show inside of the ACE command list to say that this indexes existing data into MeleeSearch database,
just like so. And then in addition to that, we also want to be able to utilize Lucid, so we want to start our AdonisJS
application to boot Lucid as well, so that we have access to our models. Inside of this index command, all that we're going to be doing is administrative operations, so we can go ahead
and set ourselves up with the admin instance of our search service. So we'll just add a search property onto this actual class
and then import our search service, and that should have added an import there from services, search service. We can reach through to our admin authorized instance of
that by calling use admin, just like so. And now anywhere inside of this command, we can do this search to get access to that MeleeSearch instance.
Perfect. To keep things inside of this index method clean, we're going to have separate private methods that will run for each one of the three indexes that we'll have inside of
our MeleeSearch database. So we'll do an async index authors method, and then we'll have an async index genres method.
And then finally, we'll have an async index books method. Whenever we call our command, the run method is what AdonisJS will actually execute. So this is where we're going to want to call each of these
methods. So we'll index our authors, then we'll index our genres, and then we'll go ahead and index our books. So let's first start with our authors index.
For this, we are going to want to query our authors. And for this application, we'll consider an author, any user inside of our database that's bound to at least one book.
So we'll set up a get authors query here that returns, and we'll import our user model to query from it, where the user
has at least one book. And then to make this happy, we need to return the query callback. We don't need to do any further filtering on the books.
We just need to ensure that the user has at least one. Then we want to pluck just the applicable information that we need to index for our authors. Now that's definitely going to be the author's full name so
that we can actually display it inside of our search results, but we're not going to actually build out any detail pages. So we're not going to dig into authors, genres, or books, but let's say that you did.
So for that, in order to be able to map that full name to an actual record inside of our database, we would also want to get back the author's ID so that we could then add that
detail page in to show all of the author's books, for example. So what we would want to do in that instance is select the ID and then of course the full name there so that we can
actually display it inside of the search results. Because whenever we actually search from Millisearch, we're just going to get back the data that we've indexed. So if we had just indexed the full name, we'd have no way to
map that full name from Millisearch's database back to our database. So that's where that ID comes into play. And then we're just going to query all of them.
If you have a boatload, you may want to paginate this and go through it in chunks so that you're not running a massive query here. But in our case, 100 will go just fine.
And the red squiggly there is just because it wants me to wrap that in parentheses. All right. So inside of our authors index, we'll go ahead and get back
our authors by awaiting this get authors. And then we need to utilize our Millisearch search instance that we've added to this class.
So we can do this search to access our admin authorized instance of Millisearch. And we want to create an index called authors.
Remember our indexes are like the tables that house individual buckets of searchable data. So by creating an authors index, we're giving ourselves
the ability to search just authors inside of our Millisearch database. Then inside of this index, we want to add documents, which
are like the tables rows of searchable data. In our case, that's going to be all of the authors that we've queried back, which are going to comprise of our ID
and full name. Now the type for this is still going to be user because Lucid is always going to return back a model instance, but whenever that serializes, it's just going to contain the ID and full name.
What this add documents method does is it's going to return back an enqueued task promise, which is a literal promise. So we do want to await this, but further yet, we can go
ahead and grab the task that this returns back as well so that we can log this into the logger in case we need to
verify its status via the Millisearch API in any way. So we can just log this as info and say something like authors and then reach through to the task to show its
status, task UID, the type of task that it is. And then we also have when it was enqueued. So we'll do status there and then we'll add the task UID,
which is how we would actually look up the status of this task after we run this command. So even though this returns back a promise, this is just
going to kick back an actual task and Millisearch is still going to run in the background outside of this promises return. So even though we get back our task, Millisearch is still
running and indexing that data. Whenever we get the task back, it just means that Millisearch has received and started the index process. Okay. We're not quite done with this index authors method,
but I do want to show why we're not done. So we'll leave it there for now. And it looks like we still have a red squiggly here, objects possibly undefined. Oh, our Millisearch service there is also possibly
undefined. So let's go see why that might be. Ah, right here, where we are getting the instance. This is going to return back a type of undefined,
but we have this inside of a has to verify that it is actually in there already. So we'll just go ahead and insert that it will be able to get a record there. Okay.
So that should fix our red squiggly on our search. Perfect. It sure did. I'm going to go ahead and fix that missing comma right up there as well within our start app. Okay.
So we can go ahead and stop here and jump back into our terminal and run node ace Millisearch index to query and index our authors. Perfect.
So you can see that we've got authors status is enqueued and the task UID is zero. We can use the Millisearch API to check to see whether or
not this enqueued task actually succeeded by sending out a get to our HTTP localhost 7, 7, 0, 0,
which is our running Millisearch instance. And then the end point for this is tasks and then followed by that task UID, which for us is zero and whoops,
we switched that to a single quote there rather than back ticks. Okay. And then we also need to authorize this with an authorization header consisting of our bearer and then our
some master key here token. And then we can pretty print this with JSON underscore PP. Okay. Run that.
And we get printed back that enqueued task and its status. So you can see the duration that it took when it was enqueued, when it finished,
whether or not it had any errors during its operation, the UID that was being indexed with this task, the type of task, and most importantly that it has succeeded.
So since this task succeeded, that should then mean that we can jump into our browser. Let's give this a refresh. And sure enough, here we go.
We have our authors index now showing up with our 100 authors. And then we have each of those documents listed here.
So we have Wanda Crushank, Patsy O'Connor, Noah Soar, so on and so forth. And we have the index data of their ID within our database
and then their full name. If we had a picture associated with this, that would be what shows up here in this pink section. And then we also have a search bar right up here that allows us to search against that data and it will mark what
is matching in our search. Perfect. Okay. So up here in this dropdown, we have our indexes. Again, these are like the tables. So we have our authors table or authors index right here
with 100 documents inside of it. The documents are like the rows. So that's each individual author that we have added a document for. And as we'll see, whenever we get to our books,
this can consist of objects and arrays as well, that we'll be able to search against too. Now, the reason why I wanted to run this and why we're not
quite done with our authors index method yet is because we've indexed the ID and by default, anything we index is going to be included in the searchable
fields. So if I were to search for three, you'll see that that's matching the ID of three, 30, 31, and then 32, 33, so on and so forth.
And whenever users search for that, that's not going to make sense for them as all that they care about is the author's full name.
So we want to exclude our IDs from the searchable fields, but we still want to index them so that we get them back with our results so that we can map them back to the data inside of our database.
And thankfully that's actually really easy to do. So if we jump back into our text editor here within our index authors method, after we have added the documents, we can go ahead and do await this and again,
reach through to our admin Melee search instance. And then we want to reach through to our authors index. And within here,
we have the ability to update searchable attributes. So the columns that we have inside of our database, like ID and full name are what Melee search is referring to
here as its attributes. So for our author, we can say that we only want the full name attribute to be searched anytime that we perform a search operation using
our authors index. So now if we give this a save, let's jump back into our terminal one more time and let's rerun our Node A's Melee search index command.
All right, now you see that that was enqueued. We got a task URD of one and by now it should have finished because 100 is going to go relatively quick. So we should be able to jump back into our browser.
Let's give this a refresh and let's try searching for three again. And you'll see that we get back no matching results despite
us having an author with an ID of three that had previously matched that search. If we still search for Wanda against that full name attribute, though,
we are getting back a result as expected, which is perfect. Now you might be asking, we just reran the same index command that we had run initially.
Why do we still only have 100 author records? That's because Melee search does actually use our ID as well as its primary key for the data that we're indexing.
And a primary key for them is required as well. So when we index a document with an ID of one, Melee search is going to say, Hey, I already have this document.
I'm going to perform an update on it rather than insert a new record. If we were to insert an author with an ID of 101, which since we've just created 100 does not exist inside of
our database nor this index, it would insert it as a new record because it doesn't exist in this index yet. Okay, great. Let's go ahead and finish out our command.
So let's jump back into our text editor. We need to do our genres next. So we'll collapse that on down and we'll create our get genres query method that returns back,
reach through to our genre model to query off of it and select its ID as well as the genre's name. And then in addition to that, if you want to,
you can also include something like a count of the number of books that each genre has. And again, just like our ID, we'll use that count just as a display only field.
We won't include it inside of our searchable attributes. All right. So we'll do const genres equals await this, and we'll use our get genres method.
Then we'll get back our task from await this search, and we'll create a new index called genres,
and then add as documents, the genres that we have queried, just like with our authors. Then we want to reach back through to that genres index to
update our searchable attributes. And for this, we'll just set the name as the searchable attribute for our genres. Then for our informational purposes,
we can log out an info of genres, print out the task status, and then the task UID is task UID just in case we should
need it for any reason. All right. That then takes us down to our books. So for our books, our query is going to be a little bit more complex than the other two.
So we'll return import our book model, and then again, reach through to query and we'll go ahead and select the book's ID.
Then we'll grab the book's title as well as the book's blurb. We could also include the excerpt in here, but that might include more than what we would want to
search against to get proper results back for our users. So we'll omit it here. Then in addition to that selected data, we also want to preload the genres.
So that'll be query. And then for this, we want to specifically select that genres ID and name. Again,
the ID here is going to be for our use case so that we can offer a link into that genre if we should wish. And then the name is for display purposes, as well as search purposes.
And then we'll also go ahead and preload the user, which is the author of the book. So we'll query. And then again, we'll select the user's ID and their full name. Perfect. All right.
So let's start by getting our books. So await this, get books. Then we'll go ahead and grab our task as await this search,
and we'll create a new index specifically for books and then add those documents into it. And again, this is going to be of type book model, but whenever the serializes is just going to contain our ID
title blurb. And then as an array, it will have our genres, which will be an array of objects consisting of the genres ID and name. And then we'll have a single object for our user consisting
of an ID and full name property. Now we aren't quite done with this line, but again, we'll show why. So we'll move onward here because we want to update this
index and it's searchable attributes. So for this, we want the title and the blurb to be searchable, but in addition to that,
it would be great if we could also search for the genres name and user full name via our book results as well. And we can specify those within these searchable attributes by reaching for them as such.
So we'll have our genres.name, and then we'll have our user.fullname, and Melee search will know to reach through to the
underlying objects within either that array or that base object to grab the underlying property that we have specified via this string here. Great.
Then we'll go ahead and again, for our informational purposes, log out books, the task status, and then the task UID is task UID there.
All right. So we should be mostly done. Again, we have one more thing left to do with their book index here, but we will see exactly what that is momentarily.
Let's first go ahead and run this index again. So we'll just clear that out, rerun our index and Oh, okay. We have two things left to do. All right. So I'm sitting here looking at my notes,
wondering why I had user ID included in the index data twice. And this sure enough makes sense. So with how AdonisJS preloads its data,
we do need to include our user ID in order for us to be able to preload that relationship to the user. So since we have omitted the user ID from the data that we're getting back,
it's not able to use that ID to preload the user. So we'll jump back into our index books or actually our get books,
because what we want to do is just add the user ID into our books, select the list there. Okay. Give that a save. And now let's try running that one more time. And there we go.
Okay. So now our authors, genres and books have all been enqueued. And if we jump back into our browser, give this a refresh, let's take a look at the various indexes that we have.
So you'll see that we have authors with 100 authors, books with zero books and genres with 25 genres. So our genres and authors look a-okay. If we take a look at our genres index here,
we can see that we have horror, poetry, Western science, fiction, so on and so forth. So those all went in. Okay. But our books are completely empty. So something went awry there. Let's jump back into our terminal here and I'm going to hit
the up arrow until we get back to that curl where we were checking our task statuses. And we'll go back over to that task ID and update that to
11, as that was the task ID of our books index. And we'll hit enter to run that. And you'll notice that it received 1000 documents,
but it indexed zero. So it received the data. Okay. And if we take a look down here at the error key, we can see that we have an object with an error here
specifying that the primary key inference failed as the engine found two fields ending with ID in their names, the ID and the user ID.
It's asking us to manually specify via a primary key property, which of those two Millie search should use as the primary key of the indexed document.
So that goes back to where we talked about how Millie search determines whether or not it needs to add or update data via the actual ID that we are indexing along with our documents.
So it's saying I found two applicable things that could serve as that primary key for the book. And I'm not sure which one's correct. So we need to inform it,
which one it should use via our add documents call. And for that, there's this primary key property. And we just need to set that to ID there as we want to use
our books, actual ID and not the user ID. So with that change, if we give this a save one more time, jump back into our terminal, I'm going to go ahead and clear this out,
hit the up arrow a couple of times to go back to our Millie search index. Let's give that another run. Okay. And let's just go ahead and double check our books task again.
So we'll switch 11 to 17 now as that's our books task ID. And we can see that it has now indexed 1000 documents and we have no error. Perfect.
So if we jump back into our browser, give this a refresh, we now have 1000 books indexed consisting of each of the documents that we have queried for. So we have our book with an ID of one,
user ID of 85, the title, a confederacy of dunces, organic pizza made with concrete for all day, courageous support. Got to love that concrete pizza.
Then we have our genres of which it has two consisting of that genres ID and name. And then we also have users,
which is an object comprising of an ID and a full name for that author. So if we were to search for the title, we can see that we do get matches there.
If we start to search for something in the blurb, so some Marvin in that one. So if we search for that and we get a match there, perfect. If we search for a user ID, we get back no results.
That would have also searched for our books ID. So that's good. And if we search for a genre, you'll see that we still get results back here,
searching for science because it's matching science fiction as our genre. And then if we search for something comprising of our user's
full name, so we'll search for Martha there. You'll see that we still get back results, despite it not matching anything inside of the book itself beyond the user's name. Perfect.

Building A Multi-Search with Meilisearch & AdonisJS

12 Lessons

1h 38m

Stop losing your best ideas!

Use our integrated note feature to capture and organize your key insights. Take a note at any timestamp and jump instantly back to that moment in the video with a single click.