Seeding Fake Data to Index for our Multi-Search

In This Lesson

We'll create a database seeder that will utilize our model factories to generate plenty of fake data to index and search against. We'll also discuss how we can easily batch insert bulk data in a single call.

Created by: @tomgobich
Published: 11 months ago

Join the Discussion 0 comments

Be the first to comment!

Transcript

the data that will be inside of our database. So let's dig back into our terminal here and do node ace make seeder and we're going to call this our fake seeder.
So we'll just put fake there and it will tack the seeder onto the end of that for us. Now in addition to that, we're going to want to utilize faker outside the realms of Adonis
JS so that we can utilize some helpers that it has to generate unique data like our user emails.
So to do that and to stay in line with AdonisJS, let's install the exact same one that they're using underneath the hood for the factories.
So we'll do NPMI at faker hyphen JS slash faker, hit enter to install that and we're good to go. So I'm going to clear that out.
Let's jump back into our text editor and we should now have this seeders folder with our fake seeder inside of it.
In here, we're going to want to create our data within this run method. But the very first thing that we'll do here is just to find some counts for the number of data that we're going to be creating.
So we're going to have users that will serve as our authors and we'll create 100 of those. And then we're going to have books. Let's create 1000 of those and we'll have genres.
We'll create 25 of those. Perfect. Next, what we're going to want to do is create a unique list of user emails for the total
number of users that we're going to be creating so that we don't get any duplicate key constraints within our insert statement there. And then we'll also do the exact same for our genres, just so that we don't have any duplicate
names generated there within that list. So we'll get an array of unique values as our first task.
So we do const emails equals, and this is where we're going to want to import and utilize faker ourselves.
So we'll import faker from at faker hyphen JS slash faker, because this has a nice utility
on it that will allow us to generate out a unique array of fake data. So we'll do faker dot helpers dot, and there's a unique array method.
And what we do with this is if we take a look at some of the examples is we pass in one of the faker methods as the first argument, and then the number of unique items that we
want to generate with that method as the second. So for our emails here, what we can do is pass in faker dot internet and the email method,
the exact same one that we've defined on our user factory for the email column. And then we'll generate a unique array of emails for the total number of users that we're going to be creating as our authors.
And whoops, never finished out the full method name there, it's unique array. There we go. Next, we'll do the exact same thing here for our genre names.
So faker helpers, unique array, faker book, and we'll utilize the genre method there and create the total number of genres that we're going to be creating.
Next, we'll create our fake user data. And to make things easier on ourselves, let's first turn a unique array of emails that we
have here, that's just a string array, into an array of objects with an email key and the unique email as the value.
So we do const user data equals loop over our emails using map, we'll get each email
there as we loop over it, and we'll just return back a new object with the email as the key and the unique email as that keys value.
What this will allow us to do is then merge that into our factory as it's creating our users. So we do const users equals await, import our user factory.
And there is a merge method here that allows us to pass in two different things. First we can pass in a direct object with data on it.
So this could be our email right there, like hello world.com.
And as we create our users, so here we'll pass in our counts.users to create our 100 users.
This, for each and every user that we're creating here, the total of 100, it would set the email as hello at world.com for each and every one of those. That's not quite what we want.
We want the emails to be unique. So instead, what we want to do is merge in our unique list of emails. So when we pass a single object in, it will use that object for each and every created
item. If we pass in an array of objects, however, it would only for this example, use hello
at world.com for the first user that it creates. So since we have created a simple list of emails as an object with our user data here,
what we can do is merge each and every one of those unique emails into the users as it's creating by just passing our user data into this merge there.
Now as it's creating our first user, it will use the zero index from our user data array to merge that into its data. Then for the second user, it will use the first index.
The third, it will use the second index, so on and so forth. Once we have created our users, to make things easier for ourselves, whenever we get down
to our books, let's go ahead and just grab a list of its IDs. So users dot map user, and then just pluck out the user's ID there.
That'll come into play a little bit later on. Next, though, let's go ahead and create our genre. So genre data. And this is going to look the exact same as our user, just now using genres.
So we'll do const genre data equals our genre names. We'll map over those for each name. We'll return back an object with that name key value pair.
Then we will create our genres using our genre factory. So genre factory there, and we'll merge in our genre data, create many, and we'll create
the number that we have defined up in our counts, which also matches the number of unique names that we've generated for our genres. Then again, to aid ourselves with our books, we'll grab just the IDs as an array.
So genres map, reach through to the genre and pluck out the genre ID and return that back. Great. Now we're ready to create our books.
So we'll create our book data next. So we'll do const books equals, and we're going to do this one different. Let's do await book factory.
And rather than creating the data and merging it in, let's instead use tap.
Tap allows us to grab the book model instance prior to it being persisted into the database
so that we can make mutations to it before it is actually persisted and created. So the first thing that we get back from tap is the actual book model.
And then we get back to factory context. This factory context contains the transaction, our faker instance, and whether or not the
model was stubbed, uh, stubbed compared to actually creating, we'll just create the model and won't actually persist it into the database.
So if it's stubbed, then that means being persisted is not the next step in this process. If it's not stubbed, then it will be persisted.
What we want to do here, however, is assign our book, a user ID, AKA an author. And for this one, rather than reaching straight through to the faker that we've imported,
we can also use it directly off of our factory context. So we have ctx.faker.
And since we've imported the exact same faker package that AdonisJS uses for its factories underneath the hood, we have the exact same option set available here.
A helper that we have not used yet is array element. Array element allows us to pass in an array of values and it will pick a random item from
that array and return it back to us. So if we were to pass it our user IDs, it's going to pick a random user ID and return
it back for setting as our books user ID. So what we're doing here is just assigning a random user as the book's author via our user ID column.
Then we'll go ahead and create many. There's also, since we mentioned it, make stubbed and make stubbed many to create stubbed books.
Again, stubs are not actually persisted to the database automatically, whereas whenever we run create many, they are. So we'll use create many here and we'll grab our counts dot and create our 1000 books.
The last step of this process is to bind our books to genres.
And unlike anything we've done here yet so far, what we want to do here is bulk insert the book and genre pair into the database in one go.
So we'll reach directly through to the pivot table to do that. Since we already have both our genres and our books created, this will allow us to use
this preexisting data rather than create brand new data for either our books or genres.
So we'll do const book genres and we'll loop over our books using a flat map. A flat map is similar to a map.
It's still going to allow us to loop over each individual book within our books array. But whenever we return data back here, if we return back another array, this flat map
will flatten it all back into a single array rather than the nested array that we'd get if we just used map.
So within each book, what we want to do is grab some genres to assign it. So we'll do const book genre IDs.
And again, we can use Faker and its helpers and specifically its array elements helper, this time using the plural version of array elements to get multiple random items from
the data that we pass in. So we can pass in our genre IDs as the first argument, and then the second argument is the number of random items that we want to get from our genre IDs.
So here I'm going to go ahead and just create two genres per book. That should give us 2,000 items within our book genre array, since we have 1,000 books and two genres per book.
Now that we have those IDs, we can go ahead and return and loop over those book genre IDs using a map to get each genre ID.
And we'll return back a new object per ID that we're mapping over with the genre ID and the book ID, which we can get from book.id.
And then let's also add in the created at using date time from Luxon, and we'll reach
for now and pass to SQL so that we get that as an actual string value for SQL there. And we can do the exact same for updated at, date time, now to SQL.
And a couple of things to note here, first, now the type for our book genres is an array of objects consisting of our genre ID of type number, book ID of type number, created at
of type string, and updated at of type string. Once we're going to be inserting these in bulk directly to our pivot table, we need
these property names to match exactly as they're defined on our pivot table, because we're no longer going to go through our lucid models.
So we want them to match exactly as we have them described on our pivot tables migration. That's book underscore ID, genre underscore ID, created at and updated at there. So perfect.
Everything's aligning so far with our book genres. Now what we need to do is bulk insert them. One thing that we could do is await import DB from AdonisJS Lucid Services DB, utilize
the table book underscore genre, and then do a multi insert and pass our book genres in that way.
However, since we're creating 2000 book genres here, we have 2000 items in this array, you
may run into an insert limitation like I would, since I'm using SQLite, it can't handle that many items in a single insert statement.
So what I need to do instead, and what you may need to do instead is to batch insert
these in bulk, you might think, okay, so we need to group each of these book genres up into batches, and then run this multi insert per batch.
And while we could do that, there's an easier option at our disposal, what we can do is reach directly for connects JS through our DB and its current connection to get connects
JS's right client. So if we check the type for this, it should just be connects JS or connects there.
What this gives us at our disposal is a utility method that it offers called batch insert.
The first argument here is the table that we're going to be batch inserting into for us, that's our book underscore genre there.
The second argument is the array of data that will be batch inserting. So that's our book genres. And then the third and final argument is the batch size or the number of records that it
would insert in a single batch. Since I'm using SQLite, I think 500 is a safe number for that.
So what this will do is it will insert 500 of our book genres, commit that to the database,
move on to the next 500, commit those to the database, the next 500, so on and so forth until it's inserted all of our book genres. So great.
With that set, we're now ready to go and ready to run our Cedar. So let's jump back into our terminal. I'm not 100% certain where we left off with our migration status.
So I'm going to go ahead and do nodeace migration status, see whether or not we ran them. And we did. Okay, great.
So that means that we're clean and clear to do nodeace db colon seed to run our Cedar and create our data. If all goes according to plan, we should get back a completed statement from that.
And we should be able to jump into nodeace REPL really quick, await load db and await
db dot from book underscore genre to see that we should get back 2000 items in here.
So we get back the first 100 as objects, as we can confirm just by the ID there. And then we get back 1900 more items that has truncated.
So that's our 2000 book genre pivot table data right there. Perfect. Let's go and exit out of there and clear out of there.
Because our next step is to create a command to actually index that data into melee search.

Building A Multi-Search with Meilisearch & AdonisJS

12 Lessons

1h 38m

Stop losing your best ideas!

Use our integrated note feature to capture and organize your key insights. Take a note at any timestamp and jump instantly back to that moment in the video with a single click.