Let's Build with AdonisJS #1.7

Building An XML Sitemap

In this stream, we'll be going over how to build a sitemap of all our site's pages for search engines using XML to help bolster SEO.

Transcript

Looking for something specific? Feel free to search the page for keywords to find a specific portion of the live stream that interests you most.

0:06

alrighty so it's been a little bit since i've posted anything so i thought it'd be about time that i live stream

0:12

something just to get something out to you guys uh so the new site is coming along great um

0:20

the home page is all populating for the user and everything um all of the sub pages and everything are

0:26

pretty much hammered out one of the last things remaining that i do need to do

0:31

is add a sitemap to it now i have the user facing sitemap already populated

0:36

within the footer there but what i need is one for search engines so if we were to go to the

0:42

current site and go to sitemap.xml we'll get an xml version of the sitemap

0:48

that contains all of the links within the actual site along with how much they

0:54

should be re-requested or we checked for and then what their priority on the site

1:00

is obviously the home page being the most priority and then series and topics

1:05

and lessons from there so i thought we'd do a little live stream where we

1:11

kind of recreate that xml sitemap within the new site um so

1:19

really only two things within the old site that took care of this at a service that really just

1:25

queried the information generated out the actual structure and then we had the

1:32

controller function uh that actually generated out what the response should look like

1:38

and send it along as xml so let's go ahead and get started with that within the current project i do believe

1:45

i have a syndication controller already created

1:53

and what this one's taken care of currently is just the user-facing site map so uh it's not quite a whole lot of

1:59

information uh the one for google will be querying a lot more um

2:05

because this user facing one is just more of a generic listing

2:10

but yeah so to start out with what we're going to want is an actual controller

2:16

endpoint to show the xml version so let's do public async xml

2:22

and this will want a response and that should actually be it

2:27

and then this will be http contacts contract and then from there

2:34

let's see so we'll probably go straight into

2:40

wanting to get the information and then we'll have our response dot

2:48

header we'll want to set the content type to

2:54

application xml response.header here header again this one will be

3:01

content encoding which will be gzip

3:06

and then let's see here so

3:17

let's start with that for right now now i believe i was using a package on the

3:22

old one um

3:27

not within there so we have zlib for creating the actual

3:32

gzip and then we have it looks like sitemap for creating the sitemap stream

3:38

so i'm going to go ahead and install those two dependencies so that was

3:43

mpmi sitemap and zlib

3:55

okay looks like we have those

4:01

all right so whenever it comes to the actual um

4:10

sitemap package what we are going to want to do is pass it in

4:15

the underlying host name so let's wrap this inside of a try catch

4:23

okay and then let's do uh const

4:29

site map stream equals new let's see if we can get this to auto

4:34

complete here site map yep there it is site map stream

4:40

and we just want to pass it the host name in this case it's going to be https

4:45

slash slash yeager.com all right and then we're going to want

4:52

to pipe into our sitemap stream so let's do

4:58

if i can spell pipeline sitemap stream oops

5:04

sitemap stream dot pipe and then we'll want to actually create

5:11

let's see if this will uh create gzip and that should be just fine just like

5:16

that and then

5:23

let's see here actually so from cash

5:30

if exists so let's try to get the information from the cash

5:36

if we can so we'll do const

5:43

sitemap equals and then for right now let's just do an empty

5:49

array and this will be from the cache we'll set the getter for that here in a second

5:56

but if sitemap and actually we just set this to null

6:01

because if it's not in the cache it won't come back as anything we will do response dot

6:08

send the segment and we want to kick that off right there

6:15

otherwise once we've created all this stuff this is where we'll want to

6:21

actually get the info if it's not cached and from here that's

6:28

where we would want to probably plop that off into a service so let's jump down to our services i

6:34

don't think i have a syndication service so let's go ahead and create that real quick

6:45

all right

6:52

actually yeah that'll be fun okay

6:59

so for the um

7:05

old one here for the syndication service i had types defined within the surface itself i'm gonna go ahead

7:11

and plop those just copy if i can oh what am i trying to do here i'm gonna go ahead and cop those into a contract file

7:17

uh so let's go down to contracts and let's create a new file here as well

7:23

called syndication and i'm just gonna plop those in there

7:29

so we have sitemap item which contains the url the change frequency and the priority which aligns

7:34

with if i still have it open what the actual xml items are

7:40

and then we just have the frequency which is what the change frequency values should be which is either daily

7:45

weekly or monthly so now that we have that in there let's go back into our syndication service

7:54

and so first and foremost we're going to want to cache this if

8:02

uh we can so let's do public static

8:09

let's do sitemap cache key i'll be if string and we'll just

8:17

cache that as sitemap and then we'll want a public static

8:25

um let's see

8:34

so on the old one here it looks like i had a getter and i had the sitemap here actually private

8:40

we could mimic that i don't think we need to and then there is how i was checking it

8:46

so i do still have that cache service it does integrate with a different cache now it integrates with redis so

8:55

let's go back up into our controller and go ahead and get that going out

9:01

so instead of that we can do

9:06

well let's do a get directly within our syndication service instead of doing it outside so

9:12

we can switch this to private i guess and then let's do public

9:18

static get sitemap from

9:23

cache or something like that um and then we will do return

9:30

cache service it looks like my autocomplete gave up on me oh no

9:37

cash service or i just can't spell nope nope it did give up

9:42

well shucks

9:48

all right let me uh switch files real quick let's go into you and then back into you see if that fixed it

9:55

it did not all right i'm going to close it real quick

10:01

and let's see if that fixed it so public static there we go all right

10:07

get sitemap from cash

10:12

um now at this point we want to return cash service there you are

10:19

dot get um and then this dot site map cache key

10:27

is this async i cannot remember

10:32

yes it is so we'll want this to be public static async and

10:38

yeah there we go

10:45

hey man um i plan on getting back to recording new things

10:51

well i'm hoping to record in between now and may but i don't intend on actually posting

10:58

any new recordings until we reach may once we reach may that's whenever i'll be in my new house and i'll have

11:06

everything set up and i won't have to deal with packing or anything like that anymore so um

11:11

just going to kind of take this time in between now and may to get some recordings in place and then we should

11:17

start seeing some new stuff in may so

11:23

should be early may

11:31

yeah so i'd say the first week of may we should get something out

11:36

um and then from there it should be pretty much weekly hopefully

11:42

so we have the get site map from cash

11:49

and we have that oop that's not where i meant to go we have that in our syndication

11:57

controller where are you here you are no we do not so let's go ahead and do that

12:04

so const site map equals await

12:10

uh syndication service dot get sitemap from

12:15

cache so now if we get the sitemap from the cache and it does exist then we can just return that cached version otherwise we

12:23

need to try and generate it and that's where this would come into play down here

12:29

so we'll start on that next so

12:34

we're going to have the intro function for this that takes care of

12:39

kind of generating and populating everything together as probably i guess generate site map would

12:45

make the most sense um and then ultimately what we'll want this

12:50

to do is return back a promise of sitemap item as an array

13:01

so first and foremost what we're going to want to do within here is actually query out all of our different information so

13:07

we're gonna have our series topics lessons

13:14

and blog posts and then from there we'll do each like

13:20

subordinate page or whatever you want to call it as individual stuff from there so like

13:26

we're going to have a section for frequently asked questions resources uses and then if we take a look at the footer

13:31

there's some stuff for like guidelines contact the visual site map cookies privacy terms and all that fun

13:37

stuff so all that stuff can go in there just manually defined but this stuff needs to be dynamically pulled

13:44

um actually

13:49

these are going to be database generated the frequently asked questions and resources um

13:55

so but for right now those are empty so i'll just put those in there manually at this point

14:04

so this will be a good starting point so for our series

14:09

we're going to have a weight and trying to think if i have something in

14:14

the series service for this that would be useful i want to say no so let's just query it

14:21

directly off of our collection model oops yes there we go all right

14:28

and then from here we specifically want to just grab the series um don't plan on populating

14:34

uh uh playlists or courses at this point in

14:40

time um and then let's see

14:49

from this all that we should need is

14:56

the slug so i have a

15:05

well i guess i should change the name of that so i have this function defined

15:13

where do i have that to find that that is within my query builder provider called select

15:20

ids now this will execute the query and then just select the id column ideally i

15:25

guess i should just be named select column since it doesn't specifically need to be

15:31

an id and then the series function will kick

15:37

off a query builder and then it will execute a where statement to make sure that it's of type series

15:43

uh so that's just a nice little shorthand that i have there so i'm gonna go ahead and just do select

15:48

ids this won't make any sense at this present point in time and name this

15:56

i think i just need the slug let's check and see whether or not

16:01

that's the right so at work they're called aliases i think i call them slugs

16:08

where am i here collections did i click on it yes i did yes slugs

16:14

all right and that should really be it for our series oh wait a minute

16:21

dot where public i have another query builder helper that will specifically do this

16:27

for the type where it is public so build that into there and we have a nice simple query for all of our series right

16:33

there and we can do the same thing for topics await top

16:40

and the model for this is called taxonomy and then

16:47

trying to think do i just want the roots of the taxonomies or do i want all of the children as well so like if we go

16:53

into topics any taxonomy is technically a topic um

16:58

but then these are all of the root level ones and then like if we were to click on adonis js there's

17:04

subtopics within it so you have like the acli authentication i guess it would make sense so let me

17:11

see do i have these as no they are root level so like if i were to go into hdb context

17:16

it's still the same url so let's just query all of them we'll get the roots first so we'll call

17:22

these i guess root topics and then we'll do child topics below that as a lower priority

17:30

so we'll do roots and that's similar to the series

17:36

it's just going to kick off query builder where it's specifically a taxonomy that doesn't have a parent id

17:44

um those are not those don't have a state id

17:50

and then so here it should just be select the slug from that as well and then we'll do const

17:57

child topics await and i don't think i have a helper

18:04

like roots for this but we can go ahead and bake that in this might be the only use case where i would need it but

18:10

see i always lose my models there they are taxonomy

18:18

i need to clean my screen off it's super dusty

18:25

where are you roots there it is all right so just a simple public static

18:36

deck it's just children and then that's just a simple we're not

18:42

null parent id

18:48

um you could probably just do the

18:53

conditional parent id in there i've typed number

19:00

default that to null actually i don't think you need a question mark an id

19:05

such as squiggly because it's not used type number

19:12

there we go and then we could do if parent id [Music]

19:18

return this query dot where

19:24

parent id is parent id otherwise just get it where it's not not null

19:31

alrighty so that should do it for that so then we could do children we don't want a parent id there

19:38

and then we can just do select ids for slug again i need to rename

19:44

select ids to just like select column or something like that hey tina

19:55

thanks for stopping by um so that should do it for topics

20:01

and then we have our lessons which

20:07

well you know what let's um kind of order to these as well i guess that would make sense with it

20:13

so let's order by

20:22

i guess let's have the newer ones first does that make sense like i guess

20:27

so created at

20:32

descending

20:39

dang it missing my shift key yeah i think that makes sense so we'll do newer ones first

20:47

yeah yeah let's do this alphabetically it'll be easier to um verify that we're getting everything

20:52

that way order by name decent wait no a something

20:59

for name there we go

21:08

okay and then for lessons

21:18

uh that goes off of the post model dot lessons

21:24

dot and then from there from there i want apply i want to apply

21:31

a scope and the scope is to ensure that the post is published

21:37

and then we'll do our order by from there uh that one would be title ascending

21:47

select ids for select yeah they're all selected

21:53

all right so there's our lessons and let's just change this to posts and

21:58

then we'll group them all within there so const lessons const

22:03

posts equals await post dot i think i named them post nope

22:08

want to name them blog blogs all right let me name this to match that

22:15

apply scope now i don't think i have any blogs populated yet uh so that will be empty

22:22

but we'll go ahead and take care of them because i intend on publishing some here soon

22:28

descending select id select okay so if i have this

22:34

working properly this should come back as just an array of slugs um

22:40

if not then we'll find out but like so if i were to go into tinker here or tinker um rebel

22:47

node ace rebel

22:52

load models and we do await

22:59

models collection dot

23:06

series dot

23:12

public dot select ids

23:18

slug we should just get back an array of slugs and that we do so it seems to be working all right

23:24

so i should be able to expect that for all of these so now that we have the

23:31

dynamic information that we're going to need to make this site map something um

23:37

next what we want to do is actually create the items and then from there we can go ahead and

23:46

excuse me kind of map everything together so we're going to have

23:52

a private static function called make

23:58

this will take in the url

24:03

of type string the change frequency of type

24:08

freak when see and that did not want to

24:16

what do i got going on here

24:25

oh i was putting that inside of the generator okay my bad

24:34

okay there we go i guess frequency already auto imported because it doesn't have a squiggly on it

24:39

so it should be okay and then the last one will be the priority which

24:48

i think that would just be type number and then we can default that to a half item and then here we'll return back a

24:55

finalized sitemap item from here so what we want to do is pretty

25:02

much just return back an object of those arguments so we'll have url change frequency and priority

25:13

i kind of prefer this approach over you know just in line creating an object

25:18

over and over again mapping or naming the keys of the object over and over again

25:23

for each item um so now that we have this we go ahead and start our actual url collection so

25:28

we'll do let your well now we can actually make this uh const urls equals

25:36

let's specify this as a sitemap item array equals

25:43

an array and then here we'll go ahead and define our top level

25:49

hard-coded urls so we can do this.make

25:54

um i'm just going to hard code the url in here since they're pretty short do weekly for that one and then the

26:01

priority for this since it is the home page will be 1.0

26:06

um and then i'm going to go ahead and i believe all of these are the same

26:13

so i'm just going to go ahead and copy and paste this instead of writing it all there we go

26:21

so we have so we're essentially just making a site map item for our home page our series

26:26

landing page topics landing page lessons landing page and our blog landing page obviously the blog landing page is going

26:32

to be less important than all of the other ones so once we have that in place then what we can do

26:39

is go through and add in all of our dynamic items um so

26:46

for that we're going to want kind of a different helper from make we're going to want something like

26:51

private static function called map and this will take the route name

26:57

as a string any params um

27:05

did what what is that that's x um it's been a while since i've wrote

27:10

this x string any i think i think that's accurate

27:16

and then the change frequency as frequency

27:23

and let's default that to weekly and then the priority

27:28

is again a type of number and we'll default that to 0.5

27:34

and then this 2 will return back a site map item for the arguments passed

27:39

in here so

27:45

we can do const url and then what we're going to want to do is use the route module to actually

27:50

just generate out this url so let's do make url we'll pass in the route name

27:56

along with those defined parameters i shouldn't need any query strings or

28:01

anything like that for this um it should just be simple to leave it as the round name and params

28:07

and then if we do not get back a url from that make

28:13

url if it's not mapped for whatever reason then we would want to just return this

28:18

dot make define it as a hard-coded url with the change frequency

28:24

and the priority

28:30

i guess we want to make that more undefined

28:38

that's how i had an old one yes

28:46

all right so that should be good for the mapping one so now we'll want to go through and actually just map each of so

28:53

our series topics and posts so that we get those items actually

28:59

created so we do series dot map and we'll just loop over each and every series that we

29:04

have and then this will be the slug since we've specifically selected just the slug

29:10

value as we've tested here so it's just be an array of strings with the slug values

29:18

and then we'll want to append this into our urls

29:24

so this will be urls dot push

29:34

this dot add oh we need to make that yet don't we

29:43

what was that wrong file

29:55

i don't think we need that i don't know why i had it in there for some reason the old one but

30:00

i guess i needed to do conditional determination for them but we should not need that in this one so it should just be a simple push

30:12

this dot map and then the

30:18

route name for our series should be series dot show

30:25

and so map is pretty much just

30:30

matching like the make url format here and then we'll have slug right in there

30:38

and it does not like that y

30:44

series came back as a number

30:50

huh that's peculiar

30:58

promise number huh what in the world is that doing it for all of us

31:04

yeah it is huh that's odd do i have that

31:11

i've used that elsewhere it's kind of odd um

31:20

okay i guess let me take a look at the contract definition for that and see exactly what i have that coming back is

31:29

worst case scenario all write it where is that

31:36

database select oh sure enough oh yeah because it's ids

31:43

right um all right so i guess that would specifically just

31:49

be a number there wouldn't it all right um tell you about let's go ahead and create

31:55

a different one then so

32:00

let's do model query builder macro

32:08

select column async function i'll leave that one it's

32:13

just the id and i want to do this one is column name string equals

32:22

i guess we'll just make that required

32:28

and then we'll do const results equals oh wait this dot select

32:36

column name return results dot map r r column name and it

32:45

doesn't like something i have here

33:07

yeah there we go parentheses in the correct spot okay

33:12

so we have that now let's go ahead and just add that into here so let's see i did that after that so this would be

33:17

select column

33:22

oops to go back and check the name of that yeah select column

33:31

column name string and then we should just be able to do

33:38

something similar to the first or up here i think equals undefined return back

33:46

promise

33:51

of t

33:57

array i think i don't know we'll test it

34:03

worst case scenario i'll just put ads in there for right now um so

34:08

change this to select column

34:16

i feel like that's wrong but let's see here is that returning back

34:22

oh an undefined array all right that makes sense

34:30

i'm just going to hard code that as a string for right now it's gonna hurt anything

34:39

yeah oh wait no that needs to be a string array

34:47

there we go okay not ideal but it'll work for right now

34:54

and we'll just map switch all of these to that

35:03

okie dokie so now this should be happy maybe argument of type site map item is

35:11

not assignable to parameter f type

35:18

oh okay oh that's why i had this dot ad because

35:23

they were nullable and if it came back as null then uh

35:29

yeah you don't want to add it do you so

35:34

okay yeah we could do that private static add

35:40

or i guess let's just do push we'll do add

35:46

switch that to a let we'll do urls equals

35:55

this if i can spell this dot add and then we'll do add here

36:02

and i think this needs to take in um

36:14

urls

36:20

right because this would return back okay um

36:27

so i think do cite my vitamins there as an array of

36:35

sitemap item and then

36:44

no condition so this is the actual array and then this is gotcha

36:52

and then conditional add as a site map

36:59

item array or undefined

37:04

there we go so then we'll do if not

37:11

conditional add and just return urls otherwise return

37:18

urls and additional evidence this is just one

37:25

item isn't it site map item yeah

37:30

there we go all right so now what's the squiggly about

37:36

expected two arguments but got one because ad needs to take in those urls all right

37:42

probably a cleaner way to do that but i'll stick with how i had it prior for right now

37:47

it'll work all right and then we'll do root topics

38:05

yeah switch this to three that way these individual items have a higher priority

38:12

than blog posts i think that makes more sense because then these will come in at the default of 0.5

38:20

so root topics slug urls equals and then we just need to pretty much do the exact same thing for this

38:30

um this one would be topics.show

38:35

slug and

38:41

all right dang it

38:47

there we go okay and then we'll do

38:53

we'll go ahead and do our lessons because those would have a higher price well now we'll go ahead and do our child

39:20

and then on the map we want to change the priority so we'll do this should be

39:26

frequency if i broke that out in the right spot i did not

39:33

weekly keep that the same and then this one

39:41

let's do actually let's leave this as 0.5 and let's change these

39:48

to 0.7 i think

39:57

oh what am i doing

40:06

okay and then this one will leave as default of 0.5 but i'll go ahead and

40:11

hard code that just so that it's easily visual all right so there's our child topics

40:17

and then we want to do our lessons and our blogs so we'll do lessons dot map slug urls equals this

40:25

dot add urls this dot map

40:34

should be lessons.show slug weekly

40:40

that one we would want to be 0.7

40:46

0.6 maybe um and then blogs dot map slug urls equals this dot

40:54

add urls this dot map

41:02

i think this is posts dot show

41:10

and then this one will go ahead and do 0.3 have to be the same as its landing page i guess

41:17

and we'll do one less okay so that should really be it for generating out all of our urls at this point in time we

41:24

should have all of our urls populated so that we can return them back so now

41:30

we can call our generate sitemap from our syndication controller

41:36

so this would be something like const urls equals

41:41

oh wait syndication service dot generate sitemap

41:50

probably make more sense to generate site might be urls

41:55

get yeah i like that better get sitemap urls

42:03

okay because you're not actually generating the sitemap in there you're really not generating it until you actually create the xml items

42:10

so let's see we have the pipeline up there

42:16

we have our urls here so let's map over our urls

42:22

and add them to our sitemap stream

42:40

and then we want to convert our stream to a promise

42:46

pass it in our pipeline which is our creating a gzip from this

42:55

and then our sitemap we would want to cache service dot set

43:04

and

43:10

no let's put that in here

43:16

public static async set sitemap

43:22

uh in cache save sitemap

43:36

not gonna get picky over the name right now cash service

43:42

dot and then we should have a set this sitemap cache key

43:47

and then this will take in

43:53

i don't even know what that would be what would that be so

43:58

and it doesn't want to give me a tight oh it's a buffer i'm just gonna do

44:11

yeah let's try setting that to a buffer and let's see whether or not i'll take it so save up cash

44:16

site map okay

44:22

and then we can get rid of this cache service and instead do that through our syndication service that way we

44:28

don't need to specify the cache key out here and it will be done within it

44:33

so set sitemaps yeah and then sm

44:38

and it looks like it accepted it okay so i guess that is indeed a buffer and then what we want to do is

44:46

sitemap stream dot end and send the stream on off so response

44:54

dot stream our pipeline

45:00

and then if we get an error at all let's go ahead and just

45:07

console.log that out and send off a 500

45:13

okay last thing that we need to do here i think is set it up to a route

45:21

so i have the public one right here i'm just going to go ahead and plop the xml version right after that so we just do

45:27

this as a get savemap.xml

45:35

i think we just named that xml

45:40

okay so if i did everything in that correctly we should be able to go to

45:56

i need to get the website under on the phone let them know about this uh let's see

46:09

looks like i got in there oh wait a minute

46:16

is my shell even started your shell um

46:25

server it would have to be because it was

46:30

clicking around before wouldn't it but where

46:36

i'm gonna go ahead and just restart this i don't know what happened

46:41

oh okay somehow i had another one open

46:47

so let's see can i read properties of null reading replace i don't think i called replace

46:54

anywhere in there did i hmm so that must mean that we have an empty

47:00

string or not an empty string but something that should be a string that's not a string

47:05

oh oh oh oh gotcha let me go into

47:15

um was answering a question on github discussion about how to

47:23

um minify html responses or the edge response and i think that

47:29

is messing up this because we it's happening within that middleware so let me

47:36

try no still no

47:42

try refreshing

47:54

okay no no still does not like that

47:59

okay i doubt the uh network tab is gonna tell me

48:05

anything here but we'll try

48:12

yeah figured out all right so we're not getting back an individual

48:19

error anymore now that we've gotten rid of the minification of the html

48:29

and it specifically says that it does not like the encoding

48:39

no root element found pretty sure i have everything set up the

48:45

way that it currently is on the other one

48:56

so let's break it down let's get rid of the try catch

49:03

let's let our error come to let's

49:08

so the response header should be happening up here setting that to application xml that looks correct and

49:14

then we have the content encoding which is specifically what it says it does not like

49:19

sitting at the gzip so what if we get rid of

49:29

that gzip encoding header

49:34

okay so now it's not well formatted so we're getting a different type of air

49:40

i think probably the air that's causing the gzip air

49:45

so from there could be

49:51

that it could be a cached issue

49:56

all right now we're just not getting anything um

50:04

so let's make sure that we're getting back some urls

50:10

and that the urls are what we expect them to be

50:19

that's a lot of console logs there all right so

50:34

scroll up to the top to make sure let's see so it's a yeah i mean that looks right

50:47

the only reason that it's in a root object is because i console plugged it that way so that should be correct

51:13

try to come to log that out i'm gonna go ahead and get rid of that

51:36

a whole lot of nothing all right well well well

51:53

i thought that would do anything yeah i figured all right

52:00

so it's definitely something in how the xml is being generated because the underlying urls

52:05

look correct um i wonder if maybe i installed the wrong

52:15

well no because i don't did that auto import wrong nope it looks right

52:39

hmm well

52:52

hmm

53:13

i don't know

53:20

i guess let's go take a look at that sitemaps documentation see if anything's changed

53:33

maybe i had a rewrite or something

53:49

um

53:57

all right so up to here it looks correct and then we get down to this stream to

54:02

promise return where it calls readable from stream

54:15

but you would think we should just be able to directly return back the response.stream

54:21

i wonder if maybe i need to i don't think that's right but

54:31

yeah all right

54:40

let's see here

54:48

let's take a look at their full example all right so you can see we're matching

54:54

everything up through here we're creating a gzip that looks correct

55:01

we're writing um our urls we're just mapping over them instead of doing it line by line

55:22

i guess we could try this

55:31

yeah yeah why don't we try that so instead of doing

55:36

this for the cash we'll stream to promise

55:51

and then instead of doing that we'll just return sitemap and then up here we'll change

55:57

sitemap from a const to led give that a save

56:07

not well formatted okay

56:18

so now that we should have something put in there to take a look at let's see

56:24

if we can take a look at maybe the structure of it

56:36

oh my yeah okay it's encoded um

57:05

this point i'm pretty much just guessing what

57:21

i don't even know where that would be coming from

57:31

well something funky is going on then huh let's see we still have the console log in there but i'm

57:38

yeah okay

57:46

well so hold on a minute that doesn't make any sense because you're not like awaiting it

58:16

well that's taking its time that probably means something's wrong

58:28

hmm

58:40

i'm gonna go ahead and stop that so it's not continuously running there

58:46

i don't know i mean we pretty much are matching what this is doing

58:52

um i'm gonna go ahead and do a sanity check here let's see

58:57

here let's see update history maybe

59:02

so i'm probably on 7.1.1 um

59:16

yep let's see what version we're on here

59:21

keep that

59:34

7.0.0 okay um

59:41

let's take a look at the specific version

59:49

yeah seven point zero point zero on the dot so um i'm going to go ahead and

59:54

mpm uninstall how to stop this so it doesn't cause much errors

1:00:00

sitemap

1:00:07

okay npmi sitemap

1:00:13

a specific version

1:00:20

okay let's get this back to how i had it initially

1:00:39

i am going to go ahead and leave the cached version off

1:00:48

okay let's try that let's see if maybe it's a version issue or something

1:00:55

yep yeah that works okay so i'll just roll with the 7.00 version um and because we were

1:01:03

following their documentation for the 7.1.1 pretty much how they had it nothing

1:01:08

changed here from the previous version so maybe it's out of date i don't know um

1:01:15

but yeah so this is the underlying this is this is what we want so we have our url set with the url the

1:01:22

location change frequency priority for the home page to the one and then everything from there goes to point

1:01:28

eight and then for posts we go to point three and then for series actual lists list

1:01:34

them all out so we have in alphabetical order all of our series

1:01:42

um oh okay so one thing i do need to take care of is i have modules coming through

1:01:49

so whenever i create a module um i've got a couple in here that are just

1:01:55

like they get default names and there's a couple i haven't changed to make it evident that you know i have a module populating

1:02:02

somewhere and so these your new collection are modules so modules would

1:02:07

be an item within a specific um so here let me go to localhost 333

1:02:14

within a specific series so if i were to go into our let's learn adonis series we have modules so you have less than

1:02:20

1.0 that's module 1 or 0 if you're going index based and then you go to 2 that's

1:02:27

another module so all of these lessons are bound to those specific modules and that's

1:02:33

probably a little bit more easier to highlight um if i were to go into

1:02:42

the actual collection here so let's see let's learn so you have the individual modules over

1:02:48

here so you have like a module for introduction routing controllers and then each module that gets the lesson

1:02:55

or the collection as a whole can get the lesson set as we are with i think like the bouncer

1:03:01

series and all that so here you can see there are no modules it's just straight direct posts

1:03:07

so i want to do a specific check within here where we're populating our data let's go ahead and get this back to how

1:03:13

we had it here so let's put our catch back in here let's do our caching

1:03:18

um i'm gonna go ahead and get rid of that

1:03:23

comment clean that up a little bit and let's go into our authentication service

1:03:33

and within here we want to do

1:03:38

an additional wear

1:03:44

null parent id i'm gonna go ahead and break this down

1:03:50

as well make it a little bit more readable should be doing that for all of

1:03:56

them but uh we'll just do it for that one for right now and it's now if we refresh

1:04:02

we're back to a content encoding here okey dokey

1:04:08

oh that's probably using the old cache i need to figure out how i can clear my cache i get a gui in here or something

1:04:13

for it yeah so it's the cache version is using the broken

1:04:19

um and so now you can see within our so now

1:04:26

we're in the topics instead of uh series so within the series now it's all just root level series and it doesn't have

1:04:32

any of those modules within that which is correct because you can't directly link to a module um well you can but

1:04:42

not within this sense it's like a hash link to the actual series page so it wouldn't

1:04:48

be good to have it here then we got all of our topics you can see that the subtopics are in there as

1:04:53

well and then it goes into the lessons and then we don't have any blog posts but those would be last if they were in

1:05:00

here and so there we go and so the last thing that i would need to do is check and make sure that all of the lessons are in

1:05:06

here but i'll worry about that later that won't drag you guys through that so

1:05:12

last thing to do would be to make sure that the cache is working

1:05:20

and that we are setting the cache to a specific time that we want this to do so

1:05:26

within here let's go ahead and set this to have a time to life of

1:05:37

eighty-six thousand four hundred uh

1:05:43

it's either 24 or 48 hours that's the best day i got

1:05:48

i'm using the old number that i had on the previous site there so um

1:05:54

yeah so that should cache it for that long and then once that happens it will clear that cached item out

1:05:59

and then it will come back through and recache it and so at this point we need to clear

1:06:04

out the actual cache version and i don't think i have a redis

1:06:12

i don't know the cli for redis that well

1:06:19

so um let me see if i can find one real quick so mac

1:06:26

redis gooey

1:06:54

come on all right i'll tell you what i'll help you out

1:07:03

what was that movie oh oh now you want okay go ahead

1:07:08

there we go all right you gotta click it twice

1:07:31

oh boy oh man yeah i haven't cleared this out in a while um

1:07:40

go ahead and delete you and so now if we come back into here and refresh it should work and there we go

1:07:47

so um let me double check i did uncomment the cache yes okay and so now if we retur or to refresh

1:07:54

this let's get rid of bull stuff come back down here we do have this and

1:07:59

it is now properly cached um so that seems to be working okay and

1:08:06

you can see it's formatted as gzip and all that fun stuff so we can test

1:08:11

this once more by adding a post and then we can do a search for that post to ensure it's not

1:08:17

in there and then we'll clear the cache so this is a test um

1:08:25

all right we'll leave all that stuff empty we'll put this in here as let's have it published as yesterday

1:08:33

and okay nothing else let's go ahead and publish

1:08:39

it it should show up on the home page it doesn't have an image so it'll be down here

1:08:44

okay and so now if we refresh this

1:08:50

bollocks why

1:08:55

all right let's clear the cash out again probably something to do with

1:09:04

oh let's see

1:09:14

hmm

1:09:20

let's try streaming instead of sending it i don't know i think the forehead said oh you know

1:09:26

what it could be that i'm directly returning the response.send

1:09:31

pardon my dog so let's kick that return off onto another line and test that

1:09:38

nope that wouldn't

1:09:50

let's confirm that it is just a caching issue thing here yep

1:09:57

all right

1:10:06

so i guess let's see what we are getting back from this

1:10:18

um yeah it wouldn't be up there it would be

1:10:23

down here

1:10:34

hmm

1:11:06

yeah so it's definitely an issue in either how it's being cast or home responding from the cache

1:11:14

and that does not look like a proper just send it back response there um

1:11:37

hmm

1:11:48

which is weird because it should be correct

1:12:21

alrighty then

1:13:50

uh

1:14:29

yeah it's definitely not it's just a string

1:14:39

it could be so a match what i have before it could be a difference between

1:14:45

so prior i was using node cache and in this app i'm using redis so it could

1:14:51

be a difference with that as well um

1:15:04

switch that back to a buffer

1:15:17

let me get rid of this let's see what the raw response is let's try that

1:15:24

yeah and then if you get rid of the cache what's the wrong response there

1:15:34

nothing did i switch that back to a stream yes

1:15:39

okay was looking to see whether or not it

1:15:45

would match

1:15:56

peculiar all right well

1:16:14

just does not want to work for me oh okay i left that in there that could be

1:16:22

part of the issue i guess maybe oh and it downloaded it

1:16:29

all right let's open that up i don't know why sublime is yeah so

1:16:35

that looks

1:17:07

all right well i don't know i'll have to play with play around with that and see and if i figure it out i will put that

1:17:13

down into the comments it could be something with the package it could be

1:17:20

i don't know um

1:17:28

is what we have definitely matches what i was using prior

1:17:34

it's over here and this works just fine

1:17:53

unless maybe the cash always came back empty i don't um

1:18:02

by the way i don't have enough information in there to where like not using the cache would

1:18:08

take a super long time so

1:18:15

i can always just leave it as this for right now oh and all right let's see i took out the headers didn't i i sure did

1:18:34

so what we could it could be

1:18:41

maybe we need to just rebuild the pipeline again and just send it down that way

1:19:04

um

1:19:09

because then it would already be gzip it wouldn't make any sense

1:19:35

you

1:19:51

i guess we could always just cache the underlying url structure um

1:20:00

wouldn't be crazy about that though but it would be something

1:20:16

so then that would look like this we would have

1:20:21

um

1:20:27

that and then we would have

1:20:33

this and then we would have that

1:20:41

and then this would be in here as well so well no that wouldn't be

1:20:48

and we would have

1:21:01

let's try that

1:21:38

yeah so there it didn't even get to the caching part um so urls might have came back with

1:21:44

something somehow even though i deleted cache

1:22:06

oh yeah

1:22:33

already

1:22:58

ah there we go okay didn't like how i had it

1:23:04

and so now that should be i'll have to play around and see if i can get get it working how i had it

1:23:10

before to where the underlying like actual xml structure was cached but this

1:23:15

would be better than nothing at least having all of the database query stuff cached

1:23:21

um so so long as that's pulling in okay then i'd be happy so

1:23:27

uh let's see if we can get rid of these shouldn't be in there anyway and then come back in here and let's make sure

1:23:35

that this all works okay still including information from the cache so let's

1:23:41

refresh again and it doesn't

1:23:46

um it's probably because it's coming through as a string at that point

1:23:51

if i can yeah so i need a parse

1:23:58

so what i'll do is let's rename this to

1:24:09

cache sitemap urls and get

1:24:15

cached urls

1:24:35

okay no oh can't spell return right here we go

1:24:42

no oh wait no wait do i need to wait yes you oh that could

1:24:47

be the whole freaking issue was i waiting in here yeah okay well that never mind

1:24:55

i think you have something all right so there we go so we'll get the item from the cache and we'll parse it we'll

1:25:02

return this back as a purse item um and then we'll do just a simple

1:25:13

nope not on the cache key on the cache item and so this will be urls

1:25:21

uh url or sitemap

1:25:26

item array

1:25:36

okay so now that will stringify and parse it for us and we need to change our names to get cached sitemap urls and

1:25:43

cache sitemap urls

1:25:49

prefix that was set okay so

1:25:54

set cache sitemap urls and get rid of this json.stringify

1:26:01

we'll make that a little bit cleaner and then

1:26:06

get cache to sitemap urls okie dokie

1:26:12

now we should be all right just get rid of that line all together save this

1:26:19

give it another go all right that seems to have worked it seems to still be working

1:26:26

just try deleting the cache

1:26:36

there we go all right so yeah the underlying issue was something to do with the buffers which

1:26:41

i really probably don't understand well enough to get into so this is a better

1:26:46

way for me anyway so it's caching the underlying urls

1:26:52

so here we have the sitemap so you can see now we're not caching the xml result but instead we're caching

1:26:58

just the underlying database information that's queried and then repopulating the actual stream

1:27:04

and streaming that off as the response so the heavy lifting is still being cached off which is rebuilding the xml

1:27:11

format for each request which isn't too big of a deal for this size of a site so

1:27:17

all right took a bit longer than i was hoping but we got the end result at the end of the day anyway so

1:27:23

yay um thank you all for watching and i will see you all in the next one um

1:27:29

i'll try to stream in between now and the

1:27:35

whenever i get back into my regular posting schedule which should be the first week of may so i'll try to do a couple of streams in

1:27:41

between then and now the last week of april was probably going to be pretty quiet well no

1:27:48

yeah the last week of april will probably be pretty quiet for me i probably won't stream that weekend

1:27:54

um because that's the actual weekend on moving so yeah

1:27:59

so i will see

Join The Discussion! (0 Comments)

Please sign in or sign up for free to join in on the dicussion.

Playing Next Lesson In
seconds