Building An XML Sitemap

In this stream, we'll be going over how to build a sitemap of all our site's pages for search engines using XML to help bolster SEO.

Published
Apr 10, 22
Duration
1h 28m

Developer, dog lover, and burrito eater. Currently teaching AdonisJS, a fully featured NodeJS framework, and running Adocasts where I post new lessons weekly. Professionally, I work with JavaScript, .Net C#, and SQL Server.

Adocasts

Burlington, KY

Transcript

Looking for something specific? Feel free to search the page for keywords to find a specific portion of the live stream that interests you most.

0:06

alrighty so it's been a little bit since i've posted anything so i thought it'd be about time that i live stream

0:12

something just to get something out to you guys uh so the new site is coming along great um

0:20

the home page is all populating for the user and everything um all of the sub pages and everything are

0:26

pretty much hammered out one of the last things remaining that i do need to do

0:31

is add a sitemap to it now i have the user facing sitemap already populated

0:36

within the footer there but what i need is one for search engines so if we were to go to the

0:42

current site and go to sitemap.xml we'll get an xml version of the sitemap

0:48

that contains all of the links within the actual site along with how much they

0:54

should be re-requested or we checked for and then what their priority on the site

1:00

is obviously the home page being the most priority and then series and topics

1:05

and lessons from there so i thought we'd do a little live stream where we

1:11

kind of recreate that xml sitemap within the new site um so

1:19

really only two things within the old site that took care of this at a service that really just

1:25

queried the information generated out the actual structure and then we had the

1:32

controller function uh that actually generated out what the response should look like

1:38

and send it along as xml so let's go ahead and get started with that within the current project i do believe

1:45

i have a syndication controller already created

1:53

and what this one's taken care of currently is just the user-facing site map so uh it's not quite a whole lot of

1:59

information uh the one for google will be querying a lot more um

2:05

because this user facing one is just more of a generic listing

2:10

but yeah so to start out with what we're going to want is an actual controller

2:16

endpoint to show the xml version so let's do public async xml

2:22

and this will want a response and that should actually be it

2:27

and then this will be http contacts contract and then from there

2:34

let's see so we'll probably go straight into

2:40

wanting to get the information and then we'll have our response dot

2:48

header we'll want to set the content type to

2:54

application xml response.header here header again this one will be

3:01

content encoding which will be gzip

3:06

and then let's see here so

3:17

let's start with that for right now now i believe i was using a package on the

3:22

old one um

3:27

not within there so we have zlib for creating the actual

3:32

gzip and then we have it looks like sitemap for creating the sitemap stream

3:38

so i'm going to go ahead and install those two dependencies so that was

3:43

mpmi sitemap and zlib

3:55

okay looks like we have those

4:01

all right so whenever it comes to the actual um

4:10

sitemap package what we are going to want to do is pass it in

4:15

the underlying host name so let's wrap this inside of a try catch

4:23

okay and then let's do uh const

4:29

site map stream equals new let's see if we can get this to auto

4:34

complete here site map yep there it is site map stream

4:40

and we just want to pass it the host name in this case it's going to be https

4:45

slash slash yeager.com all right and then we're going to want

4:52

to pipe into our sitemap stream so let's do

4:58

if i can spell pipeline sitemap stream oops

5:04

sitemap stream dot pipe and then we'll want to actually create

5:11

let's see if this will uh create gzip and that should be just fine just like

5:16

that and then

5:23

let's see here actually so from cash

5:30

if exists so let's try to get the information from the cash

5:36

if we can so we'll do const

5:43

sitemap equals and then for right now let's just do an empty

5:49

array and this will be from the cache we'll set the getter for that here in a second

5:56

but if sitemap and actually we just set this to null

6:01

because if it's not in the cache it won't come back as anything we will do response dot

6:08

send the segment and we want to kick that off right there

6:15

otherwise once we've created all this stuff this is where we'll want to

6:21

actually get the info if it's not cached and from here that's

6:28

where we would want to probably plop that off into a service so let's jump down to our services i

6:34

don't think i have a syndication service so let's go ahead and create that real quick

6:45

all right

6:52

actually yeah that'll be fun okay

6:59

so for the um

7:05

old one here for the syndication service i had types defined within the surface itself i'm gonna go ahead

7:11

and plop those just copy if i can oh what am i trying to do here i'm gonna go ahead and cop those into a contract file

7:17

uh so let's go down to contracts and let's create a new file here as well

7:23

called syndication and i'm just gonna plop those in there

7:29

so we have sitemap item which contains the url the change frequency and the priority which aligns

7:34

with if i still have it open what the actual xml items are

7:40

and then we just have the frequency which is what the change frequency values should be which is either daily

7:45

weekly or monthly so now that we have that in there let's go back into our syndication service

7:54

and so first and foremost we're going to want to cache this if

8:02

uh we can so let's do public static

8:09

let's do sitemap cache key i'll be if string and we'll just

8:17

cache that as sitemap and then we'll want a public static

8:25

um let's see

8:34

so on the old one here it looks like i had a getter and i had the sitemap here actually private

8:40

we could mimic that i don't think we need to and then there is how i was checking it

8:46

so i do still have that cache service it does integrate with a different cache now it integrates with redis so

8:55

let's go back up into our controller and go ahead and get that going out

9:01

so instead of that we can do

9:06

well let's do a get directly within our syndication service instead of doing it outside so

9:12

we can switch this to private i guess and then let's do public

9:18

static get sitemap from

9:23

cache or something like that um and then we will do return

9:30

cache service it looks like my autocomplete gave up on me oh no

9:37

cash service or i just can't spell nope nope it did give up

9:42

well shucks

9:48

all right let me uh switch files real quick let's go into you and then back into you see if that fixed it

9:55

it did not all right i'm going to close it real quick

10:01

and let's see if that fixed it so public static there we go all right

10:07

get sitemap from cash

10:12

um now at this point we want to return cash service there you are

10:19

dot get um and then this dot site map cache key

10:27

is this async i cannot remember

10:32

yes it is so we'll want this to be public static async and

10:38

yeah there we go

10:45

hey man um i plan on getting back to recording new things

10:51

well i'm hoping to record in between now and may but i don't intend on actually posting

10:58

any new recordings until we reach may once we reach may that's whenever i'll be in my new house and i'll have

11:06

everything set up and i won't have to deal with packing or anything like that anymore so um

11:11

just going to kind of take this time in between now and may to get some recordings in place and then we should

11:17

start seeing some new stuff in may so

11:23

should be early may

11:31

yeah so i'd say the first week of may we should get something out

11:36

um and then from there it should be pretty much weekly hopefully

11:42

so we have the get site map from cash

11:49

and we have that oop that's not where i meant to go we have that in our syndication

11:57

controller where are you here you are no we do not so let's go ahead and do that

12:04

so const site map equals await

12:10

uh syndication service dot get sitemap from

12:15

cache so now if we get the sitemap from the cache and it does exist then we can just return that cached version otherwise we

12:23

need to try and generate it and that's where this would come into play down here

12:29

so we'll start on that next so

12:34

we're going to have the intro function for this that takes care of

12:39

kind of generating and populating everything together as probably i guess generate site map would

12:45

make the most sense um and then ultimately what we'll want this

12:50

to do is return back a promise of sitemap item as an array

13:01

so first and foremost what we're going to want to do within here is actually query out all of our different information so

13:07

we're gonna have our series topics lessons

13:14

and blog posts and then from there we'll do each like

13:20

subordinate page or whatever you want to call it as individual stuff from there so like

13:26

we're going to have a section for frequently asked questions resources uses and then if we take a look at the footer

13:31

there's some stuff for like guidelines contact the visual site map cookies privacy terms and all that fun

13:37

stuff so all that stuff can go in there just manually defined but this stuff needs to be dynamically pulled

13:44

um actually

13:49

these are going to be database generated the frequently asked questions and resources um

13:55

so but for right now those are empty so i'll just put those in there manually at this point

14:04

so this will be a good starting point so for our series

14:09

we're going to have a weight and trying to think if i have something in

14:14

the series service for this that would be useful i want to say no so let's just query it

14:21

directly off of our collection model oops yes there we go all right

14:28

and then from here we specifically want to just grab the series um don't plan on populating

14:34

uh uh playlists or courses at this point in

14:40

time um and then let's see

14:49

from this all that we should need is

14:56

the slug so i have a

15:05

well i guess i should change the name of that so i have this function defined

15:13

where do i have that to find that that is within my query builder provider called select

15:20

ids now this will execute the query and then just select the id column ideally i

15:25

guess i should just be named select column since it doesn't specifically need to be

15:31

an id and then the series function will kick

15:37

off a query builder and then it will execute a where statement to make sure that it's of type series

15:43

uh so that's just a nice little shorthand that i have there so i'm gonna go ahead and just do select

15:48

ids this won't make any sense at this present point in time and name this

15:56

i think i just need the slug let's check and see whether or not

16:01

that's the right so at work they're called aliases i think i call them slugs

16:08

where am i here collections did i click on it yes i did yes slugs

16:14

all right and that should really be it for our series oh wait a minute

16:21

dot where public i have another query builder helper that will specifically do this

16:27

for the type where it is public so build that into there and we have a nice simple query for all of our series right

16:33

there and we can do the same thing for topics await top

16:40

and the model for this is called taxonomy and then

16:47

trying to think do i just want the roots of the taxonomies or do i want all of the children as well so like if we go

16:53

into topics any taxonomy is technically a topic um

16:58

but then these are all of the root level ones and then like if we were to click on adonis js there's

17:04

subtopics within it so you have like the acli authentication i guess it would make sense so let me

17:11

see do i have these as no they are root level so like if i were to go into hdb context

17:16

it's still the same url so let's just query all of them we'll get the roots first so we'll call

17:22

these i guess root topics and then we'll do child topics below that as a lower priority

17:30

so we'll do roots and that's similar to the series

17:36

it's just going to kick off query builder where it's specifically a taxonomy that doesn't have a parent id

17:44

um those are not those don't have a state id

17:50

and then so here it should just be select the slug from that as well and then we'll do const

17:57

child topics await and i don't think i have a helper

18:04

like roots for this but we can go ahead and bake that in this might be the only use case where i would need it but

18:10

see i always lose my models there they are taxonomy

18:18

i need to clean my screen off it's super dusty

18:25

where are you roots there it is all right so just a simple public static

18:36

deck it's just children and then that's just a simple we're not

18:42

null parent id

18:48

um you could probably just do the

18:53

conditional parent id in there i've typed number

19:00

default that to null actually i don't think you need a question mark an id

19:05

such as squiggly because it's not used type number

19:12

there we go and then we could do if parent id [Music]

19:18

return this query dot where

19:24

parent id is parent id otherwise just get it where it's not not null

19:31

alrighty so that should do it for that so then we could do children we don't want a parent id there

19:38

and then we can just do select ids for slug again i need to rename

19:44

select ids to just like select column or something like that hey tina

19:55

thanks for stopping by um so that should do it for topics

20:01

and then we have our lessons which

20:07

well you know what let's um kind of order to these as well i guess that would make sense with it

20:13

so let's order by

20:22

i guess let's have the newer ones first does that make sense like i guess

20:27

so created at

20:32

descending

20:39

dang it missing my shift key yeah i think that makes sense so we'll do newer ones first

20:47

yeah yeah let's do this alphabetically it'll be easier to um verify that we're getting everything

20:52

that way order by name decent wait no a something

20:59

for name there we go

21:08

okay and then for lessons

21:18

uh that goes off of the post model dot lessons

21:24

dot and then from there from there i want apply i want to apply

21:31

a scope and the scope is to ensure that the post is published

21:37

and then we'll do our order by from there uh that one would be title ascending

21:47

select ids for select yeah they're all selected

21:53

all right so there's our lessons and let's just change this to posts and

21:58

then we'll group them all within there so const lessons const

22:03

posts equals await post dot i think i named them post nope

22:08

want to name them blog blogs all right let me name this to match that

22:15

apply scope now i don't think i have any blogs populated yet uh so that will be empty

22:22

but we'll go ahead and take care of them because i intend on publishing some here soon

22:28

descending select id select okay so if i have this

22:34

working properly this should come back as just an array of slugs um

22:40

if not then we'll find out but like so if i were to go into tinker here or tinker um rebel

22:47

node ace rebel

22:52

load models and we do await

22:59

models collection dot

23:06

series dot

23:12

public dot select ids

23:18

slug we should just get back an array of slugs and that we do so it seems to be working all right

23:24

so i should be able to expect that for all of these so now that we have the

23:31

dynamic information that we're going to need to make this site map something um

23:37

next what we want to do is actually create the items and then from there we can go ahead and

23:46

excuse me kind of map everything together so we're going to have

23:52

a private static function called make

23:58

this will take in the url

24:03

of type string the change frequency of type

24:08

freak when see and that did not want to

24:16

what do i got going on here

24:25

oh i was putting that inside of the generator okay my bad

24:34

okay there we go i guess frequency already auto imported because it doesn't have a squiggly on it

24:39

so it should be okay and then the last one will be the priority which

24:48

i think that would just be type number and then we can default that to a half item and then here we'll return back a

24:55

finalized sitemap item from here so what we want to do is pretty

25:02

much just return back an object of those arguments so we'll have url change frequency and priority

25:13

i kind of prefer this approach over you know just in line creating an object

25:18

over and over again mapping or naming the keys of the object over and over again

25:23

for each item um so now that we have this we go ahead and start our actual url collection so

25:28

we'll do let your well now we can actually make this uh const urls equals

25:36

let's specify this as a sitemap item array equals

25:43

an array and then here we'll go ahead and define our top level

25:49

hard-coded urls so we can do this.make

25:54

um i'm just going to hard code the url in here since they're pretty short do weekly for that one and then the

26:01

priority for this since it is the home page will be 1.0

26:06

um and then i'm going to go ahead and i believe all of these are the same

26:13

so i'm just going to go ahead and copy and paste this instead of writing it all there we go

26:21

so we have so we're essentially just making a site map item for our home page our series

26:26

landing page topics landing page lessons landing page and our blog landing page obviously the blog landing page is going

26:32

to be less important than all of the other ones so once we have that in place then what we can do

26:39

is go through and add in all of our dynamic items um so

26:46

for that we're going to want kind of a different helper from make we're going to want something like

26:51

private static function called map and this will take the route name

26:57

as a string any params um

27:05

did what what is that that's x um it's been a while since i've wrote

27:10

this x string any i think i think that's accurate

27:16

and then the change frequency as frequency

27:23

and let's default that to weekly and then the priority

27:28

is again a type of number and we'll default that to 0.5

27:34

and then this 2 will return back a site map item for the arguments passed

27:39

in here so

27:45

we can do const url and then what we're going to want to do is use the route module to actually

27:50

just generate out this url so let's do make url we'll pass in the route name

27:56

along with those defined parameters i shouldn't need any query strings or

28:01

anything like that for this um it should just be simple to leave it as the round name and params

28:07

and then if we do not get back a url from that make

28:13

url if it's not mapped for whatever reason then we would want to just return this

28:18

dot make define it as a hard-coded url with the change frequency

28:24

and the priority

28:30

i guess we want to make that more undefined

28:38

that's how i had an old one yes

28:46

all right so that should be good for the mapping one so now we'll want to go through and actually just map each of so

28:53

our series topics and posts so that we get those items actually

28:59

created so we do series dot map and we'll just loop over each and every series that we

29:04

have and then this will be the slug since we've specifically selected just the slug

29:10

value as we've tested here so it's just be an array of strings with the slug values

29:18

and then we'll want to append this into our urls

29:24

so this will be urls dot push

29:34

this dot add oh we need to make that yet don't we

29:43

what was that wrong file

29:55

i don't think we need that i don't know why i had it in there for some reason the old one but

30:00

i guess i needed to do conditional determination for them but we should not need that in this one so it should just be a simple push

30:12

this dot map and then the

30:18

route name for our series should be series dot show

30:25

and so map is pretty much just

30:30

matching like the make url format here and then we'll have slug right in there

30:38

and it does not like that y

30:44

series came back as a number

30:50

huh that's peculiar

30:58

promise number huh what in the world is that doing it for all of us

31:04

yeah it is huh that's odd do i have that

31:11

i've used that elsewhere it's kind of odd um

31:20

okay i guess let me take a look at the contract definition for that and see exactly what i have that coming back is

31:29

worst case scenario all write it where is that

31:36

database select oh sure enough oh yeah because it's ids

31:43

right um all right so i guess that would specifically just

31:49

be a number there wouldn't it all right um tell you about let's go ahead and create

31:55

a different one then so

32:00

let's do model query builder macro

32:08

select column async function i'll leave that one it's

32:13

just the id and i want to do this one is column name string equals

32:22

i guess we'll just make that required

32:28

and then we'll do const results equals oh wait this dot select

32:36

column name return results dot map r r column name and it

32:45

doesn't like something i have here

33:07

yeah there we go parentheses in the correct spot okay

33:12

so we have that now let's go ahead and just add that into here so let's see i did that after that so this would be

33:17

select column

33:22

oops to go back and check the name of that yeah select column

33:31

column name string and then we should just be able to do

33:38

something similar to the first or up here i think equals undefined return back

33:46

promise

33:51

of t

33:57

array i think i don't know we'll test it

34:03

worst case scenario i'll just put ads in there for right now um so

34:08

change this to select column

34:16

i feel like that's wrong but let's see here is that returning back

34:22

oh an undefined array all right that makes sense

34:30

i'm just going to hard code that as a string for right now it's gonna hurt anything

34:39

yeah oh wait no that needs to be a string array

34:47

there we go okay not ideal but it'll work for right now

34:54

and we'll just map switch all of these to that

35:03

okie dokie so now this should be happy maybe argument of type site map item is

35:11

not assignable to parameter f type

35:18

oh okay oh that's why i had this dot ad because

35:23

they were nullable and if it came back as null then uh

35:29

yeah you don't want to add it do you so

35:34

okay yeah we could do that private static add

35:40

or i guess let's just do push we'll do add

35:46

switch that to a let we'll do urls equals

35:55

this if i can spell this dot add and then we'll do add here

36:02

and i think this needs to take in um

36:14

urls

36:20

right because this would return back okay um

36:27

so i think do cite my vitamins there as an array of

36:35

sitemap item and then

36:44

no condition so this is the actual array and then this is gotcha

36:52

and then conditional add as a site map

36:59

item array or undefined

37:04

there we go so then we'll do if not

37:11

conditional add and just return urls otherwise return

37:18

urls and additional evidence this is just one

37:25

item isn't it site map item yeah

37:30

there we go all right so now what's the squiggly about

37:36

expected two arguments but got one because ad needs to take in those urls all right

37:42

probably a cleaner way to do that but i'll stick with how i had it prior for right now

37:47

it'll work all right and then we'll do root topics

38:05

yeah switch this to three that way these individual items have a higher priority

38:12

than blog posts i think that makes more sense because then these will come in at the default of 0.5

38:20

so root topics slug urls equals and then we just need to pretty much do the exact same thing for this

38:30

um this one would be topics.show

38:35

slug and

38:41

all right dang it

38:47

there we go okay and then we'll do

38:53

we'll go ahead and do our lessons because those would have a higher price well now we'll go ahead and do our child

39:20

and then on the map we want to change the priority so we'll do this should be

39:26

frequency if i broke that out in the right spot i did not

39:33

weekly keep that the same and then this one

39:41

let's do actually let's leave this as 0.5 and let's change these

39:48

to 0.7 i think

39:57

oh what am i doing

40:06

okay and then this one will leave as default of 0.5 but i'll go ahead and

40:11

hard code that just so that it's easily visual all right so there's our child topics

40:17

and then we want to do our lessons and our blogs so we'll do lessons dot map slug urls equals this

40:25

dot add urls this dot map

40:34

should be lessons.show slug weekly

40:40

that one we would want to be 0.7

40:46

0.6 maybe um and then blogs dot map slug urls equals this dot

40:54

add urls this dot map

41:02

i think this is posts dot show

41:10

and then this one will go ahead and do 0.3 have to be the same as its landing page i guess

41:17

and we'll do one less okay so that should really be it for generating out all of our urls at this point in time we

41:24

should have all of our urls populated so that we can return them back so now

41:30

we can call our generate sitemap from our syndication controller

41:36

so this would be something like const urls equals

41:41

oh wait syndication service dot generate sitemap

41:50

probably make more sense to generate site might be urls

41:55

get yeah i like that better get sitemap urls

42:03

okay because you're not actually generating the sitemap in there you're really not generating it until you actually create the xml items

42:10

so let's see we have the pipeline up there

42:16

we have our urls here so let's map over our urls

42:22

and add them to our sitemap stream

42:40

and then we want to convert our stream to a promise

42:46

pass it in our pipeline which is our creating a gzip from this

42:55

and then our sitemap we would want to cache service dot set

43:04

and

43:10

no let's put that in here

43:16

public static async set sitemap

43:22

uh in cache save sitemap

43:36

not gonna get picky over the name right now cash service

43:42

dot and then we should have a set this sitemap cache key

43:47

and then this will take in

43:53

i don't even know what that would be what would that be so

43:58

and it doesn't want to give me a tight oh it's a buffer i'm just gonna do

44:11

yeah let's try setting that to a buffer and let's see whether or not i'll take it so save up cash

44:16

site map okay

44:22

and then we can get rid of this cache service and instead do that through our syndication service that way we

44:28

don't need to specify the cache key out here and it will be done within it

44:33

so set sitemaps yeah and then sm

44:38

and it looks like it accepted it okay so i guess that is indeed a buffer and then what we want to do is

44:46

sitemap stream dot end and send the stream on off so response

44:54

dot stream our pipeline

45:00

and then if we get an error at all let's go ahead and just

45:07

console.log that out and send off a 500

45:13

okay last thing that we need to do here i think is set it up to a route

45:21

so i have the public one right here i'm just going to go ahead and plop the xml version right after that so we just do

45:27

this as a get savemap.xml

45:35

i think we just named that xml

45:40

okay so if i did everything in that correctly we should be able to go to

45:56

i need to get the website under on the phone let them know about this uh let's see

46:09

looks like i got in there oh wait a minute

46:16

is my shell even started your shell um

46:25

server it would have to be because it was

46:30

clicking around before wouldn't it but where

46:36

i'm gonna go ahead and just restart this i don't know what happened

46:41

oh okay somehow i had another one open

46:47

so let's see can i read properties of null reading replace i don't think i called replace

46:54

anywhere in there did i hmm so that must mean that we have an empty

47:00

string or not an empty string but something that should be a string that's not a string

47:05

oh oh oh oh gotcha let me go into

47:15

um was answering a question on github discussion about how to

47:23

um minify html responses or the edge response and i think that

47:29

is messing up this because we it's happening within that middleware so let me

47:36

try no still no

47:42

try refreshing

47:54

okay no no still does not like that

47:59

okay i doubt the uh network tab is gonna tell me

48:05

anything here but we'll try

48:12

yeah figured out all right so we're not getting back an individual

48:19

error anymore now that we've gotten rid of the minification of the html

48:29

and it specifically says that it does not like the encoding

48:39

no root element found pretty sure i have everything set up the

48:45

way that it currently is on the other one

48:56

so let's break it down let's get rid of the try catch

49:03

let's let our error come to let's

49:08

so the response header should be happening up here setting that to application xml that looks correct and

49:14

then we have the content encoding which is specifically what it says it does not like

49:19

sitting at the gzip so what if we get rid of

49:29

that gzip encoding header

49:34

okay so now it's not well formatted so we're getting a different type of air

49:40

i think probably the air that's causing the gzip air

49:45

so from there could be

49:51

that it could be a cached issue

49:56

all right now we're just not getting anything um

50:04

so let's make sure that we're getting back some urls

50:10

and that the urls are what we expect them to be

50:19

that's a lot of console logs there all right so

50:34

scroll up to the top to make sure let's see so it's a yeah i mean that looks right

50:47

the only reason that it's in a root object is because i console plugged it that way so that should be correct

51:13

try to come to log that out i'm gonna go ahead and get rid of that

51:36

a whole lot of nothing all right well well well

51:53

i thought that would do anything yeah i figured all right

52:00

so it's definitely something in how the xml is being generated because the underlying urls

52:05

look correct um i wonder if maybe i installed the wrong

52:15

well no because i don't did that auto import wrong nope it looks right

52:39

hmm well

52:52

hmm

53:13

i don't know

53:20

i guess let's go take a look at that sitemaps documentation see if anything's changed

53:33

maybe i had a rewrite or something

53:49

um

53:57

all right so up to here it looks correct and then we get down to this stream to

54:02

promise return where it calls readable from stream

54:15

but you would think we should just be able to directly return back the response.stream

54:21

i wonder if maybe i need to i don't think that's right but

54:31

yeah all right

54:40

let's see here

54:48

let's take a look at their full example all right so you can see we're matching

54:54

everything up through here we're creating a gzip that looks correct

55:01

we're writing um our urls we're just mapping over them instead of doing it line by line

55:22

i guess we could try this

55:31

yeah yeah why don't we try that so instead of doing

55:36

this for the cash we'll stream to promise

55:51

and then instead of doing that we'll just return sitemap and then up here we'll change

55:57

sitemap from a const to led give that a save

56:07

not well formatted okay

56:18

so now that we should have something put in there to take a look at let's see

56:24

if we can take a look at maybe the structure of it

56:36

oh my yeah okay it's encoded um

57:05

this point i'm pretty much just guessing what

57:21

i don't even know where that would be coming from

57:31

well something funky is going on then huh let's see we still have the console log in there but i'm

57:38

yeah okay

57:46

well so hold on a minute that doesn't make any sense because you're not like awaiting it

58:16

well that's taking its time that probably means something's wrong

58:28

hmm

58:40

i'm gonna go ahead and stop that so it's not continuously running there

58:46

i don't know i mean we pretty much are matching what this is doing

58:52

um i'm gonna go ahead and do a sanity check here let's see

58:57

here let's see update history maybe

59:02

so i'm probably on 7.1.1 um

59:16

yep let's see what version we're on here

59:21

keep that

59:34

7.0.0 okay um

59:41

let's take a look at the specific version

59:49

yeah seven point zero point zero on the dot so um i'm going to go ahead and

59:54

mpm uninstall how to stop this so it doesn't cause much errors

1:00:00

sitemap

1:00:07

okay npmi sitemap

1:00:13

a specific version

1:00:20

okay let's get this back to how i had it initially

1:00:39

i am going to go ahead and leave the cached version off

1:00:48

okay let's try that let's see if maybe it's a version issue or something

1:00:55

yep yeah that works okay so i'll just roll with the 7.00 version um and because we were

1:01:03

following their documentation for the 7.1.1 pretty much how they had it nothing

1:01:08

changed here from the previous version so maybe it's out of date i don't know um

1:01:15

but yeah so this is the underlying this is this is what we want so we have our url set with the url the

1:01:22

location change frequency priority for the home page to the one and then everything from there goes to point

1:01:28

eight and then for posts we go to point three and then for series actual lists list

1:01:34

them all out so we have in alphabetical order all of our series

1:01:42

um oh okay so one thing i do need to take care of is i have modules coming through

1:01:49

so whenever i create a module um i've got a couple in here that are just

1:01:55

like they get default names and there's a couple i haven't changed to make it evident that you know i have a module populating

1:02:02

somewhere and so these your new collection are modules so modules would

1:02:07

be an item within a specific um so here let me go to localhost 333

1:02:14

within a specific series so if i were to go into our let's learn adonis series we have modules so you have less than

1:02:20

1.0 that's module 1 or 0 if you're going index based and then you go to 2 that's

1:02:27

another module so all of these lessons are bound to those specific modules and that's

1:02:33

probably a little bit more easier to highlight um if i were to go into

1:02:42

the actual collection here so let's see let's learn so you have the individual modules over

1:02:48

here so you have like a module for introduction routing controllers and then each module that gets the lesson

1:02:55

or the collection as a whole can get the lesson set as we are with i think like the bouncer

1:03:01

series and all that so here you can see there are no modules it's just straight direct posts

1:03:07

so i want to do a specific check within here where we're populating our data let's go ahead and get this back to how

1:03:13

we had it here so let's put our catch back in here let's do our caching

1:03:18

um i'm gonna go ahead and get rid of that

1:03:23

comment clean that up a little bit and let's go into our authentication service

1:03:33

and within here we want to do

1:03:38

an additional wear

1:03:44

null parent id i'm gonna go ahead and break this down

1:03:50

as well make it a little bit more readable should be doing that for all of

1:03:56

them but uh we'll just do it for that one for right now and it's now if we refresh

1:04:02

we're back to a content encoding here okey dokey

1:04:08

oh that's probably using the old cache i need to figure out how i can clear my cache i get a gui in here or something

1:04:13

for it yeah so it's the cache version is using the broken

1:04:19

um and so now you can see within our so now

1:04:26

we're in the topics instead of uh series so within the series now it's all just root level series and it doesn't have

1:04:32

any of those modules within that which is correct because you can't directly link to a module um well you can but

1:04:42

not within this sense it's like a hash link to the actual series page so it wouldn't

1:04:48

be good to have it here then we got all of our topics you can see that the subtopics are in there as

1:04:53

well and then it goes into the lessons and then we don't have any blog posts but those would be last if they were in

1:05:00

here and so there we go and so the last thing that i would need to do is check and make sure that all of the lessons are in

1:05:06

here but i'll worry about that later that won't drag you guys through that so

1:05:12

last thing to do would be to make sure that the cache is working

1:05:20

and that we are setting the cache to a specific time that we want this to do so

1:05:26

within here let's go ahead and set this to have a time to life of

1:05:37

eighty-six thousand four hundred uh

1:05:43

it's either 24 or 48 hours that's the best day i got

1:05:48

i'm using the old number that i had on the previous site there so um

1:05:54

yeah so that should cache it for that long and then once that happens it will clear that cached item out

1:05:59

and then it will come back through and recache it and so at this point we need to clear

1:06:04

out the actual cache version and i don't think i have a redis

1:06:12

i don't know the cli for redis that well

1:06:19

so um let me see if i can find one real quick so mac

1:06:26

redis gooey

1:06:54

come on all right i'll tell you what i'll help you out

1:07:03

what was that movie oh oh now you want okay go ahead

1:07:08

there we go all right you gotta click it twice

1:07:31

oh boy oh man yeah i haven't cleared this out in a while um

1:07:40

go ahead and delete you and so now if we come back into here and refresh it should work and there we go

1:07:47

so um let me double check i did uncomment the cache yes okay and so now if we retur or to refresh

1:07:54

this let's get rid of bull stuff come back down here we do have this and

1:07:59

it is now properly cached um so that seems to be working okay and

1:08:06

you can see it's formatted as gzip and all that fun stuff so we can test

1:08:11

this once more by adding a post and then we can do a search for that post to ensure it's not

1:08:17

in there and then we'll clear the cache so this is a test um

1:08:25

all right we'll leave all that stuff empty we'll put this in here as let's have it published as yesterday

1:08:33

and okay nothing else let's go ahead and publish

1:08:39

it it should show up on the home page it doesn't have an image so it'll be down here

1:08:44

okay and so now if we refresh this

1:08:50

bollocks why

1:08:55

all right let's clear the cash out again probably something to do with

1:09:04

oh let's see

1:09:14

hmm

1:09:20

let's try streaming instead of sending it i don't know i think the forehead said oh you know

1:09:26

what it could be that i'm directly returning the response.send

1:09:31

pardon my dog so let's kick that return off onto another line and test that

1:09:38

nope that wouldn't

1:09:50

let's confirm that it is just a caching issue thing here yep

1:09:57

all right

1:10:06

so i guess let's see what we are getting back from this

1:10:18

um yeah it wouldn't be up there it would be

1:10:23

down here

1:10:34

hmm

1:11:06

yeah so it's definitely an issue in either how it's being cast or home responding from the cache

1:11:14

and that does not look like a proper just send it back response there um

1:11:37

hmm

1:11:48

which is weird because it should be correct

1:12:21

alrighty then

1:13:50

uh

1:14:29

yeah it's definitely not it's just a string

1:14:39

it could be so a match what i have before it could be a difference between

1:14:45

so prior i was using node cache and in this app i'm using redis so it could

1:14:51

be a difference with that as well um

1:15:04

switch that back to a buffer

1:15:17

let me get rid of this let's see what the raw response is let's try that

1:15:24

yeah and then if you get rid of the cache what's the wrong response there

1:15:34

nothing did i switch that back to a stream yes

1:15:39

okay was looking to see whether or not it

1:15:45

would match

1:15:56

peculiar all right well

1:16:14

just does not want to work for me oh okay i left that in there that could be

1:16:22

part of the issue i guess maybe oh and it downloaded it

1:16:29

all right let's open that up i don't know why sublime is yeah so

1:16:35

that looks

1:17:07

all right well i don't know i'll have to play with play around with that and see and if i figure it out i will put that

1:17:13

down into the comments it could be something with the package it could be

1:17:20

i don't know um

1:17:28

is what we have definitely matches what i was using prior

1:17:34

it's over here and this works just fine

1:17:53

unless maybe the cash always came back empty i don't um

1:18:02

by the way i don't have enough information in there to where like not using the cache would

1:18:08

take a super long time so

1:18:15

i can always just leave it as this for right now oh and all right let's see i took out the headers didn't i i sure did

1:18:34

so what we could it could be

1:18:41

maybe we need to just rebuild the pipeline again and just send it down that way

1:19:04

um

1:19:09

because then it would already be gzip it wouldn't make any sense

1:19:35

you

1:19:51

i guess we could always just cache the underlying url structure um

1:20:00

wouldn't be crazy about that though but it would be something

1:20:16

so then that would look like this we would have

1:20:21

um

1:20:27

that and then we would have

1:20:33

this and then we would have that

1:20:41

and then this would be in here as well so well no that wouldn't be

1:20:48

and we would have

1:21:01

let's try that

1:21:38

yeah so there it didn't even get to the caching part um so urls might have came back with

1:21:44

something somehow even though i deleted cache

1:22:06

oh yeah

1:22:33

already

1:22:58

ah there we go okay didn't like how i had it

1:23:04

and so now that should be i'll have to play around and see if i can get get it working how i had it

1:23:10

before to where the underlying like actual xml structure was cached but this

1:23:15

would be better than nothing at least having all of the database query stuff cached

1:23:21

um so so long as that's pulling in okay then i'd be happy so

1:23:27

uh let's see if we can get rid of these shouldn't be in there anyway and then come back in here and let's make sure

1:23:35

that this all works okay still including information from the cache so let's

1:23:41

refresh again and it doesn't

1:23:46

um it's probably because it's coming through as a string at that point

1:23:51

if i can yeah so i need a parse

1:23:58

so what i'll do is let's rename this to

1:24:09

cache sitemap urls and get

1:24:15

cached urls

1:24:35

okay no oh can't spell return right here we go

1:24:42

no oh wait no wait do i need to wait yes you oh that could

1:24:47

be the whole freaking issue was i waiting in here yeah okay well that never mind

1:24:55

i think you have something all right so there we go so we'll get the item from the cache and we'll parse it we'll

1:25:02

return this back as a purse item um and then we'll do just a simple

1:25:13

nope not on the cache key on the cache item and so this will be urls

1:25:21

uh url or sitemap

1:25:26

item array

1:25:36

okay so now that will stringify and parse it for us and we need to change our names to get cached sitemap urls and

1:25:43

cache sitemap urls

1:25:49

prefix that was set okay so

1:25:54

set cache sitemap urls and get rid of this json.stringify

1:26:01

we'll make that a little bit cleaner and then

1:26:06

get cache to sitemap urls okie dokie

1:26:12

now we should be all right just get rid of that line all together save this

1:26:19

give it another go all right that seems to have worked it seems to still be working

1:26:26

just try deleting the cache

1:26:36

there we go all right so yeah the underlying issue was something to do with the buffers which

1:26:41

i really probably don't understand well enough to get into so this is a better

1:26:46

way for me anyway so it's caching the underlying urls

1:26:52

so here we have the sitemap so you can see now we're not caching the xml result but instead we're caching

1:26:58

just the underlying database information that's queried and then repopulating the actual stream

1:27:04

and streaming that off as the response so the heavy lifting is still being cached off which is rebuilding the xml

1:27:11

format for each request which isn't too big of a deal for this size of a site so

1:27:17

all right took a bit longer than i was hoping but we got the end result at the end of the day anyway so

1:27:23

yay um thank you all for watching and i will see you all in the next one um

1:27:29

i'll try to stream in between now and the

1:27:35

whenever i get back into my regular posting schedule which should be the first week of may so i'll try to do a couple of streams in

1:27:41

between then and now the last week of april was probably going to be pretty quiet well no

1:27:48

yeah the last week of april will probably be pretty quiet for me i probably won't stream that weekend

1:27:54

um because that's the actual weekend on moving so yeah

1:27:59

so i will see

Join The Discussion! (0 Comments)

Please sign in or sign up for free to join in on the dicussion.

robot comment bubble

Be the first to Comment!

Playing Next Lesson In
seconds