Transcript
Looking for something specific? Feel free to search the page for keywords to find a specific portion of the live stream that interests you most.
alrighty so it's been a little bit since i've posted anything so i thought it'd be about time that i live stream
something just to get something out to you guys uh so the new site is coming along great um
the home page is all populating for the user and everything um all of the sub pages and everything are
pretty much hammered out one of the last things remaining that i do need to do
is add a sitemap to it now i have the user facing sitemap already populated
within the footer there but what i need is one for search engines so if we were to go to the
current site and go to sitemap.xml we'll get an xml version of the sitemap
that contains all of the links within the actual site along with how much they
should be re-requested or we checked for and then what their priority on the site
is obviously the home page being the most priority and then series and topics
and lessons from there so i thought we'd do a little live stream where we
kind of recreate that xml sitemap within the new site um so
really only two things within the old site that took care of this at a service that really just
queried the information generated out the actual structure and then we had the
controller function uh that actually generated out what the response should look like
and send it along as xml so let's go ahead and get started with that within the current project i do believe
i have a syndication controller already created
and what this one's taken care of currently is just the user-facing site map so uh it's not quite a whole lot of
information uh the one for google will be querying a lot more um
because this user facing one is just more of a generic listing
but yeah so to start out with what we're going to want is an actual controller
endpoint to show the xml version so let's do public async xml
and this will want a response and that should actually be it
and then this will be http contacts contract and then from there
let's see so we'll probably go straight into
wanting to get the information and then we'll have our response dot
header we'll want to set the content type to
application xml response.header here header again this one will be
content encoding which will be gzip
and then let's see here so
let's start with that for right now now i believe i was using a package on the
old one um
not within there so we have zlib for creating the actual
gzip and then we have it looks like sitemap for creating the sitemap stream
so i'm going to go ahead and install those two dependencies so that was
mpmi sitemap and zlib
okay looks like we have those
all right so whenever it comes to the actual um
sitemap package what we are going to want to do is pass it in
the underlying host name so let's wrap this inside of a try catch
okay and then let's do uh const
site map stream equals new let's see if we can get this to auto
complete here site map yep there it is site map stream
and we just want to pass it the host name in this case it's going to be https
slash slash yeager.com all right and then we're going to want
to pipe into our sitemap stream so let's do
if i can spell pipeline sitemap stream oops
sitemap stream dot pipe and then we'll want to actually create
let's see if this will uh create gzip and that should be just fine just like
that and then
let's see here actually so from cash
if exists so let's try to get the information from the cash
if we can so we'll do const
sitemap equals and then for right now let's just do an empty
array and this will be from the cache we'll set the getter for that here in a second
but if sitemap and actually we just set this to null
because if it's not in the cache it won't come back as anything we will do response dot
send the segment and we want to kick that off right there
otherwise once we've created all this stuff this is where we'll want to
actually get the info if it's not cached and from here that's
where we would want to probably plop that off into a service so let's jump down to our services i
don't think i have a syndication service so let's go ahead and create that real quick
all right
actually yeah that'll be fun okay
so for the um
old one here for the syndication service i had types defined within the surface itself i'm gonna go ahead
and plop those just copy if i can oh what am i trying to do here i'm gonna go ahead and cop those into a contract file
uh so let's go down to contracts and let's create a new file here as well
called syndication and i'm just gonna plop those in there
so we have sitemap item which contains the url the change frequency and the priority which aligns
with if i still have it open what the actual xml items are
and then we just have the frequency which is what the change frequency values should be which is either daily
weekly or monthly so now that we have that in there let's go back into our syndication service
and so first and foremost we're going to want to cache this if
uh we can so let's do public static
let's do sitemap cache key i'll be if string and we'll just
cache that as sitemap and then we'll want a public static
um let's see
so on the old one here it looks like i had a getter and i had the sitemap here actually private
we could mimic that i don't think we need to and then there is how i was checking it
so i do still have that cache service it does integrate with a different cache now it integrates with redis so
let's go back up into our controller and go ahead and get that going out
so instead of that we can do
well let's do a get directly within our syndication service instead of doing it outside so
we can switch this to private i guess and then let's do public
static get sitemap from
cache or something like that um and then we will do return
cache service it looks like my autocomplete gave up on me oh no
cash service or i just can't spell nope nope it did give up
well shucks
all right let me uh switch files real quick let's go into you and then back into you see if that fixed it
it did not all right i'm going to close it real quick
and let's see if that fixed it so public static there we go all right
get sitemap from cash
um now at this point we want to return cash service there you are
dot get um and then this dot site map cache key
is this async i cannot remember
yes it is so we'll want this to be public static async and
yeah there we go
hey man um i plan on getting back to recording new things
well i'm hoping to record in between now and may but i don't intend on actually posting
any new recordings until we reach may once we reach may that's whenever i'll be in my new house and i'll have
everything set up and i won't have to deal with packing or anything like that anymore so um
just going to kind of take this time in between now and may to get some recordings in place and then we should
start seeing some new stuff in may so
should be early may
yeah so i'd say the first week of may we should get something out
um and then from there it should be pretty much weekly hopefully
so we have the get site map from cash
and we have that oop that's not where i meant to go we have that in our syndication
controller where are you here you are no we do not so let's go ahead and do that
so const site map equals await
uh syndication service dot get sitemap from
cache so now if we get the sitemap from the cache and it does exist then we can just return that cached version otherwise we
need to try and generate it and that's where this would come into play down here
so we'll start on that next so
we're going to have the intro function for this that takes care of
kind of generating and populating everything together as probably i guess generate site map would
make the most sense um and then ultimately what we'll want this
to do is return back a promise of sitemap item as an array
so first and foremost what we're going to want to do within here is actually query out all of our different information so
we're gonna have our series topics lessons
and blog posts and then from there we'll do each like
subordinate page or whatever you want to call it as individual stuff from there so like
we're going to have a section for frequently asked questions resources uses and then if we take a look at the footer
there's some stuff for like guidelines contact the visual site map cookies privacy terms and all that fun
stuff so all that stuff can go in there just manually defined but this stuff needs to be dynamically pulled
um actually
these are going to be database generated the frequently asked questions and resources um
so but for right now those are empty so i'll just put those in there manually at this point
so this will be a good starting point so for our series
we're going to have a weight and trying to think if i have something in
the series service for this that would be useful i want to say no so let's just query it
directly off of our collection model oops yes there we go all right
and then from here we specifically want to just grab the series um don't plan on populating
uh uh playlists or courses at this point in
time um and then let's see
from this all that we should need is
the slug so i have a
well i guess i should change the name of that so i have this function defined
where do i have that to find that that is within my query builder provider called select
ids now this will execute the query and then just select the id column ideally i
guess i should just be named select column since it doesn't specifically need to be
an id and then the series function will kick
off a query builder and then it will execute a where statement to make sure that it's of type series
uh so that's just a nice little shorthand that i have there so i'm gonna go ahead and just do select
ids this won't make any sense at this present point in time and name this
i think i just need the slug let's check and see whether or not
that's the right so at work they're called aliases i think i call them slugs
where am i here collections did i click on it yes i did yes slugs
all right and that should really be it for our series oh wait a minute
dot where public i have another query builder helper that will specifically do this
for the type where it is public so build that into there and we have a nice simple query for all of our series right
there and we can do the same thing for topics await top
and the model for this is called taxonomy and then
trying to think do i just want the roots of the taxonomies or do i want all of the children as well so like if we go
into topics any taxonomy is technically a topic um
but then these are all of the root level ones and then like if we were to click on adonis js there's
subtopics within it so you have like the acli authentication i guess it would make sense so let me
see do i have these as no they are root level so like if i were to go into hdb context
it's still the same url so let's just query all of them we'll get the roots first so we'll call
these i guess root topics and then we'll do child topics below that as a lower priority
so we'll do roots and that's similar to the series
it's just going to kick off query builder where it's specifically a taxonomy that doesn't have a parent id
um those are not those don't have a state id
and then so here it should just be select the slug from that as well and then we'll do const
child topics await and i don't think i have a helper
like roots for this but we can go ahead and bake that in this might be the only use case where i would need it but
see i always lose my models there they are taxonomy
i need to clean my screen off it's super dusty
where are you roots there it is all right so just a simple public static
deck it's just children and then that's just a simple we're not
null parent id
um you could probably just do the
conditional parent id in there i've typed number
default that to null actually i don't think you need a question mark an id
such as squiggly because it's not used type number
there we go and then we could do if parent id [Music]
return this query dot where
parent id is parent id otherwise just get it where it's not not null
alrighty so that should do it for that so then we could do children we don't want a parent id there
and then we can just do select ids for slug again i need to rename
select ids to just like select column or something like that hey tina
thanks for stopping by um so that should do it for topics
and then we have our lessons which
well you know what let's um kind of order to these as well i guess that would make sense with it
so let's order by
i guess let's have the newer ones first does that make sense like i guess
so created at
descending
dang it missing my shift key yeah i think that makes sense so we'll do newer ones first
yeah yeah let's do this alphabetically it'll be easier to um verify that we're getting everything
that way order by name decent wait no a something
for name there we go
okay and then for lessons
uh that goes off of the post model dot lessons
dot and then from there from there i want apply i want to apply
a scope and the scope is to ensure that the post is published
and then we'll do our order by from there uh that one would be title ascending
select ids for select yeah they're all selected
all right so there's our lessons and let's just change this to posts and
then we'll group them all within there so const lessons const
posts equals await post dot i think i named them post nope
want to name them blog blogs all right let me name this to match that
apply scope now i don't think i have any blogs populated yet uh so that will be empty
but we'll go ahead and take care of them because i intend on publishing some here soon
descending select id select okay so if i have this
working properly this should come back as just an array of slugs um
if not then we'll find out but like so if i were to go into tinker here or tinker um rebel
node ace rebel
load models and we do await
models collection dot
series dot
public dot select ids
slug we should just get back an array of slugs and that we do so it seems to be working all right
so i should be able to expect that for all of these so now that we have the
dynamic information that we're going to need to make this site map something um
next what we want to do is actually create the items and then from there we can go ahead and
excuse me kind of map everything together so we're going to have
a private static function called make
this will take in the url
of type string the change frequency of type
freak when see and that did not want to
what do i got going on here
oh i was putting that inside of the generator okay my bad
okay there we go i guess frequency already auto imported because it doesn't have a squiggly on it
so it should be okay and then the last one will be the priority which
i think that would just be type number and then we can default that to a half item and then here we'll return back a
finalized sitemap item from here so what we want to do is pretty
much just return back an object of those arguments so we'll have url change frequency and priority
i kind of prefer this approach over you know just in line creating an object
over and over again mapping or naming the keys of the object over and over again
for each item um so now that we have this we go ahead and start our actual url collection so
we'll do let your well now we can actually make this uh const urls equals
let's specify this as a sitemap item array equals
an array and then here we'll go ahead and define our top level
hard-coded urls so we can do this.make
um i'm just going to hard code the url in here since they're pretty short do weekly for that one and then the
priority for this since it is the home page will be 1.0
um and then i'm going to go ahead and i believe all of these are the same
so i'm just going to go ahead and copy and paste this instead of writing it all there we go
so we have so we're essentially just making a site map item for our home page our series
landing page topics landing page lessons landing page and our blog landing page obviously the blog landing page is going
to be less important than all of the other ones so once we have that in place then what we can do
is go through and add in all of our dynamic items um so
for that we're going to want kind of a different helper from make we're going to want something like
private static function called map and this will take the route name
as a string any params um
did what what is that that's x um it's been a while since i've wrote
this x string any i think i think that's accurate
and then the change frequency as frequency
and let's default that to weekly and then the priority
is again a type of number and we'll default that to 0.5
and then this 2 will return back a site map item for the arguments passed
in here so
we can do const url and then what we're going to want to do is use the route module to actually
just generate out this url so let's do make url we'll pass in the route name
along with those defined parameters i shouldn't need any query strings or
anything like that for this um it should just be simple to leave it as the round name and params
and then if we do not get back a url from that make
url if it's not mapped for whatever reason then we would want to just return this
dot make define it as a hard-coded url with the change frequency
and the priority
i guess we want to make that more undefined
that's how i had an old one yes
all right so that should be good for the mapping one so now we'll want to go through and actually just map each of so
our series topics and posts so that we get those items actually
created so we do series dot map and we'll just loop over each and every series that we
have and then this will be the slug since we've specifically selected just the slug
value as we've tested here so it's just be an array of strings with the slug values
and then we'll want to append this into our urls
so this will be urls dot push
this dot add oh we need to make that yet don't we
what was that wrong file
i don't think we need that i don't know why i had it in there for some reason the old one but
i guess i needed to do conditional determination for them but we should not need that in this one so it should just be a simple push
this dot map and then the
route name for our series should be series dot show
and so map is pretty much just
matching like the make url format here and then we'll have slug right in there
and it does not like that y
series came back as a number
huh that's peculiar
promise number huh what in the world is that doing it for all of us
yeah it is huh that's odd do i have that
i've used that elsewhere it's kind of odd um
okay i guess let me take a look at the contract definition for that and see exactly what i have that coming back is
worst case scenario all write it where is that
database select oh sure enough oh yeah because it's ids
right um all right so i guess that would specifically just
be a number there wouldn't it all right um tell you about let's go ahead and create
a different one then so
let's do model query builder macro
select column async function i'll leave that one it's
just the id and i want to do this one is column name string equals
i guess we'll just make that required
and then we'll do const results equals oh wait this dot select
column name return results dot map r r column name and it
doesn't like something i have here
yeah there we go parentheses in the correct spot okay
so we have that now let's go ahead and just add that into here so let's see i did that after that so this would be
select column
oops to go back and check the name of that yeah select column
column name string and then we should just be able to do
something similar to the first or up here i think equals undefined return back
promise
of t
array i think i don't know we'll test it
worst case scenario i'll just put ads in there for right now um so
change this to select column
i feel like that's wrong but let's see here is that returning back
oh an undefined array all right that makes sense
i'm just going to hard code that as a string for right now it's gonna hurt anything
yeah oh wait no that needs to be a string array
there we go okay not ideal but it'll work for right now
and we'll just map switch all of these to that
okie dokie so now this should be happy maybe argument of type site map item is
not assignable to parameter f type
oh okay oh that's why i had this dot ad because
they were nullable and if it came back as null then uh
yeah you don't want to add it do you so
okay yeah we could do that private static add
or i guess let's just do push we'll do add
switch that to a let we'll do urls equals
this if i can spell this dot add and then we'll do add here
and i think this needs to take in um
urls
right because this would return back okay um
so i think do cite my vitamins there as an array of
sitemap item and then
no condition so this is the actual array and then this is gotcha
and then conditional add as a site map
item array or undefined
there we go so then we'll do if not
conditional add and just return urls otherwise return
urls and additional evidence this is just one
item isn't it site map item yeah
there we go all right so now what's the squiggly about
expected two arguments but got one because ad needs to take in those urls all right
probably a cleaner way to do that but i'll stick with how i had it prior for right now
it'll work all right and then we'll do root topics
yeah switch this to three that way these individual items have a higher priority
than blog posts i think that makes more sense because then these will come in at the default of 0.5
so root topics slug urls equals and then we just need to pretty much do the exact same thing for this
um this one would be topics.show
slug and
all right dang it
there we go okay and then we'll do
we'll go ahead and do our lessons because those would have a higher price well now we'll go ahead and do our child
and then on the map we want to change the priority so we'll do this should be
frequency if i broke that out in the right spot i did not
weekly keep that the same and then this one
let's do actually let's leave this as 0.5 and let's change these
to 0.7 i think
oh what am i doing
okay and then this one will leave as default of 0.5 but i'll go ahead and
hard code that just so that it's easily visual all right so there's our child topics
and then we want to do our lessons and our blogs so we'll do lessons dot map slug urls equals this
dot add urls this dot map
should be lessons.show slug weekly
that one we would want to be 0.7
0.6 maybe um and then blogs dot map slug urls equals this dot
add urls this dot map
i think this is posts dot show
and then this one will go ahead and do 0.3 have to be the same as its landing page i guess
and we'll do one less okay so that should really be it for generating out all of our urls at this point in time we
should have all of our urls populated so that we can return them back so now
we can call our generate sitemap from our syndication controller
so this would be something like const urls equals
oh wait syndication service dot generate sitemap
probably make more sense to generate site might be urls
get yeah i like that better get sitemap urls
okay because you're not actually generating the sitemap in there you're really not generating it until you actually create the xml items
so let's see we have the pipeline up there
we have our urls here so let's map over our urls
and add them to our sitemap stream
and then we want to convert our stream to a promise
pass it in our pipeline which is our creating a gzip from this
and then our sitemap we would want to cache service dot set
and
no let's put that in here
public static async set sitemap
uh in cache save sitemap
not gonna get picky over the name right now cash service
dot and then we should have a set this sitemap cache key
and then this will take in
i don't even know what that would be what would that be so
and it doesn't want to give me a tight oh it's a buffer i'm just gonna do
yeah let's try setting that to a buffer and let's see whether or not i'll take it so save up cash
site map okay
and then we can get rid of this cache service and instead do that through our syndication service that way we
don't need to specify the cache key out here and it will be done within it
so set sitemaps yeah and then sm
and it looks like it accepted it okay so i guess that is indeed a buffer and then what we want to do is
sitemap stream dot end and send the stream on off so response
dot stream our pipeline
and then if we get an error at all let's go ahead and just
console.log that out and send off a 500
okay last thing that we need to do here i think is set it up to a route
so i have the public one right here i'm just going to go ahead and plop the xml version right after that so we just do
this as a get savemap.xml
i think we just named that xml
okay so if i did everything in that correctly we should be able to go to
i need to get the website under on the phone let them know about this uh let's see
looks like i got in there oh wait a minute
is my shell even started your shell um
server it would have to be because it was
clicking around before wouldn't it but where
i'm gonna go ahead and just restart this i don't know what happened
oh okay somehow i had another one open
so let's see can i read properties of null reading replace i don't think i called replace
anywhere in there did i hmm so that must mean that we have an empty
string or not an empty string but something that should be a string that's not a string
oh oh oh oh gotcha let me go into
um was answering a question on github discussion about how to
um minify html responses or the edge response and i think that
is messing up this because we it's happening within that middleware so let me
try no still no
try refreshing
okay no no still does not like that
okay i doubt the uh network tab is gonna tell me
anything here but we'll try
yeah figured out all right so we're not getting back an individual
error anymore now that we've gotten rid of the minification of the html
and it specifically says that it does not like the encoding
no root element found pretty sure i have everything set up the
way that it currently is on the other one
so let's break it down let's get rid of the try catch
let's let our error come to let's
so the response header should be happening up here setting that to application xml that looks correct and
then we have the content encoding which is specifically what it says it does not like
sitting at the gzip so what if we get rid of
that gzip encoding header
okay so now it's not well formatted so we're getting a different type of air
i think probably the air that's causing the gzip air
so from there could be
that it could be a cached issue
all right now we're just not getting anything um
so let's make sure that we're getting back some urls
and that the urls are what we expect them to be
that's a lot of console logs there all right so
scroll up to the top to make sure let's see so it's a yeah i mean that looks right
the only reason that it's in a root object is because i console plugged it that way so that should be correct
try to come to log that out i'm gonna go ahead and get rid of that
a whole lot of nothing all right well well well
i thought that would do anything yeah i figured all right
so it's definitely something in how the xml is being generated because the underlying urls
look correct um i wonder if maybe i installed the wrong
well no because i don't did that auto import wrong nope it looks right
hmm well
hmm
i don't know
i guess let's go take a look at that sitemaps documentation see if anything's changed
maybe i had a rewrite or something
um
all right so up to here it looks correct and then we get down to this stream to
promise return where it calls readable from stream
but you would think we should just be able to directly return back the response.stream
i wonder if maybe i need to i don't think that's right but
yeah all right
let's see here
let's take a look at their full example all right so you can see we're matching
everything up through here we're creating a gzip that looks correct
we're writing um our urls we're just mapping over them instead of doing it line by line
i guess we could try this
yeah yeah why don't we try that so instead of doing
this for the cash we'll stream to promise
and then instead of doing that we'll just return sitemap and then up here we'll change
sitemap from a const to led give that a save
not well formatted okay
so now that we should have something put in there to take a look at let's see
if we can take a look at maybe the structure of it
oh my yeah okay it's encoded um
this point i'm pretty much just guessing what
i don't even know where that would be coming from
well something funky is going on then huh let's see we still have the console log in there but i'm
yeah okay
well so hold on a minute that doesn't make any sense because you're not like awaiting it
well that's taking its time that probably means something's wrong
hmm
i'm gonna go ahead and stop that so it's not continuously running there
i don't know i mean we pretty much are matching what this is doing
um i'm gonna go ahead and do a sanity check here let's see
here let's see update history maybe
so i'm probably on 7.1.1 um
yep let's see what version we're on here
keep that
7.0.0 okay um
let's take a look at the specific version
yeah seven point zero point zero on the dot so um i'm going to go ahead and
mpm uninstall how to stop this so it doesn't cause much errors
sitemap
okay npmi sitemap
a specific version
okay let's get this back to how i had it initially
i am going to go ahead and leave the cached version off
okay let's try that let's see if maybe it's a version issue or something
yep yeah that works okay so i'll just roll with the 7.00 version um and because we were
following their documentation for the 7.1.1 pretty much how they had it nothing
changed here from the previous version so maybe it's out of date i don't know um
but yeah so this is the underlying this is this is what we want so we have our url set with the url the
location change frequency priority for the home page to the one and then everything from there goes to point
eight and then for posts we go to point three and then for series actual lists list
them all out so we have in alphabetical order all of our series
um oh okay so one thing i do need to take care of is i have modules coming through
so whenever i create a module um i've got a couple in here that are just
like they get default names and there's a couple i haven't changed to make it evident that you know i have a module populating
somewhere and so these your new collection are modules so modules would
be an item within a specific um so here let me go to localhost 333
within a specific series so if i were to go into our let's learn adonis series we have modules so you have less than
1.0 that's module 1 or 0 if you're going index based and then you go to 2 that's
another module so all of these lessons are bound to those specific modules and that's
probably a little bit more easier to highlight um if i were to go into
the actual collection here so let's see let's learn so you have the individual modules over
here so you have like a module for introduction routing controllers and then each module that gets the lesson
or the collection as a whole can get the lesson set as we are with i think like the bouncer
series and all that so here you can see there are no modules it's just straight direct posts
so i want to do a specific check within here where we're populating our data let's go ahead and get this back to how
we had it here so let's put our catch back in here let's do our caching
um i'm gonna go ahead and get rid of that
comment clean that up a little bit and let's go into our authentication service
and within here we want to do
an additional wear
null parent id i'm gonna go ahead and break this down
as well make it a little bit more readable should be doing that for all of
them but uh we'll just do it for that one for right now and it's now if we refresh
we're back to a content encoding here okey dokey
oh that's probably using the old cache i need to figure out how i can clear my cache i get a gui in here or something
for it yeah so it's the cache version is using the broken
um and so now you can see within our so now
we're in the topics instead of uh series so within the series now it's all just root level series and it doesn't have
any of those modules within that which is correct because you can't directly link to a module um well you can but
not within this sense it's like a hash link to the actual series page so it wouldn't
be good to have it here then we got all of our topics you can see that the subtopics are in there as
well and then it goes into the lessons and then we don't have any blog posts but those would be last if they were in
here and so there we go and so the last thing that i would need to do is check and make sure that all of the lessons are in
here but i'll worry about that later that won't drag you guys through that so
last thing to do would be to make sure that the cache is working
and that we are setting the cache to a specific time that we want this to do so
within here let's go ahead and set this to have a time to life of
eighty-six thousand four hundred uh
it's either 24 or 48 hours that's the best day i got
i'm using the old number that i had on the previous site there so um
yeah so that should cache it for that long and then once that happens it will clear that cached item out
and then it will come back through and recache it and so at this point we need to clear
out the actual cache version and i don't think i have a redis
i don't know the cli for redis that well
so um let me see if i can find one real quick so mac
redis gooey
come on all right i'll tell you what i'll help you out
what was that movie oh oh now you want okay go ahead
there we go all right you gotta click it twice
oh boy oh man yeah i haven't cleared this out in a while um
go ahead and delete you and so now if we come back into here and refresh it should work and there we go
so um let me double check i did uncomment the cache yes okay and so now if we retur or to refresh
this let's get rid of bull stuff come back down here we do have this and
it is now properly cached um so that seems to be working okay and
you can see it's formatted as gzip and all that fun stuff so we can test
this once more by adding a post and then we can do a search for that post to ensure it's not
in there and then we'll clear the cache so this is a test um
all right we'll leave all that stuff empty we'll put this in here as let's have it published as yesterday
and okay nothing else let's go ahead and publish
it it should show up on the home page it doesn't have an image so it'll be down here
okay and so now if we refresh this
bollocks why
all right let's clear the cash out again probably something to do with
oh let's see
hmm
let's try streaming instead of sending it i don't know i think the forehead said oh you know
what it could be that i'm directly returning the response.send
pardon my dog so let's kick that return off onto another line and test that
nope that wouldn't
let's confirm that it is just a caching issue thing here yep
all right
so i guess let's see what we are getting back from this
um yeah it wouldn't be up there it would be
down here
hmm
yeah so it's definitely an issue in either how it's being cast or home responding from the cache
and that does not look like a proper just send it back response there um
hmm
which is weird because it should be correct
alrighty then
uh
yeah it's definitely not it's just a string
it could be so a match what i have before it could be a difference between
so prior i was using node cache and in this app i'm using redis so it could
be a difference with that as well um
switch that back to a buffer
let me get rid of this let's see what the raw response is let's try that
yeah and then if you get rid of the cache what's the wrong response there
nothing did i switch that back to a stream yes
okay was looking to see whether or not it
would match
peculiar all right well
just does not want to work for me oh okay i left that in there that could be
part of the issue i guess maybe oh and it downloaded it
all right let's open that up i don't know why sublime is yeah so
that looks
all right well i don't know i'll have to play with play around with that and see and if i figure it out i will put that
down into the comments it could be something with the package it could be
i don't know um
is what we have definitely matches what i was using prior
it's over here and this works just fine
unless maybe the cash always came back empty i don't um
by the way i don't have enough information in there to where like not using the cache would
take a super long time so
i can always just leave it as this for right now oh and all right let's see i took out the headers didn't i i sure did
so what we could it could be
maybe we need to just rebuild the pipeline again and just send it down that way
um
because then it would already be gzip it wouldn't make any sense
you
i guess we could always just cache the underlying url structure um
wouldn't be crazy about that though but it would be something
so then that would look like this we would have
um
that and then we would have
this and then we would have that
and then this would be in here as well so well no that wouldn't be
and we would have
let's try that
yeah so there it didn't even get to the caching part um so urls might have came back with
something somehow even though i deleted cache
oh yeah
already
ah there we go okay didn't like how i had it
and so now that should be i'll have to play around and see if i can get get it working how i had it
before to where the underlying like actual xml structure was cached but this
would be better than nothing at least having all of the database query stuff cached
um so so long as that's pulling in okay then i'd be happy so
uh let's see if we can get rid of these shouldn't be in there anyway and then come back in here and let's make sure
that this all works okay still including information from the cache so let's
refresh again and it doesn't
um it's probably because it's coming through as a string at that point
if i can yeah so i need a parse
so what i'll do is let's rename this to
cache sitemap urls and get
cached urls
okay no oh can't spell return right here we go
no oh wait no wait do i need to wait yes you oh that could
be the whole freaking issue was i waiting in here yeah okay well that never mind
i think you have something all right so there we go so we'll get the item from the cache and we'll parse it we'll
return this back as a purse item um and then we'll do just a simple
nope not on the cache key on the cache item and so this will be urls
uh url or sitemap
item array
okay so now that will stringify and parse it for us and we need to change our names to get cached sitemap urls and
cache sitemap urls
prefix that was set okay so
set cache sitemap urls and get rid of this json.stringify
we'll make that a little bit cleaner and then
get cache to sitemap urls okie dokie
now we should be all right just get rid of that line all together save this
give it another go all right that seems to have worked it seems to still be working
just try deleting the cache
there we go all right so yeah the underlying issue was something to do with the buffers which
i really probably don't understand well enough to get into so this is a better
way for me anyway so it's caching the underlying urls
so here we have the sitemap so you can see now we're not caching the xml result but instead we're caching
just the underlying database information that's queried and then repopulating the actual stream
and streaming that off as the response so the heavy lifting is still being cached off which is rebuilding the xml
format for each request which isn't too big of a deal for this size of a site so
all right took a bit longer than i was hoping but we got the end result at the end of the day anyway so
yay um thank you all for watching and i will see you all in the next one um
i'll try to stream in between now and the
whenever i get back into my regular posting schedule which should be the first week of may so i'll try to do a couple of streams in
between then and now the last week of april was probably going to be pretty quiet well no
yeah the last week of april will probably be pretty quiet for me i probably won't stream that weekend
um because that's the actual weekend on moving so yeah
so i will see