How To Get Your Website Crawled By Google
When making a website it is very important to consider the overall goals for what actions you want users to ultimately take. These goals can be the core of your SEO efforts which allow you to figure out your main key terms you wish to drive traffic to.
Once you have figured out your key terms, the main goal is to get your website indexed by Google, quickly.
This week, we will go through a few steps on how to set up your WordPress website to allow for quick and consistent crawling by Google.
Google Crawling & Indexing
To understand how Google search works, let’s take a look at a simple Google search such as, “weather in Philadelphia”. When we hit enter in the search bar, what happens is Google looks into it’s very own library of known websites that match your search terms. Google will break down your search word and give you the correct match for sites that encompass these terms within the website and specific page, with quality of content in mind.
For Google to have the ability to search within their library of websites they must crawl the internet and take snapshots of every page on every website that wishes to be indexed. The indexing algorithms then arrange these websites by key terms, and quality of content. Before any of this happens, when we add a new website to the internet we must alert Google of our new website and tell them how we wish to be ranked. We do this by using our keywords in the content throughout the website pages. Along with our keywords, we must link our website to Google Search Console and submit a sitemap to be crawled. This sitemap will be used by Google to get an accurate picture of your website for indexing purposes.
Bot: A bot is the name given to the technology used by Google to scour the internet for web pages. Think of it as a set of codes that ultimately downloads and saves a website with all of its contents into a large database.
Crawling: Describes the act of Google Bots searching and downloading websites on the world wide web.
Indexing: Once a website is crawled by a bot and downloaded, Google uses indexing algorithms to store the pages according to the freshness of content, keywords, and quality of information to give users the correct pages they wish to view in the search results.
So we have our new site ready and designed with keywords implemented that ecompase your key goals. The next step is to get your website crawled by Google using Search Console and an xml-sitemap we will now create.
- Yoast SEO plugin for WordPress
- Google account
- WordPress: First what you want to do is log into your WordPress dashboard and navigate to the Plugins tab in the side panel. Once you have made your way into the Plugins page, click on Add New at the top left. Go into the search bar on the right and type, Yoast SEO. Install the plugin and make sure to hit activate once it has finished downloading. Upon activating your Yoast Plugin, the plugin will create a new xml-sitemap of your website for future use.
Yoast SEO plugin allows you to take full control of your the SEO on your website while giving you tips on how to improve your keywords, quality of content, and overall search visibility.
- WordPress: Once we have activated the Yoast plugin, we want to make sure to fill out the forms within our main pages of the site to add in our keywords and descriptions.
For more information on adding in keywords stay tuned for our next blog where we will dig deep into keyword research and implementations to your website.
- Yoast SEO: What we want to do in preparation for the next step is to find out what our new xml sitemap URL is. Navigate to the Yoast SEO plugin on your WordPress dashboard and click on the General tab within the Yoast SEO dropdown. Click on Features and make sure that xml-sitemaps is turned on. Then click on the See the XML sitemap link to view your page and url.
- Search Console: Now that we have everything set on WordPress, we can move on to the Search Console. At this point you should have your own Google account that you can sign into. Once signed in, navigate to: https://www.google.com/webmasters/tools/home?hl=en
- Search Console: You will be taken to your very own Search Console Dashboard. At the top right click on Add Property and add in your website URL. Once you have added your website, you will be asked to verify your property with a choice of a few options to do so.
- Search Console: You can simply choose to copy and paste your meta tag into your WordPress by first clicking on Alternate methods. Copy the line of code and move onto the next step.
- WordPress: Go back to your wordpress installation and navigate to Themes > Editor. Once there, open up your header.php file (Theme Header) and paste in your meta tag code within your (<header> meta code here </header>) code .
- Search Console: Navigate back to your Search Console Page. Hit verify, and now your website is a property of yours within your Google Search Console Dashboard.
- Search Console: Once your site has been verified, go to your Search Console dashboard and find your website. Click on your property and you will be taken to your panel for the specific site. Within the sidebar menu, click on Crawl > Sitemaps and then Add/Test Sitemap at the top right of the page. With the url you copied of your xml-sitemap in step four, paste it into the field and hit submit.
We have set up Yoast, verified your website on Google Search Console, and added a sitemap. Doing this will allow Google to accurately search through your website to get a better sense of how to index and prioritize your pages based off of your content and keywords. If we had not done so, Google would not prioritize the crawling and indexing of your new website leaving it to be taken care of on a later date. Meaning your website would be almost impossible to spot in the search engine results.
If you are a blogger or have frequent updates to your site, Google will use your xml-sitemap to get an updated look at your pages. If you make these changes on a consistent basis, for example every Monday at 8am, Google Bots will start to predict your updates thus allowing them to have an easier time determining if and when they should index your site again. Overall Google Bots will see that you are outputting fresh and original content that MUST BE SHOWCASED TO THE WORLD!
My name is Yeriel Martinez, web developer at Momentum Digital. I handle all website design and help oversee all website development at Momentum. Along with development I handle on page SEO to help with the overall growth and accessibility. My goal among all is to help empower small businesses and assist with spreading their brand name for a larger audience reach. While helping businesses grow, I have the amazing opportunity to learn from those we work with and am grateful to be helping hand in the world of digital marketing.