RSS FeedsHow to make sure your website or blog is indexed by the search engines
Feb 16, 2008
Much before you start expecting tons of traffic from various search engines you need to be sure that your website or your blog has been included in their databases. This may look like a minor problem but many people who wonder why they don’t get any search engine traffic haven’t had their URLs indexed.
I know the search engine crawlers are constantly scouring the length and breadth of the Internet and sooner or later they end up indexing almost all the websites and blogs, there are still chances that sometimes they get ignored or the webmasters have created, may be inadvertently, conditions that stop the search engine crawlers from crawling and consequently indexing their links. Listed below are a few steps you can take to make sure that your website or blog is crawled and indexed as soon as possible.
Make sure your robot.txt file is not creating hurdles
Robot.txt file is used to tell the search engine crawlers what files and directories should not be called and indexed. Some people prefer to do that when they keep files on the server and they don’t want people to find those files on the search engines. For instance if you don’t want the search engine crawlers to index your images folder you will use the following lines in your robot.txt file:
User-agent: * Disallow: /images/
The first line containing "User-agent: *" means that no search engine crawler should crawl and index your images folder. It so happens sometimes that instead of specifying the name of the folder your robot.txt file blocks your entire website through
User-agent: * Disallow: /
/ on its own means your root folder. Although this command in your robot.txt file should not exist unless you have put it there it is better to have a look in there. If you find such a villainous line in your robot.txt file remove it immediately.
Avoid over optimization
Some people become overzealous when optimizing their websites and blogs for keywords and key phrases. Their keywords appear everywhere like leaves of Fall. They erroneously think that repeating the keywords again and again, even when their copy sounds nonsensical, can achieve them higher search engine rankings. Search engine rankings not just depend on the appearance of your keywords they also depend on many other factors like what sort of content you have, how many people are linking to your website or blog, and what is the overall navigation structure and content organization. When you stuff your keywords all over your website the search engine crawlers cannot make sense of your content, and consequently, fail to index your link.
Create relevant content regularly
The search engines love regularly updated content and the websites that have fresh content get indexed with greater frequency. Once you have conveyed to the search engines that your website gets updated frequently they on their own start crawling your website almost on a daily basis and this I have experienced myself. There was blog I was updating multiple times daily and I did that for almost 2 months. The newly registered domain not only started appearing on the search engine result pages within 5 days my new pages appeared there usually within a few hours.
Get incoming links from respected and legitimate websites and blogs
Your incoming links not only get you clicks from the websites and blogs linking to you they also help the search engine crawlers find your links rapidly. When search engine crawlers are crawling a particular website or blog they try to visit all the links present there. So if your link exists at many places it becomes easier for the search engine crawlers to get to your website or blog.
Use legitimate search engine optimization
By using spurious SEO tactics you can get your link banned for a long time. Only use SEO techniques that are legitimate and to be frank they are much better and easier to accomplish. Some of the accepted search engine optimization steps are
- Use your keywords in your page titles
- Create search engine friendly URLs
- Organize your content under appropriate headings and subheadings
- Use your keywords in your headings and subheadings
- Use lots of interlinking
- Use your keywords as anchor text when you are interlinking
- Highlight your keywords whenever it seems appropriate
- Encourage people to link to your website or blog
- Keep your main content as near to the top of your page as possible
- Make sure there are no broken links under your URL
- Make your main navigation well-defined and use your primary keywords in your navigation; avoid using images in your navigation
- Maintain a well-structured sitemap and link to your sitemap at the top of your website so that the search engine crawlers encounter it as soon as they enter your website
These steps will make sure that you are on your way to a great search engine optimization effort.
Keep your server and file management in order
The servers that are too slow also prevent the search engine crawlers from indexing all your pages. When the server is very slow or when it is often down your web pages remain inaccessible to the search engines. Sometimes, call it bad luck or anything, your website is down exactly when the search engines are about to visit it. Host your website or blog on a decent server, it’s worth the extra cost. Also make sure that there are no missing links on your website.
Although the search engine crawlers take their own time, by making sure that you have taken care of the above-mentioned factors you will facilitate a faster and a smoother indexing. Once your website or blog has been indexed you can start improving its performance on the search engine result pages.
You can leave a response, or trackback from your own site.
February 16th, 2008 at 11:23 am
[...] Original post by Content Writing and CopyWriting Blog [...]
February 16th, 2008 at 3:11 pm
Thanks for the information. It’s a lot of things to think about.
As a self employed aspiring internet marketer, right now, I pretty much have to do everything myself. I have to be the copy writer and the webmaster and the SEO expert.
Since I have to do this all myself I am always looking for ways to help automate my tasks, SEO is no exception.
I saw a program that scores a page against a database of pages that have ranking info from the search engines. The software spits out a score and then, like some video game, you make changes and try and beat the score. By the way, the program is called RaSof.
My question is, can you really analyze the SEO of a page by comparing it to other pages?
How much can you gain, if you do the things you suggest in your post to optimize a page already?
How much affect do small tweaks in SEO make?
In other words, if the page isn’t that bad, has the usual meta tags, titles, alt text, site map and keywords in headings and body, how much difference could one make after that?
Appreciate your thoughts
Thanks,
Scott
February 18th, 2008 at 1:20 pm
Hello Scott.
I am not sufficiently familiar with the various SEO tools that help you improve your search engines ranking performance or tweak the pages to make it more search engine friendly but yes I firmly believe that small SEO tweaks do make a big difference. This is logical, because there is so much competition, I mean there are hundreds of websites competing for the same keywords and some have to rank above or below the others. Therefore even small changes can decide what is going to be your rank. But then this is my personal opinion. I think it can be a good topic for a future blog post; thanks for asking.
March 23rd, 2008 at 12:07 am
does anyone knows if there is any other information about this subject in other languages?