8. Conclusion

In this paper we addressed the problem of ordering URLs for crawling. We listed different kinds of

importance metrics, and built three models to evaluate crawlers. We experimentally evaluated several
combinations of importance and ordering metrics, using the Stanford Web pages.

In general our results show that PageRank, IR’(P), is an excellent ordering metric when either pages
with many backlinks or with high PageRank are sought. In addition, if the similarity to a driving query is
important, then it is also useful to visit earlier URLs that:

Have anchor text that is similar to the driving query;
Have some of the query terms within the URL itself; or
Have a short link distance to a page that is known to be hot.

With a good ordering strategy, it seems to be possible to build crawlers that can rather quickly obtain a
significant portion of the hot pages. This can be extremely useful when we are trying to crawl large
portions of the Web, when are resources are limited, or when we need to revisit pages often to detect
changes.

6. Similarity-Based Crawlers

In the experiments of Section 6, we compared three different backlink-based crawlers. In this section,
we present the results of our experiments on similarity-based crawlers. The similarity-based importance
metric, IS(P), measures the relevance of each page to a topic or a query that the user has in mind. There
are clearly many possible IS(P) metrics to consider, so our experiments here are not intended to be
comprehensive. Instead, our goal is to briefly explore the potential of various ordering schemes in some
sample scenarios. In particular, for our first three experiments we consider the following IS(P)
definition: A page is considered hot if it contains the word computer in its title or if it has more th an 10
occurrences of computer in its body. In our fourth experiment we consider a different definition.

Crawling algorithm (similarity-based)

Function description

enqueue(url_queue, starting_url);
while (not empty(hot_queue) and not empty(url_queue)) {
url = dequeue2(hot_queue, url_queue);
page = crawl_page(url);
enqueue(crawled_pages, (url, page));
url_list = extract_urls(page);
for each u in url_list
enqueue(links, (url, u));
if [u not in url_queue] and
[u not in hot_queue] and
[(u,-) not in crawled_pages]
if [u contains computer in anchor or url]
enqueue(hot_queue, u);
else
enqueue(url_queue, u);
reorder_queue(url_queue);
reorder_queue(hot_queue);
}

dequeue2(queue1, queue2) :
if (not empty(queue1)) dequeue(queue1);
else
dequeue(queue2);

For similarity-based crawling, the crawler of Figure 1 is not appropriate, since it does not take the
content of the page into account. To give priority to the pages mentioning computer, we modified our
crawler as shown in Figure 4. This crawler keeps two queues of URLs to visit: hot_queue stores the
URLs that have been seen in an anchor mentioning the word computer, or that have the word computer
within them. The second queue, url_queue, keeps the rest of the URLs. The crawler first takes URL to
visit from hot_queue.

The observed unexpected performance difference arises from the breadth-first crawler’s FIFO nature.
The breadth-first crawler fetches the pages in the order they are found. If a computer-related page is
crawled earlier, then the crawler discovers and visits its child pages earlier as well. These pages have a
tendency to also be computer related, so performance is better.

Thus, the observed property is that if a page has a high IS(P) value, then its children are likely to have a
higher IS(P) value too. To take advantage of this property, we modified our crawler as shown in Figure
5. This crawler places in the hot_queue URLs that have the target keyword in their anchor or within, or
that are within 3 links from a hot page.

Crawling algorithm (modified similarity-based)

enqueue(url_queue, starting_url);
while (not empty(hot_queue) and not empty(url_queue)) {
url = dequeue2(hot_queue, url_queue);
page = crawl_page(url);
if [page contains 10 or more computer in body
or one computer in title]
hot[url] = TRUE;
enqueue(crawled_pages, (url, page));
url_list = extract_urls(page);
for each u in url_list
enqueue(links, (url, u));
if [u not in url_queue] and
[u not in hot_queue] and
[(u,-) not in crawled_pages]
if [u contains computer in anchor or url]
enqueue(hot_queue, u);
else if [distance_from_hotpage(u) < 3]
enqueue(hot_queue, u);
else
enqueue(url_queue, u);
reorder_queue(url_queue);
reorder_queue(hot_queue);
}

Function description

distance_from_hotpage(u) :
return 0 if [hot[u] = TRUE];
return 1 if [hot[v] = TRUE] and [(v, u) in links]
for some v;
return 2 if [hot[v] = TRUE] and
[(v, w) in links] and [(w, u) in links]
some v, w;

5. Experimental Setup

To avoid network congestion and heavy loads on the servers, we did our experimental evaluation in two
steps. In the first step, we physically crawled all Stanford Web pages and built a local repository of the
pages. This was done with the Stanford WebBase, a system designed to create and maintain large web
repositories. It is capable of high indexing speeds (about 50 pages per second), and large data
repositories (currently 150GB of HTML is stored).

After we built the repository, we ran our virtual crawlers on it to evaluate the different crawling
schemes. Note that even though we had the complete image of the Stanford domain in the repository,
our virtual crawler based its crawling decisions only on the pages it saw for itself. In this section we
briefly discuss how the WebBase crawler operates, and how the particular database was obtained for our
experiments.

5.1 WebBase Crawler

WebBase runs several processes at a time, which crawl web pages. These processes receive a list of
URLs to be downloaded and simply return the full content of the HTML or any errors which happened
trying to get the pages. The crawling processes open several hundred connections at a time, resulting in
a crawling speed of about 25 pages/second for each process. Since servers can only handle a few tens of
hits per second at most, and slowing down servers with a web crawler is a problem, we use two different
kinds of load balancing in our system. First, our system splits all URLs which are going to be crawled
into 500 queues based on a hash of their server name. This causes all URLs from a given server to go
into the same queue. The crawlers then read one URL from each queue at a time, moving to a new queue
for each URL. This makes sure a given server is hit only once for every 500 URLs that are crawled.
Also, for servers that are slow at returning documents, only one connection is allowed from the crawler
to a particular server at a time. As is mentioned in Section 1, these kinds of load balancing requirements
affect the freedom a crawler has in deciding the crawl order.

The actual data the system is allowed to get is reduced for two reasons. The first is that many heuristics
are needed to avoid automatically generated, and potentially infinite, sets of pages. For example, any
URLs containing "/cgi-bin/" are not crawled, because they are likely to contain programs which generate
infinite sets of pages, or producing other undesirable side effects such as an unintended vote in an online
election. Several other heuristics based on the Location Metric described above are used to weed out
URLs which look undesirable. Another way the data set is reduced is through the robots exclusion
protocol[7], which allows webmasters to define pages they do not want crawled by automatic systems.

5.2 Description of Dataset

To download an image of the Stanford web pages, we started WebBase with an initial list of
"stanford.edu" URLs. These 89,119 URLs were obtained from an earlier crawl. During the crawl,
non-Stanford URLs were ignored. At the end of the process, we had 784,592 known URLs to Stanford
pages. It should be noted that 352,944 of the known URLs were on one server, www.slac.stanford.edu
[8], which has a program that generates infinite numbers of web pages. The crawl was stopped before it
was complete, but most of the uncrawled URLs were on only a few servers so we believe the dataset we
used to be a reasonable representation of the stanford.edu web. This dataset consisted of about 225,000
crawled valid HTML pages using roughly 2.5 GB of space for the raw pages.

We should stress that the virtual crawlers that will be discussed next do not use WebBase directly. As

stated earlier, they use the dataset collected by the WebBase crawler, and do their own crawling on it.
The virtual crawlers are simpler than the Web Base crawler. For instance, they can detect if a URL is
invalid simply by seeing if it is in the dataset. Similarly, they do not need to distribute the load to visited
sites. These simplifications are fine, since the virtual crawlers are only used to evaluate ordering
schemes, and not to do real crawling.

4. Ordering Metrics

A crawler keeps a queue of URLs it has seen during the crawl, and must select from this queue the next
URL to visit. The ordering metric O is used by the crawler for this selection, i.e., it selects the URL u
such that O(u) has the highest value among all URLs in the queue. The O metric can only use
information seen (and remembered if space is limited) by the crawler.

The O metric should be designed with an importance metric in mind. For instance, if we are searching
for high IB(P) pages, it makes sense to use an O(u) = IB’(P), where P is the page u points to. However,
it might also make sense to consider an O(u) = IR’(P), even if our importance metric is not weighted. In
our experiments, we explore the types of ordering metrics that are best suited for either IB(P) or IR(P).

For a location IL(P) importance metric, we can use that metric directly for ordering since the URL of P
directly gives the IL(P) value. However, for forward link IF(P) and similarity IS(P) metrics, it is much
harder to devise an ordering metric since we have not seen P yet. As we will see, for similarity, we may
be able to use the text that anchors the URL u as a predictor of the text that P might contain. Thus, one
possible ordering metric O(u) is IS(A, Q), where A is the anchor text of the URL u, and Q is the driving
query.

3. Problem Definition

Our goal is to design a crawler that if possible visits high I(P) pages before lower ranked ones, for some
definition of I(P). Of course, the crawler will only have available I’(P) values, so based on these it will
have to guess what are the high I(P) pages to fetch next.

Our general goal can be stated more precisely in three ways, depending on how we expect the crawler to
operate. (In our evaluations of Sections 6 and 7 we use the second model in most cases, but we do
compare it against the first model in one experiment. Nevertheless, we believe it is useful to discuss all
three models to understand the options.)

Crawl & Stop. Under this model, the crawler C starts at its initial page P0 and stops after visiting K
pages. At this point a perfect crawler would have visited pages R1, ..., RK , where R1 is the page with the

highest importance value, R2 is the next highest, and so on. We call pages R1 through RK the hot pages.
The K pages visited by our real crawler will contain only M pages with rank higher than or equal to
I(RK ). We define the performance of the crawler C to be PCS(C) = (M•100)/K. The performance of the
ideal crawler is of course 100%. A crawler that somehow manages to visit pages entirely at random, and
may revisit pages, would have a performance of (K•100)/T, where T is the total number of pages in the
Web. (Each page visited is a hot page with probability K/T. Thus, the expected number of desired pages
when the crawler stops is K 2 /T.)

Crawl & Stop with Threshold. We again assume that the crawler visits K pages. However, we are now
given an importance target G, and any page with I(P) >= G is considered hot. Let us assume that the
total number of hot pages is H. The performance of the crawler, PST (C), is the percentage of the H hot
pages that have been visited when the crawler stops. If K < H , then an ideal crawler will have
performance (K•100)/H. If K >= H, then the ideal crawler has 100% performance. A purely random
crawler that revisits pages is expected to visit (H/T) •K hot pages when it stops. Thus, its performance is
(K•100)/T. Only if the random crawler visits all T pages, is its performance expected to be 100%.

Limited Buffer Crawl. In this model we consider the impact of limited storage on the crawling process.
We assume that the crawler can only keep B pages in its buffer. Thus, after the buffer fills up, the
crawler must decide what pages to flush to make room for new pages. An ideal crawler could simply
drop the pages with lowest I(P) value, but a real crawler must guess which of the pages in its buffer will
eventually have low I(P) values. We allow the crawler to visit a total of T pages, equal to the total
number of Web pages. At the end of this process, the percentage of the B buffer pages that are hot gives
us the performance PBC(C). We can define hot pages to be those with I(P) >= G, where G is a target

importance, or those with I(P) >= I(RB), where RB is the page with the Bth highest importance value.
The performances of an ideal and a random crawler are analogous to those in the previous cases.

Note that to evaluate a crawler under any of these metrics, we need to compute the actual I(P) values of
pages, and this involves crawling the "entire" Web. To keep our experiments (Section 6 and 7)
manageable, we define the entire Web to be the Stanford University pages, and we only evaluate
performance in this context. That is, we assume that all pages outside Stanford have I(P) = 0, and that
links to pages outside Stanford or from pages outside Stanford do not count in I(P ) computations. In
Section 6.4 we study the implications of this assumption by also analyzing a smaller Web within the
Stanford domain, and seeing how Web size impacts performance.

2. Importance Metrics

Not all pages are of equal interest to the crawler’s client. For instance, if the client is building a
specialized database on a particular topic, then pages that refer to that topic are more important, and
should be visited as early as poss ible. Similarly, a search engine may use the number of Web URLs that
point to a page, the so-called backlink count, to rank user query results. If the crawler cannot visit all
pages, then it is better to visit those with a high backlink count, since this will give the end-user higher
ranking results.

Given a Web page P, we can define the importance of the page, I(P), in one of the following ways
(These metrics can be combined, as will be discussed later.):

1. Similarity to a Driving Query Q. A query Q drives the crawling process, and I(P) is defined to be
the textual similarity between P and Q. Similarity has been well studied in the Information
Retrieval (IR) community [Salton 1989] and has been applied to WWW environment [Yuwono
1995]. We use IS(P) to refer to the importance metric in this case. We also use IS(P, Q) when we
wish to make the query explicit.

To compute similarities, we can view each document (P or Q) as an m-dimensional vector <w1, ...,

wn>. The term wi in this vector represents the ith word in the vocabulary. If wi does not appear in
the document, then wi is zero. If it does appear, wi is set to represent the significance of the word.

One common way to compute the significance wi is to multiply the number of times the ith word

appears in the document by the inverse document frequency (idf ) of the ith word. The idf factor is
one divided by the number of times the word appears in the entire "collection," which in this case
would be the entire Web. The idf factor corresponds to the content discriminating power of a
word: a term that appears rarely in documents (e.g., "queue") has a high idf , while a term that
occurs in many documents (e.g., "the") has a low idf . (The w terms can also take into account

occurs in many documents (e.g., the ) has a low df . (The wi terms can also take into account
where in a page the word appears. For instance, words appearing in the title of an HTML page
may be given a higher weight than other words in the body.) The similarity between P and Q can
then be defined as the inner product of the P and Q vectors. Another option is to use the cosine
similarity measure, which is the inner product of the normalized vectors.

Note that if we do not use idf terms in our similarity computation, the importance of a page, IS(P),
can be computed with "local" information, i.e., P and Q. However, if we use idf terms, then we
need global information. During the crawling process we have not seen the entire collection, so we
have to estimate the idf factors from the pages that have been crawled, or from some reference idf
terms computed at some other time. We use IS’(P) to refer to the estimated importance of page P,
which is different from the actual importance IS(P), which can be computed only after the entire
Web has been crawled. If idf factors are not used, then IS’(P) = IS(P).

2. Backlink Count. The value of I(P) is the number of links to P that appear over the entire Web. We
use IB(P) to refer to this importance metric. Intuitively, a page P that is linked to by many pages is
more important than one that is seldom referenced. This type of "citation count" has been used
extensively to evaluate the impact of published papers. On the Web, IB(P) is useful for ranking
query results, giving end-users pages that are more likely to be of general interest.

Note that evaluating IB(P) requires counting backlinks over the entire Web. A crawler may
estimate this value with IB’(P), the number of links to P that have been seen so far.

3. PageRank. The IB(P) metric treats all links equally. Thus, a link from the Yahoo home page
counts the same as a link from some individual’s home page. However, since the Yahoo home
page is more important (it has a much higher IB count), it would make sense to value that link
more highly. The PageRank backlink metric, IR(P), recursively defines the importance of a page
to be the weighted sum of the backlinks to it. Such a metric has been found to be very useful in
ranking results of user queries [Page 1998.2]. We use IR’(P) for the estimated value of IR(P) when
we have only a subset of pages available.

More formally, if a page has no outgoing link, we assume that it has outgoing links to every single
Web page. Next, consider a page P that is pointed at by pages T 1, ..., T n. Let ci be the number of
links going out of page T i. Also, let d be a damping factor (whose intuition is given below). Then,
the weighted backlink count of page P is given by

IR(P) = (1-d) + d ( IR(T 1)/c1 + ... + IR(T n)/cn)

This leads to one equation per Web page, with an equal number of unknowns. The equations can
be solved for the IR values. They can be solved iteratively, starting with all IR values equal to 1.
At each step, the new IR(P) value is computed from the old IR(T i) values (using the equation
above), until the values converge. This calculation corresponds to computing the principal
eigenvector of the link matrices. PageRank is described in much greater detail in [Page 1998.2].

One intuitive model for PageRank is that we can think of a user "surfing" the Web, starting from
any page, and randomly selecting from that page a link to follow. When the user reaches a page
with no outlinks, he jumps to a random page. Also, when the user is on a page, there is some

probability, d, that the next visited page will be completely random. This damping factor d makes
sense because users will only continue clicking on one task for a finite amount of time before they
go on to something unrelated. The IR(P) values we computed above give us the probability that
our random surfer is at P at any given time.

4. Forward Link Count. For completeness we may want to consider a metric IF(P) that counts the
number of links that emanate from P. Under this metric, a page with many outgoing links is very
valuable, since it may be a Web directory. This metric can be computed directly from P, so IF’(P)
= IF(P). This kind of metric has been used in conjunction with other factors to reasonably identify
index pages [Pirolli 1996]. We could also define a weighted forward link metric, analogous to
IR(P), but we do not consider this here.

5. Location Metric. The IL(P) importance of page P is a function of its location, not of its contents. If
URL u leads to P, then IL(P) is a function of u. For example, URLs ending with ".com" may be
deemed more useful than URLs with other endings, or URL containing the string "home" may be
more of interest than other URLs. Another location metric that is sometimes used considers URLs
with fewer slashes more useful than those with more slashes. All these examples are local metrics
since they can be evaluated simply by looking at the URL u.

As stated earlier, our importance metrics can be combined in various ways. For example, we may define
a metric IC(P) = k1•IS(P, Q) + k2•IB(P), for some constants k1, k2. This combines the similarity metric
(under some given query Q) and the backlink metric. Pages that have relevant content and many
backlinks would be the highest ranked . (Note that a similar approach was used to improve the
effectiveness of a search engine [Marchiori 1997].)

1. Introduction

A crawler is a program that retrieves Web pages, commonly for use by a search engine [Pinkerton 1994]
or a Web cache. Roughly, a crawler starts off with the URL for an initial page P0. It retrieves P0,
extracts any URLs in it, and adds them to a queue of URLs to be scanned. Then the crawler gets URLs
from the queue (in some order), and repeats the process. Every page that is scanned is given to a client
that saves the pages, creates an index for the pages, or summarizes or analyzes the content of the pages.

Crawlers are widely used today. Crawlers for the major search engines (e.g., Altavista [1], InfoSeek [2],
Excite [3], and Lycos [4]) attempt to visit most text Web pages, in order to build content indexes. Other
crawlers may also visit many pages, but may look only for certain types of information (e.g., email
addresses). At the other end of the spectrum, we have personal crawlers that scan for pages of interest to
a particular user, in order to build a fast access cache (e.g. NetAttche [5], WebSnake [6]).

The design of a good crawler presents many challenges. Externally, the crawler must avoid overloading
Web sites or network links as it goes about its business [Koster 1995]. Internally, the crawler must deal
with huge volumes of data. Unless it has unlimited computing resources and unlimited time, it must
carefully decide what URLs to scan and in what order. The crawler must also decide how frequently to
revisit pages it has already seen, in order to keep its client informed of changes on the Web. Inspite of all
these challenges, and the importance of crawlers on the Internet, very little research has been done on
crawlers.

In this paper we address one of these important challenges: How should a crawler select URLs to scan
from its queue of known URLs? If a crawler intends to perform a single scan of the entire Web, and the
load placed on target sites is not an issue, then any URL order will suffice. That is, eventually every
single known URL will be visited, so the order is not critical. However, most crawlers will not be able to
visit every possible page for two main reasons:

Their client may have limited storage capacity, and may be unable to index or analyze all pages.
Currently the Web contains about 1.5TB and is growing rapidly, so it is reasonable to expect that

most clients will not want or will not be able to cope with all that data [Kahle 1997] .
Crawling takes time, so at some point the crawler may need to start revisiting previously scanned
pages, to check for changes. This means that it may never get to some pages. It is currently
estimated that over 600GB of the Web changes every month [Kahle 1997].

In either case, it is important for the crawler to visit "important" pages first, so that the fraction of the
Web that is visited (and kept up to date) is more meaningful. In this paper we present several useful
definitions of importance, and develop crawling priorities so that important pages have a higher
probability of being visited first. We also present experimental results from crawling the Stanford
University Web pages that show how effective the different crawling strategies are.

Of course, a crawler must also avoid overloading target sites, especially if they contain many important
pages. In this paper we do not address this issue. That is, we assume that URLs selected for scanning
may be delayed by a crawler component that monitors site loads, but we do not study here how this
delay component works. Similarly, we do not consider in this paper rescanning pages. To simplify our
evaluations, we assume that a crawler does not start revisiting pages until it has finished visiting pages.
In practice one may of course wish to start rescanning important pages even before a crawl is completed,
but this is beyond the scope of this paper.

Efficient Crawling Through URL Ordering

Abstract

In this paper we study in what order a crawler should visit the URLs it has seen, in order to
obtain more "important" pages first. Obtaining important pages rapidly can be very useful
when a crawler cannot visit the entire Web in a reasonable amount of time. We define
several importance metrics, ordering schemes, and performance evaluation measures for
this problem. We also experimentally evaluate the ordering schemes on the Stanford
University Web. Our results show that a crawler with a good ordering scheme can obtain
important pages significantly faster than one without.

8 Things We Learned About Google PageRank

Yesterday, Google lowered the Toolbar PageRank of many sites in many different online verticals, in what may be a permanent or temporary message to sites about selling links.

Google also lowered the Toolbar PageRank of many sites which do not sell links, so the link argument may not even be relevant. Until Google comes out with a statement on the changes in PageRank, we may be beating this argument into the ground. Especially since blog networks and major media sites were also included in the update, which seems to be hand delivered from the Google gods.

However, there has been a lot of coverage about Google PageRank over the last day, and if anything, the public is becoming more educated on the separation between Google Toolbar’s PageRank and how Google ranks a site.

Search Engine Journal is now a PageRank 4 or a PageRank 6, depending on what datacenter your toolbar is working off of. We were PageRank 7 for several years and have not, in my opinion, done much to offset this PageRank.

We have hundreds of thousands of incoming links and have never purchased a link to this site.

Here are 8 different notes about the Google PageRank ‘update’ (this could be a PageRank hiccup) from yesterday”

1. PageRank is not an indicator of Google traffic or Google Rankings.

Since May 2007, our Google search referral traffic has doubled and now stands at 45% to 50% of our incoming daily traffic. Search Engine Roundtable reports similar findings.

In fact, Search Engine Journal’s traffic from Google Search has shown a sharp increase over the past month, possibly due to some changes we have implemented (or the popularity of the subjects we cover).

2. Firing a PageRank Warning Shot

Google is not the Internet, but is all powerful. A drop in Google Toolbar page rating system leads to advertisers pulling out of sites, link deals being broken, widespread fear of paid links hurting site ranking and traffic.

In my opinion, by flipping the switch on changing the PageRank of some sites, Google is trying to send a message that they have the power to turn this search marketing industry over on its head. But they have not issued a rankings change.

3. Target the Authorities, But Not the Wrong Doers

Does lowering Search Engine Journal’s PageRank to a 4 stop a multi-million dollar industry from shutting down? No.

Do we deserve to be publicly humiliated or targeted by Google for having a couple of links in our ad section? No.

Are we still an authority site? Yes, of course.

Google targeted the PageRank of high profile sites in what is becoming a trickle down effect. ‘Engadget got hit, I better stop before I get hit too.” Yeah, but hit for what?

4. PageRank stands for PR, or Public Relations

I’ll let these Forbes quotes from Rand and Barry speak for themselves.

“The bottom line is that people are able to manipulate Google’s rankings by buying links, and Google has to do something about that,” he said. “Today, they sent a huge statement to some of the most popular blogs on the Internet, and particularly those in the search industry.”

Search engine marketer and blogger Rand Fishkin agrees that the pagerank shift was likely meant to send a message to link buyers and sellers. “Google has said in many conferences that this number isn’t accurate, that it’s a rough signal but shouldn’t make any determination about a site,” says Fishkin. “This is meant to be a more public way of saying,’we know what you’re up to.’”

5. NoFollow is a Sign of No Trust, We Trust Our Sponsors

Barry Schwartz says it best:

I trust my sponsors, I value their sponsorships and I couldn’t do what I do without their financial support. Some sponsors can’t afford huge sponsorships, so they sponsor in their ways. It is what enables this site and many other sites to function and operate on a daily basis. I turn down sponsors all the time because they are simply not relevant or useful to my reader. I hand select them and for them to be on my site, means I trust them. Why nofollow someone you trust and want to thank? Is that a slap in their face? Will I have to and will they continue to sponsor? Time will tell.

6. PageRank is not a Marketing Metric

If anything, yesterday’s PageRank change helped remind us that PageRank, like Alexa and other questionable ratings services, is not a pricing or marketing metric. A link from a PageRank 7 site is not going to have more vale than a link from a PageRank 3 site. The value is in the context of each site, the anchor text and other values. Don’t buy links based on PageRank, and don’t sell links based on PageRank. Come on, that’s just so 2002!

Doug Heil did a great job of discussing his view on this in our Search Engine Journal Comments area:

I’m don’t understand why Google displays the bar anymore. It’s purpose was to get good public relations a long time ago. It’s not needed now. It only serves “text link brokers” and those who sell pagerank text links. In other words, it’s totally useless. It actually always has been totally useless as a “rank” indicator.

Why the SEO industry has made a big deal of the green bar all these years is something I’ve never been able to understand. Oh sure; Google may have created the metric, but it’s the SEO industry who has been their very own worse culprit in spreading the BS and falsehoods about the bar.

7. It’s Time for a Better System than PageRank

SEOBook’s Aaron Wall speaks up:

Since Google is demoting PageRank’s viability as a site’s global authority score perhaps this is a time for Yahoo to bring back WebRank, or Ask to launch something like CommunityRank. The Google-webmaster relationship is fraying. This presents an opportunity for whoever wants to take it.

8. Lowered PageRank Leads to More Quality Links

Search Engine Journal was linked to yesterday from Google Blogoscoped, SEOmoz, Forbes.com, Search Engine Watch, Techmeme and Sphinn among other quality and authority sites because people’s toolbars changed around the world. Great link bait! Thanks Google :)

International markets

Optimization techniques are highly tuned to the dominant search engines in the target market. The search engines' market shares vary from market to market, as does competition. In 2003, Danny Sullivan stated that Google represented about 75% of all searches.In markets outside the United States, Google's share is often larger, and Google remains the dominant search engine worldwide as of 2007.As of 2006, Google had an 85-90% market share in Germany.While there were hundreds of SEO firms in the US at that time, there were only about five in Germany.As of June 2008, the marketshare of Google in the UK was close to 90% according to Hitwise.That market share is achieved in a number of countries.

As of 2009, there are only a few large markets where Google is not the leading search engine. In most cases, when Google is not leading in a given market, it is lagging behind a local player. The most notable markets where this is the case are China, Japan, South Korea, Russia and the Czech Republic where respectively Baidu, Yahoo! Japan, Naver, Yandex and Seznam are market leaders.

Successful search optimization for international markets may require professional translation of web pages, registration of a domain name with a top level domain in the target market, and web hosting that provides a local IP address. Otherwise, the fundamental elements of search optimization are essentially the same, regardless of language.

As a marketing strategy

SEO is not an appropriate strategy for every website, and other Internet marketing strategies can be more effective, depending on the site operator's goals.A successful Internet marketing campaign may also depend upon building high quality web pages to engage and persuade, setting up analytics programs to enable site owners to measure results, and improving a site's conversion rate.

SEO may generate an adequate return on investment. However, search engines are not paid for organic search traffic, their algorithms change, and there are no guarantees of continued referrals. Due to this lack of guarantees and certainty, a business that relies heavily on search engine traffic can suffer major losses if the search engines stop sending visitors.It is considered wise business practice for website operators to liberate themselves from dependence on search engine traffic.Seomoz.org has suggested that "search marketers, in a twist of irony, receive a very small share of their traffic from search engines." Instead, their main sources of traffic are links from other websites.

White hat versus black hat

SEO techniques are classified by some into two broad categories: techniques that search engines recommend as part of good design, and those techniques that search engines do not approve of and attempt to minimize the effect of, referred to as spamdexing. Some industry commentators classify these methods, and the practitioners who employ them, as either white hat SEO, or black hat SEO.White hats tend to produce results that last a long time, whereas black hats anticipate that their sites will eventually be banned once the search engines discover what they are doing.

An SEO tactic, technique or method is considered white hat if it conforms to the search engines' guidelines and involves no deception. As the search engine guidelines are not written as a series of rules or commandments, this is an important distinction to note. White hat SEO is not just about following guidelines, but is about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see.

White hat advice is generally summed up as creating content for users, not for search engines, and then making that content easily accessible to the spiders, rather than attempting to game the algorithm. White hat SEO is in many ways similar to web development that promotes accessibility, although the two are not identical.

White Hat SEO is merely effective marketing, making efforts to deliver quality content to an audience that has requested the quality content. Traditional marketing means have allowed this through transparency and exposure. A search engine's algorithm takes this into account, such as Google's PageRank.

Black hat SEO attempts to improve rankings in ways that are disapproved of by the search engines, or involve deception. One black hat technique uses text that is hidden, either as text colored similar to the background, in an invisible div, or positioned off screen. Another method gives a different page depending on whether the page is being requested by a human visitor or a search engine, a technique known as cloaking.

Search engines may penalize sites they discover using black hat methods, either by reducing their rankings or eliminating their listings from their databases altogether. Such penalties can be applied either automatically by the search engines' algorithms, or by a manual site review. One infamous example was the February 2006 Google removal of both BMW Germany and Ricoh Germany for use of deceptive practices.Both companies, however, quickly apologized, fixed the offending pages, and were restored to Google's list.

Additionally, many professionals in the SEO industry refer to "gray hat" tactics that may skirt the lines of black and white hat tactics. Numerous references to gray hat techniques have been published, and these usually constitute practices that are not strictly disapproved by search engines, but may go against the spirit of the regulations that search engines have laid out.

Increasing prominence

A variety of methods can increase the prominence of a webpage within the search results. Cross linking between pages of the same website to provide more links to most important pages may improve its visibility.Writing content that includes frequently searched keyword phrase, so as to be relevant to a wide variety of search queries will tend to increase traffic.[38] Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a web page's meta data, including the title tag and meta description, will tend to improve the relevancy of a site's search listings, thus increasing traffic. URL normalization of web pages accessible via multiple urls, using the "canonical" meta tag or via 301 redirects can help make sure links to different versions of the url all count towards the page's link popularity score.

Preventing crawling

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots. When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed, and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.

Methods of seo :

Getting indexed

The leading search engines, such as Google, Bing and Yahoo!, use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. Some search engines, notably Yahoo!, operate a paid submission service that guarantee crawling for either a set fee or cost per click. Such programs usually guarantee inclusion in the database, but do not guarantee specific ranking within the search results.[dead link] Two major directories, the Yahoo Directory and the Open Directory Project both require manual submission and human editorial review.Google offers Google Webmaster Tools, for which an XML Sitemap feed can be created and submitted for free to ensure that all pages are found, especially pages that aren't discoverable by automatically following links.

Search engine crawlers may look at a number of different factors when crawling a site. Not every page is indexed by the search engines. Distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled.Additionally, search engines sometimes have problems with crawling sites with certain kinds of graphic content, flash files, portable document format files, and dynamic content.

Relationship with search engines

By 1997 search engines recognized that webmasters were making efforts to rank well in their search engines, and that some webmasters were even manipulating their rankings in search results by stuffing pages with excessive or irrelevant keywords. Early search engines, such as Altavista and Infoseek, adjusted their algorithms in an effort to prevent webmasters from manipulating rankings.

Due to the high marketing value of targeted search results, there is potential for an adversarial relationship between search engines and SEO service providers. In 2005, an annual conference, AIRWeb, Adversarial Information Retrieval on the Web, was created to discuss and minimize the damaging effects of aggressive web content providers.

Companies that employ overly aggressive techniques can get their client websites banned from the search results. In 2005, the Wall Street Journal reported on a company, Traffic Power, which allegedly used high-risk techniques and failed to disclose those risks to its clients.Wired magazine reported that the same company sued blogger and SEO Aaron Wall for writing about the ban.Google's Matt Cutts later confirmed that Google did in fact ban Traffic Power and some of its clients.

Some search engines have also reached out to the SEO industry, and are frequent sponsors and guests at SEO conferences, chats, and seminars. In fact, with the advent of paid inclusion, some search engines now have a vested interest in the health of the optimization community. Major search engines provide information and guidelines to help with site optimization.Google has a Sitemaps program to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website. Google guidelines are a list of suggested practices Google has provided as guidance to webmasters. Yahoo! Site Explorer provides a way for webmasters to submit URLs, determine how many pages are in the Yahoo! index and view link information. Bing Toolbox provides a way from webmasters to submit a sitemap and web feeds, allowing users to determine the crawl rate, and how many pages have been indexed by their search engine.

History of ceo

Webmasters and content providers began optimizing sites for search engines in the mid-1990s, as the first search engines were cataloging the early Web. Initially, all webmasters needed to do was submit the address of a page, or URL, to the various engines which would send a "spider" to "crawl" that page, extract links to other pages from it, and return information found on the page to be indexed.The process involves a search engine spider downloading a page and storing it on the search engine's own server, where a second program, known as an indexer, extracts various information about the page, such as the words it contains and where these are located, as well as any weight for specific words, and all links the page contains, which are then placed into a scheduler for crawling at a later date.

Site owners started to recognize the value of having their sites highly ranked and visible in search engine results, creating an opportunity for both white hat and black hat SEO practitioners. According to industry analyst Danny Sullivan, the phrase "search engine optimization" probably came into use in 1997.The first documented use of the term Search Engine Optimization was John Audette and his company Multimedia Marketing Group as documented by a web page from the MMG site from August, 1997.

Early versions of search algorithms relied on webmaster-provided information such as the keyword meta tag, or index files in engines like ALIWEB. Meta tags provide a guide to each page's content. Using meta data to index pages was found to be less than reliable, however, because the webmaster's choice of keywords in the meta tag could potentially be an inaccurate representation of the site's actual content. Inaccurate, incomplete, and inconsistent data in meta tags could and did cause pages to rank for irrelevant searches.Web content providers also manipulated a number of attributes within the HTML source of a page in an attempt to rank well in search engines.

By relying so much on factors such as keyword density which were exclusively within a webmaster's control, early search engines suffered from abuse and ranking manipulation. To provide better results to their users, search engines had to adapt to ensure their results pages showed the most relevant search results, rather than unrelated pages stuffed with numerous keywords by unscrupulous webmasters. Since the success and popularity of a search engine is determined by its ability to produce the most relevant results to any given search, allowing those results to be false would turn users to find other search sources. Search engines responded by developing more complex ranking algorithms, taking into account additional factors that were more difficult for webmasters to manipulate.

Graduate students at Stanford University, Larry Page and Sergey Brin, developed "backrub," a search engine that relied on a mathematical algorithm to rate the prominence of web pages. The number calculated by the algorithm, PageRank, is a function of the quantity and strength of inbound links.PageRank estimates the likelihood that a given page will be reached by a web user who randomly surfs the web, and follows links from one page to another. In effect, this means that some links are stronger than others, as a higher PageRank page is more likely to be reached by the random surfer.

Page and Brin founded Google in 1998. Google attracted a loyal following among the growing number of Internet users, who liked its simple design.Off-page factors (such as PageRank and hyperlink analysis) were considered as well as on-page factors (such as keyword frequency, meta tags, headings, links and site structure) to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more difficult to game, webmasters had already developed link building tools and schemes to influence the Inktomi search engine, and these methods proved similarly applicable to gaming PageRank. Many sites focused on exchanging, buying, and selling links, often on a massive scale. Some of these schemes, or link farms, involved the creation of thousands of sites for the sole purpose of link spamming.

By 2004, search engines had incorporated a wide range of undisclosed factors in their ranking algorithms to reduce the impact of link manipulation. Google says it ranks sites using more than 200 different signals.The leading search engines, Google, Bing, and Yahoo, do not disclose the algorithms they use to rank pages. Notable SEO service providers, such as Rand Fishkin, Barry Schwartz, Aaron Wall and Jill Whalen, have studied different approaches to search engine optimization, and have published their opinions in online forums and blogs.SEO practitioners may also study patents held by various search engines to gain insight into the algorithms.

In 2005 Google began personalizing search results for each user. Depending on their history of previous searches, Google crafted results for logged in users.In 2008, Bruce Clay said that "ranking is dead" because of personalized search. It would become meaningless to discuss how a website ranked, because its rank would potentially be different for each user and each search.

In 2007 Google announced a campaign against paid links that transfer PageRank. On June 15, 2009, Google disclosed that they had taken measures to mitigate the effects of PageRank sculpting by use of the nofollow attribute on links. Matt Cutts, a well-known software engineer at Google, announced that Google Bot would no longer treat nofollowed links in the same way, in order to prevent SEO service providers from using nofollow for PageRank sculpting.As a result of this change the usage of nofollow leads to evaporation of pagerank. In order to avoid the above, SEO engineers developed alternative techniques that replace nofollowed tags with obfuscated Javascript and thus permit PageRank sculpting. Additionally several solutions have been suggested that include the usage of iframes, Flash and Javascript.

In December 2009 Google announced it would be using the web search history of all its users in order to populate search results.

The Real-time-search was introduced in late 2009 in an attempt to make search results more timely and relevant. Historically site administrators have spent months or even years optimizing a website to increase search rankings. With the growth in popularity of social media sites and blogs the leading engines made changes to their algorithms to allow fresh content to rank quickly within the search results.

Search engine optimization

Search engine optimization (SEO) is the process of improving the visibility of a website or a web page in search engines via the "natural" or un-paid ("organic" or "algorithmic") search results. In general, the earlier (or higher on the page), and more frequently a site appears in the search results list, the more visitors it will receive from the search engine's users. SEO may target different kinds of search, including image search, local search, video search, academic search,news search and industry-specific vertical search engines.

As an Internet marketing strategy, SEO considers how search engines work, what people search for, the actual search terms typed into search engines and which search engines are preferred by their targeted audience. Optimizing a website may involve editing its content and HTML and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines. Promoting a site to increase the number of backlinks, or inbound links, is another SEO tactic.

The acronym "SEOs" can refer to "search engine optimizers," a term adopted by an industry of consultants who carry out optimization projects on behalf of clients, and by employees who perform SEO services in-house. Search engine optimizers may offer SEO as a stand-alone service or as a part of a broader marketing campaign. Because effective SEO may require changes to the HTML source code of a site and site content, SEO tactics may be incorporated into website development and design. The term "search engine friendly" may be used to describe website designs, menus, content management systems, images, videos, shopping carts, and other elements that have been optimized for the purpose of search engine exposure.

Another class of techniques, known as black hat SEO, search engine poisoning, or spamdexing, uses methods such as link farms, keyword stuffing and article spinning that degrade both the relevance of search results and the quality of user-experience with search engines. Search engines look for sites that employ these techniques in order to remove them from their indices.

An Interview of the Salty Droid

Interviewer: Today we're going to interview not a person, so much as a robot, or maybe a person behind a robot. Who is the Salty Droid?
Jason Jones: Is that the first question, "Who is the Salty Droid?"

Interviewer: Yes.
Jason Jones: All right. Well, the answer is, Jason Jones.

Interviewer: Jason Jones. OK. Why did you decide to create a robot for your website or what was the idea behind that?
Jason Jones: Well, I like robots first of all, because everyone likes robots. I was just using that as my online persona, and then the whole Salty Droid project developed underneath it. The robot just came out of nowhere, out of the blue.

Interviewer: When you are writing or talking or compiling, everything you do is the Salty Droid? Do you view that as an extension of yourself? Or do you view that as something that separates yourself from what you're doing? Or how do you think of it that way?
Jason Jones: I think of it definitely as separate. I try to keep it completely depersonalized or keep a layer in between it and me, because the robot is really angry and aggressive, but those aren't healthy emotions to take on personally. The robot is the character and the blog is the project. And it's more than just me. It's more than one person. There's a whole community there. I'm just one piece of it. I definitely don't think of myself as that, as the robot.

Interviewer: That leads to two questions. One, you built a community around this, but then, two, you said that it's not good to have the anger and negative emotions. Do you view the community as being full of negative emotions?

How can you create a community that revolves around a character that has stuff you wouldn't describe as good? Can you build a community that is separate from the traits of the founder of it?

Jason Jones: Well, the hyper-aggression and the bad attitude are mostly communiqué. And I think most of the people in the community. All the legit people in the community are really caring, good people, who get that the aggression is a joke.

The targets of the aggression the things that are going on that we're pointing to are really serious things that people need to stand up and say something about. It takes an aggressive tone that I don't think anyone really tries to personify the robot's charms.

Interviewer: One of the things that's an issue online is it's really easy to point to what's bad or what's wrong. It's really easy to be cynical. But do you think there is enough good resources for people to find what will help them and what's good with the site mainly being focused on staying away from what's bad? Or can you focus on one too much or if you did both would it cause problems?
Jason Jones: Yeah. I think mixing them up would be a terrible idea because of the specific thing I'm talking about. I'm totally sure that I don't know what is the good side of making quick money online.

Interviewer: Right.
Jason Jones: How can you find the right help to do that, because that is not a real thing. You can't make quick money online. It's really hard to make money online. That is the reality of the situation. As far as how people get help in accentuating the positives. I really don't see, what are the positives?

Interviewer: From that perspective I think you hit on one of the things, is that a lot of the people have the mindset. Like, I got an email today, where the person said that they want to make something. They've been buying all these network-marketing things and they want to be able to make money really quick and easy without needing a PhD.

I've had other people say that they'd be willing to pay me a portion of the profits for whatever I taught them but nothing upfront. There's even been a person, he's offered me to pay me. They wanted me to rank someone else's page lower, a competitor. And offered to pay me after the fact.

[laughter]

Interviewer: The big thing there is there's a lot of mindset where people try to take whatever they can get and take. And the thing is a lot of them end up running into a roadblock by the view of the need fast, easy, cheap, free or placebo cost, but need it to be automated and make a lot.

Do you think the big problem is the vultures or the mindset of people?

Jason Jones: The vultures. No, it is the vultures. It's not people's mindset and people's weakness and people's vulnerability, and people's desire to have a life that's different from the life that they have. That is just how humans are.

And there's certain ways you can capitalize that that are seedy and not very respectable, but then you can prey on it. You can become a predator. And that's a fallback excuse that people use is trying to characterize the victims like that, so that it feels less painful to think about.

That they are also exploiting this idea of like, "Hey, let's make the world a better place." That is exploited just as much as this greed tactic. It exploits good people, greedy people. Anyone who has human weaknesses is exploitable.

Interviewer: What are the emotions you would say are most commonly preyed upon the "get rich" people?

Jason Jones: In the "get rich" thing, greed is a part of it. For instance, I listened to a huge batch of boiler room calls. OK. I won't mention anything specific about, but 100 hours. And it's overwhelming. I heard a few calls where it was greed and it is this stereotype person of this chaser who wants to believe the impossible.

I don't think that's the majority. I'm not sure how big of the portion that represents, but it's not that significant. It's people who are afraid, people who want a brighter tomorrow, people who things are falling apart for and who are at a moment in their life where they are particularly vulnerable.

And it's not the same people over and over. People get ground out and pushed out and in comes a new batch. They're always looking for this new batch of vulnerable people.

Interviewer: This is maybe a bit abstract or wide-reaching, but in the same way the monetary system is setup as being debt-based. To where if you have an income inequality and some people have savings, there's got to be some other people that are in debt or living right close to the edge.

Do you think how we structure our political and economic system, feeds into the people being vulnerable and desperate? Or do you think no matter how it was structured people would always be that way no matter what?

Jason Jones: No. I think part of what's making people vulnerable is they're thinking that they don't have enough. And this constant buy-buy culture and the credit, lending. And it's not just personal. Everything is based off on debt. Debt is our currency.

People's weaknesses and personalities develop inside of that. It's a microcosm, the scammy end of the spectrum what I'm writing about. It's done very basely down at the bottom, but it's a reflection of exactly how things go all the way to the top. It's in the political structure. It's in the financial system. We're structured like this.

Interviewer: Some of the patterns of the stuff you particularly don't like, is preying on people's emotions. Some of the stuff you do on your blog comes down to sleuthing and what Dereby called "investigative journalism", in a world where there is almost none. How do you get so much of the data? Is this building the community help pull on to that stuff in for you? Or are you really technically savvy? How are able to dig so much stuff up?

Jason Jones: Yeah. That's a human groundwork. It's a beat work. It just takes time. I started writing it and people started coming. And the more people come, the more people come. And then I keep quiet about who I'm talking to and I keep my sources confidential.

You'll see in the writing style I never say, "So and so says", or "This anonymous source." I never mention ever where anything is coming from. I just do it and if you read long enough you just have to come to rely on the fact that there is stuff going on behind the scenes that I'm not going to talk about. People don't want to talk about it, because it's a really cagey, dark situation. And people have their own interests and they don't want to.

But it started happening almost immediately. People started to come talk to me and I just talked to them. Keep it going. At this point I have this never-ending stream of information that just comes at me and only a tiny, tiny percentage of it ends up on the blog.

Interviewer: You did a lot of interesting graphics stuff. Did you find that hard to do? How were you able to tie in the image and audio? Let's say you put up a five-minute video or a three-minute video and you make all you custom graphics, how much work goes into that? [laughter]
Jason Jones: A lot.

Interviewer: It looks like it. Because I do the basic videos of like, "Here's the screenshot of this, and here's how it works." I make a three or a five-minute thing and I always screw-up in the middle. Then I get ticked off with myself, and start cursing on myself. I can imagine how hard it is to sequence all that together. Have you gotten more efficient with that over time? Or what did you use? Was it just a lot of hard practice till you get used to doing it?
Jason Jones: Yeah. It was just practice, because the first time I had no idea. I had a reason, a motivation to do it. I just would do it. But that big epic video they were talking about Jeff Foster and Andy Jenkins. They were talking about the Syndicate and telling me to go fuck myself. That video took 36 hours, probably.

Interviewer: Wow.
Jason Jones: It was just a long time.

Interviewer: What takes more time? Is it cutting up the audio, or creating the graphics? Or figuring out what pieces you're going to use?
Jason Jones: Yeah. Everything goes wrong, and the audio formats don't match. You just have to get a few parts in. You restart, because you feel like your idea was idiotic.

Interviewer: Have you thought about making videos about some of the stuff you do? There's one site I subscribe to. Financial advice where the guy is totally low-key, he's always questioning himself. His website's called iTulip." And he makes these amazing graphs comparing different asset classes over time.

Sometimes he's like, "Yeah, I did this pretty quick," but he's taking a long time for most of it. Have you thought about some of the stuff you do, like creating tutorials on how to do some of this?
Jason Jones: No, definitely not. Because one, that reach was the thing I'm talking about. And I don't want to do anything even close to that. Just to keep the line totally clear. Two, I have a hard time explaining the things that I'm doing to people who aren't me. It's hard. I know a lot of different little tricks. I don't know. You've got to figure out your own. Figure out your own little tricks.

Interviewer: With the stuff you're doing, if the site gets more popular, if you ever decide to do so many years down the road, so many months, so many years. At some point do you think you're eventually going to lose passion for the project? or do you see yourself doing for years to come?
Jason Jones: I don't know. I like it right now. No. I see myself doing it for a while, because no one's doing it. If I stop doing it, then what? It's something different is gone. I don't want that to happen. I'm a fan of the site. I like it. I love the site.

Interviewer: What's the hardest part with running it? Is it doing the stuff yourself? Dealing with how other people interact with it? Or dealing with what people do away from it? Or what's the hardest parts with it? You were struggling with like stuff like people taking down social media accounts.

Jason Jones: Yeah. That's the hardest part, because that's disappointing. When I first started, I expected that this project would have the support of the Internet community. Because it's the Internet community, that's creating the distribution system for this vicious scam. And people don't like it. It's not popular. Things that these people do aren't popular with the normal people you want using your websites.

I thought people would be behind me. Plus, that's how it's supposed to be. In America, there's this like illusion that you can say whatever you want. And it's all just like Wild West speech around here. But it's not like that at all. The Internet companies don't support you. It's way more work than it should've been just to keep the site existing.

That's not fun work to try to keep it up. That doesn't do anything for the cause. It doesn't help anyone. It's not helping me. It's a waste of time, and it's totally unnecessary. I'm obviously not going to lose. They're not going to be able to get rid of me, so it's a waste of their time.

I think that's the most disappointing part, getting banned from all these different social networks, getting banned from hosting sites, having to resort to...

Part of the trap is that you go onto YouTube, and you think it's an open forum where there's multiple voices. If people are getting scammed, they're going to be making YouTube videos, just like these scammers are making YouTube videos. And those two will weigh each other out, but that's not it. If someone wants to take your content down more than you want to keep it up, it's pretty hard to keep it up.

Interviewer: You mentioned something about that being an illusion. Well, you mentioned part of it being technical stuff related to that, but you also mentioned it being an illusion. Do you see that as a pattern that's always been that way in society across all cultures? Do you see the Internet making that better or worse? How do you feel about that?

Jason Jones: Well, this particular thing that's dangerous about the Internet is that there's a perception, more so than ever before, that dissent is available. When you could only distribute through the paper, you knew it wasn't open. It was incredibly limited by whatever the publication medium was. You could think about that as you were looking.

But now you get the idea from everything in the media and from most of the stuff on the Internet, that the Internet is the voice of the little guy. But then when you go and look, you find out, "No, the little guy gets silenced still, and his voice is not there"

But now there's holding out that his voice is there, and he's just not saying anything. So, he must be happy about it. He must not have just got his credit card maxed out and had his wife leave him. And suddenly drinking a fifth of Scotch a day.

It's not that those comments never pop up. It's that whenever someone gets that boldness, they get slapped right back down. And they're not in a position to fight back. I'm speaking about it in my own personal experience from this scam area , but it's obviously like that across the Web, too.

You want to talk about gas frack explosions in your back yard, like you can bet there's dozens of people who put up things. Then some company's hack lawyer came along and demanded they take them down. They didn't know their rights, and they can't afford to be availed to seek any counsel on those rights. It's just easier to take it down. Should not be like that.

Interviewer: There's also the extreme of false complaints and sites like Ripoff Report that have been called a variety of things. I don't even know what words I could use without availing myself to a lawsuit. [laughter]
Jason Jones: I'll say it, extortion racket. That's what people accuse them of, of running an extortion racket, because it looks a lot like that.

Interviewer: How does the consumer separate out? You think people are falsely confident that they have a full spectrum; how can they become more aware of stuff they should trust versus stuff they shouldn't?
Jason Jones: That is a good question. I don't know. Knowing who to trust is a hard thing to do, especially on the Internet, because of how many different channels and how many different voices there are. Because right now, at the moment, as we speak, things are running wildly out of control.

Interviewer: Things are running wildly out of control, what does that mean?

Jason Jones: If you don't know, if you're not sophisticated on the Internet, it's dangerous. It's dangerous to spend money on the Internet. It's dangerous to put your credit card on the Internet. Yeah, it's hard to tell.

You can't go to Ripoff Report and trust what's there, when you know that sometimes the complaints are false. And that no one's editing them, and the person in charge isn't at the wheel. Or is running a corporate advocacy program where he's taking the side of people who are known scammers. The Internet is turning things seedy.

Interviewer: Yeah. Part of that is that the Internet naturally has network effects built into a lot of different things, like the first person in the search result's going to get the bulk of the clicks. The leading search engine's going to get the bulk of the search traffic. And you see that with systems like....

I'm talking to you on Skype now, and it's got tons of users. Isn't it, though just how businesses run? A lot of businesses start off pure, and then grow. They get larger. Get dysfunctional through their size. Then they just have to keep making the numbers?
Jason Jones: Yes. That's clearly what's happening.

Interviewer: It's not really just a web-only phenomenon.
Jason Jones: Oh, no!

Interviewer: It's just that on the web you feel you're getting more diversity when maybe you're not. So one thing on the web...
Jason Jones: That's the big difference I'm pointing to on the web, is that there needs to be a disclaimer, so that people have the idea. But that won't work either, so there's just no way.

Interviewer: Yeah. I think the key is building good internal filters for who to trust, but anyone who's new has a hard time with that. It's almost like you have to get struck down once or twice somewhere to...
Jason Jones: Exactly. It's hard when you come right on. The sites that I trust, and the things I trust on the Internet are the ones where I can smell the person behind it. Once it gets out the point where it's so big you're not sure what you're looking at reported, like Huffington Post. It's like at first, it was a thing. Then at the end, it's just this big goulash. Then it's like, "I'm not going to look at this anymore. I can't relate to it."

Interviewer: Right. You think that it's having the character and a voice of an individual or a small group of individuals that you've learned over time is valuable. The more depersonalized it becomes, the more mushed, the less you can trust it. I had an interesting thing along that lines, when Google recently did an update called the Panda update, and a lot of larger sites from big brands got a big boost. But then, a lot of the independent sites actually end up getting crushed out that didn't have the brand. It sounds like the relevancy algos are going in the exact opposite direction of what you're saying is best for the web?
Jason Jones: Towards the bigger? No. That is bad, right? That's the old-school way. That's the thing that is so not working. Let's not put that on the Internet and do it again, where it's even easier to scale up to an irrational size, become unreasonably big and useless in all of your ways.

Interviewer: Have you seen my weight scale that posts to Twitter, is that what you're saying? No. [laughter]

Interviewer: Let's see.
Jason Jones: Although I used to like Google more, before I started reading your blog. Because then I saw them more as a heroic force, and then the way you talk about it. Yeah, I can see how so many of their tactics are little guy squeezing, which is really not what I want to see happen on the Internet.

Interviewer: It seems that offline, there's growing income inequality. And maybe technology only speeds that up as well.
Jason Jones: No. That's reflective of the offline world, too. Everything is way too big. Big groups are the worst, most unreliable. It's one of our worst human inventions, forming giant groups. And the bigger the group is, the stupider it is. Yet our whole economy is about building out the biggest possible things. I'm not an expert on this, so I don't know why I'm running my mouth about it.

Interviewer: There's a book I read called "A Thousand Years of Nonlinear History." I will admit that the reading was deep and beyond my level when I first started. But it was interesting, so I stuck with it. And one of the things he said is that it seems that we've always sacrificed variety in favor of homogenation to increase yield, as a general piece of capitalism.

Online, some people will come, to your blog and say, "F*ck you, you stupid robot!" or stuff like you would never see people do in person. Where they're really enraged. Do you deal with a lot of that? Or do you think people view you in that way? Or how do you get the humor angle across without turning people off?

Jason Jones: Well, the blog is supposed to be complicated, so it's not easy to understand. When you first get there, it's not supposed to be totally clear what's happening. Because I like it like that. Part of the message, like this looks like it's coming, this looks bad. When you just first glance at it, you're like, "Oh, this! I'm not sure this guy should be doing this, Geez! Does he have to go that far?"

But then you stay for a while, and then it's really easy to figure out. I don't think it's just like a mystery to anyone. But the trolls, that is rage. The site doesn't really get that many trolls in a traditional Internet sense, where it's like someone who's just popping in and they're just trying to get a rise out of a group. And then they thrive off of that. There is some of that, obviously.

But much more what happens on the blog is people who come to try to defend their own financial interests without disclosing that. And because this transparency and jokes, especially affect their day-to-day numbers, they have some massive overreactions. I get that in public and in private every day.

Also, people who have an idea about something that they think is possible, and it's not possible. Then they start to figure out that it's not possible. Then they want to lash out at someone about that. A lot of times, that ends up being my robot, which is a good thing to lash out at, actually.

Interviewer: What do you mean by that, that they find out something? What are you saying, are you saying like making money quickly, or having some system?

Jason Jones: Well, the specific stories I'm writing about, it follows a much more cult-like pattern, where they're trying to disrupt your normal way of thinking. They're explicitly doing that. And they're filling you with this other thing, which is convenient for them. Which ends in giving them your money and most of your time and part of your life for a while. Then they just dump you out at the end. When you're in that process, it's like a very deep, dangerous process to mess with people's personalities like that.

When you wake up out of that and you see, "Oh, I've been kind of semi-delusional here. I've been lying to my family. I've been being aggressive to my friends." "This is affecting my life," and just, "This is not as advertised," it's hard to face up to that. Lashing out at the Salty Droid is often...

And I also get, along with the death threats, I also do get a lot of apologies.

Interviewer: Do you have any way of gauging how much you help people at all? Do you get thank-you emails every day? Or do people tell you that they were in a like crappy spot, and then they came across your stuff, and it changed their way of thinking?
Jason Jones: Yes. I get those always, all the time. And it's much more in private than on the thing in public.

Interviewer: Yeah, because I would imagine people might feel a bit embarrassed to admit that they were getting ripped off or something.
Jason Jones: Yeah. A lot of the stories I hear are so personally tragic, and they contain so much just like horror that people don't... Talking about them more is painful, and people definitely are uncomfortable talking about it in public. I'm sorry, what was your question?

Interviewer: Well, continuing from where you were, is that largely what drives you to keep going with the site?
Jason Jones: Yeah. Definitely. Without that, it would be too hard, because there's not a lot. Sometimes I wonder why I'm doing it, but families especially like in the James Ray situation. Well, all of the situations. For every outrageously stupid comment, threat, or whatever that I see or get, I get 10 from the other side.

Interviewer: You mentioned a group called The Syndicate.
Jason Jones: The Syndicate.

Interviewer: What is that? Do you think there's five or 10 or 20 or 50 different groups that are aligned similarly to what you mention? And you just honed in on one group? Or do you think that one has more reach than the others? Or why so much focus on this one group? You also mentioned that it seemed like some of this stuff weaves together. Can you describe how that is? Am I making any sense or not?

Jason Jones: Yeah. You're right. It's that I focus on The Syndicate. But the idea is that this white-collar fraud that's going down in all different formats, not just the Internet, in all different transmission methods, they're using these cartels. The cartel system is one of the key features in lots of different scams that you can look up.

You get close enough and you look close at it, a key feature is cartels, false testimonial, keeping people out of the market. Having an in-group and an out-group, and a secondary tier, lieutenants. If you look close, the organizational structure is there to see across all different forms of people.

But I just focus in on The Syndicate because that's where I started. I like to say things that I have public, that I have proof of that I can put out there. That's just hard to come by.

I want to finish that story. The Syndicate is a big. They did big-time damage. They're on these audios that I have all over the website. Like just talking openly about violating the laws, and there's 1000s and 1000s of individual stories of how these people have wrecked lives and taken millions of dollars. And I'm going to keep on it until something changes.

Interviewer: Why would they record those calls? Or how would those calls end up recorded then? Like we're recording this one.
Jason Jones: Yeah, they record. They like to. They're narcissists. They like to hear the sound of their own voice, and they record their calls. They record all kinds of things they should not be recording. Even after I get some of those recordings, they keep on doing it, because they don't learn any lessons. Like the last one I released, which was that boiler room call. The boiler rooms, they record their calls, because that's what boiler rooms do.

Interviewer: Do you sometimes feel like a man trying to hold back the Sahara Desert in a way, because this stuff's everywhere? Or do you think if you make a big enough difference in one area, people will carry on and help in other areas? Like it spreads out in circles?
Jason Jones: I don't know. That's what I want to think about when I'm in a drum circle. I don't know how realistic that is. I hope that's how it is. I hope there's all these magical, beautiful butterflies being born, because of what I'm doing.

Interviewer: Part of media has always been if you go to other forms of media, stuff spreads out, and then it reconsolidates. Like there's a bunch of people experimenting. And then ultimately, a lot of that comes down to social signals. Like people trust what they think other people trust.

You did a good job marketing your site, especially on a very small budget. What were keys to getting well-known exposure, spreading your ideas?

Jason Jones: I don't know. I try not to think very much about that. Because if you're going to start a blog, and if you want to start a real blog. Not the way people I talk about, talk about starting a blog. But if you have something to talk about and you want to start a blog, then you start it. People aren't going to comment first. And you want people to read your writing. They're not, so then how do you stay motivated to keep writing? It seems like a trap.

The key is to focus on the thing that you are writing about, and have that be the thing that you care about. Because the thing that was clear immediately is that it didn't matter how big of an audience I had. If it was zero or 1000, 10,000, it didn't matter.

From the very first word, sound I made on the Internet, the people who I was talking about, they heard. That's it. You have to focus on the audience or the thing that you're talking about, and the Internet can allow you to interact with them.

If you stay focused on building a quality thing and caring about the thing that you're actually writing about. And being a responsible advocate for that thing in whatever form it is, obviously not my crazy format, but that's the important thing.

How you end up like getting popular. Now I'm popular, but I really don't know. You tell me how that happened. I don't know.

Interviewer: Did you make the number one most popular robot? Or, number two most popular robot online behind GoogleBot? Get you a little plaque made up? Three years down the road, you decide you have a great idea, or you think you could really help people. It's not big, bureaucratic, dysfunctional, large company, but you eventually want to start doing something where you think you help people. But it's going to be more of a money-making enterprise, rather than something that you're just doing for free.

Do you think the marketing lessons you learned from building the Salty Droid help you, more than any of the bad karma from people hating Salty Droid? Some of the people who you've exposed hate that obviously. And whatever you do going forward, there's going to be some connection, right, between them? In terms of people will try to connect them up?
Jason Jones: Between this project and my next project, you mean?

Interviewer: Yeah, I'm not saying that you're going to go from Salty Droid to SaltyMillionaire.com. What I'm saying is like whatever you do next, the web has a way of tying people or things together. Are you worried about that at all? Or do you never really plan on trying to make much money online, just do what you're interested in mostly?
Jason Jones: Well, I don't believe that you can make money online, number one. And no one's ever been able to convince me by showing anyone who's doing it.

Interviewer: Well, we'll do a sidebar after this.

Jason Jones: First of all, it's like an art project in my opinion. Once I'm done with it, hopefully it'll still be up. It'll be there forever, and that's one of the cool things about the Web. I'm really proud of it. I don't plan to be haunted by it in my future endeavors, although I really don't want to stop doing it.

I don't have some aversion to making money from it. I don't think it's evil to make money off of the Internet. And I'm kind of getting close now. Now I have a big audience, and if I wanted to try to do something non-exploitative, I have some ideas that I think could work. And I'm definitely not against turning it into something.

It'll be weird now if I make money online, because my primary message is that you can't make money online. But it is like that. It's like, "Yeah!" If I end up making some money doing this it's because I built up a mass audience.

But that's almost impossible, so much that you should not be thinking about that. If that's one of the specs in your business plan, "Build up mass audience," then forget it. That's not a realistic goal.

Interviewer: Unless you can keep the cost structure low while you're doing it, and you're having fun doing it. If you're having fun doing it and you don't have too high of a cost structure, then you can stumble into a business that way, because you've already built up all this leverage already. I think what harms a lot of people is they say, "Hey, I got online yesterday. I need to make a bunch of money tomorrow." It's just like a new person to the stock market that's probably going to get served and lose all their money.

But if you keep a cost structure low and do what you're interested in, it helps. You mentioned how it attracts people that are similar. And you build a community around it. That's how I started. When my blog came out, or when I first made my blog, it was a default web template. Three months in, I could afford to buy a $99 logo.

Jason Jones: Yeah. Well, that's the way to do it. And I'm not saying that. Obviously, some people have success, too. But it's not going to be instant, and it's not going to be easy. It's not going to be any of those things. And then, you don't know exactly why you're successful. Maybe you think you know, but I don't believe that you can know.

Stuff is too complicated to be extrapolating out such specific details, and there's so much chaos that surrounds you and the things that you do. Trying to predict things like that or plan for them, you've got to have some core value that you're about and focus on that. Then you can hope that it gets really successful and big. But maybe you're being useful in the interim period if that doesn't happen, or if that takes a long time.

Interviewer: A friend of mine, a close one,, I always tell him, "Hey, do something you're passionate about." They're like, "I don't want to do something I'm passionate about. I just want to make money." And then it's like, "Hey, have you been working at all on that?" and they're like, "Nah." I'm like, "That's because you're not passionate about it."
Jason Jones: Sure. Although that advice about being passionate about it, I guess that's good when you're trying to pick something to try for yourself, but most people are kind of like slaves and they're forced to, and there's not that much you can do about it. I want to do something I'm passionate about.

Interviewer: Why did you choose a .info extension? Outside of Germany, there's like three legitimate .info sites in the world. Salty Droid being one of them. What made you choose this?
Jason Jones: Yes! Because Perry Belcher and Ryan Deiss registered all our domain names.

Interviewer: They registered all your domain names?
Jason Jones: Yes. There's a post about it, it's called Deiss and Belcher's Big Mistake. I was flashing my teeth at them about it, but I really didn't care. Because it was just stupid, I just got .info. And it's worked out just fine.

Interviewer: Yeah. Have you ever made any posts that you'd later regret? For example, you saw someone doing something that was kind of shitty, crappy? And later you saw like them do something decent. And then like, "Maybe I went overboard on that." Or do you think that by the time you collect enough, that you're pretty certain that someone is what you think they are by the time you write it?

Jason Jones: Exactly. Like at the first, there at the very beginning, I was just winging it a little bit. But then I was just poking small holes at people anyway, so I don't regret any of that. Once I got going, I don't talk about someone until I know their position inside of the system. And I have to have heard someone telling me a story that's just like, "Ugh!"

Otherwise, I would never speak to someone like the robot speaks. I'm doing that on purpose, and I'm careful. Hopefully I never make that mistake.

Like someone will tell me about someone, and I'll watch them for six months before. Because you can't go off, that is a responsibility I have. You cannot do what I'm doing to just an average citizen. I wouldn't accept this kind of behavior in a different situation.

Interviewer: You think in some way ignorance is bliss? Do you think you would feel better if you didn't know all this stuff? Or do you think you feel better knowing that maybe you helped some people?
Jason Jones: Ignorance is bliss if you're ignorant and you remain ignorant of all things, so that you can't tell. If it wasn't this, it would be something else. There's bull crap going on all around, so there's plenty of reason to be depressed. I don't find this particularly depressing, because everything's depressing. Tsunamis are depressing, too. If you're reading about the world, there's lots of stuff that needs to be better. That's just the perpetual state of everything.

Interviewer: You're not going to know how you're going to become successful? When were the points when you thought that what you were doing was some little side thing, to where you really believed in it? Were there steps where you said, "OK, this was?" Do you look back where you say, "These are the five things that really made 80 percent of the difference?" Or do you think it's just going to bat every day? Or how would you describe it?

Jason Jones: I could tell from the very beginning, which I didn't know before I started doing this, but was was really clear right from the start. That what I was doing, was agitating the bad guys. It doesn't matter if I'm popular or not, this is agitating the bad guys, and that's helping. Like the way I'm doing it, I'm doing it in a particular way. And it's helping.

I also could tell right from early on that this is helping victims. This gives victims a zone to think about what's happened and like gives a chance for reasonableness to leak back in.

The people who are involved, the parties to the thing that you're talking about on the Internet, this is true. From my experience so far, no matter how small you are compared to the thing that you're talking about. If you're going to complain about YouTube's policies, then the people at YouTube are probably not going to hear that. But there's not very many settings where if you say something about someone on the Internet, they're not going to hear it.

That's powerful by itself. It doesn't matter if no one else reads it, you can talk. If you see something you don't like, some problems, you can talk about that thing. People who are involved in that thing, you can talk to them, so that isn't dependent on getting a big audience.

SEO in a Teeny Tiny Nutshell

Search Engine Optimization (SEO) is not a big secret anymore and lots and lots of businesses, big and small, are utilizing SEO and SEM strategies to increase their visibility on the internet. However, the part of the world  I still get blank looks when I tell people what I do for a living. So this is just a very, very basic write-up for those who can’t tell a meta tag from a burrito.

Very briefly, SEO is a bunch of activities that you can perform with the goal of increasing the overall visibility of your website, and in turn increasing the amount of targeted traffic that comes to your website. You might be thinking: “Hey, I have a killer website, with a killer design and a killer product, why do I need to SEO it?”

Well, granted you have the next best thing since sliced bread, but when you think about the world wide web and the number of sites that are becoming a part of it every single day, you might just get too overwhelmed to even think for a day or two. When you consider the stats, your website (and your product) is just ONE of the MILLIONS out there. So, the odds of your potential customers finding your website without any efforts on your part are pretty abysmal. But of course, there is hope.

People look for whatever they need to find on the internet through search engines (duh!) and if you are not in the top 10 results for your primary keywords, chances are you won’t get any visitors to your website (apart from your friends and acquaintances whom you might have “pushed,” subtly or otherwise, into visiting your site.)

So how do you get your site in the top 10 results? The answer is relentless SEO. There are several techniques that can be applied but they are mostly divided into two categories: on-page and off-page.

Like the names suggest, on-page optimization involves the things you do to your website in order to not only make it easier for search engine bots to crawl your site pages but also to make it appealing for your site visitors; and off-page optimization is basically “spreading the word” about your website and/or product all over the internet and getting other websites to link back to your site.

Choosing the Right Keywords for Your Website

Effective keyword selection is one of the most critical activities of successful search engine optimization. Unlike humans, all keywords are not equal; some keywords are easier to rank for than others, and some keywords tend to be almost impossible to rank for. Broad or general keywords tend to be highly competitive and therefore should represent only a small portion of your overall SEO efforts. Specific keywords, which include those keywords that describe your specific product or service and are more than three keywords in length, are less competitive and therefore should make up the bulk of your keyword generation efforts.

For example, if you own an e-commerce Web site you stand a better chance of ranking within the top search results for product-level keywords than you do for broad keywords that generally describe your business. Although broad keywords tend to generate higher levels of search volume, product level terms can generate significant search volume and tend to convert at higher rates than broad terms.

There are numerous useful keyword generation tools that can help you discover effective keywords for your Web site. Keyword generation tools such as Keyword Discovery, Wordtracker, and Google’s Keyword Suggestion Tool allow you to carefully research, analyze, and filter potential keywords. It is not enough to just generate massive lists of keywords. Instead, your keyword lists should represent a cross section of broad and specific terms that your Web site stands a legitimate chance of ranking for. Because keyword generation is so critical to your SEO success, consider making a modest financial investment by purchasing subscriptions to tools such as Keyword Discovery, Rapid Keyword, and Wordtracker even though Google, Yahoo, and other companies offer tools for free.

Keep in mind that despite the fact that keyword generation tools can quickly generate thousands upon thousands of keywords, each page of your Web site should be search-engine optimized for only one or two keywords. Therefore, focus less on generating massive lists of keywords and more on generating keyword lists that directly relate to your Web site and give you the best shot at ranking well on search engines. One of the most effective ways to generate target keywords beyond basic keyword generation is through competitive research. Readily available competitive research tools such as Seodigger, GoogSpy, Compete.com, and KeyCompete provide various data about your competition, including what keywords your competitors rank for in the organic search results, as well as what keywords your competitors are using on pay-per-click search engines such as Google. Armed with competitive research information, you can compare your success to your competition and use the information to devise a plan of attack to improve your own ranking within the organic and paid search results.

Validate HTML

Writing valid HTML is just as important as speaking proper English. Improper HTML can cause Web browser incompatibilities that result in your Web site appearing differently across different browsers. The World Wide Web Consortium has gone to great lengths to develop standards for Web development to ensure compliance across all browsers.

Create Links

Your internal linking structure leads the search-engine spiders and human visitors from one Web page to another on your Web site. The structure of your links tells the search engines what the linked Web page is about. Your internal linking structure is taken into consideration by the ranking algorithms that determine where your Web pages rank for target keywords and phrases. Making it simple for both the search engines and human visitors to find every page on your Web site is critical for optimal Web site structure.