New Google Robots.txt File Disallows Eric Schmidt from CEO Directory

My buddy Mike Smith made a funny joke about a new robots.txt file for Google in the post-Eric Schmidt CEO era. I went ahead and made it. Funny?

My buddy Mike Smith made a funny joke about a new robots.txt file or robots meta tag for Google in the post-Eric Schmidt CEO era. I went ahead and made it. Funny?

New Google Robots.txt file disallows EricSchmidtBot from CEO directory
New Google Robots.txt file disallows EricSchmidtBot from CEO directory

Now, anyone wanna give me the username and password, so I can upload this file via ftp?

PS. I was going to make a title joke, too. But I’m too busy to do all that. Maybe someone should write a post about this with the headline “Page & Brin Update Eric Schmidt’s Title Tag” 🙂

Update: Anonymous source claims Brin & Page caught Eric Schmidt buying links. (j/k)

Monitoring Historical PageRank Trends & Changes

With today’s Google PageRank update, I had to dust off a few old Excel docs. Actually, I pretty much check the PageRank for all of my sites every couple of months, so there was no ‘dusting off’ involved.

With today’s Google PageRank update, I had to dust off a few old Excel docs. Actually, I pretty much check the PageRank for all of my sites every couple of months, so there was no ‘dusting off’ involved. I know, I know. PageRank is dying and/or dead. And it’s Toolbar PageRank, so it’s not even current data. In fact, looking at Toolbar PageRank is a lot like looking into a telescope that is pointed at the center of our universe. Essentially, you are looking back in time. Don’t be surprised if Emmett Brown jumps into view with a flux capacitor, offering to have Mr. Fusion eat all your garbage.

But seriously, I thought I would share the method that I use to visually monitor historical PageRank changes and trends for a set of URLs:

Monitoring Historical PageRank Changes & Trends
Monitoring Historical PageRank Changes & Trends

Yeah. I know. It looks like a weird game of Minesweeper (SEO version). But it’s easy to set up, using conditional formatting, and it really helps to quickly identify Toolbar PageRank lottery winners and losers.

Now, do I freak out if some of my sites drop in PR? Not at all. What about the URLs that move up in PageRank? Well, I do take a little joy in that. But overall, I really don’t use this data to make any major SEO decisions. And I wouldn’t recommend using PageRank as a KPI for your SEO campaigns. However, this data can be used for diagnostic purposes if something totally random occurs with your PageRank. For instance, if you see a massive PageRank drop (i.e. PR4 –> PR1), then I recommend you find out why that might have happened, as it could be affecting your overall results.

Let me know if you have any other ways of monitoring PageRank trends. I’d love to hear about them.

Cheers!

PS. I know that ‘-1’ is not a valid PR value. I use ‘-1’ instead of ‘PageRank Unavailable’. It makes it easier to sort. Boom. Roasted.

PageRank Update January 2011

Well, it looks like Google updated the toolbar PageRank today. And now everyone is going to be checking their sites for PageRank changes. It seems like the last PageRank update was back in early 2010 (April, right?).

"It's about damn time." - Angry PageRank Bird
"It's about damn time." - Angry PageRank Bird

Well, it looks like Google updated the toolbar PageRank today. And now everyone is going to be checking their sites for PageRank changes. It seems like the last PageRank update was back in early 2010 (April, right?). Ahhh, yes. Those were the days. Pre-caffeine. Pre-MayDay. Pre-Snooki being a New York Times best-selling author (link). Welcome to 2011, Google. Thanks for joining the party. Better late than never. I guess.

You & A Keynote with Matt Cutts (SMX Advanced 2010)

Everything is better with Caffeine, right? Oh good. The metaphor joke series begins.

  • Google Caffeine is officially live: http://googlewebmastercentral.blogspot.com/2010/06/our-new-search-index-caffeine.html
  • Now your URLs are immediately processed by Google’s index
  • Caffeine is roughly the same size as the current index, but it gives Google the ability to go bigger and quickly!
  • Caffeine allows Google to process data on the order of 100 petabytes.
  • Fundamentally: As soon as a document gets published, it can be indexed.
  • Annotation: Attaching meta data to objects. For example, PageRank, FB Likes, and other meta data and just plain data. New signals.
  • Twitter introduced an Annotation API. A tweet has a lot of extra associated information. So does a document on the web. Google Caffeine will use all of that associated data when processing an object.
  • Faster indexing, bionic speed and the ability to attach meta data to objects!
  • Matt: Think about it like a car that running down the road and trying to change the engine while the car is running. It will take some time, and they are just now getting to look at a lot of the related data.
  • Mayday algo update. Ranking change. The big talk of May.
  • Matt says his team had nothing to do with the MayDay update.
  • With MayDay, Google raised the bar again. In order to get over that bar, you have to NOT be webspam.
  • But….in 2010, we look at the web as it is. We have to look at the challenges of the web as it is in 2010.
  • Maybe it’s not spam, but it’s not as good as great content.
  • Less webspam and more about the quality of the content.
  • It sounds like Cutts and his team have gone after content farm.
  • Danny is bating Matt into talking sh*t about Mahalo and/or DemandMedia.
  • What is the level of quality of these sites?
  • Danny just called out eHow on a content-free article. F’n hilarious. (Convert FLV to images)
  • Google will be looking at video sitemaps more. So get those live.
  • Crypto-404 or a soft-404. Where the page returns a 200, but Google thinks nothing is found.
  • Cached page link doesn’t get a lot of clickthrough. How can Google make it a better page and a better experience. Like where a snippet came from.
  • Danny calls him out on the DMOZ descriptions. “Just buy DMOZ and fix it.”
  • How about HTML5 and Caffeine? HTML5 is completely unrelated to Caffeine. You don’t automatically rank higher if you do W3C validating. However, Google is updating it’s parsing.
  • Matt: It doesn’t help your rankings if you buy ads in Google.
  • How to determine if you are getting traffic from real-time results? No plans in the works right now.
  • The guy who manages Google Webmaster Central is in the room. What’s his twitter account?
  • Speaking of paid links: It seems that Google has decreased the PR of some major sites that are selling links. However, the pages keep ranking high in the SERPs.
  • Matt: If we drop the PR of a site we’ve caught selling links, the people who are buying the links are not getting their value because those links don’t work anymore.
  • Matt: We have been taking strong action on paid links. In fact, we have a lot of cool new tools that let us do a lot of cool things when we find paid links. We make sure paid links don’t work.
  • There’s always news. Maybe Danny should start doing the Daily Searchcap again.
  • NoFollow: Is it completely out of our control to tell Google our most valuable pages?
  • Matt: The most important pages are the ones you link to from your homepage. It’s the pages that you submit in your sitemap. It’s the pages that you tell Google about.
  • Put a link to your most important pages at the top of your homepage.
  • Google has improved their ability to discover links in javascript, and that goes for paid links, too.
  • 3-pane thing on Google? It’s working out well for Matt Cutts. Glad to know he likes Google products.
  • Matt Cutts polls the audience on Browsers. 1) Firefox 2) Chrome 3) Explorer
  • Matt’s theory of Buzz: How many people got on Twitter just because and really had no idea what to do or how to use it? Danny created Matt Cutts twitter account as a joke and then gave it to him. With Buzz, Matt is finding out more about how he wants to use it. Much like it was with twitter in the beginning.
  • How does Matt react to accusations that Google is only promoting its own products in the SERPs? His response: Bing is worse. A lot of engines just want to show you the pretty stuff. Who’s that? A lot of engines? You mean the other 2! Brilliant. Comedy at its best.
  • Is Google favoring YouTube? Matt Cutts went to the YouTube guys to find out. The YouTube guys claim no, and they say they bend over backwards to not favor YouTube.
  • Now Matt is plugging the new preview pane in Bing where you can watch videos in the preview.
  • When will rich snippets be available for everyone because right now Yelp has an advantage? Matt says they are re-working how they use rich snippets. Time wise, look for it in the next few months. (I’m happy about that!!!!)
  • Rich snippets are great for users and clickthrough. I’m excited.
  • Bing is buying free drinks tonight, right?
  • Do we need separate flash sitemaps? No.
  • Google can determine sentiment analysis, but they don’t use that as a ranking signal.
  • Bounce rate and Google analytics are not used in general ranking alrorithm.
  • To the best of Matt’s knowledge, the Google guys don’t use bounce rate on their search rankings.
  • Sounds like Matt is wishy-washy on this question about bounce rate.
  • If paid links are bad, why are paid directories helpful? It’s the editorial discretion they provide.
  • Can you take Wikipedia out of the top 10 for 24 hours? Nevermind.
  • Put Wikipedia in the corner, so the rest of us can show up.

SEO for Google vs Bing (SMX Advanced 2010)

Similar SEO Tactics for Google & Bing:

  • Flash: Google and Bing can’t read it. Make sure you have indexable content if your site is Flash-based.
  • XML Sitemaps: Use these! Now already! They are a universal standard. Google and Bing accept sitemaps and sitemap indices. Bing does not currently accept video or news sitemaps. Be sure to register your XML sitemaps in Google and Bing.
  • Google Places & Bing Listing Center: Get your business listed.
  • GeoLocation results: Bing has great local movie results – with show times even! Wow. How can they even do that in 2010. It seems pretty impossible. Like the plot of LOST. And how about those weather results in Google and Bing. Very comprehensive. My dad would enjoy this. If only he used the internet.
  • Sitelinks: In Google, sitelinks show only on the first results. And you can remove sitelinks in Google WMC. In Bing, sitelinks also show on first results, but you cannot remove any of them.

Differences between Google & Bing for SEO:

  • Link Value: Check out Bing’s Webmaster Tools for links data. It’s pretty cool, and Bing indicates the value of backlinks to your site. How about that? Where you at Google? Oh yeah, you’re busy getting tons more traffic than Bing. Keep up the good work!
  • News: ??? I missed this one.
  • Shopping: In Google, the shopping area is completely free. Get your products in Google Base. Bing is paid only. And Bing CashBack is going away soon. 🙁

New Opportunities with Bing:

  • Social Sharing Results: You can share on Twitter and Facebook directly from Bing’s search results page. But the shared link to an image, links to Bing result. Also, to post on Facebook, you must grant access to the Bing application on Facebook. Privacy issues? Where’s Zuckerberg?
  • Swagger Wagon? Anyways, Bing has a module that you can populate with great information (marketing stuff). If you scroll over the video in the preview pane, it will play the video in the module. Another reason to put videos on YouTube.
  • Vampire Squid video is pretty cool. Oh wait. No video. But you get sitelinks and contact info in the preview pane. And you can control that data. Who knew?!?! Bing takes information from your page in this order: H1 tag (if it does not match your title tag). Bing takes the first paragraphs of information from your page. To add contact info, add contact information to the page. Add your address, phone number and email to your pages.
  • BTW, Google’s getting little face time right now.
  • To disable document preview in Bing, use a robots tag called nopreview.

Here comes @RandFish.

Ranking Factor Correlations:

  • Rand is bringing science to this topic. Correlation analysis.
  • Rand claims that H1 tags are not very important. If you believe Rand, you should go to your sites and remove all your H1 tags. Would you do that? I wouldn’t, but I get his point. Don’t freak out about H1 tags not being optimized, especially if you have other more important items that need to be optimized.
  • Correlation does not equal causation. I feel like I’m learning or something. I’d rather be spamming.
  • Shorter URLs are better. Yep. Good stuff. It’s true.
  • .COM domains have a negative correlation?
  • If your URL contains all the terms from the query, very high correlation.
  • Google has a higher correlation for exact match domains. That’s surprising. And for URLs with hyphens. BTW, Bing took a lot of heat for being easy to spam with exact match domain.It’s nice to see data like this for Google, too. To be fair, anchor text could be a factor as well, and that might be skewing the data.
  • Hyphenated matches are less powerful in Google, but more frequent in Bing. When they do occur in Google on page 1, Google has a tendency to rank hyphenated URLs higher. Hmmm… Very interesting. But what about clickthru advantages of hyphens in URL?
  • What about TLD’s such as .gov, .org or .info? Exact match .com is the way to go if you are registering domains. Other exact match TLD’s don’t correlate as well.
  • Keywords in subdomains? Google likes the keywords in the subdomains, if the subdomain appears on Page 1. Bing doesn’t like the keywords in the subdomain very much. Not influenced much at all.
  • KW’s in the subdomain not nearly as powerful as in the root domain name.
  • Bing may be rewarding subdomain keywords less than before.
  • On-page keyword usage: KWs in the body, Alt attribute, URL: Google high correlation. The Alt attribute is interesting. It seems to be influential. Putting KWs in the URL is still a best practice. Simplistic on-page optimization isn’t a huge factor. Maybe you should work on other stuff. Everyone optimizes title tags, so it’s tough to get a true difference between Google and Bing here.
  • How about links? Diversity of linking domains: very important! Diversity of link sources remains more important than raw link quantity. Now that’s important to know.
  • TLD Extensions: .ORG domains are off the charts! Negative correlation on .EDU, meaning that edu sites show up lower on page 1 than the .org and .gov counterparts. More reason to believe Matt Cutts when he says TLDs are not special cased. The .org TLD is interesting. More links? Less spam? More trusted?
  • Length of domain, URL and Content: Longer URLs mean lower performance on page 1. Domain length: not terrible to have long domain names, but is not necessarily ideal. Raw content length seems marginal in correlation.
  • Website homepages? Do homepages do better? Bing really like homepages. Homepages outperform internal pages in the SERPs. Google also has a slight preference, but for obvious reasons.
  • Anchor text link matches. Raw quantity of links with exact match anchor text: not very correlated. # of linking root domains with exact match anchor text: VERY IMPORTANT. One of the highest correlation factors we have seen. Anchor text from diverse domains appears highly correlated.
  • Features with highest correlation: exact match.com domains, domains linking with exact match, # of links
  • Google and Bing are very similar. Optimizing for Google and Bing are quite close, maybe one in the same.

Next up:

  • Sasi Parthasarathy, Program Manager, Bing, Microsoft
  • Matt Cutts, Software Engineer, Google Inc.
    First, Matt says: “This is interesting. Rather than chase after search engines, chase the user experience because that’s what search engines are chasing.” Matt also says wikipedia entries show up more in Bing. Those are fighting words. Matt points out the limited data set that Rand used, pointing out that he might have missed longtail spaces. And then some dude in the audience yells “YES!” What the hell was that? Are longtail searches that exciting? I know, I know. They convert better. Whatever. Have fun converting one out of 10 visits, while I’m converting 100 out of 10,000 visits. Sheesh.

Questions from the Audience:

  • Did Rand exclude wikipedia from his test? Nope. That probably skewed the data.
  • Reminder from Danny Sullivan: Yahoo made it clear that the algorithm at Bing and Yahoo will be identical. So the same 10 organic web results will be seen on Yahoo and Bing. Thanks for reminding me. (Rubbing my hands together. Exxxxxcccccellent. Release the hounds.)
  • If I start my domain with a number, will that help? What are we optimizing for phonebooks? Matt gets 37signals and 37folders mixed up. Matt recommends not using numbers in domains due to branding purposes.
  • What can be done to help Bing index more URLs? Yeah. Great question. Bing sucks at indexing.
  • Matt Cutts just said that Google will be looking more and more at video sitemaps. They will be more and more important, especially with the upcoming GoogleTV this fall.
  • Is Bing going to support the AJAX specs that Google uses? Sasi will look into it. No real answer. Damn.

Google AutoComplete Says SEO Is…

Could this AutoComplete data just be another way for Google to hate on SEO and SEOs? I dunno. Probably not. But after Google removed SEO companies from the local results, we know how they really feel about us. And that’s what makes it fun. Godspeed, SEOs!

Misreading Data: 1 out of 10 people think SEO is important
Misreading Data: 1 out of 10 people think SEO is important

Hey, I love the Google AutoComplete feature as much as the next guy, but this one caught me off guard. But only for a second. While the AutoComplete data is probably not the most accurate data, in this case it does give us a glimpse into the perception of SEO. Honestly, I can believe the results. This is probably what most people think about SEO. Why? I’m glad you asked.

There are a lot of bad SEOs out there. A lot of people have been burned by SEO companies and sales people who promised the world. SEO takes time. It takes work. SEO requires innovative thoughts and ideas. And research. And IT/Dev resources. In many cases, your SEO is limited by your site, product(s), business culture, goodwill, etc… Seriously. Let that sink in.

Furthermore, SEO is becoming more and more competitive, and basic SEO principles can only get you so far. In fact, many people think SEO is dead, dying and/or bullshit. Dave Snyder wrote a great post about that recently.

Could this AutoComplete data just be another way for Google to hate on SEO and SEOs? I dunno. Probably not. But after Google removed SEO companies from the local results, we know how they really feel about us. And that’s what makes it fun. Godspeed, SEOs!

Link Building Idea: Buy Some Links on Google Profiles

Hey check this out. It’s Mashable’s Pete Cashmore and his Google profile (now with more Google Buzz):

Google Profile: Mashable's Pete Cashmore on Google Buzz
Google Profile: Mashable's Pete Cashmore on Google Buzz

It appears that Google has placed a nofollow on the links within Pete’s Google Buzz feed. However, the links in the right-side column are dofollow. How about that?! Holy crap! These are dofollow links from the domain google.com!

Because Pete is obviously a smart guy, he realizes the value of linking to his own properties. Hence, he has linked to Mashable.com and various Mashable properties on Twitter, Facebook and YouTube.

At the time of this screengrab, Pete had 9,647 followers, so it’s only a matter of time before this page has a high PageRank. This Google Profile will ultimately have a lot of authority, and those dofollow links in the right-side column will drive some serious link juice. And remember: these are links from google.com!

Now, let’s be realistic. Despite the current dofollow status, I’m sure Google has discounted these links in some way or another. But the fact remains that these are dofollow links coming from google.com. Furthermore, check out the bottom of Google.com’s robots.txt file:

  • Allow: /profiles
  • Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml

When you read the XML sitemap file, you see several hundred txt files of Google profiles, such as this one. Inside each one of these txt files, there are 5,000 user profile URLs. The take home message: Google wants these profiles crawled, and they are actively making it easier for all spiders to find them. Google Profiles are obviously important in Google’s venture into social media and social search.

What about the buying links idea?

Well this should be fairly obvious. Google is allowing dofollow links on Google Profile pages. While it may be tough to get links from reputable profiles, there are many users who may be open to the idea of selling links. And some of these profile pages will end up with high PageRank, especially when people begin to link to their Google profiles. These are links from the google.com domain. Even if Google moves to disallow these links, there is still value from these links. Why not offer the profile owner(s) some money for a link in the right-side column?

Here’s a simple plan of action to acquire links from Google Profile users:

  • Google profiles have custom URLs
  • These URLs take one of two possible formats:
    1) google.com/profiles/101849747879612982297
    2) google.com/profiles/mashable
  • If you can find the username at the end of the URL, then you will know the gmail address of the profile user.
  • And that is how you can contact them.
  • Then it all comes down to your ability to not creep people out by offering them money for a link to your network of phentermine and poker websites [cuz those 2 things always go together, right?]

There it is. I hope you enjoyed that. Stay tuned for more. And stay tuned for a nofollow attribute from Google. I’m sure it’s right around the corner.

Happy Link Building!

[Full Disclosure: Neither Mashable or Pete Cashmore compensated me for the free brand exposure to my 10’s of readers. I’m just a fan, and I think everyone should be a fan of Mashable. #teammashable]

Update: Feb. 17, 2010: Nofollow attributes added to Google Profiles

It seems Google is onto us. Google has added the nofollow attribute to all links in the rightside column of Google Profiles. I wanted to see if they nofollow’d links to Google Profiles, so on my Google Profile I added a link to my Google Profile. No luck. It’s nofollow’d, too. Damn. Nice work, Google. Until we meet again…

Matt Cutts: Google Using DMOZ Info to Create ‘Better’ Title Tags in SERPs

Wow. Around 2:14 of this video, Matt Cutts says Google “can sometimes use the Open Directory Project snippets” when populating snippets for URLs displayed in the SERPs. He then goes on to talk about how Google can also use that information to create a “better” title for URLs in the SERPs:

…Webmasters are probably not as used to the idea that we’re willing to find a better title as well. So if you have a bad title or a title that we don’t think helps users as much, we can try to find a better title – and one that we think will be an informative result, so that users will know whether that’s a good result for them to click on.

So I just wanted to give people a heads-up about that because they’re used to the things below the title changing, but they’re maybe not as used to the idea that the title itself can change in our search results as well.

Thanks for this video, Matt. The last part is fascinating. We saw that Bing was doing something like this back in early June.

One question: Is Google going to do my title tag SEO for me now? That would sure save me some time. 😉 I’m sure if I was Boser or Graywolf, I would have a big problem with this. I kinda do, but I’m too busy to write much more. I guess I’d really care if a page with no title tag could outrank my page after Google created a “better” title tag for that page. That would be upsetting. But I am confident in my SEO abilities. Now where was I?

BTW, Yahoo is doing the same thing.

Fail: Google Indexes Google Voice Messages

Today in Fail News: Google has a service called Google Voice. You can get a phone number and everything! It’s like having another phone. And people can leave messages. And today I see that Google is indexing the voicemail of some Google Voice users. You can listen to some of the messages and even see the transcript of the message, who it was from, what time it was left, etc… Oops. Click here for more!

Fail: Google Indexes Google Voice Messages
Fail: Google Indexes Google Voice Messages

Today in Fail News: Google has a service called Google Voice. You can get a phone number and everything! It’s like having another phone. And people can leave messages. And today, Engadget reports that Google is indexing the voicemail of some Google Voice users. You can listen to some of the messages and even see the transcript of the message, who it was from, what time it was left, etc… Oops. Click here to see Google Voice messages indexed on Google.com!

Here are two of my favorites: Message #1, Message #2

Google PageRank Update: June 23, 2009

The most recent Google toolbar PR update occurred around May 27th. Last night as I was up to white hat type of stuff, I noticed that many June blogposts already had a toolbar PR > 1. WTF? Had Google already made a PR update again, just one month after the last one? The answer:  Yes. My buddy Frank over at Tech Jaws (a badass SEO and internet security blog) sent me a link to the SEO Round Table PR update post that confirms the update.

After checking it out for a few minutes, it looks like the PR update is affecting internal pages moreso than homepages. There’s no way I can check the entire internet to support this claim, but from just looking at a few sites, that’s what I am seeing. However, I have also seen a several homepage PR changes, and the results look to be split. Some of them are up 1 level, some of them are down 1 level, and most of them just stayed the same.

We all know that toolbar PageRank is nothing to go apeshit over, and we also know that Google updates the real PR quite often. Could it be that Google has decided to frequently publish more of their internal PageRank updates? I guess only time will tell.