Oh My God. No more keyword data.
Google finally did us all the service of stopping this charade of keyword transparency this week and finally removed them entirely from their search referrers this week.
Rand wrote a piece on his personal blog, calling it the first existential threat to the industry
That came as a bit of a surprise to be honest, as his WBF three days earlier had some great workarounds (many of which I was going to discuss in this post, but frankly you’d be as well just watching the video).
Why have they done this to us?
To trust in Google’s own reasoning – user privacy – would be highly naive.
They can’t declare privacy issues on one hand, while selling the same data through adwords, scraping personal email contents for advertising purposes, mapping the whole world’s SSID data & wifi passwords, so lets just drop that pretense.
The reason is blatantly financial, and I welcome it.
Sure, it would be great if they could just do it without sugarcoating the truth, but they need to deal with their (already increasingly flaky) reputation, so I get it.
The Cost of Google Analytics
Google Analytics (excluding Premium GA) has always been a free product.
Its main competitors in the commercial arena are the likes of Coremetrics, Webtrends, Omniture etc. All of which cost a significant amount of money. If you’re running sites big enough to warrant the investment, you’re probably paying a seven figure sum for access annually to them.
Google have provided us a robust, market leading platform in many respects, absolutely free.
I don’t know how many sites use GA at the moment, lets say its between 10 and 100 million, and closer to the second of those two figures.
That requires significant infrastructure. It requires significant data storage. It also requires significant maintenance.
None of those costs would have been covered by the premium product, while the aforementioned “enterprise analytics platforms” all made a tidy sum from their products.
The Value of Google Analytics
GA has for a long time been the best value product on earth. Bar none. That’s why we all use it and have loved it so over the years. But its time to get real folks.
Products like this need supporting with real revenue.
Why the heck should a public company continue to sink millions and millions of dollars into a product that is both market leading, and free, and not get anything out of it?
We’re going to Learn the Value of Keywords
This is truly valuable data, and Google are not deleting it. They aren’t throwing it away, that I can guarantee absolutely. They are just obfuscating it.
I don’t know how long it will take, whether its 6 months, or two years, but that is a highly monetizable resource, and google have a fiscal & legal responsibility to their shareholders to maximise profits.
We will see it coming back, but that will be in a different format and through a paid model.
In the meantime however, they are simply educating us. We have had this amazing data for ever, its always been free. That was a situation that was irrational and simply could not prevail.
It was time we learned to value SEO properly*
*So Why is this Good for Me?
Since the mid nineties when search engines first came into existence, all this stuff has been free to the webmaster.
Keywords, analytics, link data (YSE), rankings data etc. All of them were at one point, free.
That has driven our industry into the thinking that you can perform SEO for virtually nothing, and that it could be resold as a commodity, becoming driven by price alone.
This move will kill commoditization of SEO.
A different format?
Of course, just masking the referrer strings, and then parsing them in another fashion into GA wouldn’t be possible – the antitrust actions from the other analytics platform providers would guarantee that.
That’s why we needed the whole privacy charade in the first place.
My money right now is on increased integration between GA and Adwords, and when those two products are essentially melded into one, re-introducing it through “advanced organic keyword reports” or similar.
There should also be a minimum spend threshold to see this kind of data.
The (SEO) Theory of Evolution
So we’ve stripped the “little guys” ability to report on SEO.
We are still left with a whole load of other ways of getting this data, but they require a bit more effort, and a whole lot more rationale.
Crucially though, we should see less of the lowest common denominator “crap-SEO” that has dogged this industry for the past decade – simply put its going to be much tougher to scale effectiveness reporting on minimum viable product working practices.
I’ve no doubt that we will get our keyword data back eventually, and at a price, but in the meantime its about to get a whole lot harder for some of us – and if you can’t adapt, you’re not an SEO.
Some Practical Workarounds for Not Provided
In the meantime until we get our keyword data back, there are quite a few practical workarounds we can implement. I’m going to cover a few of them this coming week over in the tutorials section, so if you haven’t checked it out, now would be a good time to do so.
Top opinion and summed up perfectly at the end:
” …if you can’t adapt, you’re not an SEO”
GA has been with us for so long it is easy to forget that this is a free tool and I think you have it spot on. There is no way a company can continue providing this level of information at no cost.
It will be interesting to see what evolves from all of this and if it does eradicate the dodgy SEO companies then this is a seriously good move.
Absolutely – Aaron Wall (I think it was him) used to describe this thinking as being a “Freetard”. While its perhaps not a great word, it nicely encapsulates whats been wrong with the industry for a long time.
People get used to free. People assume “free” is low value. Its time to make SEO corporate.
I for one welcome our new (paid) Google overlords. 😀
Interesting post.
I don’t doubt that at some point, we will see a paid service GA with the keyword data plugged back in, but I think Google have made a rod for their own back by saying they were doing it for “privacy” reasons as that’s sure to come back and bite them if they then try to charge for this privacy.
I think we all would have had a lot more respect if they’d just said “You know what, we’ve been giving you this valuable information for free for years now, it’s time we monetized it”
Sure, we as an industry would bitch about it, but most of us would probably end up saying “fair enough” and paying for it. When they finally turn round and say You know the thing we stopped giving you because we thought it violated the privacy of our searchers…well we’ve decided we’re kind of OK with that, but we’re going to charge you for it.
But at the same time, it’s a master stroke, because it’s not Google Analytics that is no longer giving that information, it’s Google no longer providing it. That means that data isn’t available in any analytics product, whether it’s a free system or a 7 figure system, if they can then plug the data back into GA and not supply it to any other system, suddenly GA has a USP that the other systems which cost so much just don’t and can not ever have.
Not only can they sell this, they’ve also locked up the PPC market. Not only is Adwords the only way Google will give you any data now, it’s also stopped that data being available to other ad networks, which screws a lot of display and remarketing solutions, essentially stopping them working leaving Adwords the only viable solution.
As a search marketer, I don’t like it and think it will lead to a poorer web.
As someone who can see beyond the privacy bullshit, I think it’s a master stroke from Google.
Agree on all points Jim – regarding the privacy thing though, my thinking right now is that the master-plan has always been to provide it in a paid variant, BUT, in that paid variant you take ownership of the keyword data (which you ALREADY did in premium GA).
That therefore removes Google from legal shenanigans around ownership, as they fundamentally just become the cloud storage for other peoples data.
Yes: I would have preferred going through it without the pretence, but lets face it. Google have some reputation management issues at the moment, and this is a nice easy opt out.
This post is bang on.
Although I agree with much of your article, if your predictions are true, it makes me a bit sad.
One of the reasons I love the SEO industry is that there a fewer barriers to entry compared to other professions and I feel for people who are in the same position I was a few years ago and looking to get their foot in the door.
It’s also going to be harder to learn, when looking at GA data you get a feel for what types of keyword leads to a conversion, and sometimes it throws up some surprises, this is going to be much harder now for SEO guys and will force them to use Adwords.
Also, have Google actually came out and made an official statement about their reasoning behind this?
Great post Martin,
I will have to concour on this statement
“Why the heck should a public company continue to sink millions and millions of dollars into a product that is both market leading, and free, and not get anything out of it?”
Google are by far one of the smartest companies on the planet and they don’t sink millions of dollars into a free product. Infact by offering a free product such as GA which is at industry standard they are able to data mine all of our visitors, all of the revenue information and various other parameters. They also use Adsense as a datamining initiative to track users. This keeps Googles insight and knowledge of its users at the forefront of their industry which is why Advertisers spend Billions on investments in advertisments through Google subsideries.
The same method of thinking can be applied on Youtube Embedded videos, Gmail, Google Plus, Chrome Book, which are highly subsidised products which are there to build on data equity which is a highly profitable model for them.
“We will see it coming back, but that will be in a different format and through a paid model.”
I hope you’re right, and I also hope it’s not too expensive. I can’t see them escaping the privacy argument if this happens, but then again, I’m surprised they escaped it at all with PPC data.
Google did it because it gave them one heck of a huge amount of data
I am still trying to wrap my head around why the industry is pretending like their is no longer any way to access keywords driving traffic to their sites. Either there are allot of people in this industry who do not pay attention to Google Webmaster Tools (shame shame) or the industry is working overly hard at driving this doomsday message to weed out the newbies.
I have access to most of my clients websites keywords and I not only see how many drove traffic but I can see which page traffic was driven too, the impressions my site received from each keyword and the number of clicks directly from Google.
You can either download this information every month and chart it yourself – or you can invest $99 per month in Raven Tools which takes it a step further and charts it out offering average rank trending, impression trending and click trending.
WMT data isn’t exactly accurate though, that’s the issue…
Hey Jessica,
the problem that a lot of people in the industry have got is twofold:
1) the data in GWMT is poor quality. Right now its the best we’ve got so if we want raw keyword data, its the best we’ve got though. Not to mention frequent delays in processing by up to a week. Just ask @rjonesx on twitter 😉
2) the GWMT data is sparse. There are a couple of thousand keywords. What if you have hundreds of thousands of keywords driving traffic daily? Thats not great.
But – all this doomsday messaging ISNT the point. As you rightly say there are ways of getting this data (although GWMT isnt it), it just requires a lot more thought and planning.
While GWT offers useful data, it does not contain conversion data.
Yep. Perfectly valid point. Conversion data for SEO right now is a bit of a black box. Sure, you can fudge things, but we are effectively working with much less data than ever before. You can draw assumptions, but they are just that.
At the moment I am struggling to get upset about this and I figure there are many other ways to skin the rabbit. I imagine it’s going to be one of those frustrating things as you go to do something that was relatively simple before and realise there are new barriers in the way but over time, we will adapt and find new ways to measure the same outcomes (mostly).
We still have landing pages, funnels, multi touch, engagement metrics, conversions and we have some data from adwords and webmaster tools so I think we (collectively) will be okay once we get over the hump and find new ways to report. Heck, we may even get better at our jobs on the back of this and get more creative.
People generally attracted to SEO and Internet Marketing tend to be a fairly dynamic and resolute bunch and hey, it’s about time we all had something new and interesting to blog about.
The privacy thing is a crock though and I agree that we will likely see keyword data making some kind of return in one form or another (likely a paid version but possibly even through further improvements in webmaster tools that have anonymised the data).
Marcus
it’s about time we all had something new and interesting to blog about.
Love the spirit!
re. Privacy, yep agreed.
re. better at our job,
Yessss and as Martin suggested, this move will make it a hell lot harder for flaky SEOs to justify their work. Making it easier for us with potentially less competition while increasing the pressure on our PPC colleagues to perform ROI positive campaigns.
After the scramble we will have a gained a new level of black magic and data wizardry. On that note, must leave, I have my Voodoo scrum of the day, need to sacrifice 2 chickens to appease the Google Gods (think universal search) 😉
I think you are bang on but I think it will either weed people out or give them something to hide behind.
I think this is a great article, but of course Google gets something out of providing Google Analytics.
Bounce rate, time on site, time on page, how many pages viewed, even how many downloads and products sold, if, when and where your users click out, etc.
That information tells Google just how useful and valuable our sites are to our actual visitors. This data gave (and gives) Google a huge competitive advantage over other search engines. Google did not become the most popular search engine simply because its home page loaded fast. Google became the best at delivering what searchers wanted to find.
But how was Google going to convince millions of site owners to give them access to that valuable data? I remember when brands started implementing Google Analytics and how nervous they were about it. How many of us would have given up that data to a search engine without that compelling incentive?
They have Chrome for that now though
So basically you’re saying that if you run a small business, you’re fucked.
Not necessarily if you have a small business you’re ****ed. Although thats certainly on the cards more now than it ever has been.
I think the rational thinking is: “If you run a small business, and are not willing to think of the investment required in SEO to be a thousand times less than PPC, you need to rethink things”.
But, do you really think that small business owners are at all interested in these developments? If this reduces their visibility in the short-term and increases their costs in the long-term, they still aren’t likely to take more interest in what’s going on.
There will more than likely still be 100s of companies selling dodgy SEO and social media services.
If this kills off Omniture then I’ll be happy though 🙂
Why are SEO’s pinning their hopes that Google brings back a paid model for organic keyword data?
We seem to be talking like they will but it might be best to bet on a future where they don’t…
Am I missing something?
Google are killing keywords in search period.
They want the context of content on the page to return the best results.
This means no more keywords as we know it.
Killing the keywords is a way to force the SEO to write better content. This way, when voice search becomes the norm, google can serve results from the best content, not the best individual optimised keyword.
How this evolves Adwords remains to be seen, but ads really look like ads now on a google search, not a very subtle colour change.
Nice summation. This is how I think people should be reacting. There are things one can do, although they are all far more complex than extracting keywords from referring strings in logs. My personal opinion is that Google ought to provide them free via Webmaster Tools. I’m sure the data is way too valuable and they’ll probably end up selling it as suggested here.
This is great and right on par with something I’ve been thinking for a while. I do not condone the reasoning and rationale of Google in terms of why they claim to hide the information, but that being said, I don’t see it as a bad thing at all. For one there are work arounds as you stated in your article, but for two, this will hopefully lead to a new evolution of SEO that focuses on overall organic presence rather than keyword data. Keyword rankings and traffic are TOOLS and not GOALS. I wrote a blog post about the new keyword landscape, if it’s ok to plug it here.
Nice post, Martin. I agree with nearly everything you say, except for their being no validity in Google’s claim that security is a major factor. There are certainly other factors which (purely coincidentally, of course) work to Google’s advantage. But the fact of the matter is, by killing KWs with Not Provided, they’ve eliminated a major security risk to users AND removed a major potential liability from themselves.
Yes, you can get keywords from GA. But what you CAN’T get there is the personally identifiable information (IP). For that reason alone, I believe it was a major motivator.
That it happened to take shape in a way that may drive more people to AdWords, of course, probably didn’t cause any objections in Mountain View. 😉
uh…. “there”, not “their” (yeah, I used to be a copywriter. I’m old… gimme a break!)
Privacy might be the excuse, but the reality is that Google is still logging the keyword information, storing it together with other information about the user (like what pages you visited and what ads you saw), and selling it via its ad marketplaces.
The decision is purely business/financial – and on multiple fronts:
– Google Analytics will have monopoly on this data. Webtrends, Adobe, and the rest got locked out.
– Adwords & Ad Exchange will be the only ad marketplaces to have access to use profiles built on user search behavior. This data is highly valuable as it clearly shows intent of buying.
Obviously the biggest question here, and everyone is afraid of asking, is “Is Google using its search dominance to create a monopoly for keyword analytics and user profiles?” – and I firmly believe it is. It could be intentional, or unintentional – but the end result is the same. Marketers and Advertisers are locked to Google products and marketplaces.
If I was to make one prediction is that in the next 2 years, 99.9 percent of the sites that have implemented some web analytics tag on their site, will have Google Analytics implemented on the site.
Good post. Thank you. The only thing I’m questioning is when you say…
“…we should see less of the lowest common denominator “crap-SEO”…”
We have helped more than a few clients out of bad positions created by these companies, and from what I’ve seen, most were not providing any reporting data anyway.
They are a churn-n-burn model and one that would not usually make the time or investment in actions geared to keeping clients long term (like quality analysis and reporting). I suspect they will still find a way to stay involved.
This is all somehow leading to social search and Google+ just wait and see. #DontBeEvil
Haven’t fully thought through the technical details, but within site search (or internal search) coupled with the right setup in Google Analytics or some other analytics software should preserve keywords to target pages. This isn’t a great or perfect solution, but it’s something to throw into the mix of alternate ways of try to recover keyword referral data . . . Hat Tip catalystsearchmarketing DOT com
Agree with Alan – keyword data is a tool, not a goal in itself.
Webmaster Tools gives a very good flavour for those analysing data for SEO purposes, and any keyword data still available to AdWords customers will only relate to their campaigns so could still be a small part of the whole picture.
Google is pushing Semantic Search more and more so I’m predicting there will be some new reports available soon to tell us how our content is ranking.
Do you really think that hiding keyword data will get rid of ‘dodgy’ SEO companies? Nah…. don’t be daft! Nothing gets rid of them – they just keep on morphing. It’s up to us to make sure we communicate SEO best practice and white hat principles properly so that our customers know what’s good and what’s bad.
Interesting discussion – thanks!
If the updates and the whole slew of animals that Googleplex has unleashed on the interwebs taught me something, take their advice with a grain of salt and yes, their tools only tell half the story. My client’s and ex-clients metric of choice is how many clients of his clients has entered their stores or signed up for their deals (‘conversions’) have I made.
But then again, I have other income streams, PLR anyone? hahaha!
The more Google acts like Microsoft did when they were “on top”, the sooner Google will become passé. The current direction they are going with GA will serve as huge opportunities for new ideas… and new alternatives. Size kills.
I think Google is on a slippery slope – they are a monster totally out of control and only have one thing on their minds these days ‘shareholders’. There are plenty of good alternatives out there at the moment and some which are rising fast and I am sure over the coming 12-18 months the landscape could potentially change as so many people have just had enough of Google.
What do y’all think this means for analytics services like Omniture and Webtrends?
Honestly? Dead in the water, within 18 months.
Martin, you don’t get it. Google gains a significant advantage from the data collected (stolen) by google analytics. You also don’t realize that google doesn’t need the click data because they are already tracking that from the SERPs page.
The endgame here is thus: give up your valuable data and die slowly (reduced margins and decrease customer loyalty + engagement), or pay a little bit for private data and go on to be competitive.
As Gareth said, people are losing their trust in google. It’s only a matter of time before there is widespread understanding on this topic. There will be a shocking reversal of businesses using google “steal your data” analytics.
Superb post. spot on in my opinion.