Update – Since this was posted, Max Allan and Damion Yates have investigated further and found that BT are intercepting DNS queries and replying with doctored DNS results, which I didn’t notice when originally researching this article. The title is therefore not entirely accurate, and BT are most probably doing this without any special treatment from Google, but the point that Google is facilitating this (by performing http redirects) stands. Please keep this in mind as you read :)
–Alex 25/11/2014
As I’m currently in temporary accommodation I have found myself without a permanent internet connection. 3G service in the area is pretty spotty, so I bit the bullet and ended up purchasing a single month BT Wifi pass, effectively piggy-backing a neighbours connection. I’m guessing they see very little of the £39 I paid.
It is well-known that BT has filtering in place, supposedly for the protection of children, as required by the UK government. I don’t agree with this policy, but accept that many do.
However when it starts to affect privacy, I feel that BT’s meddling of my internet connection has gone too far.
Case in point, when using Google on BT Wifi I happened to notice a new message on the side:
SSL search is off
This network has turned off SSL search, so you cannot see personalised results.
The security features of SSL search are not available. Content filtering may be in place.
Learn More | Dismiss
After digging into it, I’ve found that statement to be demonstrably false. In actual fact what it should say is; “We have disabled SSL search on behalf of your network provider.”
To which I say, thank you for giving me another reason to use duckduckgo.
Why
I can think of a couple of reasons to block SSL search. One is child protection and filtering. Disabling SSL search allows BT to filter searches for undesirable terms, and theoretically allows them to append “?safe=active” to the search URLs. They don’t do this though, in fact doing so would require the use of a proxy server, which would be a whole new level of intrusion.
The other reason, and the one I feel is more likely to be responsible for this policy, is data mining. It’s reasonable to expect that BT knows the location of every BT Wifi router within 10-15m, because it has a home address for every one of them. Whether or not it knows who is signed in to Google (it’s reasonable to assume it doesn’t unless it’s actually inspecting the contents of the message body, and that would be *WAY* overstepping the mark), knowing what is searched by location is a marketing gold mine.
So, I’m being data-mined. Great.
Now how do I get around this!
The “learn more” page includes a link to “SSL Search for Schools“, which describes how network administrators can disable SSL search, effectively by redirecting users to specific virtual IP addresses via DNS.
Great I thought, I can just use public DNS and avoid BT’s.
Not quite.
BT doesn’t seem to meddle with Google’s DNS entries at all. I even did a query to www.google.com from my own server, put that IP in my hosts file, opened up a fresh browser profile and SSL search was still disabled!
There’s something more to it then.
Running a request with curl shows that the redirection is happening completely within Google’s network (I’ve removed some inconsequential lines to shorten it a bit):
$ curl -vL 'https://www.google.com'
* About to connect() to www.google.com port 443 (#0)
* Trying 216.239.32.20...
* Connected to www.google.com (216.239.32.20) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: www.google.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
* Server certificate: Equifax Secure Certificate Authority
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.google.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=UTF-8
< Location: https://www.google.co.uk/?gfe_rd=cr&ei=qPYmVL28J4vDcJbFgLAI
< Content-Length: 260
< Date: Sat, 27 Sep 2014 17:40:56 GMT
* Server GFE/2.0 is not blacklisted
< Server: GFE/2.0
< * Ignoring the response-body * Connection #0 to host www.google.com left intact * Issue another request to this URL: 'https://www.google.co.uk/?gfe_rd=cr&ei=qPYmVL28J4vDcJbFgLAI' * About to connect() to www.google.co.uk port 443 (#1) * Trying 216.239.32.20... * Connected to www.google.co.uk (216.239.32.20) port 443 (#1) * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA * Server certificate: www.google.co.uk * Server certificate: Google Internet Authority G2 * Server certificate: GeoTrust Global CA * Server certificate: Equifax Secure Certificate Authority > GET /?gfe_rd=cr&ei=qPYmVL28J4vDcJbFgLAI HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.google.co.uk
> Accept: */*
>
< HTTP/1.1 302 Found
< Location: http://www.google.co.uk/?gfe_rd=cr&ei=qPYmVL28J4vDcJbFgLAI&gws_rd=ssl
This above transcript is indecipherable to any one non-technical, but what it shows is that Google itself is redirecting me to http by sending a 302 “Found” header when I connect to https.
It’s important to stress that I’m connecting to a Google-owned server called “GFE/2.0” (which could stand for “Google Front End”). We’re then handed off to “gws” later, which presumably stands for Google Web Search (or Google Web Server).
Thus what I’ve shown here is that the statement “this network has turned off SSL search” is false. BT can’t be doing it unless it has re-routed some of Google’s IP addresses, spoofed its SSL certs, and hosted its own implementation of GFE before handing you back to GWS.
This is unlikely, to say the least.
What we’re witnessing therefore, is almost certainly the result of a commercial agreement between BT and Google UK. One that exchanges the privacy of my searches for BT and Google’s commercial gain.
Duckduckgo it is then.
Your assessment is slightly false. Observe CNAME vs. A record DNS replies for google subdomains like www and encrypted
Could you elaborate?
That feature was meant for schools to force safesearch, and is being abused by BT. It will be turned off http://googleonlinesecurity.blogspot.com/2014/10/an-update-to-safesearch-options-for.html
Regarding proxy servers, most mobile network providers use transparent a transparent proxy from Bytemobile. Every request and response is logged and content is changed.
:-(
Did you run a dig query on “google.com”? They might be using the magic nosslsearch DNS record: https://productforums.google.com/forum/#!topic/websearch/1l2KMUfgyo4.
Nope it’s not that, I actually covered this above and tested alternative DNS servers.
edit: I was sure it wasn’t this, but see replies below
I was recently in the same situation. And as far as I can figure the reason HTTPS is disabled, is that the BT Wifi hotspots require you to login with username/password on a custom page before you can access the internet. Most people’s default thing to do is google something, which then redirects them to the BT Wifi login page, but this only works if Google is being served up via HTTP, otherwise BT wouldn’t be able to hijack the request and redirect you to the login page.
Hence it’s probably not got much to do with privacy, and more to do with usability.
If +90% of users just got HTTPS/SSL security warnings from their browsers instead of a BT Wifi login page, they wouldn’t be able to use BT Wifi unless they’re of the minority who know and understand how HTTP/HTTPS connections work.
And regarding Google doing the redirect themselves, as far as I would guess, they probably offers certain networks the ability to disable HTTPS for various reasons. Like companies who want to actively restrict internet usage, or schools with overly strict policies, or open hotspot providers who want to improve average customer usability.
> And as far as I can figure the reason HTTPS is disabled, is that the BT Wifi hotspots require you to login with username/password on a custom page before you can access the internet.
That is a very good point, and one I hadn’t considered. For this to work though, BT would have to allow https connections to Google.com without being logged in, and they would only do this if Google forced the redirect on port 443 to port 80, which the BT AP could then hijack with the login page.
I am not near a BTWifi AP any more, almost wish I was to test! :)
There must be a better solution to this problem though, as the end result is that anyone with a smart phone can sniff your Google searches. Personally, the pain of having to hit an http site first to login pales in significance to the privacy implications.
I didn’t do extensive testing myself, but I’m fairly certain that BT specifically let HTTPS traffic through to Google as they know it’ll redirect to Google over HTTP which BT then hijack to redirect you to the login page.
And I agree that there should be better ways to do this, as personally I’d rather have some usability issues rather than privacy issues. However 98% of the population just don’t know/care, and just want internet.
Also from a privacy standpoint, the BT Wifi networks should be a last resort, since the wifi network itself is not encrypted, which alone was enough for me to use a VPN a lot of the time, so even plain text HTTP traffic was not transmitted through the air in plain text between my laptop and the access point. So either way, BT Wifi fails the privacy checklist at the most basic of points, unencrypted wifi network. Hence I consider BT Wifi a absolute last resort when it comes to getting online… lol
> Also from a privacy standpoint, the BT Wifi networks should be a last resort, since the wifi network itself is not encrypted
If they used TLS, it wouldn’t matter is the traffic is encrypted or not. Since they use plain-text, suddendly that become relevant. I think that that fact that it’s unencrypted only worsens the entire situation.
True, using TLS should keep you safe on a open wireless network. But in this case it’s only Google we’re talking about. Your bank will obviously use HTTPS. But lots of sites still don’t, despite the fact that all sites should in my opinion. You generally want to use as few plain-text transfer methods as possible. Which is why I consider any unencrypted wireless network a absolute last resort.
At first I was thinking how would people have problems unless they type in https? Unless hsts is enabled.
I think the answer is search bars that are automatically configured to use google over https. If people type in google.com, there should be no problems because it is initially http which can be redirected.
Users will still see the TLS security warning, because the redirection is done over https – eg: the https conneciton needs to be successful for the redirection to plain-text http.
According to the log you posted, the IP address you are using, 216.239.32.20, is the address for nossl.google.com. The domain name http://www.google.com normally resolves to 74.125.69.103 and similar. I think you may have made some kind of mistake, either the hosts entry is not being used, the old domain name is being cached, or your server has a DNS problem.
You can query Google’s public DNS server using
dig http://www.google.com @8.8.8.8
Access google.com using nossl IP address. Redirects to HTTP site for me.
curl https://216.239.32.20/ -H’Host: http://www.google.com‘ -k -I
Access google.com using normal IP address. No redirect
curl https://74.125.69.103/ -H’Host: http://www.google.com‘ -k -I
You’re totally right, that is nossl.google.com
Unfortunately this was a while ago now and I’m now on a permanent connection so I can’t verify, it’s entirely possible I made this mistake.
I did query against 8.8.8.8 though which I’m pretty sure gave the same answer (i.e. returned the nossl IP), so it’s possibly implemented by deep packet inspection and responding early to UDP DNS queries for google.com.
Would be interesting to test this again, but I no longer have a BTWifi account to try. And I didn’t keep any logs of the DNS queries, as I ruled out the CNAME nossl approach fairly early! (possibly incorrectly)
Looks like BT is hiding the true DNS results. This is from a BTwifi hotspot :
C:\Users\max>nslookup – 8.8.8.8
Default Server: google-public-dns-a.google.com
Address: 8.8.8.8
> http://www.google.co.uk
Server: google-public-dns-a.google.com
Address: 8.8.8.8
Non-authoritative answer:
Name: nosslsearch.google.com
Addresses: 216.239.32.20
216.239.32.20
Aliases: http://www.google.co.uk.home
>
Unfortunately I only have Windows handy on Wifi.
That would explain how I ended up using that IP. Not sure how I missed it though as it’s clearly sending the CNAME record… wish I’d noticed this!
(your comment ended up in the Akismet spam queue, sorry for the delay in approving)
I run a DNS service which logs the server doing lookups. I then looked up unique hostnames on my domain to avoid caching. I did this whilst connected to the BT access point which I can see via wifi from my house and to my surprise everything came from BT even when I explicitly requested 8.8.8.8. The reason I’m surprised is because I’m fairly sure this is new as I’m sure they didn’t do this before.
However you now have your answer. They’re definitely doing dns interception and using that to direct to the nosslsearch host to force http. Possibly so users don’t get confusing error messages, but more likely for some financial gain related to ads or something.
Whoa, that means it’s every DNS request and not just those for Google, so the DNS server logs would be interesting of themselves.
Thanks for checking this out!
Also implies that if you did a lookup directly against your own DNS server rather than 8.8.8.8, the UDP packet would never reach it.
As a webmaster you could have fun with this, and easily redirect all users from BTWifi to a different site… which injected some JS and displayed a warning…. :-)
I wouldn’t be surprised if just putting an ssl-friendly IP for http://www.google.com isn’t enough; just looking at your curl output’s redirect to https://www.google.co.uk makes me suspect if you’d had *that* in your hosts file too, you wouldn’t have got the second redirect to http://
Since I work in a school district in the US and we are required to have gateway content filters installed to prevent students from accessing inappropriate webpages. We implement a device on our network called Iboss, https://www.iboss.com. This device can decrypt https/ssl and then rencrypt it on the fly. It also has a feature to force google safe search. I suspect this company may be using a similar device. Here are some of the features of an iboss content filter.
http://www.iboss.com/web_security_suite/wss_content_management.html
Notice the second feature in the list.
That’s somewhat different than what’s being discussed here, what that device does is likely to be a dynamic CA which proxies all https requests and re-encrypts the traffic (effectively a Man in the Middle attack). It works on networks where you have control over the client devices or can get people to install a trusted CA certificate, but wouldn’t be suitable for public networks.
There’s no evidence that BT was attacking https here, they were simply forcing me to connect over http.
That is true but I suspect they might be using some kind of similar device that can force safe search. I used the iboss appliance as an example, it works as a transparent proxy between the internal network and the world at large.
Pingback: GOTCHA: Google caught STRIPPING SSL from BT WiFi users’ searches | TechDiem.com
Pingback: ste williams – GOTCHA: Google caught STRIPPING SSL from BT WiFi users’ searches
Pingback: Google commits privacy seppuku at BT's request htt... - Thej Live
Pingback: ste williams – GOTCHA: Google caught STRIPPING SSL from BT Wi-Fi users’ searches
Pingback: Google disables SSL search at BT’s request | FeedLab
wait how is this google’s fault, your title says google commits privacy seppuku at bt request, all i see is bt affecting google.
It’s Google’s servers that perform the 302 redirect to an http url, BT can not do this.
I think it has been argued here in the comments that Google is not at fault, but BT is. If you add 216.239.32.20 http://www.google.co.uk (216.239.32.20 is the IP for nossl.google.com) to your HOSTS file and go to https://www.google.co.uk, it will redirect to http. This is intentional and not by request from BT, but due to BT returning modified DNS records.
Your title is now misleading…
I somewhat agree, it’s been shown that BT is in fact intercepting and replying to DNS queries fraudulently, but Google is still facilitating this by providing a server address that intentionally disables SSL against the client’s wishes. So while the title is not as accurate as I thought, it makes a point which is still valid.
(sorry for delay, this got buried in the spam bin)
Pingback: GOTCHA: Google caught STRIPPING SSL from BT Wi-Fi users’ searches | TechDiem.com
Sad to see that BT and Google are still doing this.
Still happening- eg; http://i.imgur.com/lMIrTPb.png