[08:45 PM UTC]UPDATE : the network issue seems to be fixed now, we have removed the temporary blacklists and are monitoring closely the situation.
Since about 10:00 AM UTC this Sunday (12 Nov. 2017), Dareboost is facing issues for websites tested from Paris and Amsterdam.
Only websites using resources from Google domains are affected (Google Analytics, Google Fonts, etc). As these resources are very frequently used, numerous websites are affected by the issue. Consequently we’re communicating with details on the issue, although the issue is out of our reach (the connectivity issue does not depend on our infrastructure).
All our team members are mobilized and we’re doing our best to get this issue fixed as soon as possible.
Nature of the issue
Only Paris and Amsterdam test probes are affected. Requests emitted to Google domains (eg: fonts.google.com ou doubleclick.net) are not getting any response.
This lack of response,very frequent but not systematic (randomness of the issue) is caused by the TCP connection not being established.
Network analysis shows a common pattern between Paris and Amsterdam: for both probes, the network traffic is routed through Google’s infrastructure behind core1.ams.net.google.com.
Network traffic from our Frankfurt test probe – for example – is not using this route (so is not affected by the issue).
The probes from London, Washington DC, Seattle, Mexico, Sao Paulo, Sydney, Tokyo, Hong Kong, Chennai (and so Frankfurt) are not affected by the issue as far as we know.
For Paris and Amsterdam, the issue has resulted in an average testing time being multiplied by more than 4, causing an overload of our probes for these particular locations.
Some planned tests have reached a timeout delay, results being lost.
This overload is now properly managed by Paris and Amsterdam locations.
Actions taken to mitigate and deal with the issue
Our infrastructure provider (IBM Cloud, formerly Softlayer) has been contacted but is unfortunately clueless about the problem. As previously stated, the issue is outside of our infrastructure and out of our reach.
We’re actually looking for a point of contact at Google.
As most of the websites monitored or tested with Dareboost are using one or more dependencies from Google (third-party content), we have decided to blacklist the following domains, as they are commonly used :
- google-analytics.com
- googletagmanager.com
- fonts.google.com
- doubleclick.net
- www.googleadservices.com
Even if test results are not including anymore the resources from these domains (due to the blacklist), our customers and users will now be able to get partial results. Previously to the blacklist, lot of them were facing Single Point of Failure, preventing to get proper reports.
We want to ensure all our customers and users that we’re deploying all our efforts to manage this issue and to get a proper solution as soon and possible. Thanks for your understanding.