Cloudy Troubleshooting (2)

Unrelated to part 1 – but the same genre.

Actors this time:

  • File Cloud: A cloud service for syncing and sharing files. We won’t drop a brand name, will we?
  • Client: Another user of File Cloud.
  • [Redacted]: Once known for reliability and as The Best Network.
  • Dark Platform: Wannabe hackers’ playground.
  • elkement: Somebody who sometimes just wants to be an end user, but always ends up sniffing and debugging.

There are no dialogues with human life-forms this time, only the elkement’s stream of consciousness, interacting with the others via looking at things at a screen.

elkement: Time for a challenging Sunday hack!

elkement connects to the The Dark Platform. Hardly notices anything in the real world anymore. But suddenly elkement looks at the clock – and at File Cloud’s icon next to it.

elkement: File Cloud, what’s going on?? Seems you have a hard time Connecting… for hours now? You have not even synced my hacker notes from yesterday evening?

elkement tries to avoid to look at File Cloud, but it gets too painful.

elkement: OK – let’s consider the File Cloud problem the real Sunday hacker’s challenge…

elkement walks through the imaginary checklist:

  • File Cloud mentioned on DownDetector website? No.
  • Users tweeting about outage? No.
  • Do the other cloudy apps work fine? Yes.
  • Do other web sites work fine? Yes.
  • Does my router needs its regular reboots because it’s DNS server got stuck? No.
  • Should I perhaps try the usual helpdesk recommendation? Yes. (*)

(*) elkement turns router and firewall off and on again. Does not help.

elkement gets worried about Client using File Cloud, too. Connects to Client’s network – via another cloudy app (that obviously also works).

  • Does Client has the same issues? Yes and No – Yes at one site, No at another site.

elkement: Oh no – do I have to setup a multi-dimensional test matrix again to check for weird dependencies?

Coffee Break. Leaving the hacker’s cave. Gardening.

elkement: OK, let’s try something new!

elkement connects to super shaky mobile internet via USB tethering on the smart phone.

  • Does an alternative internet connection fix File Cloud? Yes!!

elkement: Huh!? Will now again somebody explain to me that a protocol (File Cloud) is particularly sensitive to hardly noticeable network disconnects? Is it maybe really a problem with [Redacted] this time?

elkement checks out DownDetector – and there they are the angry users and red spots on the map. They mention that seemingly random websites and applications fail. And that [Redacted] is losing packets.

elkement: Really? Only packets for File Cloud?

elkement starts sniffing. Checks IP addresses.

(elkement: Great, whois does still work, despite the anticipated issues with GDPR!)

elkement spots communication with File Cloud. File Cloud client and server are stuck in a loop of misunderstandings. File Cloud client is rude and says: RST, then starts again. Says Hello. They never shake hands as a previous segment was not captured.

elkement: But why does all the other stuff work??

elkement googles harder. Indeed, some other sites might be slower – not The Dark Platform, fortunately. Now finally Google and duckduckgo stop working, too. 

elkement: I can’t hack without Google.

elkement hacks something without Google though. Managed to ignore File Cloud’s heartbreaking connection attempts.

A few hours later it’s over. File Cloud syncs hacker notes. Red spots on DownDetector start to fade out while the summer sun is setting.

~

FIN, ACK

Cloudy Troubleshooting

Actors:

  • Cloud: Service provider delivering an application over the internet.
  • Client: Business using the Cloud
  • Telco: Service provider operating part of the network infrastructure connecting them.
  • elkement: Somebody who always ends up playing intermediary.

~

Client: Cloud logs us off ever so often! We can’t work like this!

elkement: Cloud, what timeouts do you use? Client was only idle for a short break and is logged off.

Cloud: Must be something about your infrastructure – we set the timeout to 1 hour.

Client: It’s becoming worse – Cloud logs us off every few minutes even we are in the middle of working.

[elkement does a quick test. Yes, it is true.]

elkement: Cloud, what’s going on? Any known issue?

Cloud: No issue in our side. We have thousands of happy clients online. If we’d have issues, our inboxes would be on fire.

[elkement does more tests. Different computers at Client. Different logon users. Different Client offices. Different speeds of internet connections. Computers at elkement office.]

elkement: It is difficult to reproduce. It seems like it works well for some computers or some locations for some time. But Cloud – we did not have any issues of that kind in the last year. This year the troubles started.

Cloud: The timing of our app is sensitive: If network cards in your computers turn on power saving that might appear as a disconnect to us.

[elkement learns what she never wanted to know about various power saving settings. To no avail.]

Cloud: What about your bandwidth?… Well, that’s really slow. If all people in the office are using that connection we can totally understand why our app sees your users disappearing.

[elkement on a warpath: Tracking down each application eating bandwidth. Learning what she never wanted to know about tuning the background apps, tracking down processes.]

elkement: Cloud, I’ve throttled everything. I am the only person using Clients’ computers late at night, and I still encounter these issues.

Cloud: Upgrade the internet connection! Our protocol might choke on a hardly noticeable outage.

[elkement has to agree. The late-night tests were done over a remote connections; so measurement may impact results, as in quantum physics.]

Client: Telco, we buy more internet!

[Telco installs more internet, elkement measures speed. Yeah, fast!]

Client: Nothing has changed, Clouds still kicks us out every few minutes.

elkement: Cloud, I need to badger you again….

Cloud: Check the power saving settings of your firewalls, switches, routers. Again, you are the only one reporting such problems.

[The router is a blackbox operated by Telco]

elkement: Telco, does the router use any power saving features? Could you turn that off?

Telco: No we don’t use any power saving at all.

[elkement dreams up conspiracy theories: Sometimes performance seems to degrade after business hours. Cloud running backup jobs? Telco’s lines clogged by private users streaming movies? But sometimes it’s working well even in the location with the crappiest internet connection.]

elkement: Telco, we see this weird issue. It’s either Cloud, Client’s infrastructure, or anything in between, e.g. you. Any known issues?

Telco: No, but [proposal of test that would be difficult to do]. Or send us a Wireshark trace.

elkement: … which is what I planned to do anyway…

[elkement on a warpath 2: Sniffing, tracing every process. Turning off all background stuff. Looking at every packet in the trace. Getting to the level where there are no other packets in between the stream of messages between Client’s computers and Cloud’s servers.]

elkement: Cloud, I tracked it down. This is not a timeout. Look at the trace: Server and client communicating nicely, textbook three-way handshake, server says FIN! And no other packet in the way!

Cloud: Try to connect to a specific server of us.

[elkement: Conspiracy theory about load balancers]

elkement: No – erratic as ever. Sometimes we are logged off, sometimes it works with crappy internet. Note that Client could work during vacation last summer with supper shaky wireless connections.

[Lots of small changes and tests by elkement and Cloud. No solution yet, but the collaboration is seamless. No politics and finger-pointing who to blame – just work. The thing that keeps you happy as a netadmin / sysadmin in stressful times.]

elkement: Client, there is another interface which has less features. I am going to test it…

[elkement: Conspiracy theory about protocols. More night-time testing].

elkement: Client, Other Interface has the same problems.

[elkement on a warpath 3: Testing again with all possible combinations of computers, clients, locations, internet connections. Suddenly a pattern emerges…]

elkement: I see something!! Cloud, I believe it’s user-dependent. Users X and Y are logged off all the time while A and B aren’t.

[elkement scratches head: Why was this so difficult to see? Tests were not that unambiguous until now!]

Cloud: We’ve created a replacement user – please test.

elkement: Yes – New User works reliably all the time! 🙂

Client: It works –  we are not thrown off in the middle of work anymore!

Cloud: Seems that something about the user on our servers is broken – never happened before…

elkement: But wait 😦 it’s not totally OK: Now logged off after 15 minutes of inactivity? But never mind – at least not as bad as logged off every 2 minutes in the middle of some work.

Cloud: Yeah, that could happen – an issue with Add-On Product. But only if your app looks idle to our servers!

elkement: But didn’t you tell us that every timeout ever is no less than 1 hour?

Cloud: No – that 1 hour was another timeout …

elkement: Wow – classic misunderstanding! That’s why it is was so difficult to spot the pattern. So we had two completely different problems, but both looked like unwanted logoffs after a brief period, and at the beginning both weren’t totally reproducible.

[elkement’s theory validated again: If anything qualifies elkement for such stuff at all it was experience in the applied physics lab – tracking down the impact of temperature, pressure and 1000 other parameters on the electrical properties of superconductors… and trying to tell artifacts from reproducible behavior.]

~

Cloudy

Everything as a Service

Three years ago I found a research paper that proposed a combination of distributed computing and heating as a service: A cloud provider company like Google or Amazon would install computers in users’ homes – as black-boxes providing heat to the users and computing power to their cloud.

In the meantime I have encountered announcements of startups very similar to this idea. So finally after we have been reading about the Internet of Things every day, buzz words associated with IT infrastructure enter the real world of hand-on infrastructure.

I believe that heating will indeed be offered as a service and like cloud-based IT services: The service provider will install a box in your cellar – a black-box in terms of user access, more like a home router operated by the internet provider today. It will be owned and operated by a provider you have a service contract with. There will be defined and restricted interfaces for limited control and monitoring – such as setting non-critical parameters like room temperature or viewing hourly and daily statistics.

Heating boxes will get smaller, more compact, and more aesthetically pleasing. They might rather be put in the hall rather than being tucked away in a room dedicated to technical gadgets. This is in line with a trend of smaller and smaller boiler rooms for larger and larger houses. Just like computers and routers went from ugly, clunky boxes to sleek design and rounded corners, heating boxes will more look like artistic stand-alone pillars. I remember a German startup which offered home batteries this beautiful a few years back – but they switched to another business model as they seem to have been too early.

Vendors of heating systems will try to simplify their technical and organizational interfaces with contractors: As one vendor of heat pump systems told me they were working on a new way of exchanging parts all at once so that a technician certified in handling refrigerants will not be required. Anything that can go wrong on installation will go wrong no matter how detailed the checklist for the installer is – also inlet and outlet do get confused. A vendor’s vision is rather a self-contained box delivered to the client, including heating system(s), buffer storage tanks for heating water, and all required sensors, electrical wiring, and hydraulic connections between these systems – and there are solutions like that offered today.

The vendor will have secured access to this system over the internet. They will be able to monitor continuously, detect errors early and automatically, and either fix them remotely or notify the customer. In addition, vendors will be able to optimize their designed by analyzing consolidated data gathered from a large number of clients’ systems. This will work exactly in the way vendors of inverters for photovoltaic systems deal with clients’ data already today: User get access to a cloud-based portal and show off their systems and data, and maybe enter a playful competition with other system owners – what might work for smart metering might work for related energy systems, too. The vendor will learn about systems’ performance data for different geographical regions and different usage patterns.

District heating is already offered as a service today: The user is entitled to using hot water (or cold water in case a heat pump’s heat source is shared among different users). Users sometimes dislike the lack of control and the fact they cannot opt out – as district heating only works economically if a certain number of homes in a certain area is connected to the service. But in some pilot areas in Germany and Austria combined heat and power stations have already been offered as a service and a provider-operated black-box in the user’s home.

The idea of having a third external party operating essential infrastructure now owned by an end-user may sound uncommon but we might get used to it when gasoline-powered cars in a user’s possession will be replaced by electrical vehicles and related services: like having a service contractor for a battery instead of owning it. We used to have our own computer with all our data on it, and we used to download our e-mail onto it, delete it from the server, and deal with local backups. Now all of that is stored on a server owned by somebody else and which we share with other users. The incentive is the ease of access to our data from various devices and the included backup service.

I believe that all kinds of things and products as a service will be further incentivized by bundling traditionally separate products: I used to joke about the bank account bundled with electrical power, home insurance, and an internet plus phone flat rate – until the combined bank account and green power offering was shown on my online banking’s home screen. Bundling all these services will be attractive, and users might be willing to trade in their data for a much cheaper access to services – just as a non-sniffing smart phone is more expensive than its alternatives.

Heat pump - not cloud-powered.I withhold judgement as I think there is a large grey and blurry area between allegedly evil platforms that own our lives and justified outsourcing to robust and transparent services that are easy to use also by the non tech-savvy.

Update 2016-06-02 : Seems I could not withold judgement in the comments 🙂 I better admit it here as the pingback from the book Service Innovation’s blog here might seem odd otherwise 😉

The gist of my argument made in the comments was:

I believe that artisans and craftsmen will belong in one of two categories in the future:
1) Either working as subcontractor, partner, or franchisee of large vendors, selling and installing standardized products – covering the last mile not accessible to robots and software (yet),
2) Or a lucky few will carve out a small niche and produce or customize bespoke units for clients who value luxurious goods for the sake of uniqueness or who value human imperfection as a fancy extra.

In other communication related to this post I called this platform effects Nassim Taleb’s Extremistan versus Mediocristan in action – the platform takes it all. Also ever growing regulation will help platforms rather than solo artisans as only large organizations can deal effectively with growing requirements re compliance – put forth both by government and by large clients or large suppliers.

My Google Searches Might Heat Your Home

Sorry, but this is not about Search Term Poetry!

Rather the contrary: Imagine your search terms could be utilized for something down-to-earth, for something useful.

Google states:

Energy consumption of Google internet searches

Energy consumption of Google internet searches (Google, Feb. 18, 2013)

See more comparisons for Google services here! (Though I am disappointed they did not convert to bath tubs!)

If Google’s computers run in their data centers in the middle of nowhere, this energy will be lost and contribute a bit to more carbon dioxide in the atmosphere.

What if Google would run their – virtual – data centers divided into millions tiny pieces, consisting of computers running in our homes?

This is an idea, that has been presented – though in a more serious and realistic fashion, and not focussed on Google, at the conference Hotcloud 11:

The Data Furnace: Heating Up with Cloud Computing

Cloud computing is hot, literally.

Computers can be placed directly into buildings to provide low latency cloud computing for its offices or residents, and the heat that is generated can be used to heat the building.

Thus in winter or during the night the provider of cloud services would offload / “send” more computing tasks to your heater-computer. Resembling the good old SETI screen saver searching for intelligent life in the universe (which was triggered by your being idle, not by your freezing).

I agree to all the caveats listed in the article, in particular the security related aspects. As a service provider you might not want to place your hardware in insecure, uncontrolled environment. This is like Smart Meters, just worse. In addition, Google has a point here regarding the efficiency of scale of large data centers.

But it’s geeky nonetheless!

Google, by the way, had once really planned to enter the energy business directly, as MIT Technology Review has reported last year:

[In 2007 Google] posted jobs for engineers who could speed up design of renewable-energy projects and put a team to work improving the heliostat, a mirrored device that focuses the sun’s rays to make thermal energy.

They also ventured into something Smart Meter like:

… PowerMeter, another canceled project. The software was meant to help homeowners monitor their energy use. Energy entrepreneur Kurt Brown says it had a major flaw: “Their interface was for nerds. It was something mostly a smart Googler would be intrigued by.”

Probably I would have loved this interface!

And the conclusion is:

The cancelled plans show the hazards of believing that success in computing—where products can take days to prototype—can carry over to energy.

Yes – I got it, but it will not stop me!

Currently Google is a player in the energy business, as an investor in renewable energy technology and owner of  wind farms.

I have come across other speculations how Google might enter the energy business, such as: Google might sell electricity at a competitive price, and users would provide (even more) personal data to compensate for that, or Google might offer their “algorithmic power” to utilities or compete with them with respect to aggregating data. But I cannot trace this down to a real product or service offered by Google today (E.g. on Google’s energy website).

North American users – enlighten me if Google really sells electricity to end users!

Memmingen St Martin Turmstübchen Ofen

Stove (Wikimedia) that does not require nor produce electricity, and that does not search for aliens in its idle time. Makes you 100% autonomous (or only dependent on the coal industry) and neither Google, nor the NSA or your utility can track down what you are doing right now.