Cloudy Troubleshooting


  • Cloud: Service provider delivering an application over the internet.
  • Client: Business using the Cloud
  • Telco: Service provider operating part of the network infrastructure connecting them.
  • elkement: Somebody who always ends up playing intermediary.


Client: Cloud logs us off ever so often! We can’t work like this!

elkement: Cloud, what timeouts do you use? Client was only idle for a short break and is logged off.

Cloud: Must be something about your infrastructure – we set the timeout to 1 hour.

Client: It’s becoming worse – Cloud logs us off every few minutes even we are in the middle of working.

[elkement does a quick test. Yes, it is true.]

elkement: Cloud, what’s going on? Any known issue?

Cloud: No issue in our side. We have thousands of happy clients online. If we’d have issues, our inboxes would be on fire.

[elkement does more tests. Different computers at Client. Different logon users. Different Client offices. Different speeds of internet connections. Computers at elkement office.]

elkement: It is difficult to reproduce. It seems like it works well for some computers or some locations for some time. But Cloud – we did not have any issues of that kind in the last year. This year the troubles started.

Cloud: The timing of our app is sensitive: If network cards in your computers turn on power saving that might appear as a disconnect to us.

[elkement learns what she never wanted to know about various power saving settings. To no avail.]

Cloud: What about your bandwidth?… Well, that’s really slow. If all people in the office are using that connection we can totally understand why our app sees your users disappearing.

[elkement on a warpath: Tracking down each application eating bandwidth. Learning what she never wanted to know about tuning the background apps, tracking down processes.]

elkement: Cloud, I’ve throttled everything. I am the only person using Clients’ computers late at night, and I still encounter these issues.

Cloud: Upgrade the internet connection! Our protocol might choke on a hardly noticeable outage.

[elkement has to agree. The late-night tests were done over a remote connections; so measurement may impact results, as in quantum physics.]

Client: Telco, we buy more internet!

[Telco installs more internet, elkement measures speed. Yeah, fast!]

Client: Nothing has changed, Clouds still kicks us out every few minutes.

elkement: Cloud, I need to badger you again….

Cloud: Check the power saving settings of your firewalls, switches, routers. Again, you are the only one reporting such problems.

[The router is a blackbox operated by Telco]

elkement: Telco, does the router use any power saving features? Could you turn that off?

Telco: No we don’t use any power saving at all.

[elkement dreams up conspiracy theories: Sometimes performance seems to degrade after business hours. Cloud running backup jobs? Telco’s lines clogged by private users streaming movies? But sometimes it’s working well even in the location with the crappiest internet connection.]

elkement: Telco, we see this weird issue. It’s either Cloud, Client’s infrastructure, or anything in between, e.g. you. Any known issues?

Telco: No, but [proposal of test that would be difficult to do]. Or send us a Wireshark trace.

elkement: … which is what I planned to do anyway…

[elkement on a warpath 2: Sniffing, tracing every process. Turning off all background stuff. Looking at every packet in the trace. Getting to the level where there are no other packets in between the stream of messages between Client’s computers and Cloud’s servers.]

elkement: Cloud, I tracked it down. This is not a timeout. Look at the trace: Server and client communicating nicely, textbook three-way handshake, server says FIN! And no other packet in the way!

Cloud: Try to connect to a specific server of us.

[elkement: Conspiracy theory about load balancers]

elkement: No – erratic as ever. Sometimes we are logged off, sometimes it works with crappy internet. Note that Client could work during vacation last summer with supper shaky wireless connections.

[Lots of small changes and tests by elkement and Cloud. No solution yet, but the collaboration is seamless. No politics and finger-pointing who to blame – just work. The thing that keeps you happy as a netadmin / sysadmin in stressful times.]

elkement: Client, there is another interface which has less features. I am going to test it…

[elkement: Conspiracy theory about protocols. More night-time testing].

elkement: Client, Other Interface has the same problems.

[elkement on a warpath 3: Testing again with all possible combinations of computers, clients, locations, internet connections. Suddenly a pattern emerges…]

elkement: I see something!! Cloud, I believe it’s user-dependent. Users X and Y are logged off all the time while A and B aren’t.

[elkement scratches head: Why was this so difficult to see? Tests were not that unambiguous until now!]

Cloud: We’ve created a replacement user – please test.

elkement: Yes – New User works reliably all the time! 🙂

Client: It works –  we are not thrown off in the middle of work anymore!

Cloud: Seems that something about the user on our servers is broken – never happened before…

elkement: But wait 😦 it’s not totally OK: Now logged off after 15 minutes of inactivity? But never mind – at least not as bad as logged off every 2 minutes in the middle of some work.

Cloud: Yeah, that could happen – an issue with Add-On Product. But only if your app looks idle to our servers!

elkement: But didn’t you tell us that every timeout ever is no less than 1 hour?

Cloud: No – that 1 hour was another timeout …

elkement: Wow – classic misunderstanding! That’s why it is was so difficult to spot the pattern. So we had two completely different problems, but both looked like unwanted logoffs after a brief period, and at the beginning both weren’t totally reproducible.

[elkement’s theory validated again: If anything qualifies elkement for such stuff at all it was experience in the applied physics lab – tracking down the impact of temperature, pressure and 1000 other parameters on the electrical properties of superconductors… and trying to tell artifacts from reproducible behavior.]



Give the ‘Thing’ a Subnet of Its Own!

To my surprise, the most clicked post ever on this blog is this:

Network Sniffing for Everyone:
Getting to Know Your Things (As in Internet of Things)

… a step-by-step guide to sniff the network traffic of your ‘things’ contacting their mothership, plus a brief introduction to networking. I wanted to show how you can trace your networked devices’ traffic without any specialized equipment but being creative with what many users might already have, by turning a Windows PC into a router with Internet Connection Sharing.

Recently, an army of captured things took down part of the internet, and this reminded me of this post. No, this is not one more gloomy article about the Internet of Things. I just needed to use this Internet Sharing feature for the very purpose it was actually invented.

The Chief Engineer had finally set up the perfect test lab for programming and testing freely programmable UVR16x2 control systems (successor of UVR1611). But this test lab was a spot not equipped with wired ethernet, and the control unit’s data logger and ethernet gateway, so-called CMI (Control and Monitoring Interface), only has a LAN interface and no WLAN.

So an ages-old test laptop was revived to serve as a router (improving its ecological footprint in passing): This notebook connects to the standard ‘office’ network via WLAN: This wireless connection is thus the internet connection that can be shared with a device connected to the notebook’s LAN interface, e.g. via a cross-over cable. As explained in detail in the older article the router-laptop then allows for sniffing the traffic, – but above all it allows the ‘thing’ to connect to the internet at all.

This is the setup:

Using a notebook with Internet Connection Sharing enabled as a router to connect CMI (UVR16x2's ethernet gatway) to the internet

The router laptop is automatically configured with IP address and hands out addresses in the 192.168.137.x network as a DHCP server, while using an IP address provided by the internet router for its WLAN adapter (indicated here as commonly used 192.168.0.x addresses). If Windows 10 is used on the router-notebook, you might need to re-enable ICS after a reboot.

The control unit is connected to the CMI via CAN bus – so the combination of test laptop, CMI, and UVR16x2 control unit is similar to the setup used for investigating CAN monitoring recently.

The CMI ‘thing’ is tucked away in a private subnet dedicated to it, and it cannot be accessed directly from any ‘Office PC’ – except the router PC itself. A standard office PC (green) effectively has to access the CMI via the same ‘cloud’ route as an Internet User (red). This makes the setup a realistic test for future remote support – when the CMI plus control unit has been shipped to its proud owner and is configured on the final local network.

The private subnet setup is also a simple workaround in case several things can not get along well with each other: For example, an internet TV service flooded CMI’s predecessor BL-NET with packets that were hard to digest – so BL-NET refused to work without a further reboot. Putting the sensitive device in a private subnet – using a ‘spare part’ router, solved the problem.

The Chief Engineer's quiet test lab for testing and programming control units

Watching TV Is Dangerous

I am not talking about humans.

But TV-sets might threaten other devices in the smart home; this was a recent puzzle submitted by a blog reader.

Two unrelated devices / services met on the user’s local computer network:

  • IP-TV provided by a large German telco.
  • a data logger for monitoring the heating system.

This user had one of the solutions in place that I mentioned in my previous post on data logging: The logger BL-NET connects to the controller UVR1611 via CAN bus, and to the computer network via ethernet, and it acts as a ‘CAN-ethernet gateway’ to allow for logging data to database server on the network, hosting the application UVR Data Logger Pro.

Data Logger BL-NET, with ethernet, CAN, and USB interfaces (My attempt at 'organic tech product photography.)

Data Logger BL-NET, with ethernet, field bus, and USB interfaces (My attempt at ‘organic tech product photography’.)

The issue: Every time the user turned on the TV, BL-NET suddenly refused to work – communicating its predicament via a red LED. The IP-TV did not use up all the band width; so my suspicion was that the TV (or TV service) sends a network packet that the logger does not like; perhaps a special unicast or broadcast message too sci-fi-like for it. Or any of the parties involved does not strictly comply with standards. Or standards might be ambiguous.

It would have been interesting to do network analysis and trace the network traffic and spot that packet of death. But the logger BL-NET had been superseded by its successor – called C.M.I. – Control and Monitoring Interface – which has better out-of-the box logging, cloud support etc.. Since the open source UVR Data Logger Pro does not yet speak CMI’s protocol so I understand that users do not want to change their solution immediately. But it is unlikely that BL-NET will get firmware updates, and it is very unlikely that a large internet services provider will change its IP-TV communications protocol.

My suggestion was to shield the logger from packets sent by the TV – by tucking BL-NET away in its private subnet – using a spare internet router or the sniffer-router PC I had described in Network Sniffing for Everyone:

The ‘spare’ internet router was placed behind the main internet router, connecting its ‘WAN’ port to the main LAN, and BL-NET was connected to a LAN port of this second router.  If the router is a PC with sniffer software this configuration would also allow for researching the evil packet.

In order to avoid running yet another box consuming electrical power, one might also…

  • add another network interface to the the UVR Data Logger Pro database server and use this one as that router.
  • replace the internet router by one that can be configured for more than one virtual LAN (in case the current one does not have this option).
Bundesarchiv Bild 183-H0812-0031-001, Werbung, RFT Color 20, Fernseher

At that time, TV-sets did not yet jeopardize the integrity of other devices in the dumb home. Ad for color TV set, 1969. (Bundesarchiv – German national archive, Wikimedia)

Network Sniffing for Everyone – Getting to Know Your Things (As in Internet of Things)

Simple Sniffing without Hubs or Port Mirroring for the Curious Windows User
[Jump to instructions and skip intro]

Your science-fiction-style new refrigerator might go online to download the latest offers or order more pizza after checking your calendar and noticing that you have to finish a nerdy project soon.

It may depend on your geekiness or faith in things or their vendors, but I absolutely need to know more about the details of this traffic. How does the device authenticate to the external partner? Is the connection encrypted? Does the refrigerator company spy on me? Launch the secret camera and mic on the handle?

In contrast to what the typical hacker movie might imply you cannot simply sniff traffic all on a network even if you have physical access to all the wiring.

In the old days, that was easier. Computers were connected using coaxial cables:

10base2 t-pieceCommunications protocols are designed to deal with device talking to any other device on the network any time – there are mechanisms to sort out collisions. When computers want to talk to each other the use (logical) IP addresses that need to get translated to physical device (MAC) addresses. Every node in the network can store the physical addresses of his peers in the local subnet. If it does not know the MAC address of the recipient of a message already it shouts out a broadcasting message to everybody and learns MAC addresses. But packets intended for one recipient are still visible to any other party!

A hub does (did) basically the same thing as coaxial cables, only the wiring was different. My very first ‘office network’ more than 15 years ago was based on a small hub that I have unfortunately disposed.

Nowadays even the cheapest internet router uses a switch – it looks similar but works differently:

A switch minimizes traffic and collisions by memorizing the MAC addresses associated with different ports (‘jacks’). If a notebook wants to talk to the local server this packet is sent from the notebook to the switch who forwards it to that port the server is connected to. Another curious employee’s laptop could not see that traffic.

This is fine from the perspective of avoiding collisions and performance but a bad thing if you absolutely want to know what’s going on.

I could not resist using the clichéd example of the refrigerator but there are really more and more interesting devices that make outbound connections – or even effectively facilitate inbound ones – so that you can connect to your thing from the internet.

Using a typical internet connection and router, a device on the internet cannot make an unsolicited inbound connection unless you open up respective ports on your router. Your internet provider may prevent this: Either you don’t have access to your router at all, or your router’s external internet address is still not a public one.

In order to work around this nuisance, some devices may open a permanent outbound connection to a central rendezvous server. As soon as somebody wants to connect to the device behind your router, the server utilizes this existing connection that is technically an outbound one from the perspective of the device.

Remote support tools such as Teamviewer use technologies like that to allow helping users behind firewalls. Internet routers doing that: DLink calls their respective series Cloud Routers (and stylish those things have become, haven’t they?).

How to: Setup your Windows laptop as a sniffer-router

If you want to sniff traffic from a blackbox-like device trying to access a server on the internet you would need a hub which is very hard to get these days – you may find some expensive used ones on ebay. Another option is to use a switch that supports Port Mirroring: All traffic on the network is replicated to a specific port, and connecting to that with your sniffer computer you could inspect all the packets

But I was asking myself for the fun of it:

Is there a rather simple method a normal Windows user could use though – requiring only minimal investment and hacker skills?

My proposed solution is to force the interesting traffic to go through your computer – that is turning this machine into a router. A router connects two distinct subnets; so the computer needs two network interfaces. Nearly every laptop has an ethernet RJ45 jack and wireless LAN – so these are our two NICs!

I am assuming that the thing to be investigated rather has wired than wireless LAN so we want…

  • … the WLAN adapter to connect to your existing home WLAN and then the internet.
  • … the LAN jack to connect to a private network segment for your thing. The thing will access the internet through a cascade of two routers finally.

Routing is done via a hardly used Windows feature experts will mock – but it does the job and is built-in: So-called Internet Connection Sharing.

Additional hardware required: A crossover cable: The private network segment has just a single host – our thing. (Or you could use another switch for the private subnet – but I am going for the simplest solution here.)

Software required: Some sniffer such as the free software Wireshark.

That’s the intended network setup (using 192.168.0.x as a typical internal LAN subnet)

|    Thing    |       |      Laptop Router      |      |Internet Router
|     LAN     |-cross-|     LAN     |    WLAN   |-WLAN-|Internal LAN
||       |||      |
  • Locate the collection of network adapters, in Windows 7 this is under
    Control Panel
    –Network and Internet
    —-View Network Status and Tasks
    ——Change Adapter Settings
  • In the Properties of the WLAN adapter click the Sharing tab and check the option Allow other network users to connect through this computer’s Internet connection.
  • In the drop-down menu all other network adapters except to one to be shared should be visible – select the one representing the RJ45 jack, usually called Local Internet Connection.

Internet Connection Sharing

  • Connect the RJ45 jack of the chatty thing (usually tagged LAN) to the LAN jack of your laptop with the crossover cable.
  • If it uses DHCP (most devices do), it will be assigned an IP address in the 192.168.137.x network. If it doesn’t i it needs a fixed IP address you should configure it for an address in this network with x other than 1. The router-computer will be assigned and is the DHCP server, DNS server, and the default gateway.
  • Start Wireshark, click Capture…, Interfaces, locate the LAN adapter with IP address and click Start

Now you see all the packets this device may send to the internet.

I use an innocuous example now:

On connecting my Samsung Blu-ray player, I see some interesting traffic:

Samsung bluray, packets

The box gets an IP address via DHCP (only last packet shown – acknowledgement of the address), then tries to find the MAC address for the router-computer – a Dell laptop – as it needs to consult the DNS service there and ask for the IP address corresponding to an update server whose name is obviously hard-coded. It receives a reply, and the – fortunately non-encrypted – communication with the first internet-based address is initiated.

Follow TCP stream shows more nicely what is going on:

Samsung bluray player wants to update

The player sends an HTTP GET to the script liveupdate.jsp, appending the model, version number of location in the European Union. Since the player is behind two routers – that is NAT devices – Samsung now sees this coming from my Austrian IP address.

The final reply is a page reading [NO UPDATE], and they sent me a cookie that is going to expire 3,5 years in the past 😉 So probably this does not work anymore.

As I said – this was an innocuous example. I just wanted to demonstrate that you never know what will happen if you can’t resist connecting your things to your local computer network. You might argue that normal computers generate even more traffic trying to contact all kinds of update servers – but in this case you 1) can just switch on the sniffer and see that traffic without any changes to be made to the network and 2) as an owner of your computers you could on principle control it.

Edit: Added the ASCII ‘networking diagram’ based on feedback!


Further reading:

Peer-to-Peer Communication Across Network Address Translators – an overview of different technique to allow for communications of devices behind NAT devices such as firewalls or internet routers.

Ethernet and Address Resolution Protocol (ARP) on Wikipedia

Sniffing Tutorial part 1 – Intercepting Network Traffic: Overview on sniffing options: dumb hubs, port mirroring, network tap.