Give the ‘Thing’ a Subnet of Its Own!

To my surprise, the most clicked post ever on this blog is this:

Network Sniffing for Everyone:
Getting to Know Your Things (As in Internet of Things)

… a step-by-step guide to sniff the network traffic of your ‘things’ contacting their mothership, plus a brief introduction to networking. I wanted to show how you can trace your networked devices’ traffic without any specialized equipment but being creative with what many users might already have, by turning a Windows PC into a router with Internet Connection Sharing.

Recently, an army of captured things took down part of the internet, and this reminded me of this post. No, this is not one more gloomy article about the Internet of Things. I just needed to use this Internet Sharing feature for the very purpose it was actually invented.

The Chief Engineer had finally set up the perfect test lab for programming and testing freely programmable UVR16x2 control systems (successor of UVR1611). But this test lab was a spot not equipped with wired ethernet, and the control unit’s data logger and ethernet gateway, so-called CMI (Control and Monitoring Interface), only has a LAN interface and no WLAN.

So an ages-old test laptop was revived to serve as a router (improving its ecological footprint in passing): This notebook connects to the standard ‘office’ network via WLAN: This wireless connection is thus the internet connection that can be shared with a device connected to the notebook’s LAN interface, e.g. via a cross-over cable. As explained in detail in the older article the router-laptop then allows for sniffing the traffic, – but above all it allows the ‘thing’ to connect to the internet at all.

This is the setup:

Using a notebook with Internet Connection Sharing enabled as a router to connect CMI (UVR16x2's ethernet gatway) to the internet

The router laptop is automatically configured with IP address 192.168.137.1 and hands out addresses in the 192.168.137.x network as a DHCP server, while using an IP address provided by the internet router for its WLAN adapter (indicated here as commonly used 192.168.0.x addresses). If Windows 10 is used on the router-notebook, you might need to re-enable ICS after a reboot.

The control unit is connected to the CMI via CAN bus – so the combination of test laptop, CMI, and UVR16x2 control unit is similar to the setup used for investigating CAN monitoring recently.

The CMI ‘thing’ is tucked away in a private subnet dedicated to it, and it cannot be accessed directly from any ‘Office PC’ – except the router PC itself. A standard office PC (green) effectively has to access the CMI via the same ‘cloud’ route as an Internet User (red). This makes the setup a realistic test for future remote support – when the CMI plus control unit has been shipped to its proud owner and is configured on the final local network.

The private subnet setup is also a simple workaround in case several things can not get along well with each other: For example, an internet TV service flooded CMI’s predecessor BL-NET with packets that were hard to digest – so BL-NET refused to work without a further reboot. Putting the sensitive device in a private subnet – using a ‘spare part’ router, solved the problem.

The Chief Engineer's quiet test lab for testing and programming control units

Watching TV Is Dangerous

I am not talking about humans.

But TV-sets might threaten other devices in the smart home; this was a recent puzzle submitted by a blog reader.

Two unrelated devices / services met on the user’s local computer network:

  • IP-TV provided by a large German telco.
  • a data logger for monitoring the heating system.

This user had one of the solutions in place that I mentioned in my previous post on data logging: The logger BL-NET connects to the controller UVR1611 via CAN bus, and to the computer network via ethernet, and it acts as a ‘CAN-ethernet gateway’ to allow for logging data to database server on the network, hosting the application UVR Data Logger Pro.

Data Logger BL-NET, with ethernet, CAN, and USB interfaces (My attempt at 'organic tech product photography.)

Data Logger BL-NET, with ethernet, field bus, and USB interfaces (My attempt at ‘organic tech product photography’.)

The issue: Every time the user turned on the TV, BL-NET suddenly refused to work – communicating its predicament via a red LED. The IP-TV did not use up all the band width; so my suspicion was that the TV (or TV service) sends a network packet that the logger does not like; perhaps a special unicast or broadcast message too sci-fi-like for it. Or any of the parties involved does not strictly comply with standards. Or standards might be ambiguous.

It would have been interesting to do network analysis and trace the network traffic and spot that packet of death. But the logger BL-NET had been superseded by its successor – called C.M.I. – Control and Monitoring Interface – which has better out-of-the box logging, cloud support etc.. Since the open source UVR Data Logger Pro does not yet speak CMI’s protocol so I understand that users do not want to change their solution immediately. But it is unlikely that BL-NET will get firmware updates, and it is very unlikely that a large internet services provider will change its IP-TV communications protocol.

My suggestion was to shield the logger from packets sent by the TV – by tucking BL-NET away in its private subnet – using a spare internet router or the sniffer-router PC I had described in Network Sniffing for Everyone:

The ‘spare’ internet router was placed behind the main internet router, connecting its ‘WAN’ port to the main LAN, and BL-NET was connected to a LAN port of this second router.  If the router is a PC with sniffer software this configuration would also allow for researching the evil packet.

In order to avoid running yet another box consuming electrical power, one might also…

  • add another network interface to the the UVR Data Logger Pro database server and use this one as that router.
  • replace the internet router by one that can be configured for more than one virtual LAN (in case the current one does not have this option).
Bundesarchiv Bild 183-H0812-0031-001, Werbung, RFT Color 20, Fernseher

At that time, TV-sets did not yet jeopardize the integrity of other devices in the dumb home. Ad for color TV set, 1969. (Bundesarchiv – German national archive, Wikimedia)

Data Logging with UVR1611 – FAQ

I have received several questions related to my article on data logging on this blog, or to my postings on monitoring and control on our German blog.

Thus I have decided to write the article I would have wanted to read when I once made myself familiar with this. The target audience for this article are IT guys / web developers / ambitious DIY enthusiasts trying to make sense of the interfaces provided by the freely programmable controller UVR1611.

We use this device as the main controller of heat pump systems we design, and for monitoring and optimizing heating systems in general.

UVR1611, customized welcome screen

This control unit receives data from sensors (temperature, flow, irradiation,…) and controls pumps and valves accordingly.

You interact with the unit via programming it directly – using a scroll wheel and buttons, but this should only be used for changing parameters such as a temperature set point. The control logic should rather be developed with a graphical programming application, called TAPPS. This software creates the functional data which have to be uploaded to the controller.

Snapshot of a part of the control logic of our heat pump system, as designed with TAPPS.

Sensors and manageable devices as valves talk to the control unit via traditional field buses, such as CAN bus (e.g. also used in cars’ internal networks) and so-called DL Bus. In order to access UVR1611 via a standard TCP/IP computer network you need a kind of gateway. This device does not only convert the field bus communication, but also serves as a repository for the logged measurement data. 

In our control network there are two different kinds of loggers, for ‘research purposes’. According to the vendor Technische Alternative GmbH, using two loggers in parallel is not supported and discouraged at is might cause issues. For us it works fine, but only try at your own risk:

Data loggers CMI and BL-NET

Data loggers by Technische Alternative: CMI (Control and Monitoring Interface) to the left, BL-NET (Boot Loader)  in the middle, standard ethernet switch to the right. The CAN bus cable is connected to both of them via the blue connector.

Two loggers – CMI and BL-NET – are connected to UVR1611 via CAN bus – a linear bus that needs to be terminated on both ends. Each of the devices is connected to the local computer network via standard ethernet wiring.

CMI is BL-NET’s successor although there might be no immediate reason to upgrade. Starting from scratch now, I would recommend CMI though.

This is how our local CAN bus looks like now, as displayed in CMI’s web interface:

Devices on CAN bus, displayed by CMI

Devices on CAN bus, displayed by CMI: Loggers CMI and BL-NET, plus and energy counter (CAN-EZ) and an extension of inputs/outputs (CAN-IO), UVR1611’s successor UVR16x2.

So there is a web server on CMI, which can be accessed locally. As described in the previous article, you can also access it via a ‘cloud-based’ portal.

In summary, this logger / gateway allows for the following:

1) Uploading functional data (programming logic) to UVR1611, by uploading the file from  computer onto the SD card inserted into the device and then dragging the file to the control unit’s icon in this web interface. This is an improvement over BL-NET which required an additional software application called Memory Manager to transfer functional data to the logger first. Existing functional data can be downloaded and inspected with the recent versions of TAPPS.

2) Accessing the control unit as if you would use the scroll wheel and buttons, replicating its physical interfaces to a virtual version. The layout and menu shown here is defined by programming (functional data).

This is a web view of the configurable items exposed by the control unit UVR1611, as seen via CMI’s web interface.  The language of the web interface itself can be changed but the menu of the control unit depends on the operating system of the device (DE).

UVR1611 welcome screen, as seen via CMI.

The custom welcome screen (also shown in the photo above), as ‘forwarded’ via CMI’s web interface. The highlighted ‘DE’ indicates German firmware.

BL-NET basically does the same: it also ‘forwards the hardware interface to a web page.

Managing UVR1611 via BL-NET – same ‘MENUE’ as available on the physical device or via CMI. Here also the navigation of the web interface itself (left pane) is language-dependent, as tied to the device’s firmware.

Reader’s question: When you click BL-NET’s icon on the CMI website, you just see an error – why? It is expected as BL-NET operates at the same logical level as CMI, and thus cannot be managed via CMI (and BL-NET’s firmware predates the release of CMI).

Result of clicking the BL-NET icon in CMI’s display of the devices on the CAN bus.

3) Storing the logged data. In contrast to BL-NET and its scarce storage CMI’s storage card often does not need to be cleared often. We log data every 1,5 minutes, in total a few MB every month. An SD card with up to 32 GB capacity could be used, capable of holding several years of logging data.

Log files can be downloaded / ‘dragged’ from the SD card – but these files are not readable text files. To get CSV text files you would use Technische Alternative’s software Winsol, a Windows software, not a web application.The Winsol PC can communicate with CMI on the local network and having installed most recent firmware, also with other users’ devices via the cloud portal. But the software can also interpret data gathered from other loggers, e.g. files sent by clients.

Winsol: Heat pump operations, temperatures of heating water and brine

Screenshot of Winsol’s display of logged data: a custom view of temperatures of the heating water (curves in the middle 30°C to 50°C, and temperature of brine at different points (bottom curves, below 0°C. Zooming in on an interesting part of the curves is done by selecting a rectangular area anywhere in the plot with the mouse.

We use Winsol for digging into the data to spot glitches and evaluate heating systems’ performance – for optimization. Using Winsol and logfiles ‘sent’ by whatever transmission method will always work, no matter which logger a client uses, how their firewall is configured, or if they use the cloud portal.

The ‘logging architecture’ was the same with BL-NET, but from checking the networking traffic between the Winsol PC and the logger I conclude that the communication protocol was different. CMI now seems to use more straight-forward HTTP calls.

4) Providing visualization of the data measured right now. In contrast to BL-NET, you cannot show your system to anonymous visitors on the internet. Viewers need to register with Technische Alternative’s online (‘cloud’) portal and be given Guest access. With BL-NET system owners forwarded port 80 at their local firewalls and kept the Guest User’s password blank. Perhaps not always on purpose as the same was often true for the Expert’s password. Theoretically, you can still do this with CMI but I would not recommend it as the port for web access is now the same port as for fetching the log files.

Hydraulic Schema, as displayed by CMI

Hydraulic schema with dynamic values, as displayed by CMI to 1) local network users and 2) cloud users given Guest access. The green numbers are the current sensor values – a subset of all columns in the log file. CMI’s web server allows for creating different pages, and versions for different languages.

The software TA Designer creates the web view based on an image file of the hydraulic layout, and on a list of sensors and controlled devices read from functional data:

Hydraulic schematic for CMI in TA Designer

First steps when creating a dynamic visualization: You need to provide a  drawing of your hydraulic layout. Status and readings of sensors, valves,, pumps etc. can be dragged in the right place from the ‘tree’ in the left pane – which has been created from the imported functional data file.

What web developers like to add or improve is related to the last two points: Logging data into a database directly, and providing a custom web interface – with the option to give anonymous users view-schematic-only access.

Recent questions:

  • Is there a (standardized, XML-based) web service I can use to poll the data?
  • (Why) do I need an additional box like BL-NET?
  • You stated you log and analyze your data on your local network – how do you do it?

No, there is no web service. But I have been pointed to this open source web application: UVR1611 Data Logger Pro. Data Logger Pro uses the same port as Winsol to talk to the BL-NET (40000), so the same protocol. Data are polled and stored in a MySQL database – working around BL-NET’s limited storage capacity. You still need the logger hardware, as data gathered from communicating over the CAN bus have to be converted. In this case BL-NET operates as a CAN-Ethernet gateway only.

If you google for UVR1611 Data Logger Pro, you will find lots of websites on the internet: They all use nice domain names, like heater.surname.com, so I suppose these are accessible on purpose.

This solution does not yet work with CMI due to the different communication protocol. But somebody might work on this already, so this information might be outdated soon.

Update, autumn 2015: CMI and UVR16x2 are now supported by UVR Data Logger Pro.

We also use our own database (Microsoft SQL Server), but we create it from the CSV files exported with Winsol.

SQL scripts import data from the CSV export files created with Winsol to a database. Custom views are used to consolidate data (daily, monthly, per season), and to merge them with data measured manually about every day.

Since 2012, we have added sensors, and we calculated new parameters from these sensors’ readings. Sometimes you need to exclude non-meaningful sensor values from calculations, e.g. when the tank is drained or changes are made to the collector.  The custom SQL application keeps track of different calculations to be applied to different periods.

Recently I have also developed an Excel application – to calculate the most important performance parameters only, directly from a bunch of CSV files. The latter is surprisingly performant if you resist the temptation to mix VBA and those really huge spreadsheet formulas.

All the plots I had inserted into blog posts or into our PDF summary of key data had been created with Excel – as a frontend to SQL Server. For the Ice Storage Challenge plot, we picked the columns with daily averages of temperatures and the volume of ice as calculated from the increase in water level:

Volume of ice in the water tank over time, 2015-03-06

A plot created from our database of measured sensor values. Excel connects to SQL server – to a view with daily averages and lots of calculated values, such as the volume of ice.

___________________________

Re UVR1611’s successor, UVR16x2: We have it installed, but we are waiting for the firmware update that will allow logging via CMI.

Google and Heating Systems (2)

I googled our company name. Then I found this:

What should not be online

Auftrag means order and the obfuscated parts contain our full company name, the Chief Engineer’s name, the URL of a vendor we ordered material from recently, invoice total, and a comment like The client said we should…

The now inaccessible URL had pointed to a comma-separated text related to statistics for orders. Obviously they had put company-internal data on an internet-facing system without knowing it. If you are familiar with the details of the URL and keywords you can actively search for such systems on the internet.

This is in essence what Google Hacking is about – here is a detailed manual, a presentation from a security conference. The infamous list of orders is used as a prime example on p.10.

If you wonder why this is called Google and Heating (2). This was on Google and heating, too, though there is not much relation to the topics covered.

Search engine Shodan takes this a step further: It allows for searching specifically for devices who are listening for incoming connections on the internet. Analyzing the standardized headers of the responses tells you if this is a traffic light, web cam, an internet router … or some home owner’s heating system.

These are search results for ADSL modems used by a large telco.

shodan-search-resultThose devices have a web server listening on HTTP. Not necessarily an issue if passwords have been set, there are no known vulnerabilities, and in case there is those systems are updated. As an end user you would not have a chance to interfere here as the modems are managed by the provider.

But it definitely should not look like this.

This is the passwords page of of data logger (BL-NET by Technische Alternative) for a heater accessible via the internet, showing that none of the passwords for guests, normal and expert user had been set. You could maliciously change control parameters or set passwords and lock the owner out.

But in contrast to a provider’s modem you need to take action to make such loggers and their web interfaces available on the internet. Vulnerabilities aside, any typical internet router (a device doing Network Address Translation) does not allow unsolicited incoming connections from the the internet to a device on the local network, that is behind the provider’s access device and/or your router. Only traffic that is recognized as the response to an outgoing request, such as browsing a public web pages, will be relayed by the router. In order to show off your heater’s performance to your friend you need to open up your router’s firewall and configure a rule for so-called port forwarding.

The problem with this approach is that some people don’t know exactly what they are doing (see inquiries via forums along the lines: I have no idea at all what VPN, TCP/IP, ports, DNS etc. means – but could you explain me briefly how to access my heater from the internet?), and there might be lots of running systems never touched again, once configured by the computer-savvy friend.

Then there might be hidden risks related to undetected vulnerabilities in the embedded web servers used. A German vendor of heating systems had caused a stir last year: Their clients’ systems had been accessible from the internet via port-forwarding. Their naming conventions for the dyndns names of such hosts could easily be guess – so attackers could find the systems. Passwords have been set; but sending a specifically crafted URL to the device you could force the web server to respond with the list of all passwords in clear text. The vendor reacted quickly and referred the issue to the supplier of the underlying control software – which was used with larger and more critical systems and residential heating. It turned out that the software vendor had never recommended to use the system in that way – only protected by passwords, but a VPN tunnel should be provided instead – wrapping the insecure traffic within a channel equipped with stronger protection. Adding a VPN is a major change and required the installation of a new physical module at clients’ site.

Apart from opening up your network up to the internet or VPNs there is another class of solutions to the Internet of Things issue: Things may actively connect to a server on the internet, and this server will relay or mediate the connection. I have written about Things unexpected phoning home and how to sniff the traffic before, and I add some more links at the end of this post. If the owner of the thing is given some control over the communication I still think it is the best option.

We now use such a Thing as our latest data logger for our heat pump system.

That’s the Thing – C.M.I., Control and Monitoring Interface – a failed attempt at innovative tech product photography:

(The usual disclaimer: I don’t make money from reselling or recommending products, I just like them. Vendors beware, I might change my mind anytime.)

It does not get better if I try to capture The Things in their natural habitats – CMI to the left, BL-NET in the middle, and a simple ethernet switch to the right.

CMI and BL-NEZ data loggers, by Technische Alternative

This is the ‘data center’. The control system (UVR1611) is in the ‘boiler room’, connected via CAN bus (blue connectors) to both loggers. We operate them in parallel, on the same CAN bus – for ‘research purposes’ and fun, though this is discouraged by Technische Alternative. Both loggers are connected to the local network.

We haven’t opened our firewall for BL-NET but CMI is allowed to make an outbound connection to the vendor’s portal https://cmi.ta.co.at/. You are required to create a user at this portal (that is running on amazon’s cloud BTW), and associate your CMI’s unique serial number and key with your user online. Other portal users may be given permission to view or manage your device – which is how we do online support of clients’ devices. It is not possible to allow anonymous users to view your current data and hydraulic layout.

The CMI is keeping a permanent outbound connection to the portal server who relays ‘incoming’ requests that technically aren’t incoming.

What I find important is:

You can access the device locally and directly, too. All your logged data are stored on an SD card – the slot and the blue card are visible in the photos. You can turn off the device’s connection to the portal and perhaps only turn it on if you required support.

The networking settings are similar to that of any computer on the local network. Turning off the portal is equivalent to not running Teamviewer, VNC, or similar remote support tools.

CMI settings, turn off connection to online portal.Unfortunately this cannot be said for any appliance that sends data to a portal. Actually, this article had in part been triggered by my researching the data logging capabilities of inverters of photovoltaic generators. Some of those send data to their clouds while giving the user no local access to the data at all.

Ambitious users build tools (e.g. running on Raspberry Pi) that intercept and store the traffic that was intended for the portal. A user reported that his battery did not work for weeks after the inverter vendor had upgraded the firmware. The new firmware used different temperature thresholds when determining if the battery was operating normally – and decided that the battery was much too cold. It took some time to persuade the vendor to restore the previous version of the firmware.

Remote firmware upgrade is subject to heated discussions, and can cause legal issues. Vendors of smart meters have to to separate the software that is required for ‘features’ – to be upgraded later, following ever changing standards and advances in technology – and the software associated with the data used in billing – subject to official calibration.

In case the vendor of the modems shown in the Shodan screenshot detects a vulnerability we would probably happy if they patch it immediately. Our favorite Things can be updated automatically and it went well so far.

____________________________________________________

Further reading:

Security Statement for Teamviewer – which also happens to be the software I am using for remote connections to clients’ computer systems and for remote meetings.

The Internet of Things, and how those Things phone home. An accessible and brief explanation of the different ways things allow for connections leveraged by a server on the internet.

Peer to Peer – Hole Punching – more detailed explanations.

Peer-to-Peer Communication Across Network Address Translators – even more detailed explanations, similar to this RFC by the same authors.