I googled our company name. Then I found this:
Auftrag means order and the obfuscated parts contain our full company name, the Chief Engineer’s name, the URL of a vendor we ordered material from recently, invoice total, and a comment like The client said we should…
The now inaccessible URL had pointed to a comma-separated text related to statistics for orders. Obviously they had put company-internal data on an internet-facing system without knowing it. If you are familiar with the details of the URL and keywords you can actively search for such systems on the internet.
This is in essence what Google Hacking is about – here is a detailed manual, a presentation from a security conference. The infamous list of orders is used as a prime example on p.10.
If you wonder why this is called Google and Heating (2). This was on Google and heating, too, though there is not much relation to the topics covered.
Search engine Shodan takes this a step further: It allows for searching specifically for devices who are listening for incoming connections on the internet. Analyzing the standardized headers of the responses tells you if this is a traffic light, web cam, an internet router … or some home owner’s heating system.
These are search results for ADSL modems used by a large telco.
Those devices have a web server listening on HTTP. Not necessarily an issue if passwords have been set, there are no known vulnerabilities, and in case there is those systems are updated. As an end user you would not have a chance to interfere here as the modems are managed by the provider.
But it definitely should not look like this.
This is the passwords page of of data logger (BL-NET by Technische Alternative) for a heater accessible via the internet, showing that none of the passwords for guests, normal and expert user had been set. You could maliciously change control parameters or set passwords and lock the owner out.
But in contrast to a provider’s modem you need to take action to make such loggers and their web interfaces available on the internet. Vulnerabilities aside, any typical internet router (a device doing Network Address Translation) does not allow unsolicited incoming connections from the the internet to a device on the local network, that is behind the provider’s access device and/or your router. Only traffic recognized as the response to an outgoing request, such as browsing a public web pages, will be relayed by the router. In order to show off your heater’s performance to your friend you need to open up your router’s firewall and configure a rule for so-called port forwarding.
The problem with this approach is that some people don’t know exactly what they are doing (see inquiries via forums along the lines: I have no idea at all what VPN, TCP/IP, ports, DNS etc. means – but could you explain me briefly how to access my heater from the internet?), and there might be lots of running systems never touched again, once configured by the computer-savvy friend.
Then there might be hidden risks related to undetected vulnerabilities in the embedded web servers used. A German vendor of heating systems had caused a stir last year: Their clients’ systems had been accessible from the internet via port-forwarding. Their naming conventions for the dyndns names of such hosts could easily be guess – so attackers could find the systems. Passwords have been set; but sending a specifically crafted URL to the device you could force the web server to respond with the list of all passwords in clear text. The vendor reacted quickly and referred the issue to the supplier of the underlying control software – which was used with larger and more critical systems and residential heating. It turned out that the software vendor had never recommended to use the system in that way – only protected by passwords, but a VPN tunnel should be provided instead – wrapping the insecure traffic within a channel equipped with stronger protection. Adding a VPN is a major change and required the installation of a new physical module at clients’ site.
Apart from opening up your network up to the internet or VPNs there is another class of solutions to the Internet of Things issue: Things may actively connect to a server on the internet, and this server will relay or mediate the connection. I have written about Things ‘phoning home’ and how to sniff the traffic before, and I add some more links at the end of this post. If the owner of the thing is given some control over the communication I still think it is the best option.
We now use such a Thing as our latest data logger for our heat pump system.
That’s the Thing – C.M.I., Control and Monitoring Interface – here is my failed attempt at innovative tech product photography:
(The usual disclaimer: I don’t make money from reselling or recommending products, I just like them. Vendors beware, I might change my mind anytime.)
It does not get better if I try to capture The Things in their natural habitats – CMI to the left, BL-NET in the middle, and a simple ethernet switch to the right.
This is the ‘data center’. The control system (UVR1611) is in the ‘boiler room’, connected via CAN bus (blue connectors) to both loggers. We operate them in parallel, on the same CAN bus – for ‘research purposes’ and fun, though this is discouraged by Technische Alternative. Both loggers are connected to the local network.
We haven’t opened our firewall for BL-NET but CMI is allowed to make an outbound connection to the vendor’s portal https://cmi.ta.co.at/. You are required to create a user at this portal (that is running on amazon’s cloud BTW), and associate your CMI’s unique serial number and key with your user online. Other portal users may be given permission to view or manage your device – this is how we do online support of clients’ devices. It is not possible to allow anonymous users to view your current data and hydraulic layout.
The CMI is keeping a permanent outbound connection to the portal server who relays ‘incoming’ requests that technically aren’t incoming.
What I find important is:
You can access the device locally and directly, too. All your logged data are stored on an SD card – the slot and the blue card are visible in the photos. You can turn off the device’s connection to the portal and perhaps only turn it on if you required support.
The networking settings are similar to that of any computer on the local network. Turning off the portal is equivalent to not running Teamviewer, VNC, or similar remote support tools.
Unfortunately this cannot be said for any appliance that sends data to a portal. Actually, this article had in part been triggered by my researching the data logging capabilities of inverters of photovoltaic generators. Some of those send data to their clouds while giving the user no local access to the data at all.
Ambitious users build tools (e.g. running on Raspberry Pi) that intercept and store the traffic that was intended for the portal. A user reported that his battery did not work for weeks after the inverter vendor had upgraded the firmware. The new firmware used different temperature thresholds when determining if the battery was operating normally – and decided that the battery was much too cold. It took some time to persuade the vendor to restore the previous version of the firmware.
Remote firmware upgrade is subject to heated discussions, and can cause legal issues. Vendors of smart meters have to to separate the software that is required for ‘features’ – to be upgraded later, following ever changing standards and advances in technology – and the software associated with the data used in billing – subject to official calibration.
In case the vendor of the modems shown in the Shodan screenshot detects a vulnerability we would probably happy if they patch it immediately. Our favorite Things can be updated automatically and it went well so far.
____________________________________________________
Further reading:
Security Statement for Teamviewer – which also happens to be the software I am using for remote connections to clients’ computer systems and for remote meetings.
The Internet of Things, and how those Things phone home. An accessible and brief explanation of the different ways things allow for connections leveraged by a server on the internet.
Peer to Peer – Hole Punching – more detailed explanations.
Peer-to-Peer Communication Across Network Address Translators – even more detailed explanations, similar to this RFC by the same authors.
elkemental Force 
awhile back I received a AT&T M-Cell. the 1st thing i was annoyed with was the fact that there was no management interface or even a status page that could be accessed locally. the 2nd was the fact that it took like 5 hours to negotiate. a 3rd it’s a mysterious white box that you can’t open because it will self-destruct.
and 4th with no user configurable settings, it automatically sets the LAN IP address to 192.168.0.1 because it is essentially a router. and now it will conflict with any other router on that same IP without the users knowledge, whether it was upstream or downstream.
a couple years later they have since updated it, and now usually it will reconnect within an hour or less if it still maintains a GPS location when needing to be power cycled. and if not it’s about 1 ½ to 2 hours.
still no status information other than the blinking lights on the front.
they have since sort of fixed the IP address conflict by transparently passing the upstream IP address through, and it works better behind a router ( or 2 of mine and one of there’s ).
it doesn’t use too much data unless of course your on a cell phone call, but every so often it does occasionally download a update, I can see a spike in data usage with my ( open source, repurposed computer ) router.
I have since settled in on wireless cellular Internet for our home Internet, which means I pay attention to my data ( because I have to, and thus a need for a router with software for monitoring ). but of course I’m triple NATing because not only am I running my own router, the hotspot is also a router and the network is also a proxy’ed NAT.
so to add to all this, last year we bought a smart TV. it doesn’t download too much data ( unless you use Netflix ), just a few kilobytes here and there. and i am certainly not concerned about things coming in to my network.
BUT the TV uploads about 100 to 200 MB a month, just watching TV over the air!
I have to dig out wireshark one of these days and figure out what’s going on.
Thanks for sharing this! I think the difficult decisions for vendors of services is whether to give their power users more access permissions to a device or if this would put the service at risk. I am not familiar with micro-cells – I suppose the owner of the network considers the device part of their network and thus will not give end-users access (as with a residential internet router)?
But often it is debatable who should own the device or the data – I researched inverters for photovoltaic panels recently, and now all vendors run a web portal – so the devices send their data to the cloud service. I’d say the difference is that the primary service offered as such has nothing to do with IT but it is the solar power harvested by the user, so users should own their data and should be able to decide what is sent to the cloud. However, some vendors give the users only access to a subset of the data while they use all the detailed data for optimizing their system (and perhaps learning about the performance of their inverters under different conditions). So if you don’t like the cloud you cannot log your own data as there are no local interfaces.
As for the triple NATting: I have accidentally noticed recently that one of our (Austrian) mobile carriers have re-structured their networks. While it was possible to use a public IP address if you picked a specific access point, your device was suddenly hidden behind a NAT although no changes had been made to the contract or the configuration.
Always interesting to read these kind of articles. Just cleaning a network I inherited and found a telnet listening on the internet, … with not so good credentials. Difficult not to talk about incompetence of the predecessor. We have to remain polite.
15 years ago a customer bought gadget ip-cameras from a store around the corner, to protect his business against burglars, … but those used the cameras to monitor the guards in stead …
Didn’t know about Shodan. Don’t know how to use it, too lazy yet to read on it …
Perhaps I could construct an internet of things, with every tile and stone and window and door an ipv6 address built in. It’s possible, so sooner or later this will be implemented. Probably not by me :-)
Yes, RaspBerries everywhere. My thermostate is giving up, and I want to built a new one with the raspPi, … if i have the time …
Silence takes a lot of time these days. A raspPi is not essential thus skipped, for the moment. You keep your flow a.o. with your scythe. :-) I sometimes enjoy the days with its 2-takt version(s).
Have a great Sunday!!
Thanks, Bert!
I haven’t used the Raspberry Pi for heating control as we use the mentioned solution (which is also quite popular among tinkerers…). I have mainly played with Wolfram Alpha on the Pi so far.
But a friend recently showed me his R Pi web admin panel for his heater on his smart phone – he is an IT sysadmin but said about himself he had no programming background but found it suprisingly easy to build that solution.
Increasingly our lives are defined by the vast multitude of online systems and devices that we have surrounded ourselves with. Computers, smartphones and now the many, many tiny cheap gadgets that can integrate some functionality thanks to a $2 WiFi or bluetooth radio. Each time there’s some new little ability that bring. A fitness tracker, a system that warns you of intruders, systems that remind you when you need supplies whether it’s toner for the printer or bread for the fridge. the list is endless. Each time something new is added, though, just a bit of our privacy is removed, a little less skill and memory is required and, so it seems, a new opportunity is provided for some nefarious third party to profit at our expense. In so many ways it’s still the wild, wild west, isn’t it. I admit that a part of me likes it. I enjoy it when things are not neatly buttoned down, when you can still exercise some creative choices and do interesting and often novel things. i do fear, though, that for every interesting useful application there are ten of the other sort–vulnerabilities that can be creatively exploited by criminal organizations and always at my expense. Then again, I suppose that’s life–cause and effect, push and response, cat and mouse. At some point, though, when the vast amount of data that the likes of Google have been gathering on us start to be exploited to evil ends–and that day will come–things will surely start to get truly scary.
Haha, I hate the reminders for ink – the ones who require me to click away 10 legal disclaimers and if I am really sure I want to proceed with expired cartridges :-)
There are many interesting trends, some of the them conflicting: User interfaces keep getting simpler and more intuitive but the overall system keeps getting more complex due to an increasing number of interfaces. Then everything is turned into a subscription today, including the user’s physical devices, and there is all that legal complexity – even assuming there are no evil corporations and spying agencies.
Also providing heat energy has been turned into “subscription-based services” – it had become popular in recent years here: A contractor owns and maintains the heater at the user’s premises. What I learned about these contracts is that defining the legal framework is more complex than the technical infrastructure – which is already demanding enough as the user should be prevented from misconfiguring the system and from sabotaging the goals the contractor is legally bound to (e.g. in terms of energy demands)
I enjoyed this post; I read it earlier today and was thinking of it while we were getting things ready for installing new water lines inside the house. I’ve seen something like this, when running a Google search, and coming up with something that looked like it belonged in a private network, then not being able to find it again. I assumed it was something weird I had done, or misinterpreted.
Then, yesterday on WordPress it seemed that some one’s smart phone left a message in my comments:
a:hover { color: red; } a { text-decoration: none; color: #0088cc; } a.primaryactionlink:link, a.primaryactionlink:visited { background-color: #2585B2; color: #fff; } a.primaryactionlink:hover, a.primaryactionlink:active { background-color: #11729E !important; color: #fff !important; }
/* @media only screen and (max-device-width: 480px) { .post { min-width: 700px !important; } } */ WordPress.com
Thanks, Michelle! Some years ago it was even easier to find files with passwords and the like. I can also recall a journalist having been sued over downloading files (and utilized them as a professional) that had been online in error.
I wonder if the snippet of the stylesheet had probably been added to the comment because the commentator had responded via e-mail? Though it is rare sometimes multiple conversions of e-mail (when passing several mail servers, I suppose) result in HTML tags being displayed in the final e-mail.
That snippet actually was the entire comment, although I excluded personal data including the brand of phone, who it belongs to, email address and their own WordPress site url. The owner of the phone has never commented on this blog and I don’t know who it is…. there seemed to be no comment except the one made by the phone. I didn’t approve it to be published, owing to the fact that there was personal information in it and it had no relevance to the post.
I hadn’t heard about the journalist before; years ago, the comparable might be finding confidential documents in the trash that is left in a public space.
So my theory is now: A silent e-mail follower replied to a notification e-mail in error, leaving an empty message… and due to conversion issues it was not really empty but contained the meta data. Perhaps it is also a glitch in e-mail commenting and/or the settings of the e-mail client on that phone left the phone brand and the e-mail address as a signature – just in the right place WP would recognize this as a comment (“reply above this line”…)
The comparison with documents in the trash is spot-on – I don’t know if laws allow for exploiting those.
I like your theory, as it makes sense. I thought it was interesting when it happened, and I like that you posted on something that made it relevant for me to share it!
I think I’m worried! No … I shouldn’t be worried … big business should, right? And how about our power grid? Yup … I’m worried again! D
It is hard to calculate risks for businesses versus private users. Big business might be a more attractive target if the goal of an attacker is to steal specific data – but on the other hand big business worries more anyway.
But there are lots of interdependencies and new risks – e.g. a major vulnerability in residential routers owned by home users would make those an attractive target for attackers who could turn them into a botnet (http://en.wikipedia.org/wiki/Botnet). The problem are blurring boundaries between the ‘public’ and the ‘private’ infrastructure as more and more devices and appliances that the user believes to own are actually just part of a service agreement or subscription and effectively owned by a third party.
I still think it is possible – legally and technically – to define reasonable safeguards and well defined interfaces … that allow the user to stay in control. Of course more personal control over one’e devices might reduce convenience.