Modbus Server on Raspberry Pi as Babelfish for UVR16x2

Our main data logger is the Control and Monitoring Interface of the freely programmable controller UVR16x2. There are two pieces of hardware you need for logging – the actual control unit and the logger connected to the controller via the CAN bus. This ‘architecture’ might be due to historical reasons, but I like the separation of critical control functions and nice-to-have logging. An operating system for control should do just that – I don’t want a web server and TCP/IP on the controller itself.

But sometimes it might be useful to send the controller a non-critical ‘nice-to-have’ signal. If you want to optimize the usage of photovoltaic energy, the controller should know about the PV ‘production’ and current consumption. As the CMI supports Modbus inputs and our Fronius Symo inverter has a Modbus interface, PV ‘generation’ data can be logged via Modbus. However, our smart meter has ‘only’ a JSON interface – and you can always scrape data from the website as last resort. But while the CMI is a web server with visualization a JSON API, it cannot log ‘from’ JSON.

My Data Kraken is a SQL Server that is fed log files from different data loggers – the largest one from the CMI, and others are from our smart power meter’s logger, from our photovoltaic inverter, and from the heat pump’s internal CAN bus. Separate log files provide for built-in backup – but what if I could send these data to the UVR16x2 in real time in addition?

Currently different loggers log data at different points of time, so you cannot simply join all these tables together. Of course you can simply overlay the data without the time stamps being exactly equal. As I am mainly interested in daily consumption or performance data, the Data Kraken creates a series of nested views, finally spitting out one row per day, and those views for different loggers can be joined again. So far I could not, say, calculate the photovoltaic power used directly at each instant.

I was looking for … a babel fish! A universal protocol translator that …

  • collects data from other sources which UVR cannot directly process.
  • processes these data and calculates derived values, without having to ‘waste’ UVR16x2’s building blocks for functions on that.
  • and provides these data to UVR16x2 in a proper format as an input relevant for control.

As I had already tested logging from Modbus sources, the logical solution was:

Starting my own Modbus server at the Raspberry Pi we already use for logging from the heat pump’s CAN bus.

The main units of our control infrastructure – after the Chief Engineer’s heroic rebuild of the technical room last year. Top right: Raspberry Pi 3 B – with a USB stick for logging (This extends the life time of the SD card a lot ;-)), second ‘row’ UVR16x2, third row: UVR1611 – most of its former functions have been migrated to UVR16x2.

There is a great python library called pymodbus which makes this task easy:

You write values to your own registers by building a payload from values in different formats. Here is an example using the payload builder to create a ‘stack’ of registers containing payloads of different types. The payload builder can be fed values retrieved from another logger, or created by whatever complex calculation. These values are then ‘sent out’ via your Modbus server interface. That infamous usage of weather forecast data from some public API for optimizing battery storage could theoretically be one application for that.

I update the values ‘internally’ using the setvalues method of the context, like this. Another option would be to create a client connecting to your own local server and writing to the registers.

In order to change the values while the Modbus server runs without interruption, Python threading comes in handy. The ‘context’ is a global object, and one thread constantly updates the values in an infinite loop, and the other thread is for running the actual TCP server. Using log.debug lets you follow the details of the communication on the server – and helps with sorting out any byte ordering issues 😉

On the CMI you configure the related inputs connecting to your Raspberry Pi Modbus server:

Configuration of Modbus inputs at the CMI. Input 1 is provided by the the PV inverter, inputs >= 10 by the Modbus server running on Raspberry Pi. Values are either ‘forwarded’ from the smart meter EM210 (via JSON), forwarded again from the PV inverter (just for comparison), or they are calculated. E.g. the direct consumption of PV power is calculated from the current PV ‘production’ and the power delivered to the grid as measured by the meter.

Currently the CMI reads the following from my ‘fake’ Modbus device:

Net electrical power measured by the smart meter: Total power for all three AC phases, and each phase separately (OBIS object model – “*2.8.0” and “*1.8.0”).

PV output power: The CMI also logs data from the PV inverter directly. This is a double Modbus hop ‘for fun’!

PV power used immediately and electrical power used in total in the building: Calculated from the smart meter’s output values (via JSON) and the PV inverter’s output (via Modbus).

PV utilization: The directly used PV power is related to either power consumed or power ‘generated’ right now. Over a season these values are ~30% (self-sufficiency quota) and ~35 (self-consumption quota).

Calculations based on the ‘current’ PV power should not be taken too seriously as you might encounter those alien “>100%” spikes. The default log files with 5-minute averages still makes sense.

To actually log these values to CMI’s log file and to use them as control input requires the configuration of a CAN input at the UVR, as described here.

The Babelfish is now translating from Modbus to Modbus, from JSON to Modbus, and from ‘internal calculations’ to Modbus – and it can be extended to translate from a foreign CAN bus to UVR’s in a sense.

Frequently asked question:

How can I log from the heat pump’s internal CAN bus to UVR16x2? It’s all CAN buses, why don’t you simply connect them instead of having a separate CAN logger installed on a Pi?

‘Connected to CAN bus’ does not necessarily mean that those devices speak the same dialect at all network levels of the protocol. While UVR uses CANopen, the heat pump’s CAN protocol is proprietary (though similar to CANopen). In addition, you might want to separate the control bus for the heat pump from your standard CAN bus. What if an exotic issue with one of the devices confuses the heat pump? I remember how an IP TV solution broke the CMI’s predecessor BL-NET, and the solution was to put the CMI into a different subnet, behind another router.

My Modbus Babel fish gathers the heat pump’s values from its internal can bus and saves them to a CSV file (… which is then fed to the SQL Server Kraken). The values could as well be served via my Modbus server.

The final glorious chain looks like this 🙂

  • CAN bus heat pump –> CAN sniffer on Raspberry Pi
  • Python Modbus Server on Raspberry Pi –> Modbus input on CMI
  • CAN output on the CMI –> CAN input on UVR16x2
  • –> UVR uses CAN input in a control function
  • –> CMI logs this value from the CAN bus ‘again’ –> into its log file

A Color Box. Lost in Translation

It was that time again.

The Chief Engineer had rebuilt the technical room from scratch. Each piece of heavy equipment had a new place, each pipe and wire was reborn in a new incarnation (German stories here.)

The control system was turned upset down as well, and thus the Data Kraken was looking at its entangled tentacles, utterly confused. The fabric of spacetime was broken again – the Kraken was painfully reminded of its last mutation in 2016.

Back then a firmware update had changed the structure of log files exported by Winsol. Now these changes were home-made. Sensors have been added. Sensor values have been added to the logging setup. Known values are written to different places in the log file. The log has more columns. The electrical input power of the heat pump has now a positive value, finally. Energy meters have been reset during the rebuild. More than once. And on and on.

But Data Kraken had provided for such disruptions! In a classical end-of-calendar-year death march project its software architecture had been overturned 2016. Here is a highly illustrative ‘executive level’ diagram:

The powerful SQL Server Kraken got a little companion – the Proto Kraken. Proto Kraken proudly runs on Microsoft Access. It comprises the blueprint of Big Kraken – its DNA: a documentation of all measured values, and their eternal history: When was the sensor installed, at which position in the log file do you find its values from day X to day Y, when was it retired, do the values need corrections …

A Powershell-powered tentacle crafts the Big Kraken from this proto version. It’s like scientists growing a mammoth from fossils: The database can be rebuilt from the original log files, no matter how the file structure mutates over time.

A different tentacle embraces the actual functions needed for data analysis – which is Data Kraken’s true calling. Kraken consolidates data from different loggers, and it has to do more than just calculating max / min / totals / averages. For example, the calculation of the heat pump’s performance factor had to mutate over time. Originally energy values had been read off manually from displays, but then the related meters were automated. Different tentacles need to reach out into different tables at different points of time.

Most ‘averages’ only make sense under certain conditions: The temperature at different points in the brine circuits should only contribute to an average temperature when the brine circulation pump is active. If you calculate the performance factor from heat source and target temperature (using a fit function), only time intervals may contribute when the heat pump did actually run.

I live in fear of Kraken’s artificial intelligence – will it ever gain consciousness? Will I wake up once in a science fiction dystopia? Fortunately, there is one remaining stumbling block: We have not yet fully automated genetic engineering. How could that ever work? A robot or a drone trying to follow the Chief Engineer’s tinkering with sensor wiring … and converting this video stream into standardized change alerts sent to Data Kraken?

After several paragraphs laden with silly metaphors, I finally come to the actual metaphor in the title of this post. The

Color Box

Once you came up with a code name for your project, you cannot get it out of your head. That also happened to the Color Box (Farbenkastl).

Here, tacky tasteless multi-colored things are called a color box. Clothes and interior design for example. Or the mixture of political parties in parliament. That’s probably rather boring, but the Austrian-German term Farbenkastl has a colorful history: It had been used in times of the monarchy to mock the overly complex system of color codes applied to the uniforms of the military.

What a metaphor for a truly imperial tool: As a precursor to the precursor to the Kraken Database … I use the Color Box! Brought to me by Microsoft Excel! I can combine my artistic streak, coloring categories of sensors and their mutations. Excel formulas spawn SQL code.

The antediluvian 2016 color box was boring:

But trying to display the 2018 color box I am hitting the limit of Excel’s zooming abilities:

I am now waiting now for the usual surprise nomination for an Science & Arts award. In the meantime, my Kraken enjoys its new toys. Again, the metaphoric power of this video is lost in translation as in German ‘Krake’ means octopus.

(We are still working at automating PVC piping via the Data Kraken, using 3D printing.)

Where Are the Files? [Winsol – UVR16x2]

Recently somebody has asked me where the log files are stored. This question is more interesting then it seems.

We are using the freely programmable controller UVR16x2 (and its predecessor) UVR1611) …

.. and their Control and Monitoring Interface – CMI:The CMI is a data logger and runs a web server. It logs data from the controllers (and other devices) via CAN bus – I have demonstrated this in a contrived example recently, and described the whole setup in this older post.

IT / smart home nerds asked me why there are two ‘boxes’ as other solutions only use a ‘single box’ as both controller and logger. I believe separating these functions is safer and more secure: A logger / web server should not be vital to run the controller, and any issues with these auxiliary components must impact the controller’s core functions.

Log files are stored on the CMI in a proprietary format, and they can retrieved via HTTP using the software Winsol. Winsol lets you visualize data for 1 or more days, zoom in, define views etc. – and data can be exported as CSV files. This is the tool we use for reverse engineering hydraulics and control logic (German blog post about remote hydraulics surgery):

In the latest versions of Winsol, log files are per default stored in the user’s profile on Windows:
C:\Users\[Username]\Documents\Technische Alternative\Winsol

I had never paid much attention to this; I had always changed that path in the configuration to make backup and automation easier. The current question about the log files’ location was actually about how I managed to make different users work with the same log files.

The answer might not be obvious because of the historical location of the log files:

Until some version of Winsol in use in 2017 log files were by stored in the Program Files folder, or at least Winsol tried to use that folder. Windows does not allow this anymore for security reasons.

If Winsol is upgraded from an older version, settings might be preserved. I did my tests  with Winsol 2.07 upgraded from an earlier version. I am a bit vague about versions as I did not test different upgrade paths in detail My point is users of control system’s software tend to be conservative when it comes to changing a running system – an older ‘logging PC’ with an older or upgraded version of Winsol is not an unlikely setup.

I started debugging on Windows 10 with the new security feature Controlled Folder Access enabled. CFA, of course, did not know Winsol, considered it an unfriendly app … to be white-listed.

Then I was curious about the default log file folders, and I saw this:

In the Winsol file picker dialogue (to the right) the log folders seem to be in the Program Files folder:
C:\Program Files\Technische Alternative\Winsol\LogX
But in Windows Explorer (to the left) there are no log files at that location.

What does Microsoft Sysinternals Process Monitor say?

There is a Reparse Point, and the file access is redirected to the folder:
C:\Users\[User]\AppData\Local\VirtualStore\Program Files\Technische Alternative\Winsol
Selecting this folder directly in Windows Explorer shows the missing files:

This location can be re-configured in Winsol to allow different users to access the same files (Disclaimer: Perhaps unsupported by the vendor…)

And there are also some truly user-specific configuration files in the user’s profile, in
C:\Users\[User]\AppData\Roaming\Technische Alternative\Winsol

Winsol.xml is e.g. for storing the list of ‘clients’ (logging profiles) that are included in automated processing of log files, and cookie.txt is the logon cookie for access to the online logging portal provided by Technische Alternative. If you absolutely want to switch Windows users *and* switch logging profiles often *and* sync those you have to tinker with Winsol.xml, e.g. by editing it using a script (Disclaimer again: Unlikely to be a supported way of doing things ;-))

As a summary, I describe the steps required to migrate Winsol’s configuration to a new PC and prepare it for usage by different users.

  • Install the latest version of Winsol on the target PC.
  • If you use Controlled Folder Access on Windows 10: Exempt Winsol as a friendly app.
  • Copy the contents of C:\Users\[User]\AppData\Roaming\Technische Alternative\Winsol from the user’s profile on the old machine to the new machine (user-specific config files).
  • If the log file folder shows up at a different path on the two machines – for example when using the same folder via a network share – edit the path in Winsol.xml or configure it in General Settings in Winsol.
  • Copy your existing log data to this new path. LogX contains the main log files, Infosol contain clients’ data. The logging configuration for each client, e.g. the IP address or portal name of the logger, is included in the setup.xml file in the root of each client’s folder.

Note: If you skip some Winsol versions on migrating/upgrading the structure of files might have changed – be careful! Last time that happened by the end of 2016 and Data Kraken had to re-configure some tentacles.

Logging Fun with UVR16x2: Photovoltaic Generator – Modbus – CAN Bus

The Data Kraken wants to grow new tentacles.

I am playing with the CMI – Control and Monitoring Interface – the logger / ‘ethernet gateway’ connected to our control units (UVR1611, UVR16x2) via CAN bus. The CMI has become a little Data Kraken itself: Inputs and outputs can be created for CAN bus and Modbus, and data from most CAN devices can also be logged via JSON.

Are these features useful to integrate the datalogger of our photovoltaics inverter – Fronius Symo 4.5-3-M? I am now logging data to an USB stick and feed the CSV files to the SQL Server Data Kraken. The USB logger’s logging interval is 5 minutes whereas Modbus TCP allows for logging every few seconds. The inverter has built-in energy management features, but it can only ‘signal’ via a relay which also requires proper wiring. Modbus TCP, on the other hand, could use the existing WLAN connection of the inverter and the control unit could do something smarter with the sensor reading of the output power.

My motivation is to test if you – as an UVR16x2 user – can re-use a logger you  already have – the CMI – as much as possible, avoiding the need to run another ‘logging server’ all the time (Also my SQL Server is for analysis, not for real-time logging). I know that there are many open source Modbus clients available and that it is easy to write a Python script.

Activate Modbus on the inverter: I prefer floating numbers to integers plus a scaling factor, and I turn off the option to make changes via Modbus:

Modbus settings, Fronius Datalogger, inverter’s local web server. 502 is the TCO standard port. The alternative to floating numbers is integers plus a varying scaling factor (SF), to be found in another register.

Check Fronius documentation of its Modbus registers: The document is currently available here (Link last checked and updated 2019-01). There are different sets of registers related to the inverter or associated with one string of PV panels:

PDF p.30, Common Inverter Model. For logging AC output power you need:  Address 40092, type of register 3 (read and hold), datatype float32 (‘corresponds’ to two 16bit integer register, thus size 2).

The address to be configured on a Modbus client is smaller than this address by 1 – so 40091 needs to be set to log AC power.

Using these configuration parameters an analog Modbus Input is added at the CMI. The signal is ‘digital’ – but in field-bus-language everything that is not a single bit – 0/1 – seems to be ‘analog’.

Modbus input at the CMI. Input value:  32bits read from the bus interpreted as an integer. Actual value: Integer part of the ‘true value’ = the 32bits interpreted as 32-bit float.

Yes, I checked the network trace 😉 as the byte order dropdown menu confused me: According to the Modbus protocol specification Big Endian is required, not an option.

Factors and data types: Only integer values are understood by CAN devices. Decimal places might be indicated by a scaling factor. The PV power value in Watt has enough significant digits; so the integer part of the float number is fine. But for current in Ampere – typically about 15A maximum – a Factor of 10 would be better. It would not have helped to select int + scaling factor at the inverter: The scaling factor would be stored in a second register, there is a different factor for every parameter, and you cannot configure another ‘scaling factor register’ per input at the CMI. Theoretically you could log the scaling factor separately and re-scale the value in a custom application – but then I would use a separate, custom logger.

In any case, if you screw this up, you see non-sensible numbers of the CAN bus: Slowly evolving positive values – like PV power on a sunny day – are displayed as wild variations of signed integers between -32000 and 32000 😉

Where are the ‘logged’ data? The CMI is first and foremost the data logger for the control units. The CMI does not immediately store the data from Modbus inputs in a  local ‘logging database’. All I have achieved so far is to display the value on the Settings page. The CMI can only log values from the CAN bus or DL bus. So we need an…

… Analog CAN Output at the CMI:

The CMI has the default node number 56 on the CAN bus. Other CAN devices on the bus can query it for this parameter by specifying node 56 and output no 1.

These are the devices on our CAN bus:

CAN bus displayed on the CMI’s website. UVR1611 and UVR16x2 controllers can be managed by clicking their icons – which brings up a web page that resembles the controller’s local display.

The CMI’s Logging page looks tempting – can we simply select the CMI itself as a CAN logging source – CAN 56?

Configuration of the devices the CMI logs data from, via CAN bus. CAN 1 – UVR1611, CAN 2 – UVR16x3, CAN 41 – energy meter CAN-EZ.

Nothing stops you from selecting CAN 56 in this dropdown menu, but it does not end well:

CAN error message displayed at the logger CMI when you try to configure the CMI also as a logging source.

We need a round-trip: Data needs to be sent to a supported device first – one of the controllers on the CAN bus. We need an…

Analog CAN input / network variable at the UVR16x2:

Configuration of a CAN input at the controller UVR16x2 (via CMI’s web interface to the controller).

The value of AC power is displayed as integer without scaling. Had a factor of 10 been used at the Modbus input it would be ‘corrected’ here, using the Unit called dimensionless,1.

logging-uvr16x2-can-network-input-can-value-display

Values received by the controller UVR16x2 over CAN bus.

Result of all this: UVR16x2 knows PV power and can use it do magic smart things when controlling the heat pump. On the other hand, CMI can log this value – in the same way it logs all other sensor readings (after an update of the logging settings in the controller’s functional data, using TAPPS).

Log files are retrieved by Winsol, the free logging software for the CMI …

Logged visualized with Winsol. Logfiles are downloaded from the CMI on the internal LAN or via Technsche Alternative’s web portal. PV power (PV.Leistung.Watt) is displayed together with global radiation on a vertical plane (GBS, at the solar/air collector for the theat pump), ambient temperature (red), temperature of solar/air collector (orange)

… or logging is configured at the web portal cmi.ta.co.at …

Configuration of logging at cmi.ta.co.at: Supported loggers are UVR1611 and UVR16x2. Values to be logged are selected from all direct inputs / outputs / functions and from CAN network inputs and outputs.

… and data can be viewed online:

Data visualized at cmi.ta.co.at. Data logged via CAN are sent from the CMI to the web portal.

Using this kind of logging for all values the inverter provides would be costly: It’s not just a column you add to a log file, but you occupy one of the limited inputs and outputs at the CMI and the controller. If you really need to know the voltage between phase 1 and 2 or apparent power you better stick with the USB file or use a separate Modbus logger like a Rasbperry Pi. This project is great and documented very well – data acqusition from a Symo inverter using Python plus a web front end.

Sending Modbus data back and forth from the CMI to UVR controllers is only worth the efforts if you need them for control, not for ‘nice-to-have’ logging.

Data for the Heat Pump System: Heating Season 2016-2017

I update the documentation of measurement data [PDF] about twice a year. This post is to provide a quick overview for the past season.

The PDF also contains the technical configuration and sizing data. Based on typical questions from an ‘international audience’ I add a summary here plus some ‘cultural’ context:

Building: The house is a renovated, nearly 100-year old building in Eastern Austria: a typical so-called ‘Streckhof’ – an elongated, former small farmhouse. Some details are mentioned here. Heating energy for space heating of two storeys (185m2) and hot water is about 17.000-20.000kWh per year. The roof / attic had been rebuilt in 2008, and the facade was thermally insulated. However, the major part of the house is without an underground level, so most energy is lost via ground. Heating only the ground floor (75m2) with the heat pump reduces heating energy only by 1/3.

Climate: This is the sunniest region of Austria – the lowlands of the Pannonian Plain bordering Hungary. We have Pannonian ‘continental’ climate with low precipitation. Normally, monthly average temperatures in winter are only slightly below 0°C in January, and weeks of ‘ice days’ in a row are very rare.

Heat energy distribution and storage (in the house): The renovated first floor has floor loops while at the ground floor mainly radiators are used. Wall heating has been installed in one room so far. A buffer tank is used for the heating water as this is a simple ‘on-off’ heat pump always operating at about its rated power. Domestic hot water is heated indirectly using a hygienic storage tank.

Heating system. An off-the-shelf, simple brine-water heat pump uses a combination of an unglazed solar-air collector and an underwater water tank as a heat source. Energy is mainly harvested from rather cold air via convection.

Addressing often asked questions: Off-the-shelf =  Same type of heat pump as used with geothermal systems. Simple: Not-smart, not trying to be the universal energy management system, as the smartness in our own control unit and logic for managing the heat source(s). Brine: A mixture of glycol and water (similar to the fluid used with flat solar thermal collectors) = antifreeze as the temperature of brine is below 0°C in winter. The tank is not a seasonal energy storage but a buffer for days or weeks. In this post hydraulics is described in detail, and typical operating conditions throughout a year. Both tank and collector are needed: The tank provides a buffer of latent energy during ‘ice periods’ and it allows to harvest more energy from air, but the collector actually provides for about 75% of the total ambient energy the heat pump needs in a season.

Tank and collector are rather generously sized in relation to the heating demands: about 25m3 volume of water (total volume +10% freezing reserve) and 24m2 collector area.

The overall history of data documented in the PDF also reflects ongoing changes and some experiments, like heating the first floor with a wood stove, toggling the effective area of the collector used between 50% and 100%, or switching off the collector to simulate a harsher winter.

Data for the past season

Finally we could create a giant ice cube naturally. 14m3 of ice had been created in the coldest January since 30 years. The monthly average temperature was -3,6°C, 3 degrees below the long-term average.

(Re the oscillations of the ice volume are see here and here.)

We heated only the ground floor in this season and needed 16.600 kWh (incl. hot water) – about the same heating energy as in the previous season. On the other hand, we also used only half of the collector – 12m2. The heating water inlet temperatures for radiators was even 37°C in January.

For the first time the monthly performance factor was well below 4. The performance factor is the ratio of output heating energy and input electrical energy for heat pump and brine pump. In middle Europe we measure both energies in kWh 😉 The overall seasonal performance factor was 4,3.

The monthly performance factor is a bit lower again in summer, when only hot water is heated (and thus the heat pump’s COP is lower because of the higher target temperature).

Per day we needed about 100kWh of heating energy in January, while the collector could not harvest that much:

In contrast to the season of the Ice Storage Challenge, also the month before the ‘challenge’ (Dec. 2016) was not too collector-friendly. But when the ice melted again, we saw the usual large energy harvests. Overall, the collector could contribute not the full ‘typical’ 75% of ambient energy this season.

(Definitions, sign conventions explained here.)

But there was one positive record, too. In a hot summer of 2017 we consumed the highest cooling energy so far – about 600kWh. The floor loops are used for passive cooling; the heating buffer tank is used to transfer heat from the floor loops to the cold underground tank. In ‘colder’ summer nights the collector is in turn used to cool the tank, and every time hot tap water is heated up the tank is cooled, too.

Of course the available cooling power is just a small fraction of what an AC system for the theoretical cooling load would provide for. However, this moderate cooling is just what – for me – makes the difference between unbearable and OK on really hot days with more than 35°C peak ambient temperature.

My Data Kraken – a Shapeshifter

I wonder if Data Kraken is only used by German speakers who translate our hackneyed Datenkrake – is it a word like eigenvector?

Anyway, I need this animal metaphor, despite this post is not about facebook or Google. It’s about my personal Data Kraken – which is a true shapeshifter like all octopuses are:

(… because they are spineless, but I don’t want to over-interpret the metaphor…)

Data Kraken’s shapeability is a blessing, given ongoing challenges:

When the Chief Engineer is fighting with other intimidating life-forms in our habitat, he focuses on survival first and foremost … and sometimes he forgets to inform the Chief Science Officer about fundamental changes to our landscape of sensors. Then Data Kraken has to be trained again to learn how to detect if the heat pump is on or off in a specific timeslot. Use the signal sent from control to the heat pump? Or to the brine pump? Or better use brine flow and temperature difference?

It might seem like a dull and tedious exercise to calculate ‘averages’ and other performance indicators that require only very simple arithmetics. But with the exception of room or ambient temperature most of the ‘averages’ just make sense if some condition is met, like: The heating water inlet temperature should only be calculated when the heating circuit pump is on. But the temperature of the cold water, when the same floor loops are used for cooling in summer, should not be included in this average of ‘heating water temperature’. Above all, false sensor readings, like 0, NULL or any value (like 999) a vendor chooses to indicate as an error, have to be excluded. And sometimes I rediscover eternal truths like the ratio of averages not being equal to the average of ratios.

The Chief Engineer is tinkering with new sensors all the time: In parallel to using the old & robust analog sensor for measuring the water level in the tank…

Level sensor: The old way

… a multitude of level sensors was evaluated …

Level sensors: The precursors

… until finally Mr. Bubble won the casting …

blubber-messrohr-3

… and the surface level is now measured via the pressure increasing linearly with depth. For the Big Data Department this means to add some new fields to the Kraken database, calculate new averages … and to smoothly transition from the volume of ice calculated from ruler readings to the new values.

Change is the only constant in the universe, paraphrasing Heraclitus [*]. Sensors morph in purpose: The heating circuit, formerly known (to the control unit) as the radiator circuit became a new wall heating circuit, and the radiator circuit was virtually reborn as a new circuit.

I am also guilty of adding new tentacles all the time, too, herding a zoo of meters added in 2015, each of them adding a new log file, containing data taken at different points of time in different intervals. This year I let Kraken put tentacles into the heat pump:

Data Kraken: Tentacles in the heat pump!

But the most challenging data source to integrate is the most unassuming source of logging data: The small list of the data that The Chief Engineer had recorded manually until recently (until the advent of Miss Pi CAN Sniffer and Mr Bubble). Reason: He had refused to take data at exactly 00:00:00 every single day, so learned things I never wanted to know about SQL programming languages to deal with the odd time intervals.

To be fair, the Chief Engineer has been dedicated at data recording! He never shunned true challenges, like a legendary white-out in our garden, at the time when measuring ground temperatures was not automated yet:

The challenge

White Out

Long-term readers of this blog know that ‘elkement’ stands for a combination of nerd and luddite, so I try to merge a dinosaur scripting approach with real-world global AI Data Krakens’ wildest dream: I wrote scripts that create scripts that create scripts [[[…]]] that were based on a small proto-Kraken – a nice-to-use documentation database containing the history of sensors and calculations.

The mutated Kraken is able to eat all kinds of log files, including clients’ ones, and above all, it can be cloned easily.

I’ve added all the images and anecdotes to justify why an unpretentious user interface like the following is my true Christmas present to myself – ‘easily clickable’ calculated performance data for days, months, years, and heating seasons.

Data Kraken: UI

… and diagrams that can be changed automatically, by selecting interesting parameters and time frames:

Excel for visualization of measurement data

The major overhaul of Data Kraken turned out to be prescient as a seemingly innocuous firmware upgrade just changed not only log file naming conventions and publication scheduled but also shuffled all the fields in log files. My Data Kraken has to be capable to rebuild the SQL database from scratch, based on a documentation of those ever changing fields and the raw log files.

_________________________________

[*] It was hard to find the true original quote for that, as the internet is cluttered with change management coaches using that quote, and Heraclitus speaks to us only through secondary sources. But anyway, what this philosophy website says about Heraclitus applies very well to my Data Kraken:

The exact interpretation of these doctrines is controversial, as is the inference often drawn from this theory that in the world as Heraclitus conceives it contradictory propositions must be true.

In my world, I also need to deal with intriguing ambiguity!

Same Procedure as Every Autumn: New Data for the Heat Pump System

October – time for updating documentation of the heat pump system again! Consolidated data are available in this PDF document.

In the last season there were no special experiments – like last year’s Ice Storage Challenge or using the wood stove. Winter was rather mild, so we needed only ~16.700kWh for space heating plus hot water heating. In the coldest season so far – 2012/13 – the equivalent energy value was ~19.700kWh. The house is located in Eastern Austria, has been built in the 1920s, and has 185m2 floor space since the last major renovation.

(More cross-cultural info:  I use thousands dots and decimal commas).

The seasonal performance factor was about 4,6 [kWh/kWh] – thus the electrical input energy was about 16.700kWh / 4,6 ~ 3.600kWh.

Note: Hot water heating is included and we use flat radiators requiring a higher water supply temperature than the floor heating loops in the new part of the house.

Heating season 2015/2016: Performance data for the 'ice-storage-/solar-powered' heat pump system

Red: Heating energy ‘produced’ by the heat pump – for space heating and hot water heating. Yellow: Electrical input energy. Green: Performance Factor = Ratio of these energies.

The difference of 16.700kWh – 3.600kWh = 13.100kWh was provided by ambient energy, extracted from our heat source – a combination of underground water/ice tank and an unglazed ribbed pipe solar/air collector.

The solar/air collector has delivered the greater part of the ambient energy, about 10.500kWh:

Heating season 2015/2016: Energy harvested from air by the collector versus heating-energy

Energy needed for heating per day (heat pump output) versus energy from the solar/air collector – the main part of the heat pump’s input energy. Negative collector energies indicate passive cooling periods in summer.

Peak Ice was 7 cubic meters, after one cold spell of weather in January:

Heating season 2015/2016: Temperature of ambient air, water tank (heat source) and volume of water frozen in the tank.

Ice is formed in the water tank when the energy from the collector is not sufficient to power the heat pump alone, when ambient air temperatures are close to 0°C.

Last autumn’s analysis on economics is still valid: Natural gas is three times as cheap as electricity but with a performance factor well above three heating costs with this system are lower than they would be with a gas boiler.

Is there anything that changed gradually during all these years and which does not primarily depend on climate? We reduced energy for hot tap water heating – having tweaked water heating schedule gradually: Water is heated up once per day and as late as possible, to avoid cooling off the hot storage tank during the night.

We have now started the fifth heating season. This marks also the fifth anniversary of the day we switched on the first ‘test’ version 1.0 of the system, one year before version 2.0.

It’s been about seven years since first numerical simulations, four years since I have been asked if I was serious in trading in IT security for heat pumps, and one year since I tweeted: