A Color Box. Lost in Translation

It was that time again.

The Chief Engineer had rebuilt the technical room from scratch. Each piece of heavy equipment had a new place, each pipe and wire was reborn in a new incarnation (German stories here.)

The control system was turned upset down as well, and thus the Data Kraken was looking at its entangled tentacles, utterly confused. The fabric of spacetime was broken again – the Kraken was painfully reminded of its last mutation in 2016.

Back then a firmware update had changed the structure of log files exported by Winsol. Now these changes were home-made. Sensors have been added. Sensor values have been added to the logging setup. Known values are written to different places in the log file. The log has more columns. The electrical input power of the heat pump has now a positive value, finally. Energy meters have been reset during the rebuild. More than once. And on and on.

But Data Kraken had provided for such disruptions! In a classical end-of-calendar-year death march project its software architecture had been overturned 2016. Here is a highly illustrative ‘executive level’ diagram:

The powerful SQL Server Kraken got a little companion – the Proto Kraken. Proto Kraken proudly runs on Microsoft Access. It comprises the blueprint of Big Kraken – its DNA: a documentation of all measured values, and their eternal history: When was the sensor installed, at which position in the log file do you find its values from day X to day Y, when was it retired, do the values need corrections …

A Powershell-powered tentacle crafts the Big Kraken from this proto version. It’s like scientists growing a mammoth from fossils: The database can be rebuilt from the original log files, no matter how the file structure mutates over time.

A different tentacle embraces the actual functions needed for data analysis – which is Data Kraken’s true calling. Kraken consolidates data from different loggers, and it has to do more than just calculating max / min / totals / averages. For example, the calculation of the heat pump’s performance factor had to mutate over time. Originally energy values had been read off manually from displays, but then the related meters were automated. Different tentacles need to reach out into different tables at different points of time.

Most ‘averages’ only make sense under certain conditions: The temperature at different points in the brine circuits should only contribute to an average temperature when the brine circulation pump is active. If you calculate the performance factor from heat source and target temperature (using a fit function), only time intervals may contribute when the heat pump did actually run.

I live in fear of Kraken’s artificial intelligence – will it ever gain consciousness? Will I wake up once in a science fiction dystopia? Fortunately, there is one remaining stumbling block: We have not yet fully automated genetic engineering. How could that ever work? A robot or a drone trying to follow the Chief Engineer’s tinkering with sensor wiring … and converting this video stream into standardized change alerts sent to Data Kraken?

After several paragraphs laden with silly metaphors, I finally come to the actual metaphor in the title of this post. The

Color Box

Once you came up with a code name for your project, you cannot get it out of your head. That also happened to the Color Box (Farbenkastl).

Here, tacky tasteless multi-colored things are called a color box. Clothes and interior design for example. Or the mixture of political parties in parliament. That’s probably rather boring, but the Austrian-German term Farbenkastl has a colorful history: It had been used in times of the monarchy to mock the overly complex system of color codes applied to the uniforms of the military.

What a metaphor for a truly imperial tool: As a precursor to the precursor to the Kraken Database … I use the Color Box! Brought to me by Microsoft Excel! I can combine my artistic streak, coloring categories of sensors and their mutations. Excel formulas spawn SQL code.

The antediluvian 2016 color box was boring:

But trying to display the 2018 color box I am hitting the limit of Excel’s zooming abilities:

I am now waiting now for the usual surprise nomination for an Science & Arts award. In the meantime, my Kraken enjoys its new toys. Again, the metaphoric power of this video is lost in translation as in German ‘Krake’ means octopus.

(We are still working at automating PVC piping via the Data Kraken, using 3D printing.)

My Data Kraken – a Shapeshifter

I wonder if Data Kraken is only used by German speakers who translate our hackneyed Datenkrake – is it a word like eigenvector?

Anyway, I need this animal metaphor, despite this post is not about facebook or Google. It’s about my personal Data Kraken – which is a true shapeshifter like all octopuses are:

(… because they are spineless, but I don’t want to over-interpret the metaphor…)

Data Kraken’s shapeability is a blessing, given ongoing challenges:

When the Chief Engineer is fighting with other intimidating life-forms in our habitat, he focuses on survival first and foremost … and sometimes he forgets to inform the Chief Science Officer about fundamental changes to our landscape of sensors. Then Data Kraken has to be trained again to learn how to detect if the heat pump is on or off in a specific timeslot. Use the signal sent from control to the heat pump? Or to the brine pump? Or better use brine flow and temperature difference?

It might seem like a dull and tedious exercise to calculate ‘averages’ and other performance indicators that require only very simple arithmetics. But with the exception of room or ambient temperature most of the ‘averages’ just make sense if some condition is met, like: The heating water inlet temperature should only be calculated when the heating circuit pump is on. But the temperature of the cold water, when the same floor loops are used for cooling in summer, should not be included in this average of ‘heating water temperature’. Above all, false sensor readings, like 0, NULL or any value (like 999) a vendor chooses to indicate as an error, have to be excluded. And sometimes I rediscover eternal truths like the ratio of averages not being equal to the average of ratios.

The Chief Engineer is tinkering with new sensors all the time: In parallel to using the old & robust analog sensor for measuring the water level in the tank…

Level sensor: The old way

… a multitude of level sensors was evaluated …

Level sensors: The precursors

… until finally Mr. Bubble won the casting …


… and the surface level is now measured via the pressure increasing linearly with depth. For the Big Data Department this means to add some new fields to the Kraken database, calculate new averages … and to smoothly transition from the volume of ice calculated from ruler readings to the new values.

Change is the only constant in the universe, paraphrasing Heraclitus [*]. Sensors morph in purpose: The heating circuit, formerly known (to the control unit) as the radiator circuit became a new wall heating circuit, and the radiator circuit was virtually reborn as a new circuit.

I am also guilty of adding new tentacles all the time, too, herding a zoo of meters added in 2015, each of them adding a new log file, containing data taken at different points of time in different intervals. This year I let Kraken put tentacles into the heat pump:

Data Kraken: Tentacles in the heat pump!

But the most challenging data source to integrate is the most unassuming source of logging data: The small list of the data that The Chief Engineer had recorded manually until recently (until the advent of Miss Pi CAN Sniffer and Mr Bubble). Reason: He had refused to take data at exactly 00:00:00 every single day, so learned things I never wanted to know about SQL programming languages to deal with the odd time intervals.

To be fair, the Chief Engineer has been dedicated at data recording! He never shunned true challenges, like a legendary white-out in our garden, at the time when measuring ground temperatures was not automated yet:

The challenge

White Out

Long-term readers of this blog know that ‘elkement’ stands for a combination of nerd and luddite, so I try to merge a dinosaur scripting approach with real-world global AI Data Krakens’ wildest dream: I wrote scripts that create scripts that create scripts [[[…]]] that were based on a small proto-Kraken – a nice-to-use documentation database containing the history of sensors and calculations.

The mutated Kraken is able to eat all kinds of log files, including clients’ ones, and above all, it can be cloned easily.

I’ve added all the images and anecdotes to justify why an unpretentious user interface like the following is my true Christmas present to myself – ‘easily clickable’ calculated performance data for days, months, years, and heating seasons.

Data Kraken: UI

… and diagrams that can be changed automatically, by selecting interesting parameters and time frames:

Excel for visualization of measurement data

The major overhaul of Data Kraken turned out to be prescient as a seemingly innocuous firmware upgrade just changed not only log file naming conventions and publication scheduled but also shuffled all the fields in log files. My Data Kraken has to be capable to rebuild the SQL database from scratch, based on a documentation of those ever changing fields and the raw log files.


[*] It was hard to find the true original quote for that, as the internet is cluttered with change management coaches using that quote, and Heraclitus speaks to us only through secondary sources. But anyway, what this philosophy website says about Heraclitus applies very well to my Data Kraken:

The exact interpretation of these doctrines is controversial, as is the inference often drawn from this theory that in the world as Heraclitus conceives it contradictory propositions must be true.

In my world, I also need to deal with intriguing ambiguity!