Other People Have Lives – I Have Domains

These are just some boring update notifications from the elkemental Webiverse.

The elkement blog has recently celebrated its fifth anniversary, and the punktwissen blog will turn five in December. Time to celebrate this – with new domain names that says exactly what these sites are – the ‘elkement.blog‘ and the ‘punktwissen.blog‘.

Actually, I wanted to get rid of the ads on both blogs, and with the upgrade came a free domain. WordPress has a detailed cookie policy – and I am showing it dutifully using the respective widget, but they have to defer to their partners when it comes to third-party cookies. I only want to worry about research cookies set by Twitter and Facebook, but not by ad providers, and I am also considering to remove social media sharing buttons and the embedded tweets. (Yes, I am thinking about this!)

On the websites under my control I went full dinosaur, and the server sends only non-interactive HTML pages sent to the client, not requiring any client-side activity. I now got rid of the last half-hearted usage of a session object and the respective cookie, and I have never used any social media buttons or other tracking.

So there are no login data or cookies to protect, but yet I finally migrated all sites to HTTPS.

It is a matter of principle: I of all website owners should use https. Since 15 years I have been planning and building Public Key Infrastructures and troubleshooting X.509 certificates.

But of course I fear Google’s verdict: They have announced long ago to HTTPS is considered a positive ranking by its search engine. Pages not using HTTPS will be tagged as insecure using more and more terrifying icons – e.g. http-only pages with login buttons already display a striked-through padlock in Firefox. In the past years I migrated a lot of PKIs from SHA1 to SHA256 to fight the first wave of Insecure icons.

Finally Let’s Encrypt has started a revolution: Free SSL certificates, based on domain validation only. My hosting provider uses a solution based on Let’s Encrypt – using a reverse proxy that does the actual HTTPS. I only had to re-target all my DNS records to the reverse proxy – it would have been very easy would it not have been for all my already existing URL rewriting and tweaking and redirecting. I also wanted to keep the option of still using HTTP in the future for tests and special scenario (like hosting a revocation list), so I decided on redirecting myself in the application(s) instead of using the offered automated redirect. But a code review and clean-up now and then can never hurt 🙂 For large complex sites the migration to HTTPS is anything but easy.

In case I ever forget which domains and host names I use, I just need to check out this list of Subject Alternative Names again:

(And I have another certificate for the ‘test’ host names that I need for testing the sites themselves and also for testing various redirects ;-))

WordPress.com also uses Let’s Encrypt (Automattic is a sponsor), and the SAN elkement.blog is lumped together with several other blog names, allegedly the ones which needed new certificates at about the same time.

It will be interesting what the consequences for phishing websites will be. Malicious websites will look trusted as being issued certificates automatically, but revoking a certificate might provide another method for invalidating a malicious website.

Anyway, special thanks to the WordPress.com Happiness Engineers and support staff at my hosting provider Puaschitz IT. Despite all the nerdiness displayed on this blog I prefer hosted / ‘shared’ solutions when it comes to my own websites because I totally like it when somebody else has to patch the server and deal with attacks. I am an annoying client – with all kinds of special needs and questions – thanks for the great support! 🙂

Diffusion of iTechnology in Corporations (or: Certificates for iPhones)

[Jump to technical stuff]

Some clichés are true. One I found confirmed often is about how technologies are adopted within organizations: One manager meets another manager at a conference / business meeting / CIO event. Manager X show off the latest gadget and/or brags about presents a case-study of successful implementation of Y.

Another manager becomes jealous inspired, and after returning home he immediately calls upon his poor subordinates and have them implement Y – absolutely, positively, ASAP.

I suspect that this is the preferred diffusion mechanism for implementing SAP at any kind of organization or for the outsourcing hype (probably also the insourcing-again movement that followed it).

And I definitely know this works that way for iSomething such as iPhones and iPads. Even if iSomething might be not the officially supported standard. But no matter how standardized IT and processes are – there is always something like VIP support. I do remember vividly how I was one told that we (the IT guys) should not be so overly obliging when helping users –  unless I (the top manager) needs something.

So trying to help those managers is the root cause for having to solve a nice puzzle: iThings need to have access to the network and thus often need digital certificates. Don’t tell me that certificates might not be the perfect solution – I know that. But working in some sort of corporate setting you are often not in the position to bring up these deep philosophical questions again and again, so let’s focus on solving the puzzle:

[Technical stuff – I am trying a new format to serve different audiences here]

Certificates for Apple iPhone 802.1x / EAP-TLS WLAN Logon

The following is an environment you would encounter rather frequently: Computer and user accounts are managed in Microsoft Active Directory – providing both Kerberos authentication infrastructure and LDAP directory. Access to Wireless LAN is handled by RADIUS authentication using Windows Network Protection Server, and client certificates are mandatory as per RADIUS policies.

You could require 802.1x to be done by either user accounts and/or machine accounts (though it is a common misunderstanding that in this way you can enforce a logon by 1) the computer account and then 2) the user account at the same machine.) I am now assuming that computers (only) are authenticated. This the iDevice needs to present itself as a computer to the logon servers.

Certificates contain lots of fields and standards either don’t enforce clearly what should go into those fields and/or applications interpret standards in weird ways. Thus the pragmatic approach is to tinker and test.

This is the certificate design that works for iPhones according to my experience:

  • We need a ‘shadow account’ in Active Directory whose properties will match fields in the certificates. Two LDAP attributes needto be set
    1. dnsHostName: machine.domain.com
      This is going to be mapped onto the DNS name in the Subject Alternative Name of the certificate.
    2. servicePrincipalNames: HOST/machine.domain.com
      This makes the shadow account a happy member of the Kerberos realm.

    According to my tests, the creation of an additional name mapping – as recommended here – is not required. We are using Active Directory default mapping here – DNS machine names work just as user’s UPNs (User Principal Name – the logon name in user@dmain syntax. See e.g. Figure 21 – Certificate Processing Logic – in this white paper for details.)

  • Extensions and fields in the certificate
    1. Subject Alternative Name: machine.domain.com (mapped to the DNS name dnsHostName in AD)
    2. Subject CN: host/machine.domain.com. This is different from Windows computers – as far as I understood what’s going on from RADIUS logging the Apple 802.1x client sends the string just as it appears in the CN. Windows clients would add the prefix host/ automatically.
    3. If this is a Windows Enterprise PKI: Copy the default template Workstation Authentication, and configure the Subject Name as to be submitted with the Request. The CA needs to accept custom SANs via enabling the  EDITF_ATTRIBUTESUBJECTALTNAME2 flag. Keys need to be configured as exportable to carry them over to the iDevice.
  • Create the key, request and certificate on a dedicated enrollment machine. Note that this should be done in the context of the user rather than the local machine. Certificates/key could be transported to another machines as PKCS#12 (PFX files).
  • Import the key and certificate to the iPhone using the iPhone Configuration Manager – this tools allows for exporting directly from the current user’s store. So if the user does not enroll for those certificates himself (which makes sense as the enrollment procedure is somewhat special, given the custom names), the PFX files would be first imported to the user’s store and then exported from there to the iPhone.

The point I like to stress in relation to certificates is that logon against AD is based on matching strings – containing the DNS names – not on a binary comparison of a file presented by the client versus a certificate file in the directory.

I have encountered that misconception often as there is an attribute in AD – userCertificate – that is actually designed for holding users’ (or machines’) certificates. But this is more of a Alice-tries-to-get-Bob’s-public-key-phonebook-style attribute, and it is not intended to be used for authentication but rather for encryption – Outlook is searching for S/MIME e-mail recipients’ public keys there. Disclaimer: I cannot vouch for any custom application that may exist.

Authentication is secure nonetheless as the issuing CA’s certificate needs to be present in a special LDAP object, the so-called NTAuth object in Active Directory’s Configuration Container, and per default it can only be edited by Enterprise Admins – the ‘root admins’ of AD. In addition you have to configure the CA for accepting arbitrary SANs in requests.

IPhone Fashion Valley

Happy iPhone users with their iPhones, when the product was released in 2007. I have never owned any iThing so I need to borrow an image from Wikimedia (user 1DmkIIN).

The Strange World of Public Key Infrastructure and Certificates

An e-mail discussion related to my recent post on IT security has motivated me to ponder about issues with Public Key Infrastructure once more. So I attempt – most likely in vain – to merge a pop-sci introduction to certificates with sort of an attachment to said e-mail discussion.

So this post might be opaque to normal users and too epic and introductory for security geeks. I apologise for the inconvenience.

I mentioned the failed governmental PKI pilot project in that post – a hardware security device destroying the key and there was no backup. I would have made fun of this – hadn’t I experienced it often that it is the so-called simple processes and logistics that can go wrong.

Ponte Milvio love padlocks

I didn’t expect to find such a poetic metaphor for “security systems” rendered inaccessible. Padlocks at Ponte Milvio in Italy – legend has it that lovers attaching a padlock to the bridge and throwing the key into the water will be together forever.

When compiling the following I had in mind what I call infrastructure PKIs – company-internal systems to be used mainly for internal purposes and very often for use by devices rather than by humans. (Ah, the internet of things.)

Issues often arise due to a combination of the following:

  • Human project resources assigned to such projects are often limited.
  • Many applications simply demand certificates so you need to create them.

Since the best way to understand certificates is probably by comparing them to passports or driver licenses I will nonetheless use one issued to me as a human life-form:

Digital Certificate

In Austria the chipcards used to authorize you to medical doctors as a patient can also be used as digital ID cards. That is, the card’s chip also holds the cryptographic private key, and the related certificate ties your identity as a citizen to the corresponding public key. A certificate is a file digitally signed by a Certificate Authority which in this case has the name a-sign-Token-03. The certificate can be downloaded here or searched for in the directory (German sute).

Digital X.509 Certificate: Details

The Public key related to my identity as a citizen (or better a database record representing myself as a citizen). As a passport, the certificate has an end of life and requires renewal.

Alternatives to Hardware Security Modules

An HSM is protecting that sacred private key of the certification authority. It is often a computer, running a locked-down version of an operating system, and it is equipped with sensors that detect and attempt to access the key store physically – it should actually destroy the key rather than having an attacker gain access to it.

It allows for implementing science-fiction-style (… Kirk Alpha 2Spock Omega 3 …) split administration and provides strong key protection that cannot be provided if the private key is stored in software – somewhere on the hard disk of the machine running the CA.

Captain Jean-Luc Picard transfers command of the USS Enterprise-D to Captain Edward Jellico

Yes, a key ceremony – the initiation of a certification authority – sometimes feel like that (memory-alpha.org). Here is the definitive list of Star Trek authorization codes.

Modern HSMs have become less cryptic in terms of usage but still: It is a hardware device not used on a daily basis, and requires additional training and management. Storage of physical items like the keys for unlocking the device and the corresponding password(s) is a challenge as is keeping the know-how of admins up to date.

Especially for infrastructure CAs I propose a purely organizational split administration for offline CAs such as a Root CA: Storing the key in software, but treating the whole CA machine as a device to be protected physically. You could store the private key of the Root CA or the virtual machine running the Root CA server on removable media (and at least one backup). The “protocol” provides spilt administration: E.g. one party has the key to the room, the other party has the password to decrypt the removable medium. Or the unencrypted medium is stored in a location protected by a third party – which in turn does only allow two persons to enter the room together.

But before any split administration is applied an evaluation of risks it should be made sure that the overall security strategy does not look like this:

Steps to nowhere^ - geograph.org.uk - 666960

From the description on Wikimedia: The gate is padlocked, though the fence would not prevent any moderately determined person from gaining access.

You might have to question the holy order (hierarchy) and the security implemented at the lowest levels of CA hierarchies.

Hierarchies and Security

In the simplest case a certification authority issues certificates to end-entities – users or computers. More complex PKIs consist of hierarchies of CAs and thus tree-like structures. The theoretical real-world metaphor would be an agency issuing some license to a subordinate agency that issues passports to citizens.

Chain of certificates associated with this blog

Chain of certificates associated with this blog: *.wordpress.com is certified by Go Daddy Secure Certification Authority which is in turn certified by Go Daddy Class 2 Certification Authority. The asterisk in the names makes it usable with any wordpress.com site – but it defies the purpose of denoting one specific entity.

The Root CA at the top of the hierarchy should be the most secure as if it is compromised (that is: it’s private key has – probably – been stolen) all certificates issued somewhere in the tree should be invalidated.

However, this logic only makes sense:

  • if there is or will with high probability be at least a second Issuing CA – otherwise the security of the Issuing CA is as important as that of the Root CA.
  • if the only purpose of that Root CA is to revoke the certificate of the Issuing CA. The Root CA’s key is going to sign a blacklist referring to the Issuing CA. Since the Root should not revoke itself its key signing the revocation list should be harder to compromise than the key of the to-be-revoked Issuing CA.
Certificate Chain

The certificate chain associated with my “National ID” certificate. Actually, these certificates stored on chipcards are invalidated every time the card (which serves another purpose primarily) is retired as a physical item. Invalidation of tons of certificates can create other issues I will discuss below.

Discussions of the design of such hierarchies focus a lot on the security of the private keys and cryptographic algorithms involved

But yet the effective security of an infrastructure PKI in terms of Who will be able to enroll for certificate type X (that in turn might entitle you to do Y) is often mainly determined by typical access control lists in databases or directories system that are integrated with an infrastructure PKI. Think would-be subscribers logging on to a web portal or to a Windows domain in order to enroll for a certificates. Consider e.g. Windows Autoenrollment (licensed also by non-Windows CAs) or the Simple Certificate Enrollment Protocol used with devices.

You might argue that it should be a no-no to make allegedly weak  software-credential-based authentication the only prerequisite for the issuance of certificates that are then considered strong authentication. However, this is one of the things that distinguish primarily infrastructure-focused CAs from, say, governmental CAs, or “High Assurance” smartcard CAs that require a face-to- face enrollment process.

In my opinion certificates are often deployed because their is no other option to provide platform-independent authentication – as cumbersome as it may be to import key and certificate to something like a printer box. Authentication based on something else might be as secure, considering all risks, but not as platform-agnostic. (For geeks: One of my favorites is 802.1x computer authentication via PEAP-TLS versus EAP-TLS.)

It is finally the management of group memberships or access control lists or the like that will determine the security of the PKI.

Hierarchies and Cross-Certification

It is often discussed if it does make sense to deploy more intermediate levels in the hierarchy – each level associated with additional management efforts. In theory you could delegate the management of a whole branch of the CA tree to different organizations, e.g. corresponding to continents in global organizations. Actually, I found that the delegation argument is often used for political reasons – which results in CA-per-local-fiefdom instead of the (in terms of performance much more reasonable) CA-per-continent.

I believe the most important reason to introduce the middle level is for (future) cross-certification: If an external CA cross-certifies yours it issues a certificate to your CA:

Cross Certification

Cross Certification between two CA hierarchies, each comprising three levels. Within a hierarchy each CA issues a certificate for its subordinate CA (orange lines). In addition the middle-tier CAs in each hierarchy issue certificates to the Root CAs of the other hierarchy – effectively creating logical chains consisting of 4 CAs. Image credits mine.

Any CA on any level could on principle be cross-certified. It would be easier to cross-certificate the Root CA but then the full tree of CAs subordinate to it will also be certified (For the experts: I am not considering name or other constraints here). If a CA an intermediate level is issued the cross-certificate trust is limited to this branch.

Cross-Certification constitutes a bifurcation in the CA tree and its consequences can be as weird and sci-fi as this sounds. It means that two different paths exists that connect an end-entity certificate to a different Root CA. Which path is actually chosen depends on the application validating the certificate and the protocol involved in exchanging or collecting certificates.

In an SSL handshake (which happens if you access your blog via https://yourblog.wordpress.com, using the certificate with that asterisk) happens if you access the web server is so kind to send the full certificate chain – usually excl. the Root CA – to the client. So the path finally picked by the client depends on the chain the server knows or that takes precedence at the server.

Cross-certification is usually done by CAs considered external, and it is expected that an application in the external world sees the path chaining to the External CAs.

Tongue-in-cheek I had once depicted the world of real PKI hierarchies and their relations as:

CA hierarchies in the real world.

CA hierarchies in the real world. Sort of. Image credits mine.

Weird things can happen if a web server is available on an internal network and accessible by the external world (…via a reverse proxy. I am assuming there is no deliberate termination of the SSL connection at the proxy – what I call a corporate-approved man-in-the-middle attack). This server knows the internal certificate chain and sends it to the external client – which does not trust the corresponding internal-only Root CA. But the chain sent in the handshake may take precedence over any other chain found elsewhere so the client throws an error.

How to Really Use “Cross-certification”

As confusing cross-certification is – it can be  used in a peculiar way to solve other PKI problems – those with applications that cannot deal with the validation of a hierarchy at all or who can deal with only a one-level hierarchy. This is interesting in particular in relation to devices such as embedded industry systems or iPhones.

Assuming that only the needed certificates can be safely injected to the right devices and that you really know what you are doing the fully pesky PKI hierarchy can be circumvented by providing an alternative Root CA certificate to the CA at the bottom of the hierarchy:

The real, full blown hierarchy is

  1. Root CA issued a root certificate for Root CA (itself). It contains the key 1234.
  2. Root CA issues a certificate to Some Other CA related to key 5678.

… then the shortcut hierarchy for “dumb devices” looks like:

  1. Some Other CA issues a root certificate to itself, thus to Subject named Some Other CA. The public key listed in this certificate is 5678 the same as in certificate (2) of the extended hierarchy.

Client certificates can then use either chain – the long chain including several levels or the short one consisting of a single CA only. Thus if certificates have been issued by the full-blown hierarchy they can be “dumbed-down to devices” by creating the “one-level hierarchy” in addition.

Names and Encoding

In the chain of certificates the Issuer field in the certificate of the Subordinate CA needs to be the same as the Subject field of the Root CA – just as the Subject field in my National ID certificate contains my name and the Issuer field that of the signing CA. And it depends on the application how names with be checked. In a global world, names are not simple ASCII strings anymore, but encoding matters.

Certificates are based on an original request sent by the subordinate CA, and this request most often contains the name – the encoded name. I have sometimes seen that CAs changed the encoding of the names when issuing the certificates, or they reshuffled the components of the name – the order of tags like organization and country. An application may except that or not, and the reasons for rejections can be challenging to troubleshoot if the application is running in a blackbox-style device.

Revocation List Headaches

Certificates (X.509) can be invalidated by adding their respective serial number to a blacklist. This list is – or actually: may – be checked by relying parties. So full-blown certificate validation comprises collecting all certificates in the chain up to a self-signed Root CA (Subject=Issuer) and then checking each blacklist signed by each CA in the chain for the serial number of the entity one level below:

Certificate Validation

Validation of a certificate chain (“path). You start from the bottom and locate both CA certificates and the revocation lists via URLs in each subordinate certificate. Image credits mine.

The downside: If the CRL isn’t available at all applications following the recommended practices will for example deny network access to thousands of clients. With infrastructure PKIs that means that e.g. access to WLAN or remote access via VPN will fail.

This makes desperate PKI architects (or rather the architects accountable for the application requiring certificate based logon) build all kinds of workarounds, such as switching off CRL checking in case of an emergency or configuring grace periods. Note that this is all heavily application dependent and has to be figured out and documented individually for emergencies for all VPN servers, exotic web servers, Windows domain controllers etc.

A workaround is imperative if a very important application is dependent on a CRL issued by an “external” certificates’ provider. If I would use my Austrian’s digital ID card’s certificate for logging on to server X, that server would need tp have a valid version of this CRL which only lives for 6 hours.

Certificate Revocation List

A Certificate Revocation List (CRL) looks similar to a certificate. It is a file signed the Certification Authority that also signed the certificates that might be invalidated via that CRL. From downloading this CRL frequently I conclude that it a current version is published every hour – so there are 5 hours of overlap.

The predicament is that CRLs may be cached for performance reasons. Thus if you publish short-lived CRLs frequently you might face “false negative” outages due to operational issues (web server down…) but if the CRL is too long-lived it does not serve its purpose.

Ideally, CRLs would be valid for a few days, but a current CRL would be published, say every day, AND you could delete the CRL at the validating application every day. That’s exactly how I typically try to configure it. VPN servers, for example, have allowed to delete the CRL cache for a long time and Windows has a supported way to do that since Vista. This allows for reasonable continuity but revocation information would still be current.

If you cannot control the CRL issuance process one workaround is: Pro-active fetching of the CRL in case it is published with an overlap – that is: the next CRL is published while the current one is still valid – and mirroring the repository in question.

As an aside: It is more difficult as it sounds to give internal machines access to a “public” external URL. Machines not necessarily use the proxy server configured for user (which cause false positive results – Look, I tested it by accessing it in the browser and it works), and/or machines in the servers’ network are not necessarily allowed to access “the internet”.

CRLs might also simply be too big – for some devices with limited processing capabilities. Some devices of a major vendor used to refuse to process CRLs larger than 256kB. The CRL associated with my sample certificate is about 700kB:

LDAP CDP URL

How the revocation is located – via a URL embedded in the certificate. For the experts: OCSP is supported, too, and it is the recommended method. However considering older devices it might be necessary to resort to CRLs.

CRL Details - Blacklist

The actual blacklist part of the CRL. The scrollbar is misleading – the list contains about 20.000 entries (best viewed with openssl or Windows certutil).

Emergency Revocation List

In case anything goes wrong – HSM inaccessible, passwords lost, datacenter 1 flooded abd backup datacenter 2 destroyed by a meteorite – there is one remaining option to keep PKI-dependent applications happy:

Prepare a revocation list in advance whose end of life (NextUpdate date) is after the end of validity of the CA certificate. In contrast to any backup of key material this CRL can be “backed up” by pasting the BASE64 string to the documentation as it does not contain sensitive information.

In an emergency this CRL will be published to the locations embedded in certificates. You will never be able to revoke anything anymore as CRLs might be cached – but business continuity is secured.

Emergency CRL

An Emergency CRL for my home-grown CA. It seems 9999 days is the maximum I can use with Windows certutil. Actually, the question of How many years should the lifetime be so that I will not be bothered anymore until retirement? comes up often in relation to all kinds of validity dates.