Certificates and PKI. The Prequel.

Some public key infrastructures run quietly in the background since years. They are half forgotten until the life of a signed file has come to an end – but then everything is on fire. In contrast to other seemingly important deadlines (Management needs this until XY or the world will come to an end!) this deadline really can’t be extended. The time of death is included in the signed data since a long time.

The entire security ‘ecosystem’ changes while these systems sleep in the background. Now we have Let’s Encrypt (I was late to that party), HTTPS is everywhere, and the green padlock as an indicator of a secure site is about to die.

Recently I stumbled upon a whirlwind tour of the history of PKI and SSL/TLS – covering important events in the evolution of standards and technologies, from shipping SSLv2 in Netscape Navigator 1.1 in 1995 to Chrome marking HTTP pages as ‘not secure’ in 2018. Scrolling down the list of years I could not avoid waxing nostalgic. I had written about PKI before at length before, but this time I do what the Hollywood directors of blockbusters do – I write a prequel.

I remember the first times I created a Certificate Signing Request (CSR) and submitted it to a Certificate Authority (CA). It was well before 2000, and it was an adventure!

I was a scientist turned freelance IT consultant – I went from looking at Transmission Electron Microscope images to troubleshooting why Outlook did not start on small business owners’ computers. And I was daring enough to give trainings, based on the little I knew (with hindsight) about IT and networking. I also developed some classes from scratch – creating wiki-style training material, using Microsoft FrontPage 1998.

One class was  called networking and security of the like, and it was part of a vocational retraining curriculum – to turn former factory workers and admin assistants into computer technicians. For reasons I cannot remember I included a brief explanation of the RSA algorithm in my clunky FrontPage site. It was maybe a pretext to justify an exciting lab exercise: As the PKI history timeline shows, SSL was still rather new. Press releases by Austrian IT companies highlighted the military-grade protection from eavesdropping. It felt like Star Trek. One of the early Austrian National CAs offered ‘light’ test certificates. The description of the enrollment process was targeted to business users, but it was pure geek speak: A mysterious multi-step procedure explained in hacker terms like Secure Vault.

I don’t remember if my students found it that exciting or if the test enrolling a lots of certificates simultaneously did work so well at all. But I was hooked.

As a freelancer I started working with my former colleagues again – supporting the sciencists to subvert re-interpert the central IT department’s policies by setting up their own server, or by circumventing the firewall by dialing in to their own modem. This were the days of IT hype in the late 1990s before the dotcom bust. The research center had a new CEO with an IT background, and to get your projects approved you had to tack the label virtual onto anything. So I helped with creating a Virtual Materials Science Lab – which meant we used Microsoft Netmeeting.

Despite or because of such activities I also started working for the IT department. It was the time when The Cluetrain Manifesto told us that hyperlinks were subversive. As a ‘manager’ I should have disciplined shadow IT admins purchasing their own domains and running their shadow servers, but I could not stop tinkering with the web servers myself. It was also the time when I learned that to make things work in larger organizations – or a combination of several of those – you often need to social engineer someone.

We needed a SSL certificate – and I was the super qualified person for that task, based on my experience playing with the Secure Vault. But creating and submitting the CSR, and installing the certificate was the easy part. There were unexpected challenges:

The research center had a long legal name – 65 characters including the final dot in the indication of the legal entity. Common Names in X.509 certificates are limited to 64 characters, so I could not enter the final dot in IIS’s (Internet Information Server’s) wizard for CSRs. The legal name was cross-checked against the Dun&Bradstreet database. One would think that the first 64 characters of a peculiar German name would have been sufficient, but no. It took several phone calls – and faxes! – to prove to the US-based CA company that we were who we claimed to be.

The fact I called a CA company in the US might highlight a mistake: If I recall correctly Big CA had partners in Europe already at that time, but I missed that, or I wanted to talk to the mothership for some reason.

To purchase the certificate from the US-based company you needed a credit card, to be entered exactly when you submit the CSR. This process was disrupting the usual purchasing procedures and I had to social engineer somebody from the procurement department to join me in my adventure, bringing the corporate credit card.

The research center was a company owned 51% by government – so you had SAP and insane management deadlines as well as conferences and academic publication records. The Internet in general was still dominated by its academic roots. Not long ago, there had been a single web page listing All WWW servers in Austria, and that page was run by the academic internet backbone. Domain registration data were tied to a person, to the wrong person, or to the wrong entity – which came back to bite you later.

Fortunately the domain assigned to the SSL certificate belonged to us – so I did not have to  social engineer a DNS admin this time. But it was assigned to a person and not to the organization. The person was an employee in charge of the network, but how should you prove that? More faxes and phone calls were required to sort out the fine legal points.

I did not keep records of that period, so I don’t know if this web server is alive or if at least the domain still exists. Maybe unlikely, given the rapid decay of rotting links. But while researching history for this post – randomly googling for early versions of Microsoft’s web servers – I discovered interesting things. There is a small change it may be alive!

The first version of the Windows Certificate Authority had been released as an add-on to Windows NT 4, as part of the so-called Windows NT 4 Option Pack – the same add-on that also contained the webserver (IIS) itself. It was the time when I learned ASP programming by going online via dial-up and browsing through MSDN as quick as possible not to overspend my precious monthly online time.

I wanted to relive the setup of Internet Information Server 4.0 as and the Option Pack – and found lots of support articles and how-to’s, like this one.

However, I also found live websites like this:

This is only the setup CD, so no danger yet, but you can as well find sites with the welcome page of the operating web server online – including sample ASP applications – which I don’t show deliberately. (Image credits: Microsoft.)

I wonder why I had been frantically re-developing my websites in ASP.NET from scratch – ‘just because’ ASP was outdated technology, even though there were no known vulnerabilities and the sites were running on a modern operating system.

Time to quote from Peter Gutmann’s book Engineering Security:

A great many of today’s security technologies are “secure” only because no-one has ever bothered attacking them.

… which is also true for yesterday’s technology still online!

Other People Have Lives – I Have Domains

These are just some boring update notifications from the elkemental Webiverse.

The elkement blog has recently celebrated its fifth anniversary, and the punktwissen blog will turn five in December. Time to celebrate this – with new domain names that says exactly what these sites are – the ‘elkement.blog‘ and the ‘punktwissen.blog‘.

Actually, I wanted to get rid of the ads on both blogs, and with the upgrade came a free domain. WordPress has a detailed cookie policy – and I am showing it dutifully using the respective widget, but they have to defer to their partners when it comes to third-party cookies. I only want to worry about research cookies set by Twitter and Facebook, but not by ad providers, and I am also considering to remove social media sharing buttons and the embedded tweets. (Yes, I am thinking about this!)

On the websites under my control I went full dinosaur, and the server sends only non-interactive HTML pages sent to the client, not requiring any client-side activity. I now got rid of the last half-hearted usage of a session object and the respective cookie, and I have never used any social media buttons or other tracking.

So there are no login data or cookies to protect, but yet I finally migrated all sites to HTTPS.

It is a matter of principle: I of all website owners should use https. Since 15 years I have been planning and building Public Key Infrastructures and troubleshooting X.509 certificates.

But of course I fear Google’s verdict: They have announced long ago to HTTPS is considered a positive ranking by its search engine. Pages not using HTTPS will be tagged as insecure using more and more terrifying icons – e.g. http-only pages with login buttons already display a striked-through padlock in Firefox. In the past years I migrated a lot of PKIs from SHA1 to SHA256 to fight the first wave of Insecure icons.

Finally Let’s Encrypt has started a revolution: Free SSL certificates, based on domain validation only. My hosting provider uses a solution based on Let’s Encrypt – using a reverse proxy that does the actual HTTPS. I only had to re-target all my DNS records to the reverse proxy – it would have been very easy would it not have been for all my already existing URL rewriting and tweaking and redirecting. I also wanted to keep the option of still using HTTP in the future for tests and special scenario (like hosting a revocation list), so I decided on redirecting myself in the application(s) instead of using the offered automated redirect. But a code review and clean-up now and then can never hurt :-) For large complex sites the migration to HTTPS is anything but easy.

In case I ever forget which domains and host names I use, I just need to check out this list of Subject Alternative Names again:

(And I have another certificate for the ‘test’ host names that I need for testing the sites themselves and also for testing various redirects ;-))

WordPress.com also uses Let’s Encrypt (Automattic is a sponsor), and the SAN elkement.blog is lumped together with several other blog names, allegedly the ones which needed new certificates at about the same time.

It will be interesting what the consequences for phishing websites will be. Malicious websites will look trusted as being issued certificates automatically, but revoking a certificate might provide another method for invalidating a malicious website.

Anyway, special thanks to the WordPress.com Happiness Engineers and support staff at my hosting provider Puaschitz IT. Despite all the nerdiness displayed on this blog I prefer hosted / ‘shared’ solutions when it comes to my own websites because I totally like it when somebody else has to patch the server and deal with attacks. I am an annoying client – with all kinds of special needs and questions – thanks for the great support! :-)

All My Theories Have Been Wrong. Fortunately!

I apologize to Google. They still like my blog.

This blog’s numbers plummeted as per Webmaster Tools, here and here you find everything you never wanted to know about it. I finally figured that my blog was a victim of Google’s latest update Panda 4.1. Sites about ‘anything’ had suffered, and the Panda rollout matched the date of the onset of the decline.

Other things happened in autumn, too: I had displayed links to latest WordPress blog posts on my other websites, but my feed parser suddenly refused to work. The root cause was the gradual migration of all WP.com blogs and feeds to https:// only. Only elkement’s blog had been migrated at that time; our German blog’s feed was affected two months later.

Recently also the German blog started its descent in impressions and clicks, again two months after elkement’s blog. I pondered about https URLs again – the correlation was too compelling. Then suddenly the answer came to me:

!

!!

!!!

You need to add the https URL as an additional site in Webmaster Tools.

!!!

!!

!

It was that simple. All the traffic I missed was here all the time – tucked away in the statistics for https://elkement.wordpress.com. This also answers the question I posed in my last Google rant post: Why do I see more Search Engine referrers in WordPress stats than clicks in Webmaster Tools? I had just looked in the wrong place.

I had briefly considered the https thing last year but ruled it out as I misinterpreted Webmaster Tools – falsely believing that one entry for a site would cover both the http and the https version. These are the results for both URLs – treated like separate entities by Webmaster Tools:

Results for http : // elkement.wordpress.com  – abysmal:

(Edit: I cannot use a link here and have to add those weird blanks – otherwise WP will always convert both URL and text to https automatically even if the prefix is displayed as http in the editor.)

Google traffic for http version of this blogResults for https://elkement.wordpress.com – better by a factor of 100: Way more Google traffic for the https version of this blog URLPopular pages were the first to ‘move’ over to the https entry. This explains why my top page was missing first from http pages impressions – the book review which I assumed to have been penalized by Panda as an alleged cross-link scam. In full paranoia mode I was also concerned of my adding random Wikimedia images to my poetry.

But now I will do it again as I feel relieved. And relaxed – as this Panda. Giant panda01 960______________________________

You have read a post in my new category Make a Fool of Myself. I tried to top the self-sabotaging effect of writing about my business website being hacked – as a so-called security expert.

Yet the theory was all too compelling. I found numerous examples of small sites penalized by Panda in a weird way. See this discussion (Link now broken): A shop’s webmaster makes a product database with succinct descriptions available online and is penalized for ‘key word spamming’ – as his key words are part of each product name. Advice by SEO experts: Circumscribe your product names.

Legend has it that Panda was named after a Google engineer. I figured it was because the Panda is so choosy, insisting on bamboo eucalyptus (*), just as Google scrutinizes our sites more and more. (*) One more theory I got wrong, now edited! Thanks to a commentator for pointing out the mistake.