FreeIPA Identity Management planet - technical blogs

December 08, 2018

William Brown

Work around docker exec bug

Work around docker exec bug

There is currently a docker exec bug in Centos/RHEL 7 that causes errors such as:

# docker exec -i -t instance /bin/sh
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

As a work around you can use nsenter instead:

PID=docker inspect --format {{.State.Pid}} <name of container>
nsenter --target $PID --mount --uts --ipc --net --pid /bin/sh

For more information, you can see the bugreport here.

December 08, 2018 02:00 PM

November 30, 2018

Fraser Tweedale

Diagnosing Dogtag cloning failures

Diagnosing Dogtag cloning failures

Sometimes, creating a Dogtag clone or a FreeIPA CA replica fails. I worked with Dogtag and FreeIPA for nearly five years. Over these years I’ve analysed a lot of these clone/replica installation failures and internalised a lot of knowledge about how cloning works, and how it can break. Often when I read a problem report and inspect the logs I quickly get a “gut feeling” about the cause. The purpose of this post is to externalise my internal intuition so that others can benefit. Whether you are an engineer or not, this post explains what you can do to get to the bottom of Dogtag cloning failures faster.

How Dogtag clones are created

Some notes about terminology: in FreeIPA we talk about replicas, but in Dogtag we say clones. These terms mean the same thing. When you create a FreeIPA CA replica, FreeIPA creates a clone of the Dogtag CA instance behind the scenes. I will use the term master to refer to the server from which the clone/replica is being created.

The pkispawn(8) program, depending on its configuration, can be used to create a new Dogtag subsystem or a clone. pkispawn, a Python program, manages the whole clone creation process, with the possible exception of setting up LDAP database and replication. But some stages of the configuration are handled by the Dogtag server itself (thus implemented in Java). Furthermore, the Dogtag server on the master must service some requests to allow the new clone to integrate into the topology.

The high level procedure of CA cloning is roughly:

  1. (ipa-replica-install) Create temporary Dogtag admin user account and add to relevant groups
  2. (ipa-replica-install or pkispawn) Establish LDAP replication of the Dogtag database
  3. (pkispawn) Extract private keys and system certifiates into Dogtag’s NSSDB
  4. (pkispawn) Lay out the Dogtag instance on the filesystem
  5. (pkispawn) Start the pki-tomcatd instance
  6. (pkispawn) Send a configuration request to the new Dogtag instance
    1. (pki-tomcatd on clone) Send security domain login request to master (using temporary admin user credentials)
    2. (pki-tomcatd on master) Authenticate user, return cookie.
    3. (pki-tomcatd on clone) Send number range requests to master
    4. (pki-tomcatd on master) Service number range requests for clone
  7. (ipa-replica-install) remove temporary admin user account

There are several places a problem could occur: in pkispawn, pki-tomcatd on the clone, or pki-tomcatd on the master. Therefore, depending on what failed, the best data about the failure could be in pkispawn output/logs, the Dogtag debug log on the replica, or the master, or even the system journal on either of the hosts. Recommendation: when analysing Dogtag cloning or FreeIPA CA replica installation failures, inspect all of these logs. It is often not obvious where the error is occurring, or what caused it. Having all these log files helps a lot.

Case studies

Failure to set up replication

Description: ipa-replica-install or pkispawn fail with errors related to replication (failure to establish). I don’t know how common this is in production environments. I’ve encountered it in my development environments. I think it is usually caused by stale replication agreements or something of that nature.

Workaround: A “folk remedy”: uninstall and clean up the instance, then try again. Most often the error does not recur.

Replication races

Description: pkispawn fails; replica debug log indicates security domain login failure; master debug log indicates user unknown; debug log indicates token/session unkonwn

During cloning, the clone adds LDAP objects in its own database. It then performs requests against the master, assuming that those objects (or effects of other LDAP operations) have been replicated to the master. Due to replication lag, the data have not been replicated and as a consequence, a request fails.

In the past couple of years several replication races were discovered and fixed (or mitigated) in Dogtag or FreeIPA:

updateNumberRange failure due to missing session object


Description: After security domain login (locally on the replica) the session object gets replicated to the master. The cookie/token conveyed in the updateNumberRange range referred to a session that the master did not yet know about.

Resolution: the replica sleeps (duration configuration; default 5s) after security domain login, giving time for replication. This is not guaranteed the avoid the problem: the complete solution (yet to be implemented) will be to use a signed/MACed token.

Security domain login failure due to missing user or group membership


Description: This bug was actually in FreeIPA, but manifested in pki-tomcatd on master as a failure to log into the security domain. This could occur for one of two reasons: either the user was unknown, or the user was not a member of a required group. FreeIPA performs the relevant LDAP operations on the replica, but they have not replicated to master yet. The pkispawn/ipa-replica-install error message looks something like:

com.netscape.certsrv.base.PKIException: Failed to obtain
installation token from security domain:
com.netscape.certsrv.base.UnauthorizedException: User
admin-replica1.ipa.example is not a member of Enterprise CA
Administrators group.

Workaround: no supported workaround. (You could hack in a sleep though).

Resolution: The user creation routine was already waiting for replication but the wait routine had a timeout bug causing false positives, and the group memberships were not being waited on. The timeout bug was fixed. The wait routine was enhanced to support waiting for particular attribute values; this feature was used to ensure group memberships have been replicated before continuing.

Other updateNumberRange failures


Description: When creating a clone from a master that was itself a clone, an updateNumberRange request fails at master with status 500. A NullPointerException backtrace appears in the journal for the pki-tomcatd@pki-tomcat unit (on master). The problem arises because the initial number range assignment for the second clone is equal to the range size of the first clone (range transfer size is a fixed number). This scenario was not handled correctly, leading to the exception.

Workaround: Ensure that each clone services one of each kind of number (e.g. one full certificate request and issuance operation). This ensures that the clone’s range is smaller than the range transfer size, so that a subsequent updateNumberRange request will be satisfied from the master’s “standby” range.

Resolution: detect range depletion due to updateNumberRange requests and eagerly switch to the standby range. A better fix (yet to be implemented) will be to allocate each clone a full-sized range from the unallocated numbers.


Dogtag subsystem cloning is a complex procedure. Even more so in the FreeIPA context. There are lots of places failure can occur.

The case studies above are a few examples of difficult-to-debug failures where the cause was non-obvious. Often the error occurs on a different host (the master) from where the error was observed. And the important data about the true cause may reside in ipareplica-install.log, pkispawn log output, the Dogtag CA debug log (on replica or master) or the system journal (again on replica or master). Sometimes the 389DS logs can be helpful too.

Normally the fastest way to understand a problem is to gather all these sources of data and look at them all around the time the error occurred. When you see one failure, don’t assume that that is the failure. Cross-reference the log files. If you can’t see anything about an error, you probably need to look in a different file…

…or a different part of the file! It is important to note that Dogtag time stamps are in local time, whereas most other logs are UTC. Different machines in the topology can be in different timezones, so you could be dealing with up to three timezones across the log files. Check carefully what timezone the timestamps are in when you are “lining up” the logfiles. Many times I have seen (and often erred myself) an incorrect conclusion that “there is no error in the debug log” because of this trap.

In my experience, the most common causes of Dogtag cloning failure have involved Security Domain authentication issues and number range management. Over time I and others have fixed several bugs in these areas, but I am not confident that all potential problems have been fixed. The good news is that checking all the relevant logs usually leads to a good theory about the root cause.

What if you are not an engineer or not able to make sense of the Dogtag codebase? (This is fine by the way—Dogtag is a huge, gnarly beast!) The best thing you can do to help us analyse and resolve the issue is to collect all the logs (from the master and replica) and prune them to the relevant timeframe (minding the timezones) before passing them on for an engineer to analyse.

In this post I only looked at Dogtag cloning failures. I have lots of other Dogtag “gut knowledge” that I plan to get out in upcoming posts.

November 30, 2018 12:00 AM

November 27, 2018

Rob Crittenden

Setting up a Mac (OSX) as an IPA client

I periodically see people trying to setup a Mac running OSX as an IPA client. I don’t have one myself so can’t really provide assistance.

There is this guide which seems to be pretty thorough,

This upstream ticket also has some information on setting up a client, though it isn’t always directly related to simply configuring a client,

So I record this here so I know where to look later 🙂

by rcritten at November 27, 2018 10:10 PM

November 26, 2018

Rob Crittenden

How do I get a certificate for my web site with IPA?

That’s a bit of a loaded question that begs additional questions:

  1. Is the web server enrolled as an IPA client?
  2. What format does the private key and certificate need to be in? (OpenSSL-style PEM, NSS, other?)

If the answer to question 1 is YES then you can do this on that client machine (to be clear, this first step can be done anywhere or in the UI):

$ kinit admin$ ipa service-add HTTP/

You can use certmonger to request and manage the certificate which includes renewals (bonus!).

If you are using NSS and let’s say mod_nss you’d do something like after creating the database and/or putting the password into /etc/httpd/alias/pwdfile.txt:

# ipa-getcert request -K HTTP/ -d /etc/httpd/alias -n MyWebCert -p /etc/httpd/alias/pwdfile.txt -D

Let’s break down what these options mean:

  • -K is the Kerberos principal to store the certificate with. You do NOT need to get a keytab for this service
  • -d the NSS database directory. You can use whatever you want but be sure the service has read access and SELinux permission access to it.
  • -n the NSS nickname. This is just a shortcut name to your cert, use what you want.
  • -p the path to the pin/password for the NSS database
  • -D creates a DNS SAN for the hostname This is current best practice.

You’ll also need to add the IPA cert chain to the NSS database using certutil.

If you are using OpenSSL and say mod_ssl you’d do something like:

# ipa-getcert request -K HTTP/ -k /etc/pki/tls/private/httpd.key -f /etc/pki/tls/certs/httpd.pem -D

Similar options as above but instad -f -d and -n:

  • -k path to the key file
  • -f path to the certificate file

To check on the status of your new request you can run:

# ipa-getcert list -n <numeric id that was spit out before>

It should be in status MONITORING.

If the answer to #1 is NO then you have two options: use certmonger on a different machine to generate the key and certificate and transfer them to the target or generate a CSR manually.

For the first case, using certmonger on a different machine, the steps are similar to the YES case.

Create a host and service for the web server:

$ kinit admin$ ipa host-add$ ipa service-add HTTP/

Now we need to grant the rights to the current machine to get certificates for the HTTP service of

$ ipa service-add-host --hosts <your current machine FQDN> HTTP/

Now run the appropriate ipa-getcert command above to match the format you need and check the status in a similar way.

Once it’s done you need to transfer the cert and key to the webserver.

Finally, if you want to get certificates on an un-enrolled system the basic steps are:

  1. Create a host entry and service as above
  2. Generate a CSR, see (or the next section)
  3. Submit that CSR per the above docs

If your webserver is not registered in DNS then you can use the –force option to host-add and service-add to force their creation.

This should pretty generically apply to all versions of IPA v4+, and probably to v3 as well.


by rcritten at November 26, 2018 09:48 PM

November 20, 2018

Fraser Tweedale

FreeIPA CA renewal master explained

FreeIPA CA renewal master explained

Every FreeIPA deployment has a critical setting called the CA renewal master. In this post I explain how this setting is used, why it is important, and the consequences of improper configuration. I’ll also discuss scenarios which cause the value to change, and why and how you would change it manually.

What is the CA renewal master?

The CA renewal master configuration controls which CA replica is responsible for renewing some important certificate used within a FreeIPA deployment. I will call these system certificates.

Unlike service certificates (e.g. for HTTP and LDAP) which have different keypairs and subject names on different servers, FreeIPA system certificates, and their keys, are shared by all CA replicas. These include the IPA CA certificate, OCSP certificate, Dogtag subsystem certificates, Dogtag audit signing certificate, IPA RA agent certificate and KRA transport and storage certificates.

The current CA renewal master configuration can be viewed via ipa config-show:

[f28-1] ftweedal% ipa config-show | grep 'CA renewal master'
  IPA CA renewal master: f28-1.ipa.local

Under the hood, this configuration is a server role attribute. The CA renewal master is indicated by the presence of an (ipaConfigString=caRenewalMaster) attribute value on an IPA server’s CA role object. You can determine the renewal master via a plain LDAP search:

[f28-1] ftweedal% ldapsearch -LLL \
      -D "cn=Directory Manager" \
dn: cn=CA,cn=f28-1.ipa.local,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=local
objectClass: nsContainer
objectClass: ipaConfigObject
objectClass: top
cn: CA
ipaConfigString: startOrder 50
ipaConfigString: caRenewalMaster
ipaConfigString: enabledService

The configuration is automatically set to the first master in the topology on which the CA role was installed. Unless you installed without a CA, this is the original master set up via ipa-server-install.

What problem is solved by having a CA renewal master?

All CA replicas have tracking requests for all system certificates. But if all CA replicas renewed system certificates independently, they would end up with different certificates. This is especially a problem for the CA certificate, and the subsystem and IPA RA certificates which get stored in LDAP for authentication purposes. The certificates must match exactly, otherwise there will be authentication failures between the FreeIPA framework and Dogtag, and between Dogtag and LDAP.

Appointing one CA replica as the renewal master allows the system certificates to be renewed exactly once, when required.

How do other replicas acquire the updated certificates?

The Certmonger tracking requests on all CA replicas use the dogtag-ipa-ca-renew-agent renewal helper. This program reads the CA renewal master configuration. If the current host is the renewal master, it performs the renewal, and stores the certificate in LDAP under cn=<nickname>,cn=ca_renewal,cn=ipa,cn=etc,{basedn}. Additionally, if the certificate is the IPA RA or the Dogtag CA subsystem certificate, the new certificate gets added to the userCertificate attribute of the corresponding LDAP user entry

If the renewal master is a different host, the latest certificate is retrieved from the ca_renewal LDAP entry and returned to Certmonger. Due to non-determinism in exactly when Certmonger renewal attempts will occur, the non-renewal helper could attempt to “renew” the certificate before the renewal master has actually renewed the certificate. So it is not an error for the renewal helper to return the old (soon to expire) certificate. Certmonger will keep attempting to renew the certificate (with some delay between attempts) until it can retrieve the updated certificate (which will not expire soon).

What can go wrong?

If it wasn’t clear already, a (CA-ful) FreeIPA deployment must at all times have exactly one CA replica configured as the renewal master. That server must be online, operating normally, and replicating properly with other servers. Let’s look at what happens if these conditions are not met.

If the CA renewal master configuration refers to a server that has been decommissioned, or is offline, then no server will actually renew the certificates. All the non-renewal master servers will happily reinstall the current certificate, until they expire, and things will break. The troublesome thing about certificates is even one expired certificate can cause renewal failures for other certificates. The problems cascade and eventually the whole deployment is busted.

FreeIPA has a simple protection in place to ensure the renewal master configuration stays valid. Servers can be deleted from the topology via the ipa server-del, ipa-replica-manage del, ipa-csreplica-manage del or ipa-server-install --uninstall command. In these commands, if the server being deleted is the current CA renewal master, a different CA replica is elected as the new CA renewal master.

These protections only go so far. If the renewal master is still part of the topology but is offline for an extended duration it may miss a renewal window, causing expired certificates. If there are replication problems between the renewal master and other CA replicas, renewal might succeed, but the other CA replicas might not be able to retrieve the updated certificates before they expire. All of these problems (and more) have been seen in the wild.

I have seen cases where a CA renewal master was simply decommissioned without formally removing it from the FreeIPA topology. I have also seen cases where there was no CA renewal master configured (I do not know how this situation arose). Both of these scenarios have similar consequences to the “offline for extended duration” scenario.

What would happen if you had two (or more) CA replicas with (ipaConfigString=caRenewalMaster)? I haven’t seen this one in the wild, but I would not be surprised if one day I did see it. In this case, multiple CA replicas will perform renewals. Will clobber each others’ certificates, and will result in some replicas having RA Agent or Dogtag subsystem certificates out of sync with the corresponding user entries in LDAP. This is a less catastrophic consequence than the aforementioned scenarios, but still serious. It will result in Dogtag or IPA RA authentication failures on some (or most) CA replicas.

Why and how to change the CA renewal master

Why would you need to change the renewal master configuration? Assuming the existing configuration is valid, the main reason you would need to change it is in anticipation of the decommissioning of the existing CA renewal master. You may wish to appoint a particular server as the new renewal master. As discussed above, the commands that remove servers from the topology will do this automatically, but which server will be chosen is out of your hands. So you can get one step ahead and change the renewal master yourself.

In my test setup there are two CA replicas:

[f28-1] ftweedal% ipa server-role-find --role 'CA server'
2 server roles matched
  Server name: f28-0.ipa.local
  Role name: CA server
  Role status: enabled

  Server name: f28-1.ipa.local
  Role name: CA server
  Role status: enabled
Number of entries returned 2

The current renewal master is f28-1.ipa.local:

[f28-1] ftweedal% ipa config-show | grep 'CA renewal master'
  IPA CA renewal master: f28-1.ipa.local

The preferred way to change the renewal master configuration is via the ipa config-mod command:

[f28-1] ftweedal% ipa config-mod \
      --ca-renewal-master-server f28-0.ipa.local \
      | grep 'CA renewal master'
  IPA CA renewal master: f28-0.ipa.local

You can also use the ipa-csreplica-manage command. This requires the Directory Manager passphrase:

[f28-1] ftweedal% ipa-csreplica-manage \
                    set-renewal-master f28-1.ipa.local
Directory Manager password: XXXXXXXX

f28-1.ipa.local is now the renewal master

If for whatever reason the current renewal master configuration is invalid, you can use these same commands to reset it. As a last resort, you can modify the LDAP objects directly to ensure that exactly one CA role object has (ipaConfigString=caRenewalMaster). Note that both the attribute name (ipaConfigString) and value (caRenewalMaster) are case-insensitive.

Finally, let’s observe what happens when we remove a server from the topology. I’ll remove f28-1.ipa.local (the current renewal master) using the ipa-server-install --uninstall command. After this operation, the CA renewal master configuration should point to f28-0.ipa.local (the only other CA replica in the topology).

[f28-1:~] ftweedal% sudo ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data
and configuration!
It is highly recommended to take a backup of existing data
and configuration using ipa-backup utility before proceeding.

Are you sure you want to continue with the uninstall procedure? [no]: yes
Forcing removal of f28-1.ipa.local
Failed to cleanup f28-1.ipa.local DNS entries: DNS is not configured
You may need to manually remove them from the tree
Deleted IPA server "f28-1.ipa.local"
Shutting down all IPA services
Unconfiguring CA
... (snip!)
Client uninstall complete.
The ipa-client-install command was successful
The ipa-server-install command was successful

Jumping across to f28-0.ipa.local, I confirm that f28-0.ipa.local has become the renewal master:

[f28-0] ftweedal% ipa config-show |grep 'CA renewal master'
  IPA CA renewal master: f28-0.ipa.local

Explicit CA certificate renewal

There is one more scenario that can cause the CA renewal master to be changed. When the IPA CA certificate is explicitly renewed via the ipa-cacert-manage renew command the server on which the operation is performed becomes the CA renewal master. This is to cause the CA replica that was the renewal master to retrieve the new CA certificate from LDAP instead of renewing it.


In this post I explained what the CA renewal master configuration is for and what it looks like under the hood. For FreeIPA/Dogtag system certificates, the CA renewal master configuration controls which CA replica actually performs renewal. The CA renewal master stores the renewed certificates in LDAP, and all other CA replicas look for them there. The dogtag-ipa-ca-renew-agent Certmonger renewal helper implements both of these behaviours, using the CA renewal master configuration to decide which behaviour to execute.

There must be exactly one CA renewal master in a topology and it must be operational. I discussed the consequences of various configuration or operational problems. I also explained why you might want to change the CA renewal master, and how to do it.

The CA renewal master is a critical configuration and incorrect renewal master configuration is often a factor in complex customer cases involving FreeIPA’s PKI. Commands that remove servers from the topology should elect a new CA renewal master when necessary. But misconfigurations do arise (if only we could know all the ways how!)

The upcoming FreeIPA Healthcheck feature will, among other checks, confirm that the CA renewal master configuration is sane. It will not (in the beginning at least) be able to diagnose availability or connectivity issues. But it should be able to catch some misconfigurations before they lead to catastrophic failure of the deployment.

November 20, 2018 12:00 AM

October 31, 2018

William Brown

High Available RADVD on Linux

High Available RADVD on Linux

Recently I was experimenting again with high availability router configurations so that in the cause of an outage or a failover the other router will take over and traffic is still served.

This is usually done through protocols like VRRP to allow virtual ips to exist that can be failed between. However with ipv6 one needs to still allow clients to find the router, and in the cause of a failure, the router advertisments still must continue for client renewals.

To achieve this we need two parts. A shared Link Local address, and a special RADVD configuration.

Because of howe ipv6 routers work, all traffic (even global) is still sent to your link local router. We can use an address like:


This doesn’t clash with any reserved or special ipv6 addresses, and it’s easy to remember. Because of how link local works, we can put this on many interfaces of the router (many vlans) with no conflict.

So now to the two components.


Keepalived is a VRRP implementation for linux. It has extensive documentation and sometimes uses some implementation specific language, but it works well for what it does.

Our configuration looks like:

#  /etc/keepalived/keepalived.conf
global_defs {
  vrrp_version 3

vrrp_sync_group G1 {
 group {

vrrp_instance ipv6_ens256 {
   interface ens256
   virtual_router_id 62
   priority 50
   advert_int 1.0
   virtual_ipaddress {
   garp_master_delay 1

Note that we provide both a global address and an LL address for the failover. This is important for services and DNS for the router to have the global, but you could omit this. The LL address however is critical to this configuration and must be present.

Now you can start up vrrp, and you should see one of your two linux machines pick up the address.


For RADVD to work, a feature of the 2.x series is required. Packaging this for el7 is out of scope of this post, but fedora ships the version required.

The feature is that RADVD can be configured to specify which address it advertises for the router, rather than assuming the interface LL autoconf address is the address to advertise. The configuration appears as:

# /etc/radvd.conf
interface ens256
    AdvSendAdvert on;
    MinRtrAdvInterval 30;
    MaxRtrAdvInterval 100;
    AdvRASrcAddress {
    prefix 2001:db8::/64
        AdvOnLink on;
        AdvAutonomous on;
        AdvRouterAddr off;

Note the AdvRASrcAddress parameter? This defines a priority list of address to advertise that could be available on the interface.

Now start up radvd on your two routers, and try failing over between them while you ping from your client. Remember to ping LL from a client you need something like:

ping6 fe80::1:1%en1

Where the outgoing interface of your client traffic is denoted after the ‘%’.

Happy failover routing!

October 31, 2018 02:00 PM

October 19, 2018

Fraser Tweedale

Should FreeIPA ship a subordinate CA profile?

Should FreeIPA ship a subordinate CA profile?

In my previous post I discussed how to issue subordinate CA (sub-CA) certificates from FreeIPA. In brief, the administrator must create and import a profile configuration for issuing certificates with the needed characteristics. The profile must add a Basic Constraints extension asserting that the subject is a CA.

After publishing that post, it formed the basis of an official Red Hat solution (Red Hat subscription required to view). Subsequently, an RFE was filed requesting a sub-CA profile to be included by default in FreeIPA. In this short post I’ll outline the reasons why this might not be a good idea, and what the profile might look like if we did ship one.

The case against

The most important reason not to include a sub-CA profile is that it will not be appropriate for many use cases. Important attributes of a sub-CA certificate include:

  • validity period (how long will the certificate be valid for?)
  • key usage and extended key usage (what can the certificate be used for?)
  • path length constraint (how many further subordinate CAs may be issued below this CA?)
  • name constraints (what namespaces can this CA issue certificates for?)

If we ship a default sub-CA profile in FreeIPA, all of these attributes will be determined ahead of time and fixed. There is a good chance the values will not be appropriate, and the administrator must create a custom profile configuration anyway. Worse, there is a risk that the profile will be used without due consideration of its appropriateness.

If we do nothing, we still have the blog post and official solution to guide administrators through the process. The administrator has the opportunity to alter the profile configuration according to their security or operational requirements.

The case for

The RFE description states:

Signing a subordinate CA’s CSR in IdM is difficult and requires tinkering. This functionality should be built in and present with the product. Please bundle a subordinate CA profile like the one described in the [blog post].

I agree that Dogtag profile configuration is difficult, even obtuse. It is not well documented and there is limited sanity checking. There is no “one size fits all” when it comes to sub-CA profiles, but can there be a “one size fits most”? Such a profile might have:

  • path length constraint of zero (the CA can only issue leaf certificates)
  • name constraints limiting DNS names to the FreeIPA domain (and subdomains)
  • a validity period of two years

In terms of security these are conservative attributes but they still admit the most common use case. Two years may or may not be a reasonable lifetime for the subordinate CA, but we have to choose some fixed value. The downside is that customers could use this profile without being aware of its limitations (path length, name constraints). The resulting issues will frustrate the customer and probably result in some support cases too.

Alternatives and conclusion

There is a middle road: instead of shipping the profile, we ship a “profile assistant” tool that asks some questions and builds the profile configuration. Questions would include the desired validity period, whether it’s for a CA (and if so the path length constraint), name constraints (if any), and so on. Then it imports the configuration.

There may be merit to this option, but none of the machinery exists. The effort and lead time are high. The other options: do-nothing (really improve and maintain documentation), or shipping a default sub-CA profile—are low effort and lead time.

In conclusion, I am open to either leaving sub-CA profiles as a documentation concern, or including a conservative default profile. But because there is no one size fits all, I prefer to leave sub-CA profile creation as a documented process that administrators can perform themselves—and tweak as they see fit.

October 19, 2018 12:00 AM

October 18, 2018

William Brown

Rust RwLock and Mutex Performance Oddities

Rust RwLock and Mutex Performance Oddities

Recently I have been working on Rust datastructures once again. In the process I wanted to test how my work performed compared to a standard library RwLock and Mutex. On my home laptop the RwLock was 5 times faster, the Mutex 2 times faster than my work.

So checking out my code on my workplace workstation and running my bench marks I noticed the Mutex was the same - 2 times faster. However, the RwLock was 4000 times slower.

What’s a RwLock and Mutex anyway?

In a multithreaded application, it’s important that data that needs to be shared between threads is consistent when accessed. This consistency is not just logical consistency of the data, but affects hardware consistency of the memory in cache. As a simple example, let’s examine an update to a bank account done by two threads:

acc = 10
deposit = 3
withdrawl = 5

[ Thread A ]            [ Thread B ]
acc = load_balance()    acc = load_balance()
acc = acc + deposit     acc = acc - withdrawl
store_balance(acc)      store_balance(acc)

What will the account balance be at the end? The answer is “it depends”. Because threads are working in parallel these operations could happen:

  • At the same time
  • Interleaved (various possibilities)
  • Sequentially

This isn’t very healthy for our bank account. We could lose our deposit, or have invalid data. Valid outcomes at the end are that acc could be 13, 5, 8. Only one of these is correct.

A mutex protects our data in multiple ways. It provides hardware consistency operations so that our cpus cache state is valid. It also allows only a single thread inside of the mutex at a time so we can linearise operations. Mutex comes from the word “Mutual Exclusion” after all.

So our example with a mutex now becomes:

acc = 10
deposit = 3
withdrawl = 5

[ Thread A ]            [ Thread B ]
mutex.lock()            mutex.lock()
acc = load_balance()    acc = load_balance()
acc = acc + deposit     acc = acc - withdrawl
store_balance(acc)      store_balance(acc)
mutex.unlock()          mutex.unlock()

Now only one thread will access our account at a time: The other thread will block until the mutex is released.

A RwLock is a special extension to this pattern. Where a mutex guarantees single access to the data in a read and write form, a RwLock (Read Write Lock) allows multiple read-only views OR single read and writer access. Importantly when a writer wants to access the lock, all readers must complete their work and “drain”. Once the write is complete readers can begin again. So you can imagine it as:

Time ->

T1: -- read --> x
T3:     -- read --> x                x -- read -->
T3:     -- read --> x                x -- read -->
T4:                   | -- write -- |
T5:                                  x -- read -->

Test Case for the RwLock

My test case is simple. Given a set of 12 threads, we spawn:

  • 8 readers. Take a read lock, read the value, release the read lock. If the value == target then stop the thread.
  • 4 writers. Take a write lock, read the value. Add one and write. Continue until value == target then stop.

Other conditions:

  • The test code is identical between Mutex/RwLock (beside the locking costruct)
  • –release is used for compiler optimisations
  • The test hardware is as close as possible (i7 quad core)
  • The tests are run multiple time to construct averages of the performance

The idea being that X target number of writes must occur, while many readers contend as fast as possible on the read. We are pressuring the system of choice between “many readers getting to read fast” or “writers getting priority to drain/block readers”.

On OSX given a target of 500 writes, this was able to complete in 0.01 second for the RwLock. (MBP 2011, 2.8GHz)

On Linux given a target of 500 writes, this completed in 42 seconds. This is a 4000 times difference. (i7-7700 CPU @ 3.60GHz)

All things considered the Linux machine should have an advantage - it’s a desktop processor, of a newer generation, and much faster clock speed. So why is the RwLock performance so much different on Linux?

To the source code!

Examining the Rust source code , many OS primitives come from libc. This is because they require OS support to function. RwLock is an example of this as is mutex and many more. The unix implementation for Rust consumes the pthread_rwlock primitive. This means we need to read man pages to understand the details of each.

OSX uses FreeBSD userland components, so we can assume they follow the BSD man pages. In the FreeBSD man page for pthread_rwlock_rdlock we see:


 To prevent writer starvation, writers are favored over readers.

Linux however, uses different constructs. Looking at the Linux man page:

  This is the default.  A thread may hold multiple read locks;
  that is, read locks are recursive.  According to The Single
  Unix Specification, the behavior is unspecified when a reader
  tries to place a lock, and there is no write lock but writers
  are waiting.  Giving preference to the reader, as is set by
  PTHREAD_RWLOCK_PREFER_READER_NP, implies that the reader will
  receive the requested lock, even if a writer is waiting.  As
  long as there are readers, the writer will be starved.

Reader vs Writer Preferences?

Due to the policy of a RwLock having multiple readers OR a single writer, a preference is given to one or the other. The preference basically boils down to the choice of:

  • Do you respond to write requests and have new readers block?
  • Do you favour readers but let writers block until reads are complete?

The difference is that on a read heavy workload, a write will continue to be delayed so that readers can begin and complete (up until some threshold of time). However, on a writer focused workload, you allow readers to stall so that writes can complete sooner.

On Linux, they choose a reader preference. On OSX/BSD they choose a writer preference.

Because our test is about how fast can a target of write operations complete, the writer preference of BSD/OSX causes this test to be much faster. Our readers still “read” but are giving way to writers, which completes our test sooner.

However, the linux “reader favour” policy means that our readers (designed for creating conteniton) are allowed to skip the queue and block writers. This causes our writers to starve. Because the test is only concerned with writer completion, the result is (correctly) showing our writers are heavily delayed - even though many more readers are completing.

If we were to track the number of reads that completed, I am sure we would see a large factor of difference where Linux has allow many more readers to complete than the OSX version.

Linux pthread_rwlock does allow you to change this policy (PTHREAD_RWLOCK_PREFER_WRITER_NP) but this isn’t exposed via Rust. This means today, you accept (and trust) the OS default. Rust is just unaware at compile time and run time that such a different policy exists.


Rust like any language consumes operating system primitives. Every OS implements these differently and these differences in OS policy can cause real performance differences in applications between development and production.

It’s well worth understanding the constructions used in programming languages and how they affect the performance of your application - and the decisions behind those tradeoffs.

This isn’t meant to say “don’t use RwLock in Rust on Linux”. This is meant to say “choose it when it makes sense - on read heavy loads, understanding writers will delay”. For my project (A copy on write cell) I will likely conditionally compile rwlock on osx, but mutex on linux as I require a writer favoured behaviour. There are certainly applications that will benefit from the reader priority in linux (especially if there is low writer volume and low penalty to delayed writes).

October 18, 2018 02:00 PM

Powered by Planet