Once in a while people come and ask for FreeIPA servers to work in multi-homed environments. A multi-homed environment in this context is a deployment where the same server is accessible through multiple network interfaces which connect together networks which are not routable to each other. This is typical for administrative and operational networks but there are other types of environments which employ disconnected networks for their operations. FreeIPA server right now has a single host name that resolves to the same IP address in all networks and if one cannot reach the server through that IP address, access to IPA server would not be possible. This typically assumes use of unicast networking as well.
A solution many people look for is to be able to access IPA servers by their
interface-specific addresses. Since all secure communication over HTTPS and other
protocols (LDAP, Kerberos, etc.) uses name-based resolution in the first place,
use of different host names is implied. For example, if FreeIPA is deployed at
DNS domain example.test
, it would be using Kerberos realm EXAMPLE.TEST
and
then the original IPA server would be deployed at a host named
ipa.example.test
(the server host name is not that important here, rather the
fact that is is an individual host name). Let’s look at possible communications
with this server in a non-multihomed environment first.
An IPA client uses HTTPS to communicate with IPA management API, SSSD on the
IPA client would use LDAP(S) and Kerberos protocols. In both HTTPS and LDAP(S)
cases TLS negotiation would force checking server TLS certificate correctness.
A hostname of the host we connect to (ipa.example.test
) would have to be
present as a dNS SAN record in the TLS certificate presented by the IPA server.
In Kerberos protocol case a different mechanism is used. Yet, Kerberos KDC must
know the name of the service principal that a client is asking a service ticket
for. If the client wants to acquire a service ticket to
ldap/ipa.example.test@EXAMPLE.TEST
, this service principal must exist in the
Kerberos database that KDC is looking up at.
There are multiple ways of exposing a single hostname in multi-homed
environment but they generally involve use of DNS views specific to the
individual networking. In such cases DNS serves visible to clients in one
network would resolve ipa.example.test
to an IP address in that specific
network. FreeIPA DNS integration does not support DNS views; this means any of
DNS manipulations would have to be done externally to FreeIPA. This is, of
course possible, but it really is not then different from a single-homed
environment from FreeIPA perspective.
Thus, we would have to have not a single ipa.example.test
hostname but for
each independent network’s address present on the IPA server a different host
name must be present. Let’s assume these are ipa1.example.test
and
ipa2.example.test
. Had we not done this split and simply added multiple
addresses for the same ipa.example.test
name, clients might resolve the name
to an IP address which they could not reach through their own networking routing.
Immediately we get a set of requirements here:
TLS certificates issued by IPA CA for HTTPS and LDAP(S) use on IPA server must include dNS SAN records for each hostname represented by the multi-homed server.
Kerberos principals for at least LDAP (ldap/
), HTTP (HTTP/
), and the
system (host/
) service principals must have aliases for all multi-homed
hostnames.
The latter requirement means that if ipa1.example.test
is the primary name,
then ldap/ipa1.example.test
should have an alias of ldap/ipa2.example.test
,
HTTP/ipa1.example.test
should have an alias of HTTP/ipa2.example.test
, and
host/ipa1.example.test
should have an alias of host/ipa2.example.test
.
The same would apply to any other service hosted on IPA server: SMB (cifs/..
)
or DNS services would need those aliases as well. However, this is not enough.
FreeIPA does additional checks when issuing certificiates prior to passing the
request to the Dogtag CA that is integrated into FreeIPA. For hosts and
services on those hosts we also check whether a requestor is granted to issue
these certificates. In FreeIPA terms, a host ipa1.example.test
would be
allowed to issue certificates with dNS SAN record of ipa2.example.test
if a
host object of ipa1.example.test
manages the host object ipa2.example.test
in FreeIPA.
Here lies our first problem. A host object in FreeIPA represents the host
principal in Kerberos, host/ipa1.example.test
. If we created two host
objects, ipa1.example.test
and ipa2.example.test
, then they cannot be
aliases to each other on Kerberos level because they’d be two completely
different objects from FreeIPA perspective.
Perhaps, we can avoid creating two different host objects? DNS records for hosts are different from the host objects themselves, we only need to have different IP addresses for the hostnames represented by these host entries, not the host entries themselves. May be we could mark one hostname an alias of the other host object?
On Kerberos level FreeIPA does have Kebreros principal name aliasing already.
However, it does not exist for hosts as this task has never appeared in past. We
would need to add a way to add multiple names to the host object. One way to
achieve that is to rely on the fact that fqdn
LDAP attribute is multi-valued.
Unfortunately, it is also enforced to be a primary key in IPA API – while the
underlying LDAP attribute is a multi-valued one, IPA API will enforce its single
value:
$ ipa host-mod ipa1.example.test --addattr fqdn=ipa2.example.test
ipa: ERROR: fqdn: Only one value allowed.
This happens because in IPA API any parameter which could be a multi-valued one
should explicitly set multivalue=True
in its definition. We probably would
need to change the multi-valued state for fqdn
parameter:
...
Str('fqdn', hostname_validator,
cli_name='hostname',
label=_('Host name'),
primary_key=True,
normalizer=normalize_hostname,
>>>>>> multivalue=True,
),
...
and review countless places where its single value is assumed through the code,
like in the resolve_fqdn()
helper below. LDAP does not guarantee a particular
order of returned values for the multi-valued attributes. From LDAP protocol
point of view they all equal, there is no particular order.
def resolve_fqdn(name):
hostentry = api.Command['host_show'](name)['result']
return hostentry['fqdn'][0]
An alternative would be to introduce a separate attribute purely for the hostname alias management. We don’t need to use it anywhere else at Kerberos level because there we use Kerberos-specific attributes to handle Kerberos principal names and aliases.
A big part of the FreeIPA access control mechanism relies on 389-ds LDAP server
access control interface. Permissions and roles in FreeIPA effectively define a
set of ACIs for 389-ds to check access rights. IPA servers verified to be
present in certain resource groups (like cn=masters,cn=ipa,cn=etc,$BASEDN
).
For host aliases this means they should be present in the same groups to be able
to operate as their own entities when checking permissions. This is important
for internal logic in IPA LDAP plugins and in KDC driver. For the cases when
authentication would be done via GSSAPI a resulting Kerberos principal will be
normalized to he primary name of the system anyway.
Issuing a certificate is a whole separate topic which still awaits its write up. An abridged version of it can be found in my freeipa-users@ mailing list response from May 2022. I need to turn that into a proper document one day.
From the perspective of aliases, we would need to teach the certificate request processing code to look at the host and service aliases when validating SAN records.
There are two approaches to setting up this multi-homed environment. We can provide all information upfront or we can add a tool that adds individual aliases after deployment. This would mean to (re-)generate certificates, create host aliases and services, create configuration snippets and other details which are required to handle multiple host names for the same host from different networks.
The after-deployment case would cause re-issuance of certificates. For external
CA providers it could be handled with existing tool that allows to replace
existing TLS certificates with externally provided ones. We need to create a
checker to verify that all required dNS SAN names were added and are available.
In general, troubleshooting this environment would be non-trivial so a special
module for ipa-healthcheck
definitely would help.
Multi-homed environments are hard to automate as many assumptions aren’t
actually known to us. They are partly implicit to system and network
administrator’s work and cannot be derived merely from the system state. It
means administrators would need to aid IPA installers with an information. At
this point, it is unclear how to structure this information and which of it
going to be useful enough. In contemporary Linux environments you might have
DNS resolution depend on a specific network interface thanks to
systemd-resolved
or VPN connection properties. We might not have that
information for introspection in advance. While automatically issuing
certificates with required names to cover multi-homed setup is not going to be
easy, writing down requirements for external CAs and verifying those
certificates before applying them at the second stage of external CA enrollment
would further complicate things.
All these problems could be solved, of course. Prioritization of this work against other, more urging tasks, is what we need to figure first…
On August 2nd-4th, 2023, Fedora Project ran its annual contributors conference, Flock to Fedora, in Cork, Ireland. After a previous successful Flock in 2019 in Budapest, Fedora contributors did not meet in person due to rough pandemia years and had created Nest with Fedora online event instead. Nest ran for three years but online meetings aren’t a full replacement for face to face collaboration. Cork’s Flock was supposed to combine both online and offline events together.
I have been attending and presenting at various Flock and Nest events over past seven years. I was looking forward to see and collaborate with many other project participants and users and get to know new people as well.
My travel to Cork was unremarkable. I took a direct flight from Helsinki to Dublin and then an Aircoach bus to Cork. The ‘unremarkable’ part was really about the unexpected delays people did report over the Matrix channel. The only ‘trouble’ I had was to catch a taxi at 11pm after arrival to Cork to get to my B&B. The Aircoach bus from Dublin airport is very popular in summer and whatever taxi fleet is in Cork was DDoSed by the passengers.
Cork is hilly. I stayed in excellent B&B across the road from the conference hotel. The hotel and its conference facilities are in separate buildings; the events building is up hill from the hotel. Walking is helpful, climbing harder but given we are sitting most of the time, was a welcoming ‘struggle’. Perhaps, my stay outside of the conference hotel has also helped to avoid COVID-19 which few other participants, sadly, contracted. It is hit or miss every time.
Unfortunately, not everyone made to Cork. Marina Zhurakhinskaya passed away in June 2022. Ben Cotton, Fedora Program Manager, has been let go as a part of Red Hat’s layoffs earlier this year. Both had definitely changed Fedora project dramatically, in many ways, both leading to openness and friendliness Fedora is known for. Many presenters remembered both Marina and Ben during their sessions.
2023’s edition of Flock to Fedora was also the first Fedora Project’s event collocated with CentOS Connect. As a result, it brought together Red Hat Enterprise Linux distribution upstreams and downstreams together.
In total, there were up to four parallel tracks, dedicated to different areas of a distribution development and a project’s life and spanned over three days. That, unsurprisingly, made it challenging to visit all talks and activities. It is a common trait shared by many successful events. And for those who wanted to continue discussions after a talk has ended, there is always a ‘hallway track’.
The first talk was ‘State of Fedora 2023’ by the project leader, Matthew Miller. Recording is available here. I am linking to the re-take of the talk as the original streaming was off by 20 minutes and Matthew had to reprise it again.
A major announcement made during the talk was a hiring one. Fedora Operations Architect role has been introduced after a program manager role that Ben Cotton so masterfully executed was eliminated. Hopefully, this new role will be filled soon and will allow to capture the same benefits that Ben brought to Fedora. The role is a bit different, though, as it is focused on cross-project and cross-distro impact across Fedora and RHEL.
Fedora contributors’ survey results were also unveiled by Matthew in the talk. In general, contributors keep trust in the project and continue their participation at the pre-pandemia levels. Recent social networking turmoil around Red Hat actions hasn’t influenced the results too much. The screenshots below are from the video stream as the talk’s slides aren’t yet available.
The talk went into details on what Matthew and Fedora Council aim for Fedora Project’s future. Growing a project with thousands of contributors spread around the world and representing different cultures is hard. A lot of effort is put into making Fedora a welcoming place to everyone who is willing to work together towards a common goal.
rpminspect
: Lessons from three distributionsDavid Cantrell created and maintains a tool that helps RHEL maintainers to keep their packages sane over years of maintenance. It is run as a part of CentOS Stream merge request process, as part of Fedora gating and pull request testing, and as a gating test for RHEL.
The talk itself is an excellent retrospective on what one should consider when creating a new Open Source project while working on it full time. David provided observations on how to sell the idea to your management, how to get people interested in becoming a community for your project, how to sustain development in a long run. This is one of rare gems of a ‘lone wolf’ maintainership stories that everybody needs to absorb when they start their new journey. Believe me, it is worth it.
Tim Flink from Fedora QE team decided to apply AI/ML to a problem of identifying hanging or failing jobs in OpenQA. OpenQA runs full-VM tests and records screencasts of everything what is shown on a VM screen. A crash of a graphical environment is abruptly visible there as graphics would be replaced by a terminal with a Wayland’s stacktrace. What followed is an experiment on processing these screens to reliably detect a particular type of a crash.
We spent some time with Tim discussing how these experiments can be applied to finding out possible issues in other system reports. Since Fedora is upstream of CentOS Stream or RHEL, it means certain issues – and their fixes – would often appear in Fedora first. If we could train a model on those issues in Fedora, can we detect automatically whether a particular fix is required in RHEL later? This is quite relevant to FreeIPA and SSSD as we do run their tests in OpenQA as a part of Fedora Server release criteria.
Another possible use case is to do a reverse training. Since we do know how a potential failure could look like, we can intentionally build an OpenQA test environment that would reproduce the failure and then train a model to recognize logs from such failures in real life scenarios. For example, establishing trust to Active Directory in FreeIPA is reliant on a working DNS setup, working firewall, etc. Failure to communicate through an incomplete firewall would be reflected with timeouts in the logs which we could train to recognize. There are endless possibilities here to aid through known errors…
On a similar note, I had discussion with Amazon’s David Duncan in the ‘hallway track’ which started from an observation that Cloud SIG would really benefit from our passwordless work: distributing VMs with pre-set passwords is not ideal, an ability to inject FIDO2 passkey information and have everything obey it at login in cloud would be great to have. Somewhere along this way, discussion switched to CoreOS-based environments and I realised my experiments with Fedora Silverblue to develop passwordless support for FreeIPA would probably be a subject to a talk that would be interesting to others as well.
I am running my own Silverblue images which source SSSD and FreeIPA upstream test builds to allow me easily to switch between different potential options in one go, without messing with an installation environment. It is quite important for the integration work we do and would be crucial for end-to-end testing of upcoming GNOME changes.
This also provided me an insight into what container-based environments need from FreeIPA and overall from enterprise domains to fit nicely. I should have submitted a talk about that to Flock! Well, I will do one next year, for sure. (And, TODO: file issues to track for that integration to FreeIPA upstream!)
Another interesting discussion we had with Jonathan Dieter. Jonathan is a long term Fedora contributor and FreeIPA user. For past several years Jonathan works with a local Irish company that provides services around the world to test local phone numbers. They maintain an infrastructure in more than 80 countries where there might be no global cloud providers at all. To keep that infrastructure reliable, they use FreeIPA (not alone, of course) and OStree-based images.
It is one of the talks that I missed to attend in person as conflicts are inevitable: Mo Duffy’s Podman Desktop talk and Adam Williamson’s Fedora CI state talk were running at the same time.
Asahi Linux is a project which aims to upstream support for Apple’s ARM64 architecture, best known through Apple’s M1 and M2 systems. At the Flock Asahi Linux project members have announced that not only Fedora Asahi Remix will be the flagship distribution for the project, but also Fedora Discourse instance will be used to handle Asahi Linux community collaborations.
Asahi’s announcement also an example of how friendly has become Fedora Project as a community over years. I am definitely looking forward to see the remix to become one of official builds of Fedora.
Mo Duffy gave an outstanding talk about using Podman desktop to deliver workloads for non-technical people. It was a highlight of the conference, for sure. She also made few interesting points. For one, running cloud-based workloads locally to allow offline operations is nice. Mo demonstrated a Penpot instance, which is a design and prototyping application. Running it locally helps to maintain the same workflow while on an intercontinental flight. However, even more interesting is that this approach also allows to use a cloud software that otherwise is considered insecure. For example, running a Wordpress setup locally to benefit from its nice UI in a local browser and export static web site content to push to the actual web hosting.
By lowering a barrier to use containerised applications through Podman Desktop we may hope to get more people join our community and contribute. Starting with Podman Desktop’s friendliness would allow these newcomers to discover other Fedora flavors and features. It is certainly an interesting aspect we could expand further in a way similar how this ‘F’ in Fedora got expanded in Mo’s presentation.
Another conference highlight was the panel that brought representatives of Fedora, RHEL, Rocky Linux, Alma Linux, and CentOS Stream together on stage. Distributions upstream and downstream of RHEL presented their views on various development and community topics. It is worth to watch the stream.
Trow Dawson and Carl George did present another state of EPEL. EPEL has a solid contributors’ base who keep thousands of packages available to RHEL and downstream distributions’ users. EPEL is using Fedora infrastructure and for many packages it shares maintainers with Fedora (EPEL branches are branches in Fedora dist-git for the same package, if this package is not in RHEL). So all EPEL contributors are Fedora contributors. ;)
One interesting aspect in every “State of EPEL” talk is a long tail of the EPEL demographics. Much like “State of Fedora” shows demographics of Fedora releases, EPEL statistics includes details on who is running the lowest number of downstream systems:
My talk was on the morning of the second day. People were still recovering from the night of International Candy Swap and table games so at start I had may be a couple of attendies. Eventually, we’ve got more people in the room and there were also online attendees so it wasn’t so feeling so lonely.
My talk was similar to previous ones at FOSDEM and SambaXP. What was new is a demo from Ray Strode on how potentially a user experience could look like in GNOME for a passwordless login. Ray implemented a prototype of external identity provider login flow that Allan Day has shared recently. This flow could be used for login through Microsoft’s Entra ID (a.k.a. Azure AD) or any OAuth2 provider supported by FreeIPA. We aren’t fully there yet but the goal is to do this work once for GNOME and reuse for various passwordless authentication approaches supported through SSSD.
I also showed an old demo from my FOSDEM and Flock 2016. It shows how we integrated 2FA tokens (Yubikeys in this example) with FreeIPA to authenticate and obtain Kerberos tickets through a KDC proxy over HTTPS. These tickets then were used to login onto a VPN. This is something that is possible in Fedora and RHEL for almost a decade now.
Before Flock, Adam Williamson started to work on integrating Samba AD tests into OpenQA for Fedora. It almost worked well but there were few issues Adam wasn’t able to resolve so we set down at the Flock and figured out at least few of those. The only remaining one was an apparent race condition within a test that enrolls a system to Samba AD using kickstart. SSSD, it seems, starts before networking is up and stable, and decides that it is offline. When the test tries to resolve an Active Directory user, SSSD fails to do so as it thinks to be offline.
Interestingly, the same test against FreeIPA works fine. The same test done past kickstart works fine as well, for both FreeIPA and Samba AD. There is probably a need to add a waiting period to settle a network state. We saw this in past too but never found a good way to trigger a proper event for SSSD to recover.
I am trying to reduce candy consumption so I skipped the social events on the first day but attended the conference dinner on the second day. All social events during the Flock well organized and this one wasn’t exception either. We had interesting discussions with Fedora and Rocky folks, getting to know there is a lot of similarity in how people do their lives across the world.
On Friday’s night another social event was a Ghost Tour. However, we skipped it and together with few other people went to do a bit of memorabilia road through another Mexican place and (of course!) a local bar. Life in IT and development in 90’s and early 2000’s weren’t that much different in US and Europe, really. Thanks to Spot and Amazon for covering the dinner, thanks to other folks for beer and a company.
I left on Saturday at noon using the same Aircoach bus towards Dublin airport. The bus was full – make sure you have booked your seat online in advance. My flight back to Finland was uneventful as well. Overall, it was a great conference, as usual. I’d like to say thank you to all volunteers and organizers who keep Flock so wonderful and Fedora project so welcoming. Thank you!