Configuring SystemD with YAML makes me long for Bash

Evidently, if you deploy the Nutanix CSI Operator on Red Hat’s OpenShift Container Platform the StorageClass can claim PVs and PVCs but when it comes to binding the storage to a running container you’ll run into a very interesting error when viewing the Pod Events stream:

MountVolume.SetUp failed for volume "pvc-a139d12a-036a-490c-bc2d-214e8da078e3" : rpc error: code = Internal desc = iscsi/lvm failure, last err seen: iscsi: failed to sendtargets to portal 192.168.42.58:3260 output: Failed to connect to bus: No data available iscsiadm: can not connect to iSCSI daemon (111)! iscsiadm: Cannot perform discovery. Initiatorname required. iscsiadm: Could not perform SendTargets discovery: could not connect to iscsid , err exit status 20

Basically, what that means is that the iSCSId service isn’t running on the Red Hat CoreOS hosts.

So what can you do to enable this service?

You could SSH into each of the OCP Application nodes and do a system enable --now iscsid.service but the problem with that is that it is an unsupported anti-pattern and as soon as the system reboots or is reconfigured the service would be reset back to its normal state due to how the read-only composition of CoreOS works.

The best way to enable the needed SystemD service on all of the Application nodes would be with a MachineConfig.

A MachineConfig and the Machine Config Operator will allow you to set machine/node/RHCOS configuration as a traditional Kubernetes object. Upon updating the state of a MachineConfig the operator will compose all the known configuration and apply it to the nodes, executing a rolling reboot to apply the updated configuration.

So this instance, when you need to simply add the MachineConfig to enable the iSCSId SystemD service all you need is something like this:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-custom-enable-iscsid-worker
spec:
  config:
    ignition:
      version: 2.2.0
    systemd:
      units:
      - enabled: true
        name: iscsid.service

You don’t need to specify that the service is started/stopped, just enabled - the nodes will reboot with the newly assembled configuration specifications so whatever state they have on start up is what is used.

This can easily be used to enable or disable other SystemD services as well with little effort.

Once the MachineConfig is applied to your OpenShift cluster you’ll notice that the nodes will be cordoned off, which is a sort of taint that will disable scheduling of any new workloads on that node - then it will drain the node of any application workloads and reboot. You can see this by running something like:

oc get nodes

…which would look something like this:

NAME    STATUS                        ROLES    AGE   VERSION
app-1   Ready                         worker   12h   v1.22.3+4dd1b5a
app-2   Ready                         worker   12h   v1.22.3+4dd1b5a
app-3   NotReady,SchedulingDisabled   worker   12h   v1.22.3+4dd1b5a
...

The application nodes will transition between Ready/SchedulingDisabled, NotReady/SchedulingDisabled, Ready/SchedulingDisabled, and Ready states as they reboot to apply the new configuration.