Immutable Infrastructure

Creating Clusters on Huawei Cloud Stack

This document provides comprehensive instructions for creating Kubernetes clusters on the Huawei Cloud Stack platform using Cluster API.

TOC

Prerequisites1. Required Plugin Installation2. HCS Infrastructure Input PreparationCluster Creation OverviewControl Plane ConfigurationConfigure HCS AuthenticationConfigure Machine Configuration PoolConfigure Machine TemplateConfigure KubeadmControlPlaneConfigure HCSClusterConfigure ClusterCluster VerificationUsing kubectlExpected ResultsAdding Worker NodesUpgrading Clusters

Prerequisites

Before creating clusters, ensure all of the following prerequisites are met:

1. Required Plugin Installation

Install the following plugins on the 's global cluster:

  • Alauda Container Platform Kubeadm Provider
  • Alauda Container Platform HCS Infrastructure Provider

For detailed installation instructions, refer to the Installation Guide.

2. HCS Infrastructure Input Preparation

Prepare all HCS-specific inputs before writing any YAML in this document:

  • HCS credential Secret values
  • Provider-recognized compute values such as imageName, flavorName, and availabilityZone
  • Cluster network inventory, including the subnets and free IP ranges used by the cluster
  • Control plane ELB address planning, including vipAddress, vipSubnetName, and optional fixed L4 and L7 IPs
  • Static IP pool planning for control plane and worker nodes

See Infrastructure Resources for Huawei Cloud Stack for the complete checklist, source information, and constraints.

Cluster Creation Overview

At a high level, you'll create the following Cluster API resources in the 's global cluster to provision infrastructure and bootstrap a functional Kubernetes cluster.

Before you write any YAML in this page, complete the preparation checklist in Infrastructure Resources for Huawei Cloud Stack. This checklist covers the values that the provider expects, where to get them, and which values must be planned before you fill the manifests.

WARNING

Important Namespace Requirement

To ensure proper integration with the as business clusters, all resources must be deployed in the cpaas-system namespace. Deploying resources in other namespaces may result in integration issues.

The cluster creation process follows this order:

  1. Configure HCS authentication (Secret)
  2. Create machine configuration pool (HCSMachineConfigPool)
  3. Configure machine template (HCSMachineTemplate)
  4. Configure KubeadmControlPlane
  5. Configure HCSCluster
  6. Create the Cluster

Control Plane Configuration

The control plane manages cluster state, scheduling, and the Kubernetes API. This section shows how to configure a highly available control plane.

WARNING

Configuration Parameter Guidelines

When configuring resources, exercise caution with parameter modifications:

  • Replace only values enclosed in <> with your environment-specific values
  • Preserve all other parameters as they represent optimized or required configurations
  • Modifying non-placeholder parameters may result in cluster instability or integration issues

Configure HCS Authentication

HCS authentication information is stored in a Secret resource.

apiVersion: v1
kind: Secret
metadata:
  name: <credential-secret-name>
  namespace: cpaas-system
type: Opaque
data:
  accessKey: <base64-encoded-access-key>
  secretKey: <base64-encoded-secret-key>
  projectID: <base64-encoded-project-id>
  region: <base64-encoded-region>
  externalGlobalDomain: <base64-encoded-domain>
ParameterDescription
.data.accessKeyHCS access key ID from My Settings > Access Keys (base64-encoded)
.data.secretKeyHCS secret access key from My Settings > Access Keys (base64-encoded)
.data.projectIDResource Space ID from My Settings > Resource Spaces (base64-encoded)
.data.regionHCS region API value used by the provider (base64-encoded). Tenant administrators cannot retrieve this value from the HCS UI; get it from the HCS administrator
.data.externalGlobalDomainHCS platform access domain used by the provider (base64-encoded)

You can reuse an existing HCS credential Secret. Its name does not need to match the cluster name, but HCSCluster.spec.identityRef.name must reference this Secret.

Configure Machine Configuration Pool

The HCSMachineConfigPool defines pre-configured hostnames and static IP addresses for VMs.

WARNING

Pool Size Requirement

The configuration pool must include at least as many entries as the number of control plane nodes you plan to deploy.

Use one subnet selector per networks[] entry. For new manifests, set either subnetName or subnetId, but not both. Existing manifests may keep the deprecated subenetName field; if you also add subnetName while updating that manifest, its value must exactly match subenetName. Do not supply conflicting values across subenetName, subnetName, and subnetId.

If you use subnetName in the machine configuration pool, include the same subnet name in HCSCluster.spec.network.subnets.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineConfigPool
metadata:
  name: <cluster-name>
  namespace: cpaas-system
spec:
  configs:
    - hostname: master-1
      networks:
        - subnetName: <subnet-name>
          ipAddress: 192.168.1.11
    - hostname: master-2
      networks:
        - subnetName: <subnet-name>
          ipAddress: 192.168.1.12
    - hostname: master-3
      networks:
        - subnetName: <subnet-name>
          ipAddress: 192.168.1.13
ParameterTypeRequiredDescription
.spec.configs[]arrayYesNon-empty list of machine configurations
.spec.configs[].hostnamestringYesVM hostname. Use lowercase letters, numbers, hyphens (-), or dots (.); the value must start and end with a lowercase letter or number and must not exceed 253 characters
.spec.configs[].networks[]arrayYesNon-empty list of network configurations for the VM
.spec.configs[].networks[].subnetNamestringNo*Recommended subnet name field for new manifests
.spec.configs[].networks[].subnetIdstringNo*Subnet ID. Use this field instead of subnetName when the subnet name is ambiguous
.spec.configs[].networks[].ipAddressstringYesStatic IP address for the VM

*For new manifests, set either subnetName or subnetId. Existing manifests may continue to use subenetName, and may add subnetName only if both fields use the same value. Do not provide conflicting subnet selector values.

Note: The CRD schema lists subnetName, subenetName, and subnetId as optional fields and does not express their allowed combinations. Follow the provider-level rules above when writing manifests.

Configure Machine Template

The HCSMachineTemplate defines the VM specifications for control plane nodes.

WARNING

Storage Requirements

The following data disk mount points are recommended for control plane nodes:

  • /var/lib/etcd - etcd data (10GB+)
  • /var/lib/kubelet - kubelet data (100GB+)
  • /var/lib/containerd - container runtime data (100GB+)
  • /var/cpaas - platform data and logs (40GB+)
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineTemplate
metadata:
  name: <cluster-name>-control-plane
  namespace: cpaas-system
spec:
  template:
    spec:
      imageName: <vm-image-name>
      flavorName: <instance-flavor>
      availabilityZone: <availability-zone>
      rootVolume:
        type: SSD
        size: 100
      configPoolRef:
        name: <cluster-name>
      dataVolumes:
        - size: 10
          type: SSD
          mountPath: /var/lib/etcd
          format: xfs
        - size: 100
          type: SSD
          mountPath: /var/lib/kubelet
          format: xfs
        - size: 100
          type: SSD
          mountPath: /var/lib/containerd
          format: xfs
        - size: 40
          type: SSD
          mountPath: /var/cpaas
          format: xfs
ParameterTypeRequiredDescription
.spec.template.spec.imageNamestringYesHCS image name such as microos-4.2.1-new
.spec.template.spec.flavorNamestringYesProvider-recognized HCS API value matched against Flavor.Name. Do not use the tenant UI display name
.spec.template.spec.availabilityZonestringNoProvider-recognized HCS API value matched against ZoneName. Do not use the tenant UI display name
.spec.template.spec.rootVolume.typestringYesVolume type (SSD or SATA)
.spec.template.spec.rootVolume.sizeintYesSystem disk size in GB
.spec.template.spec.configPoolRef.namestringYesReferenced HCSMachineConfigPool name
.spec.template.spec.dataVolumes[]arrayNoData volume configurations
.spec.template.spec.dataVolumes[].sizeintYes*Data disk size in GB
.spec.template.spec.dataVolumes[].typestringYes*Volume type
.spec.template.spec.dataVolumes[].mountPathstringYes*Mount path
.spec.template.spec.dataVolumes[].formatstringYes*File system format (xfs or ext4)

*Required when dataVolumes is specified.

Note: Do not set runtime identity fields such as providerID or serverId in HCSMachineTemplate manifests. The provider assigns these values when it creates HCS instances.

Note: Tenant administrators cannot retrieve the provider-recognized flavorName and availabilityZone values from the HCS UI. Get the exact values from the HCS administrator before you apply the manifest.

Configure KubeadmControlPlane

The KubeadmControlPlane defines the Kubernetes control plane configuration.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <cluster-name>
  namespace: cpaas-system
spec:
  replicas: 3
  version: <kubernetes-version>
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
  kubeadmConfigSpec:
    files:
      - path: /etc/kubernetes/admission/psa-config.yaml
        owner: root:root
        permissions: "0644"
        content: |
          apiVersion: apiserver.config.k8s.io/v1
          kind: AdmissionConfiguration
          plugins:
          - name: PodSecurity
            configuration:
              apiVersion: pod-security.admission.config.k8s.io/v1
              kind: PodSecurityConfiguration
              defaults:
                enforce: "privileged"
                enforce-version: "latest"
                audit: "baseline"
                audit-version: "latest"
                warn: "baseline"
                warn-version: "latest"
              exemptions:
                usernames: []
                runtimeClasses: []
                namespaces:
                - kube-system
                - cpaas-system
      - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
        owner: root:root
        permissions: "0644"
        content: |
          {
            "apiVersion": "kubelet.config.k8s.io/v1beta1",
            "kind": "KubeletConfiguration",
            "protectKernelDefaults": true,
            "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt",
            "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key",
            "streamingConnectionIdleTimeout": "5m",
            "clientCAFile": "/etc/kubernetes/pki/ca.crt"
          }
      - path: /etc/kubernetes/encryption-provider.conf
        owner: root:root
        permissions: "0600"
        content: |
          apiVersion: apiserver.config.k8s.io/v1
          kind: EncryptionConfiguration
          resources:
          - resources:
            - secrets
            providers:
            - aescbc:
                keys:
                - name: key1
                  secret: bootstrap-placeholder
      - path: /etc/kubernetes/audit/policy.yaml
        owner: root:root
        permissions: "0644"
        content: |
          apiVersion: audit.k8s.io/v1
          kind: Policy
          rules:
          - level: Metadata
    clusterConfiguration:
      imageRepository: <image-repository>
      dns:
        imageTag: <dns-image-tag>
      etcd:
        local:
          imageTag: <etcd-image-tag>
      apiServer:
        extraArgs:
          audit-log-format: json
          audit-log-mode: batch
          audit-log-path: /etc/kubernetes/audit/audit.log
          audit-policy-file: /etc/kubernetes/audit/policy.yaml
          admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml
          encryption-provider-config: /etc/kubernetes/encryption-provider.conf
          kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt
          profiling: "false"
          tls-min-version: VersionTLS12
        extraVolumes:
          - name: vol-dir-0
            hostPath: /etc/kubernetes
            mountPath: /etc/kubernetes
            pathType: Directory
      controllerManager:
        extraArgs:
          bind-address: "::"
          flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
          profiling: "false"
          tls-min-version: VersionTLS12
      scheduler:
        extraArgs:
          bind-address: "::"
          profiling: "false"
          tls-min-version: VersionTLS12
    postKubeadmCommands:
      - chmod 600 /var/lib/kubelet/config.yaml
    initConfiguration:
      patches:
        directory: /etc/kubernetes/patches
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
          protect-kernel-defaults: "true"
          volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
    joinConfiguration:
      patches:
        directory: /etc/kubernetes/patches
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
          volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
  machineTemplate:
    nodeDrainTimeout: 1m
    nodeDeletionTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: HCSMachineTemplate
      name: <cluster-name>-control-plane

The HCS controller also injects files while resolving cloud-init data. It writes /etc/kubernetes/pki/kubelet.crt, /etc/kubernetes/pki/kubelet.key, and /etc/kubernetes/encryption-provider.conf for control plane machines. For the first control plane machine, the controller generates the encryption provider configuration. After the control plane is initialized, it tries to reuse the existing kube-apiserver encryption provider configuration. If you include a bootstrap file at /etc/kubernetes/encryption-provider.conf, treat it as a placeholder because the controller-generated or synchronized file takes precedence.

Note: Configure apiServer.extraArgs and apiServer.extraVolumes together. If the volume is not mounted, kube-apiserver cannot read the files written under /etc/kubernetes.

Note: For HCS control planes that use a fixed-size static IP pool, keep rolloutStrategy.rollingUpdate.maxSurge: 0 so replacements happen in a scale-down-then-scale-up order. This default upgrade path usually does not require additional control plane IPs. If you plan to increase control plane replicas or set maxSurge greater than 0, first extend the referenced HCSMachineConfigPool with additional hostname and static IP entries.

For component versions (DNS image tag, etcd image tag), refer to the OS Support Matrix.

Configure HCSCluster

The HCSCluster resource defines the HCS infrastructure configuration.

The HCS provider creates an ELB on the HCS platform for the Kubernetes API server. This ELB must use mixed load balancing so cluster nodes can also reach the API server through the ELB address.

If you want all ELB-related addresses to be fixed, provide vipAddress, elbVirsubnetL4Ips, and elbVirsubnetL7Ips. Each elbVirsubnetL4Ips[].ips and elbVirsubnetL7Ips[].ips entry must contain two IPs. If you omit the L4 or L7 virtual subnet IPs, HCS allocates them randomly.

If you set vipDomainName, configure HCS Cloud DNS Private Zones so that the domain resolves to vipAddress.

List every cluster subnet in spec.network.subnets before you reference it anywhere else. vipSubnetName, elbVirsubnetL4Ips[].subnetName, elbVirsubnetL7Ips[].subnetName, and the subnetName values used by HCSMachineConfigPool must all exist in spec.network.subnets.

Do not disable Hybrid Load Balancing on the provider-created ELB after the cluster is created. The cluster depends on that ELB mode so nodes can reach the API server through the ELB address.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSCluster
metadata:
  name: <cluster-name>
  namespace: cpaas-system
spec:
  controlPlaneLoadBalancer:
    vipAddress: <control-plane-vip-address>
    vipSubnetName: <vip-subnet-name>
    vipDomainName: <control-plane-vip-domain-name>
    elbVirsubnetL4Ips:
      - subnetName: <subnet-name>
        ips:
          - <l4-ip-1>
          - <l4-ip-2>
    elbVirsubnetL7Ips:
      - subnetName: <subnet-name>
        ips:
          - <l7-ip-1>
          - <l7-ip-2>
  networkType: kube-ovn
  network:
    vpc:
      name: <vpc-name>
    subnets:
      - name: <subnet-name>
    securityGroup:
      name: <security-group-name>
  identityRef:
    name: <credential-secret-name>
ParameterTypeRequiredDescription
.spec.networkTypestringYesNetwork type, currently supports kube-ovn
.spec.network.vpc.namestringYesVPC name
.spec.network.subnets[].namestringYesComplete subnet inventory used by this cluster. Every subnet referenced elsewhere in the cluster manifests must appear in this list
.spec.network.securityGroup.namestringYesSecurity group name
.spec.identityRef.namestringYesNon-empty credential Secret name referenced by HCSCluster; this value does not need to match the cluster name
.spec.controlPlaneLoadBalancerobjectYesELB settings for the provider-created control plane ELB
.spec.controlPlaneLoadBalancer.vipAddressstringYesFixed VIP for the control plane ELB
.spec.controlPlaneLoadBalancer.vipSubnetNamestringYesSubnet name that contains the ELB VIP. This subnet must also appear in .spec.network.subnets
.spec.controlPlaneLoadBalancer.vipDomainNamestringNoDomain name for the VIP. Configure HCS Cloud DNS Private Zones so it resolves to vipAddress
.spec.controlPlaneLoadBalancer.elbVirsubnetL4Ips[]arrayNo*L4 virtual subnet IP groups used by Hybrid Load Balancing
.spec.controlPlaneLoadBalancer.elbVirsubnetL4Ips[].subnetNamestringYes*Subnet name for the L4 virtual subnet IPs. This subnet must also appear in .spec.network.subnets
.spec.controlPlaneLoadBalancer.elbVirsubnetL4Ips[].ips[]stringYes*Two fixed L4 virtual subnet IPs
.spec.controlPlaneLoadBalancer.elbVirsubnetL7Ips[]arrayNo*L7 virtual subnet IP groups used by Hybrid Load Balancing
.spec.controlPlaneLoadBalancer.elbVirsubnetL7Ips[].subnetNamestringYes*Subnet name for the L7 virtual subnet IPs. This subnet must also appear in .spec.network.subnets
.spec.controlPlaneLoadBalancer.elbVirsubnetL7Ips[].ips[]stringYes*Two fixed L7 virtual subnet IPs

*Required when you want to fix all ELB-related addresses. If you omit the L4 or L7 virtual subnet IPs, HCS allocates them randomly.

Do not include spec.controlPlaneEndpoint in the create manifest. In the HCS create flow, the controller derives and populates this field from spec.controlPlaneLoadBalancer after the HCSCluster is created. Do not set controlPlaneEndpoint manually, and do not add an empty controlPlaneEndpoint object. If controlPlaneEndpoint is explicitly present in the manifest, it must include both host and port.

Configure Cluster

The Cluster resource in Cluster API declares the cluster and references the control plane and infrastructure resources.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster-name>
  namespace: cpaas-system
  annotations:
    cpaas.io/sentry-deploy-type: Baremetal
    cpaas.io/alb-address-type: ClusterAddress
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: HCSCluster
    cpaas.io/kube-ovn-join-cidr: <kube-ovn-join-cidr>
    cpaas.io/kube-ovn-version: <kube-ovn-version>
  labels:
    cluster-type: HCS
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - <pods-cidr>
    services:
      cidrBlocks:
        - <services-cidr>
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: <cluster-name>
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: HCSCluster
    name: <cluster-name>
AnnotationDescription
cpaas.io/sentry-deploy-typeDeployment type, set to Baremetal
cpaas.io/alb-address-typeALB address type, set to ClusterAddress
capi.cpaas.io/resource-group-versionResource group version
capi.cpaas.io/resource-kindResource type, set to HCSCluster
cpaas.io/kube-ovn-join-cidrKube-OVN join subnet CIDR
cpaas.io/kube-ovn-versionKube-OVN version

Cluster Verification

After deploying all cluster resources, verify that the cluster has been created successfully.

Using kubectl

# Check cluster status
kubectl get cluster -n cpaas-system <cluster-name>

# Verify control plane nodes
kubectl get kubeadmcontrolplane -n cpaas-system <cluster-name>

# Check machine status
kubectl get machines -n cpaas-system

# Verify cluster deployment status
kubectl get clustermodule <cluster-name> -o jsonpath='{.status.base.deployStatus}'

Expected Results

A successfully created cluster should show:

  • Cluster status: Running or Provisioned
  • All control plane machines: Running
  • Kubernetes nodes: Ready
  • Cluster Module Status: Completed

Adding Worker Nodes

For instructions on adding worker nodes to the cluster, refer to Managing Nodes.

Upgrading Clusters

For instructions on upgrading cluster components, refer to Upgrading Clusters.