Table of Contents

Search

  1. Preface
  2. Advanced clusters
  3. Setting up AWS
  4. Setting up Google Cloud
  5. Setting up Microsoft Azure
  6. Setting up a self-service cluster
  7. Setting up a local cluster
  8. Advanced configurations
  9. Troubleshooting
  10. Appendix A: Command reference

Advanced Clusters

Advanced Clusters

Configure role permissions

Configure role permissions

Configure permissions for the Kubernetes ClusterRole to create and manage resources in the Kubernetes cluster.
The following table describes each resource that the ClusterRole needs to access:
Resource
Description
Services
Used to communicate across Kubernetes Pods.
Pods
Used to run Spark drivers and Spark executors.
Secrets
Used to pass sensitive metadata to Kubernetes Pods.
Configmaps
Used to pass Spark configurations to Kubernetes Pods.
DaemonSets
Used to deploy the Spark shuffle service.
Deployments
Used to deploy a keystore on the cluster so that Kubernetes Pods can use keys to access the Secure Agent.
The permissions required depend on whether the cluster runs mappings with the Spark shuffle service. The Spark shuffle service helps the cluster perform dynamic allocation for Spark jobs. The service is responsible for persisting shuffle files beyond the lifetime of the executors, allowing the number of executors to scale up and down without losing computation.
A ClusterRole is global and not associated with a namespace. If you need to restrict permissions to a specific namespace, you can split the ClusterRole permissions into two different roles.

Minimum permissions to run a mapping with the Spark shuffle service

The following code snippet shows the minimum permissions required to run a mapping with the Spark shuffle service:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: optimized-cluster-role rules: - apiGroups: [""] resources: ["services","pods","secrets","configmaps"] verbs: ["watch","list","get","create","update","patch","delete","deletecollection"] - apiGroups: ["apps"] resources: ["daemonsets","deployments"] verbs: ["watch","list","get","create","update","patch","delete","deletecollection"] - apiGroups: [""] resources: ["nodes"] verbs: ["watch","list","get","patch"] - apiGroups: [""] resources: ["namespaces","persistentvolumeclaims"] verbs: ["watch","list","get"]
The permissions in this code snippet applies to all namespaces.
If you need to limit the permissions to certain namespaces only, split these permissions into two roles: Role and ClusterRole. ClusterRole contains permissions for resources that are global, while Role contains permissions for resources that are specific to a namespace.
The following code snippet shows the permissions for Role:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbac-informatica-np-admin namespace: informatica rules: - apiGroups: [""] resources: ["services","pods","secrets","configmaps"] verbs: ["watch","list","get","create","update","patch","delete","deletecollection"] - apiGroups: ["apps"] resources: ["daemonsets","deployments"] verbs: ["watch","list","get","create","update","patch","delete","deletecollection"]
The following code snippet shows the permissions for ClusterRole:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: rbac-informatica-global-admin rules: - apiGroups: [""] resources: ["nodes"] verbs: ["watch","list","get","patch"] - apiGroups: [""] resources: ["namespaces", "persistentvolumeclaims"] verbs: ["watch","list","get"]

Minimum permissions to run a mapping without the Spark shuffle service

The following code snippet shows the minimum permissions required to run a mapping without the Spark shuffle service:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: optimized-cluster-role rules: - apiGroups: [""] resources: ["services","pods","secrets","configmaps"] verbs: ["watch","list","get","create","update","patch","delete","deletecollection"] - apiGroups: [""] resources: ["nodes"] verbs: ["watch","list","get"] - apiGroups: [""] resources: ["namespaces","persistentvolumeclaims"] verbs: ["watch","list","get"]
The permissions in this code snippet applies to all namespaces.
If you need to limit the permissions to certain namespaces only, split these permissions into two roles: Role and ClusterRole. ClusterRole contains permissions for resources that are global, while Role contains permissions for resources that are specific to a namespace.
The following code snippet shows the permissions for Role:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbac-informatica-np-admin namespace: informatica rules: - apiGroups: [""] resources: ["services","pods","secrets","configmaps"] verbs: ["watch","list","get","create","update","patch","delete","deletecollection"]
The following code snippet shows the permissions for ClusterRole:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: rbac-informatica-global-admin rules: - apiGroups: [""] resources: ["nodes"] verbs: ["watch","list","get"] - apiGroups: [""] resources: ["namespaces", "persistentvolumeclaims"] verbs: ["watch","list","get"]

Permissions to enable job priority (optional)

Optionally, you can enable job priority to allow the cluster role to schedule jobs on the self-service cluster according to the job priority that a developer sets for a
mapping
task in
Data Integration
.
To enable job priority, complete the following tasks:
  1. Grant the following permissions to the cluster role:
    rules: - apiGroups: ["scheduling.k8s.io"] resources: ["priority classes"] verbs: ["list","create","update","patch","delete"]
  2. Set the following custom property in the
    advanced configuration
    :
    ccs.enable.app.priority=true

0 COMMENTS

We’d like to hear from you!