Flink

flink使用 本文讲解的是在k8s中使用flink,摆脱yarn, hdfs等大数据组件 k8s 部署(手动部署HA模式) 想要实现K8S的HA模式必须要有几个条件 要有能查看,添加,修改,删除configmap权限的账户 # flink-service-account.yaml apiVersion: v1 kind: ServiceAccount metadata: name: flink-sa namespace: flink-namespace --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: flink-role namespace: flink-namespace rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["create", "get", "list", "watch", "update", "delete", "patch"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["services"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: flink-role-binding namespace: flink-namespace subjects: - kind: ServiceAccount name: flink-sa namespace: flink-namespace roleRef: kind: Role name: flink-role apiGroup: rbac.authorization.k8s.io 配置文件开启高可用模式为kubernetes和配置集群id,检查点和重启策略 #高可用配置 high-availability.type: kubernetes high-availability.storageDir: file:///flink/storage/ha #集群信息 kubernetes.cluster-id: flink-cluster-001 kubernetes.namespace: flink-namespace #状态后端,检查点保存点默认位置,可以配置为oss state.backend.type: filesystem state.checkpoints.dir: file:///flink/storage/checkpoints state.savepoints.dir: file:///flink/storage/savepoints #检查点执行配置 execution.checkpointing.interval: 20s execution.checkpointing.max-concurrent-checkpoints: 1 execution.checkpointing.mode: EXACTLY_ONCE execution.checkpointing.timeout: 1min #重启策略 restart-strategy.type: fixed-delay restart-strategy.fixed-delay.attempts: 10 restart-strategy.fixed-delay.delay: 30s session模式 创建命名空间 我们把flink部署到名称为flink-namespace的命名空间中(创建命名空间kubectl create namespace flink-namespace) ...