DolphinScheduler 实战
一、伪集群部署
伪集群部署目的是在单台机器部署 DolphinScheduler 服务，该模式下master、worker、api server、logger server都在同一台机器上。
前置准备工作
伪分布式部署 DolphinScheduler 需要有外部软件的支持
- JDK：下载JDK (1.8+)，并将 JAVA_HOME 配置到以及 PATH 变量中。如果你的环境中已存在，可以跳过这步。
- 二进制包：在下载页面下载 DolphinScheduler 二进制包
- 数据库：PostgreSQL (8.2.15+) 或者 MySQL (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
- 注册中心：ZooKeeper (3.4.6+)，下载地址
- 进程树分析
- macOS安装pstree
- Fedora/Red/Hat/CentOS/Ubuntu/Debian安装psmisc
注意: DolphinScheduler 本身不依赖 Hadoop、Hive、Spark，但如果你运行的任务需要依赖他们，就需要有对应的环境支持。
准备 DolphinScheduler 启动环境
配置用户免密及权限
创建部署用户，并且一定要配置 sudo 免密。以创建 dolphinscheduler 用户为例
# 创建用户需使用 root 登录
useradd dolphinscheduler
# 添加密码
echo "dolphinscheduler" | passwd --stdin dolphinscheduler
# 配置 sudo 免密
sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
# 修改目录权限，使得部署用户对二进制包解压后的 apache-dolphinscheduler-*-bin 目录有操作权限
chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin操作步骤：
wget --no-check-certificate https://www.apache.org/dyn/closer.lua/dolphinscheduler/2.0.5/apache-dolphinscheduler-2.0.5-bin.tar.gz
# 解压缩
tar -zxvf apache-dolphinscheduler-2.0.5-bin.tar.gz -C /opt/modules/
# 修改目录权限，使得部署用户对二进制包解压后的 apache-dolphinscheduler-*-bin 目录有操作权限
chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin
[root@homaybd04 modules]# ls -al
total 0
drwxr-xr-x. 3 root root 47 Apr 16 10:55 .
drwxr-xr-x. 5 root root 52 Apr 16 10:54 ..
drwxr-xr-x. 9 dolphinscheduler dolphinscheduler 138 Apr 16 10:55 apache-dolphinscheduler-2.0.5-bin
注意:因为任务执行服务是以
sudo -u {linux-user}切换不同 linux 用户的方式来实现多租户运行作业，所以部署用户需要有 sudo 权限，而且是免密的。初学习者不理解的话，完全可以暂时忽略这一点。如果发现/etc/sudoers文件中有 "Defaults requirett" 这行，也请注释掉
配置机器SSH免密登陆
由于安装的时候需要向不同机器发送资源，所以要求各台机器间能实现SSH免密登陆。配置免密登陆的步骤如下
su dolphinscheduler
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys注意: 配置完成后，可以通过运行命令 ssh localhost 判断是否成功，如果不需要输入密码就能ssh登陆则证明成功
启动zookeeper
进入 zookeeper 的安装目录，将 zoo_sample.cfg 配置文件复制到 conf/zoo.cfg，并将 conf/zoo.cfg中 dataDir 中的值改成 dataDir=./tmp/zookeeper
# 启动 zookeeper
./bin/zkServer.sh start修改相关配置
完成了基础环境的准备后，在运行部署命令前，还需要根据环境修改配置文件。配置文件在路径在conf/config/install_config.conf 下，一般部署只需要修改 INSTALL MACHINE、DolphinScheduler ENV、Database、Registry Server 部分即可完成部署，下面对必须修改参数进行说明
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# 因为是在单节点上部署master、worker、API server，所以服务器的IP均为机器IP或者localhost
ips="localhost"
masters="localhost"
workers="localhost:default"
alertServer="localhost"
apiServers="localhost"
pythonGatewayServers="localhost"
# DolphinScheduler安装路径，如果不存在会创建
installPath="~/dolphinscheduler"
# 部署用户，填写在 **配置用户免密及权限** 中创建的用户
deployUser="dolphinscheduler"
# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME 的路径，是在 **前置准备工作** 安装的JDK中 JAVA_HOME 所在的位置
javaHome="/your/java/home/here"
# ---------------------------------------------------------
# Database
# ---------------------------------------------------------
# 数据库的类型，用户名，密码，IP，端口，元数据库db。其中 DATABASE_TYPE 目前支持 mysql, postgresql, H2
# 请确保配置的值使用双引号引用，否则配置可能不生效
DATABASE_TYPE="mysql"
SPRING_DATASOURCE_URL="jdbc:mysql://ds1:3306/ds_201_doc?useUnicode=true&characterEncoding=UTF-8"
# 如果你不是以 dolphinscheduler/dolphinscheduler 作为用户名和密码的，需要进行修改
SPRING_DATASOURCE_USERNAME="dolphinscheduler"
SPRING_DATASOURCE_PASSWORD="dolphinscheduler"
# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# 注册中心地址，zookeeper服务的地址
registryServers="localhost:2181"修改后的：
[dolphinscheduler@homaybd04 config]$ cat install_config.conf
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="homaybd04"
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="homaybd04"
# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="homaybd04:default"
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="homaybd04"
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="homaybd04"
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
pythonGatewayServers="homaybd04"
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="~/dolphinscheduler"
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="dolphinscheduler"
# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="/tmp/dolphinscheduler"
# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/usr/local/jdk1.8.0_131"
# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"
# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
DATABASE_TYPE=${DATABASE_TYPE:-"mysql"}
------------------------------------------------------------------------------------------------------------------
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL="jdbc:mysql://homaybd05:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# Spring datasource username
SPRING_DATASOURCE_USERNAME="root"
# Spring datasource password
SPRING_DATASOURCE_PASSWORD="homaytech"
# ---------------------------------------------------------
# Registry Serverdolphinschedulerdolphinscheduler
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# Registry Server address.
registryServers="homaybd03:2181"
# Registry Namespace
registryNamespace="dolphinscheduler"
# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="NONE"
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS，defaultFS write namenode address，HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://mycluster:8020"
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps="192.168.xx.xx,192.168.xx.xx"
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="yarnIp1"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# worker tenant auto create
workerTenantAutoCreate="false"
[dolphinscheduler@homaybd04 config]$
初始化数据库
DolphinScheduler 元数据存储在关系型数据库中，目前支持 PostgreSQL 和 MySQL，如果使用 MySQL 则需要手动下载 mysql-connector-java 驱动 (8.0.16) 并移动到 DolphinScheduler 的 lib目录下。下面以 MySQL 为例，说明如何初始化数据库
这里导入的MySQL驱动为 mysql-connector-java-5.1.34-bin.jar：
您已经为 DolphinScheduler 创建一个新数据库，现在你可以通过快速的 Shell 脚本来初始化数据库
sh script/create-dolphinscheduler.sh初始化后的数据库：
启动 DolphinScheduler
使用上面创建的部署用户运行以下命令完成部署，部署后的运行日志将存放在 logs 文件夹内
[dolphinscheduler@hei apache-dolphinscheduler-2.0.5-bin]$ ls -l
total 164
drwxr-xr-x. 2 dolphinscheduler dolphinscheduler 100 Apr 16 10:55 bin
drwxr-xr-x. 5 dolphinscheduler dolphinscheduler 4096 Apr 16 10:55 conf
-rwxrwxr-x. 1 dolphinscheduler dolphinscheduler 5190 Feb 25 15:45 install.sh
drwxr-xr-x. 2 dolphinscheduler dolphinscheduler 12288 Apr 16 12:07 lib
-rw-rw-r--. 1 dolphinscheduler dolphinscheduler 42482 Mar 2 20:11 LICENSE
drwxrwxr-x. 3 dolphinscheduler dolphinscheduler 12288 Feb 25 15:45 licenses
-rw-rw-r--. 1 dolphinscheduler dolphinscheduler 73996 Feb 25 15:45 NOTICE
drwxr-xr-x. 2 dolphinscheduler dolphinscheduler 239 Apr 16 10:55 script
drwxr-xr-x. 3 dolphinscheduler dolphinscheduler 17 Apr 16 10:55 sql
drwxr-xr-x. 8 dolphinscheduler dolphinscheduler 113 Apr 16 10:55 ui
[dolphinscheduler@homaybd04 apache-dolphinscheduler-2.0.5-bin]$
启动
sh install.sh登录 DolphinScheduler
浏览器访问地址 http://homaybd04:12345/dolphinscheduler 即可登录系统UI。默认的用户名和密码是 admin/dolphinscheduler123
查看安装目录：
[dolphinscheduler@homaybd04 dolphinscheduler]$ pwd
/home/dolphinscheduler/dolphinscheduler
[dolphinscheduler@homaybd04 dolphinscheduler]$ ls -l
total 32
drwxr-xr-x. 2 dolphinscheduler dolphinscheduler 100 Apr 16 12:24 bin
drwxr-xr-x. 5 dolphinscheduler dolphinscheduler 4096 Apr 16 12:24 conf
-rwxrwxr-x. 1 dolphinscheduler dolphinscheduler 5190 Apr 16 12:24 install.sh
drwxr-xr-x. 2 dolphinscheduler dolphinscheduler 12288 Apr 16 12:24 lib
drwxrwxr-x. 2 dolphinscheduler dolphinscheduler 4096 Apr 16 12:24 logs
drwxrwxr-x. 2 dolphinscheduler dolphinscheduler 262 Apr 16 12:24 pid
drwxr-xr-x. 2 dolphinscheduler dolphinscheduler 239 Apr 16 12:24 script
drwxr-xr-x. 3 dolphinscheduler dolphinscheduler 17 Apr 16 12:24 sql
drwxr-xr-x. 8 dolphinscheduler dolphinscheduler 113 Apr 16 12:24 ui
[dolphinscheduler@homaybd04 dolphinscheduler]$
启停服务
# 切换用户
su dolphinscheduler
[dolphinscheduler@homaybd04 dolphinscheduler]$ pwd
/home/dolphinscheduler/dolphinscheduler
# 一键停止集群所有服务
sh ./bin/stop-all.sh
# 一键开启集群所有服务
sh ./bin/start-all.sh
# 启停 Master
sh ./bin/dolphinscheduler-daemon.sh stop master-server
sh ./bin/dolphinscheduler-daemon.sh start master-server
# 启停 Worker
sh ./bin/dolphinscheduler-daemon.sh start worker-server
sh ./bin/dolphinscheduler-daemon.sh stop worker-server
# 启停 Api
sh ./bin/dolphinscheduler-daemon.sh start api-server
sh ./bin/dolphinscheduler-daemon.sh stop api-server
# 启停 Logger
sh ./bin/dolphinscheduler-daemon.sh start logger-server
sh ./bin/dolphinscheduler-daemon.sh stop logger-server
# 启停 Alert
sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server
# 启停 Python Gateway
sh ./bin/dolphinscheduler-daemon.sh start python-gateway-server
sh ./bin/dolphinscheduler-daemon.sh stop python-gateway-serverHive 数据源创建
jdbc:hive2://homaybd04:2181,homaybd05:2181> create database ds_db;
INFO : Compiling command(queryId=hive_20220416222640_6b825504-726b-4a3a-b900-ac3b44e721fd): create database ds_db（1）hive的多用户配置
hive本身不创建用户，用户就是linux的用户，（2）打开hive的server服务
第一步：运行hive下 bin/hiveserver2 脚本
# ./hiveserver2
hive --service hiveserver2 &[root@homaybd04 ~]# beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://homaybd04:2181,homaybd05:2181,homaybd03:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Enter username for jdbc:hive2://homaybd04:2181,homaybd05:2181,homaybd03:2181/default: ds
Enter password for jdbc:hive2://homaybd04:2181,homaybd05:2181,homaybd03:2181/default: ********
22/04/16 22:56:22 [main]: INFO jdbc.HiveConnection: Connected to homaybd04:10000
Connected to: Apache Hive (version 3.1.0.3.1.0.0-78)
Driver: Hive JDBC (version 3.1.0.3.1.0.0-78)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
0: jdbc:hive2://homaybd04:2181,homaybd05:2181>
可以看到连接的是：
INFO jdbc.HiveConnection: Connected to homaybd04:10000第二步：新建xshell窗口
beeline
# 2、连接hive2
!connect jdbc:hive2://192.168.6.102:10000
# 3、输入用户名ds, 和密码:ds123456
Enter username for jdbc:hive2://homaybd04:2181,homaybd05:2181,homaybd03:2181/default: ds
Enter password for jdbc:hive2://homaybd04:2181,homaybd05:2181,homaybd03:2181/default: ********
（3）海豚调度器创建资源
MySQL数据源
需要将MySQL驱动升级为 (8.0.16)，使用 MySQL 则需要手动下载 mysql-connector-java 驱动 (8.0.16) 并移动到 DolphinScheduler 的 lib目录下。
然后重启DS：
# 切换用户
su dolphinscheduler
[dolphinscheduler@homaybd04 dolphinscheduler]$ pwd
cd /home/dolphinscheduler/dolphinscheduler
# 一键停止集群所有服务
sh ./bin/stop-all.sh
# 一键开启集群所有服务
sh ./bin/start-all.sh
日志查看：
[dolphinscheduler@homaybd04 logs]$ pwd
/home/dolphinscheduler/dolphinscheduler/logs
[dolphinscheduler@homaybd04 logs]$ tail -n 100 dolphinscheduler-api.log查看运行异常日志：
[dolphinscheduler@hoybd04 logs]$ tail -n 100 dolphinscheduler-master.log
相关文章：
DolphinScheduler文档
第三章 dolphinscheduler基础使用之数据源配置（包括hive，spark，mysql，oracle）
为者常成，行者常至
自由转载-非商用-非衍生-保持署名（创意共享3.0许可证）
