手摸手极简docker部署rocketmq单机实战

Simon included in 编程开发

2021-12-17 678 words 4 minutes

Contents

自己本地搭建rocketmq开发环境还是用docker方便点，之前搭建一直有些小问题，最近专项解决了下，终于搞定了，这里整理分享下

环境准备

因笔者用的MacBook，所以会主要以Mac为主讲解，其他环境区别不大

首先请准备好docker环境，推荐docker desktop ，无他，点点点就完事了，不用的时候不启动，开发机紧张的内存问题得到了心理上的安慰。

还需要安装好docker-compose，请自行参考安装教程点这里

注：docker desktop自带了docker-compose，所以不需要再单独安装compose了

编译rocketmq镜像

环境搞定了，我们开始搭建rocketmq，直接用官方推荐的docker镜像即可。镜像下载地址：https://github.com/apache/rocketmq-docker

我们先通过git把代码拉下来，打开命令行复制粘贴

git clone https://hub.fastgit.org/apache/rocketmq-docker.git

clone完成后，继续cd rocketmq-docker/image-build到image-build目录下

目前最新版本为4.9.2,我们就以这个版本为例：继续输入下列指令

sh build-image.sh 4.9.2 alpine

等待完成就即可，如果报错，请自行根据错误查找原因。只要环境没问题，编译完成后我们能得到apacherocketmq/rocketmq镜像

至此，我们已经完成了一半的工作。

编写docker-compose配置

鲁迅：授人以鱼不如授人以渔和鱼有经验的同学此时已经急不可耐了，鱼拿去。完整的docker-compose.yml文件

version: '3'
services:
  namesrv:
    image: apacherocketmq/rocketmq:4.9.2-alpine
    container_name: rmqnamesrv
    ports:
      - 9876:9876
    volumes:
      - ./data/namesrv/logs:/home/rocketmq/logs
    command: sh mqnamesrv
    restart: always
  broker:
    image: apacherocketmq/rocketmq:4.9.2-alpine
    container_name: rmqbroker
    links:
      - namesrv
    ports:
      - 10909:10909
      - 10911:10911
      - 10912:10912
    environment:
      - NAMESRV_ADDR=namesrv:9876
    volumes:
      - ./data/broker/logs:/home/rocketmq/logs
      - ./data/broker/store:/home/rocketmq/store
      - ./data/broker/conf/broker.conf:/home/rocketmq/rocketmq-4.9.2/conf/broker.conf
      - ./data/broker/runbroker.sh:/home/rocketmq/rocketmq-4.9.2/bin/runbroker.sh
    command: sh mqbroker -c /home/rocketmq/rocketmq-4.9.2/conf/broker.conf
    depends_on:
      - namesrv
    restart: always
  rmqconsole:
    image: candice0630/rocketmq-console-ng:2.0
    container_name: rmqconsole
    ports:
      - 9001:8080
    environment:
      JAVA_OPTS: -Drocketmq.namesrv.addr=namesrv:9876
        -Dcom.rocketmq.sendMessageWithVIPChannel=false
    depends_on:
      - namesrv
    restart: always

上面一共三个images，namesrv、broker和rmqconsole 熟悉docker-compose的同学可以直接跳过这部分了本部分主要结合上面的配置来说，不适应的根据自己的习惯自行更改先建立一个rocketmq文件夹，然后在文件夹下面创建docker-compose.yml文件，复制上面的内容进去。在docker-compose.yml同级目录下创建data文件夹，在data文件夹下分别创建broker和namesrv文件夹，broker目录下分别创建conf、logs和store文件夹。namesrv目录下创建logs目录，logs目录下创建rocketmqlogs目录，这样所有的目录创建完成。自定义的broker.conf文件放到data/broker/conf目录下。

一切就绪后，就可以用过docker-compose up -d启动了。

测试

通过`rmqconsole`测试

访问http://localhost:9001/#/topic,通过主题-TopicTest，点击发送消息，随便填写,点击提交，出现SEND_OK说明发送成功，可以通过消息tab，选择TopicTest搜索来查看刚发送的消息。

通过命令行测试

docker exec -it 容器id /bin/sh 利用docker exec分别进入namesrv和broker 在namesrc容器里面输入命令ip addr查询容器的内网IP 然后回到broker容器，先导入环境变量export NAMESRV_ADDR=namesrv的容器内网ID:9876 然后通过./tools.sh org.apache.rocketmq.example.quickstart.Producer来测试消息发送，出现SendResult [sendStatus=SEND_OK, msgId=7F00000100A说明发送成功，如果是connect failed，请检查配置是否正确。如果出现磁盘满的提示，需要修改diskMaxUsedSpaceRatio的值，可以通过修改conf或者sh脚本的方式。上面的yml文件已经定义好了，你只需要把自己的配置文件放入就行，两种方式二选一。提供下我的文件

broker.conf

brokerClusterName = DefaultCluster
brokerName = broker-a
brokerId = 0
deleteWhen = 04
fileReservedTime = 48
brokerRole = ASYNC_MASTER
flushDiskType = ASYNC_FLUSH
diskMaxUsedSpaceRatio=99

runbroker.sh

#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit() {
    echo "ERROR: $1 !!"
    exit 1
}

find_java_home() {
    case "$(uname)" in
    Darwin)
        JAVA_HOME=$(/usr/libexec/java_home)
        ;;
    *)
        JAVA_HOME=$(dirname $(dirname $(readlink -f $(which javac))))
        ;;
    esac
}

find_java_home

[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}

#===========================================================================================
# JVM Configuration
#===========================================================================================
calculate_heap_sizes() {
    case "$(uname)" in
    Linux)
        system_memory_in_mb=$(free -m | sed -n '2p' | awk '{print $2}')
        system_cpu_cores=$(egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo)
        ;;
    FreeBSD)
        system_memory_in_bytes=$(sysctl hw.physmem | awk '{print $2}')
        system_memory_in_mb=$(expr $system_memory_in_bytes / 1024 / 1024)
        system_cpu_cores=$(sysctl hw.ncpu | awk '{print $2}')
        ;;
    SunOS)
        system_memory_in_mb=$(prtconf | awk '/Memory size:/ {print $3}')
        system_cpu_cores=$(psrinfo | wc -l)
        ;;
    Darwin)
        system_memory_in_bytes=$(sysctl hw.memsize | awk '{print $2}')
        system_memory_in_mb=$(expr $system_memory_in_bytes / 1024 / 1024)
        system_cpu_cores=$(sysctl hw.ncpu | awk '{print $2}')
        ;;
    *)
        # assume reasonable defaults for e.g. a modern desktop or
        # cheap server
        system_memory_in_mb="2048"
        system_cpu_cores="2"
        ;;
    esac

    # some systems like the raspberry pi don't report cores, use at least 1
    if [ "$system_cpu_cores" -lt "1" ]; then
        system_cpu_cores="1"
    fi

    # set max heap size based on the following
    # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
    # calculate 1/2 ram and cap to 1024MB
    # calculate 1/4 ram and cap to 8192MB
    # pick the max
    half_system_memory_in_mb=$(expr $system_memory_in_mb / 2)
    quarter_system_memory_in_mb=$(expr $half_system_memory_in_mb / 2)
    if [ "$half_system_memory_in_mb" -gt "1024" ]; then
        half_system_memory_in_mb="1024"
    fi
    if [ "$quarter_system_memory_in_mb" -gt "8192" ]; then
        quarter_system_memory_in_mb="8192"
    fi
    if [ "$half_system_memory_in_mb" -gt "$quarter_system_memory_in_mb" ]; then
        max_heap_size_in_mb="$half_system_memory_in_mb"
    else
        max_heap_size_in_mb="$quarter_system_memory_in_mb"
    fi
    MAX_HEAP_SIZE="${max_heap_size_in_mb}M"

    # Young gen: min(max_sensible_per_modern_cpu_core * num_cores, 1/4 * heap size)
    max_sensible_yg_per_core_in_mb="100"
    max_sensible_yg_in_mb=$(expr $max_sensible_yg_per_core_in_mb "*" $system_cpu_cores)

    desired_yg_in_mb=$(expr $max_heap_size_in_mb / 4)

    if [ "$desired_yg_in_mb" -gt "$max_sensible_yg_in_mb" ]; then
        HEAP_NEWSIZE="${max_sensible_yg_in_mb}M"
    else
        HEAP_NEWSIZE="${desired_yg_in_mb}M"
    fi
}

calculate_heap_sizes

# Dynamically calculate parameters, for reference.
Xms=$MAX_HEAP_SIZE
Xmx=$MAX_HEAP_SIZE
Xmn=$HEAP_NEWSIZE
MaxDirectMemorySize=$MAX_HEAP_SIZE
# Set for `JAVA_OPT`.
JAVA_OPT="${JAVA_OPT} -server -Xms${Xms} -Xmx${Xmx} -Xmn${Xmn}"
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=8"
JAVA_OPT="${JAVA_OPT} -verbose:gc -Xloggc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy"
JAVA_OPT="${JAVA_OPT} -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
JAVA_OPT="${JAVA_OPT} -XX:+AlwaysPreTouch"
JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=${MaxDirectMemorySize}"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages -XX:-UseBiasedLocking"
JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
#JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"
JAVA_OPT="${JAVA_OPT} -Drocketmq.broker.diskSpaceWarningLevelRatio=0.99"

numactl --interleave=all pwd >/dev/null 2>&1
if [ $? -eq 0 ]; then
    if [ -z "$RMQ_NUMA_NODE" ]; then
        numactl --interleave=all $JAVA ${JAVA_OPT} $@
    else
        numactl --cpunodebind=$RMQ_NUMA_NODE --membind=$RMQ_NUMA_NODE $JAVA ${JAVA_OPT} $@
    fi
else
    $JAVA ${JAVA_OPT} $@
fi

通过指令./tools.sh org.apache.rocketmq.example.quickstart.Consumer来查看发送的消息