上海交大超算容器制作-CFANZ编程社区

1.登录PI2.0的login节点

2.转到PI2.0的容器制作节点

ssh build@container-x86

3.编写docker的容器定义文件siyuan1.DockerFile，siyuan1是名字可以随便起

我的siyuan1.DockerFile文件内容

# siyuan1.DockerFile
FROM centos:8 
# make essential
RUN sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-* && \
sed -i -e "s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g" /etc/yum.repos.d/CentOS-* 
# basic tools
RUN yum install -y \
        tree \
        vim \
        wget && \
    rm -rf /var/cache/yum/*
RUN echo "basic tools completed !!" 
# gmsh basic libs
RUN yum install -y \
        mesa-libGLU \
        libXrender-0.9.10-7.el8.x86_64 \
        libXcursor-1.1.15-3.el8.x86_64 \
        libXft-2.3.3-1.el8.x86_64 \
        libXinerama-1.1.4-1.el8.x86_64 \
        libgomp-8.5.0-4.el8_5.x86_64 && \
    rm -rf /var/cache/yum/*
RUN echo "gmsh basic libs completed !!" 
# install gmsh
RUN wget http://www.gmsh.info/bin/Linux/gmsh-4.4.0-Linux64.tgz && \
    tar -xzf gmsh-4.4.0-Linux64.tgz && \
    rm -f gmsh-4.4.0-Linux64.tgz 
RUN echo "gmsh 4.4.0 completed !!" 
# openfoam06 basic tools
RUN yum install -y \
        epel-release \
        cmake3 \
        make \
        git \
        wget && \
    # ln -s /usr/bin/cmake3 /usr/bin/cmake && \
    rm -rf /var/cache/yum/* 
RUN echo "openfoam06 basic tools completed !!" 
# OpenFOAM 6.0 tools 2
RUN yum install -y \
        patch \
        flex \
        bison \
        zlib-devel \
        boost-system \
        boost-thread \
        readline-devel \
        ncurses-devel \
        openssl && \
    rm -rf /var/cache/yum/*
RUN echo "OpenFOAM 6.0 tools 2 completed !!" 
# OpenFOAM 6.0 download
RUN mkdir -p /opt && \
    export BUILD_DIR=/opt/OpenFOAM && \
    mkdir -p $BUILD_DIR && \
    cd $BUILD_DIR && \
    git clone https://gitee.com/diyboy/OpenFOAM-6 && \
    git clone https://gitee.com/diyboy/ThirdParty-6 && \
    # git clone https://github.com/OpenFOAM/OpenFOAM-6 && \
    # git clone https://github.com/OpenFOAM/ThirdParty-6 && \
    pwd 
RUN echo "OpenFOAM 6.0 download completed !!" 
# OpenFOAM 6.0 libs
RUN yum install -y \
        gcc-8.5.0-4.el8_5.x86_64 \
        gcc-c++-8.5.0-4.el8_5.x86_64 \
        openmpi-4.1.1-2.el8.x86_64 && \
    rm -rf /var/cache/yum/* 
RUN echo "OpenFOAM 6.0 libs completed !!" 
# OpenFOAM 6.0 make
RUN export PATH=$PATH:/usr/lib64/openmpi/bin && \
    cd /opt/OpenFOAM/OpenFOAM-6 && \
    sed -i '47d' etc/bashrc && \
    sed -i '46a export FOAM_INST_DIR=/opt/\$WM_PROJECT' etc/bashrc && \
    source etc/bashrc && \
    ./Allwmake -j 
RUN echo "OpenFOAM 6.0 make completed !! " 
RUN echo "export PATH="$PATH":/usr/lib64/openmpi/bin:/usr/lib64/openmpi/bin/mpicc " >> /etc/bashrc && \
    echo "source /opt/OpenFOAM/OpenFOAM-6/etc/bashrc " >> /etc/bashrc 
# ThirdParty libs
RUN yum groupinstall -y 'Development Tools' && \
    yum install -y \
        qt5-qtbase-devel \
        libXt-devel \
        zlib-devel \
        libXext-devel \
        libGLU-devel \
        libXrender-devel \
        libXinerama-devel \
        libpng-devel \
        libXrandr-devel \
        libXi-devel \
        libXft-devel \
        libjpeg-turbo-devel \
        libXcursor-devel \
        readline-devel \
        ncurses-devel \
        python38 \
        cmake \
        mpfr-devel \
        gmp \
        gmp-devel && \
    rm -rf /var/cache/yum/* 
RUN echo "ThirdParty libs completed !! " 
# ThirdParty-6 make 
RUN source /etc/bashrc && \
    cd $WM_THIRD_PARTY_DIR && \
    export PATH=$PATH:/usr/lib64/qt5/bin 
    # ./makeParaView && \
    # wMRefresh && \
    # cd $FOAM_UTILITIES/postProcessing/graphics/PVReaders && \
    # ./Allwclean && \
    # ./Allwmake -j 
RUN echo "make ThirdParty-6 completed !! " 

RUN echo "----end-----" 

CMD /bin/bash

注意：我是从centos8开始制作的，很多库不成熟，建议大家从centos7开始制作容器，我优先安装了gmsh这款软件，这是不需要的，可以直接安装OpenFoam，我OpenFoam-6编译成功了，但是还没有编译ThirdParty-6，因为centos8这个系统太新了，不太熟

注意这几行：

RUN echo "export PATH="$PATH":/usr/lib64/openmpi/bin:/usr/lib64/openmpi/bin/mpicc " >> /etc/bashrc && \
    echo "source /opt/OpenFOAM/OpenFOAM-6/etc/bashrc " >> /etc/bashrc

代表把mpicc和OF6的bashrc添加进容器系统的bashrc，我只安装了OF6如果安装更多的OF版本，需要

RUN echo "export PATH="$PATH":/usr/lib64/openmpi/bin:/usr/lib64/openmpi/bin/mpicc " >> /etc/bashrc && \
    echo "alias of60=\"/opt/OpenFOAM/OpenFOAM-6/etc/bashrc\" " >> /etc/bashrc

然后在使用容器的时候，记得先加载/etc/bashrc

注意一下这里：

RUN export PATH=$PATH:/usr/lib64/openmpi/bin && \
    cd /opt/OpenFOAM/OpenFOAM-6 && \
    sed -i '47d' etc/bashrc && \
    sed -i '46a export FOAM_INST_DIR=/opt/\$WM_PROJECT' etc/bashrc && \
    source etc/bashrc && \
    ./Allwmake -j

这里通过sed命令更改一下OF6的安装路径，把他安装在/opt/下面

4.制作siyuan1的docker镜像

docker build -f ./container/siyuan1/siyuan1.DockerFile -t siyuan1 .

这里需要等他自己下载源代码，然后安装编译啥的，首次大概俩小时

5.把制作好的docker容器转换成sif文件

singularity build --force siyuan1.sif docker-daemon://siyuan1:latest

6.把sif文件通过scp命令传到思源一号上

scp -o StrictHostKeyChecking=no ./siyuan1.sif naoxb@sylogin1.hpc.sjtu.edu.cn:~/tongxin/

注意：修改成你自己的路径

7.退出，然后登录思源一号，给大家看一下我的slurm文件

#!/bin/bash

#SBATCH --job-name=myCase
#SBATCH --partition=64c512g
#SBATCH -N 1
#SBATCH --ntasks-per-node=default
#SBATCH --output=%j.out
#SBATCH --error=%j.err

module purge &&
module --ignore-cache load "openmpi/4.1.1-gcc-9.3.0" &&

export IMAGE_NAME=../../../siyuan1.sif &&
export myScript=". /etc/bashrc " &&
export myScript=$myScript"&&"$myAppBin" -parallel " &&
echo "begin mpi running " &&
mpirun -n $SLURM_NTASKS singularity exec $IMAGE_NAME /bin/bash -c "$myScript" &&
echo "end mpi running "

注意：先加载一下openmpi模组

运行容器的时候先source /etc/bashrc 再加你的命令，当然，你需要先在容器里编译你的求解器

export script=". /etc/bashrc " &&
export script=$script"&&wclean "$mySolverPath" " &&
export script=$script"&&wmake "$mySolverPath" " &&
export script=$script"&&wclean "$mySolverPath" " &&
singularity exec ../../../siyuan1.sif /bin/bash -c "$script" &&
export FOAM_USER_APPBIN='/dssg/home/acct-naoxb/naoxb/OpenFOAM/naoxb-6/platforms/linux64GccDPInt32Opt/bin/' &&
cp $FOAM_USER_APPBIN/$mySolverName ./dict/$mySolverName

大致流程就是这样，思源一号比pi2.0快了三倍，不需要排队，还是比较好用的，编译求解器需要先申请一个计算节点