搭建第三方开发者代码/服务托管系统

 

公司新业务, 希望支持第三方开发者开发插件, 并实现插件代码的托管. 开发者可以上传代码, 审核通过后自动做资源分配, 代码发布等.

支撑公司内部开发人员和第三方开发者也需要关注更多的内容.

  1. 安全风险, 对于未知的第三方开发者. 要做好资源隔离并避免污染.
  2. 保证业务连续性.
  3. 自动实现代码发布, 支持回退/指定tag发布, 减少人工成本.
  4. 方便查看第三方应用产生的日志.
  5. 支持区分测试环境/生产环境.

业务上也需要支持框架, 如权限鉴定/一套k/v存储接口等.

 

运维的架构, 从来都应该是面向需求, 要满足业务需求, 但是又不能过早引入复杂的设计, 一步一步进行演变.

每个应用会分配给开发者一个仓库地址. 开发人员提交代码后, 在运维平台提交发布请求(仓库地址, 生成/测试环境), 管理员审核通过/拒绝, 自动发布代码到指定环境., 采用的实现方案如下:

通过docker来实现环境隔离.

通过nginx upstream机制做故障转移.

通过gitlab ci/cd实现代码发布接口.

通过运维平台提供资源分配, 生成配置文件, 调用发布接口, 提供日志接口, 修改子域名dns.

以第三方开发者的插件是nginx/php实现, 平台提供一个标准模板.

docker资源隔离:

提供Dockerfile

[nginx Dockerfile]

FROM alpine:3.3

MAINTAINER NGINX Docker Maintainers "docker-maint@nginx.com"

ENV NGINX_VERSION 1.10.1

ENV GPG_KEYS B0F4253373F8F6F510D42178520A9993A1C052F8
ENV CONFIG "\
  --prefix=/etc/nginx \
  --sbin-path=/usr/sbin/nginx \
  --modules-path=/usr/lib/nginx/modules \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --pid-path=/var/run/nginx.pid \
  --lock-path=/var/run/nginx.lock \
  --http-client-body-temp-path=/var/cache/nginx/client_temp \
  --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
  --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
  --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
  --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
  --user=nobody \
  --group=nobody \
  --with-http_ssl_module \
  --with-http_realip_module \
  --with-http_addition_module \
  --with-http_sub_module \
  --with-http_dav_module \
  --with-http_flv_module \
  --with-http_mp4_module \
  --with-http_gunzip_module \
  --with-http_gzip_static_module \
  --with-http_random_index_module \
  --with-http_secure_link_module \
  --with-http_stub_status_module \
  --with-http_auth_request_module \
  --with-http_xslt_module=dynamic \
  --with-http_image_filter_module=dynamic \
  --with-http_geoip_module=dynamic \
  --with-http_perl_module=dynamic \
  --with-threads \
  --with-stream \
  --with-stream_ssl_module \
  --with-http_slice_module \
  --with-mail \
  --with-mail_ssl_module \
  --with-file-aio \
  --with-http_v2_module \
  --with-ipv6 \
  "

RUN \
  apk add --no-cache --virtual .build-deps \
    gcc \
    libc-dev \
    make \
    openssl-dev \
    pcre-dev \
    zlib-dev \
    linux-headers \
    curl \
    gnupg \
    libxslt-dev \
    gd-dev \
    geoip-dev \
    perl-dev \
  && curl -fSL http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz -o nginx.tar.gz \
  && curl -fSL http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz.asc  -o nginx.tar.gz.asc \
  && export GNUPGHOME="$(mktemp -d)" \
  && gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEYS" \
  && gpg --batch --verify nginx.tar.gz.asc nginx.tar.gz \
  && rm -r "$GNUPGHOME" nginx.tar.gz.asc \
  && mkdir -p /usr/src \
  && mkdir -p /var/log/nginx \
  && mkdir -p /var/cache/nginx/client_temp \
  && mkdir -p /var/cache/nginx/proxy_temp \
  && mkdir -p /var/cache/nginx/fastcgi_temp \
  && mkdir -p /var/cache/nginx/uwsgi_temp \
  && mkdir -p /var/cache/nginx/scgi_temp \
  && tar -zxC /usr/src -f nginx.tar.gz \
  && rm nginx.tar.gz \
  && cd /usr/src/nginx-$NGINX_VERSION \
  && ./configure $CONFIG --with-debug \
  && make \
  && mv objs/nginx objs/nginx-debug \
  && mv objs/ngx_http_xslt_filter_module.so objs/ngx_http_xslt_filter_module-debug.so \
  && mv objs/ngx_http_image_filter_module.so objs/ngx_http_image_filter_module-debug.so \
  && mv objs/ngx_http_geoip_module.so objs/ngx_http_geoip_module-debug.so \
  && mv objs/ngx_http_perl_module.so objs/ngx_http_perl_module-debug.so \
  && ./configure $CONFIG \
  && make \
  && make install \
  && rm -rf /etc/nginx/html/ \
  && mkdir /etc/nginx/conf.d/ \
  && mkdir -p /usr/share/nginx/html/ \
  && install -m644 html/index.html /usr/share/nginx/html/ \
  && install -m644 html/50x.html /usr/share/nginx/html/ \
  && install -m755 objs/nginx-debug /usr/sbin/nginx-debug \
  && install -m755 objs/ngx_http_xslt_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_xslt_filter_module-debug.so \
  && install -m755 objs/ngx_http_image_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_image_filter_module-debug.so \
  && install -m755 objs/ngx_http_geoip_module-debug.so /usr/lib/nginx/modules/ngx_http_geoip_module-debug.so \
  && install -m755 objs/ngx_http_perl_module-debug.so /usr/lib/nginx/modules/ngx_http_perl_module-debug.so \
  && ln -s ../../usr/lib/nginx/modules /etc/nginx/modules \
  && strip /usr/sbin/nginx* \
  && strip /usr/lib/nginx/modules/*.so \
  && runDeps="$( \
    scanelf --needed --nobanner /usr/sbin/nginx /usr/lib/nginx/modules/*.so \
      | awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
      | sort -u \
      | xargs -r apk info --installed \
      | sort -u \
  )" \
  && apk add --virtual .nginx-rundeps $runDeps \
  && apk del .build-deps \
  && rm -rf /usr/src/nginx-$NGINX_VERSION \
  && apk add --no-cache gettext \
  \
  # forward request and error logs to docker log collector
  && ln -sf /dev/stdout /var/log/nginx/access.log \
  && ln -sf /dev/stderr /var/log/nginx/error.log

COPY nginx.conf /etc/nginx/nginx.conf
COPY nginx.vh.default.conf /etc/nginx/conf.d/default.conf

EXPOSE 80 443

CMD ["nginx", "-g", "daemon off;"]

docker用到的nginx配置文件:

user  nobody;
worker_processes  4;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    use epoll;
    worker_connections  10240;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

 

nginx.vh.default.conf:

server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/log/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

[php Dockerfile]

FROM php:7-fpm-alpine

MAINTAINER foobar Docker Maintainers "docker@foobar.com"
# install php-core-extensions
RUN apk add --no-cache freetype libpng libjpeg-turbo freetype-dev libpng-dev libjpeg-turbo-dev libmcrypt-dev && \
  docker-php-ext-configure gd \
    --with-gd \
    --with-freetype-dir=/usr/include/ \
    --with-png-dir=/usr/include/ \
    --with-jpeg-dir=/usr/include/ && \
  NPROC=$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) && \
  docker-php-ext-install -j${NPROC} gd mcrypt && \
  apk del --no-cache freetype-dev libpng-dev libjpeg-turbo-dev

生成环境nginx故障转移的实现方式:

默认使用docker环境处理请求, 当docker挂掉后, 可以切换到宿主机进行处理(宿主机处理有安全风险),用这个方式实施简单,不需要引入docker集群相关的内容.

upstream app1000239 {

     server 10.47.49.233:11239;

     server 127.0.0.1:10239 backup;

}

server {

    listen 443 ssl;

    server_name app1000239.foobar.com;

    server_tokens off;

    proxy_set_header           Host $host;

    proxy_set_header           X-Real-IP $remote_addr;

    proxy_set_header           X-Forwarded-For $proxy_add_x_forwarded_for;

    location / {

        proxy_pass      http://app1000239;

    }

    access_log /data/logs/nginx/${host}_access.log combined;

}

server {

        listen 127.0.0.1:10239;

        root   /data/web/app1000239/web/;

        index  index.html index.htm index.php;

        location ~* \.php$ {

                fastcgi_pass   127.0.0.1:9000;

                fastcgi_index  index.php;

                fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;

                include        fastcgi_params;

               

        }

        location / {

            index index.php;

            try_files $uri $uri/ /index.php$is_args$args;

        }

        access_log /data/logs/nginx/app1000239_vm_access.log combined;

    }

gitlab 实现ci/cd

gitlab支持通过在代码仓库中放置.gitlab-ci.yml来实现, 但是如果开发者直接支持管理.gitlab-ci.yml会有学习成本和较大的安全风险, 因此可以通过2个仓库来达到目的.

开发者可以上传代码到仓库A. 然后在运维平台提交发布申请.

仓库A对应的管理仓库AA. AA可以存放.gitlab-ci.yml, php/nginx的配置文件等. 运维平台可以调用仓库AA的代码发布来完成发布.

AA仓库的目录结构:

.gitlab-ci.yml (存放发布相关)

codes/ (这里是用的gitlab的子模块功能, 把仓库A当做子模块)

conf/ 这里存放渲染好的nginx/php配置文件, docker-compose文件

仓库AA可以设置一些默认参数, 并开启一个webhooks监听pipline event, 向运维平台发送发布成功/失败的结果.

附录:

[docker-compose.yaml文件]

version: '2'
services:
  app-nginx:
    image: registry.foobar.com/images/foobar-nginx
    volumes_from:
      - app-php
    ports:
     - "1.1.1.1:11038:80"
    command: nginx -g 'daemon off;'
    links:
      - app-php
    extra_hosts:
      - "api.foobar.com:2.3.4.5"
  app-php:
    image: registry.foobar.com/images/foobar-php-fpm-gd
    extra_hosts:
      - "api.foobar.com:2.3.4.5"
    volumes:
     - /data/web/foobar_app1000038:/usr/share/nginx/html
     - ./app-php.conf:/usr/local/etc/php-fpm.d/www.conf
     - ./app-nginx.conf:/etc/nginx/conf.d/default.conf
     - /etc/localtime:/etc/localtime:ro
     - /etc/resolv.conf:/etc/resolv.conf:ro

[app-php.conf]

[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 100
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20

[app-nginx.conf]

server {
    listen      80;
    server_tokens  off;

    root /usr/share/nginx/html/web/;
    index  index.html index.htm;
    location ~* \.php$ {
            fastcgi_pass   app-php:9000;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
            include        fastcgi_params;
    }
    location / {
        index index.php;
        try_files $uri $uri/ /index.php$is_args$args;
    }
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
    error_page   500 502 503 504  /50x.html;
    access_log  /var/log/nginx/access.log  main;
}

[.gitlab-ci.yml]

image: foobar/open_docs_php:latest
before_script:
  - ping git.foobar.com -w 2
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - ssh-add <(echo "$GIT_CLONE_PRIVATE_KEY")
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  - cd $CI_PROJECT_DIR && git checkout . && git submodule update --init --force
  - cd $CI_PROJECT_DIR/codes && echo ${commit_ref} && git fetch --all && git checkout ${commit_ref} .
stages:
  - test
  - build
  - deploy
deploy_stage:
  stage: deploy
  script:
    - if [ "${deploy_to}" == "development" ]; then echo "development" && rsync -vzrtog --delete $CI_PROJECT_DIR/codes/ $SSH_USER@$SSH_HOST:$DEV_BASE_DIR/ --exclude=.git; elif [ "${deploy_to}" == "production" ]; then echo "production" && rsync -vzrtog --delete $CI_PROJECT_DIR/codes/ $SSH_USER@$SSH_HOST:$DEV_BASE_DIR/ --exclude=.git; else echo "no variable provided"; exit 1; fi

 

运维平台实现:

开发者创建新应用后分配一个仓库账号,地址

分配域名:

创建针对测试/正式环境用的域名.

调用域名商接口配置对应的A记录.

创建对应的管理仓库.

初始化管理仓库:

创建.gitlab-ci.yml

创建子模块并放在codes/目录

渲染出的配置文件放到conf目录

创建仓库的变量(.gitlab-ci.yml里面会用到)

创建webhooks

调用gitlab ci发布代码, 推送conf里配置文件到宿主服务器, 使用docker-compose启动docker镜像, 并启动/重启外部nginx服务.

运维平台用的flask + sqlalchemy配合一些模块如python-gitlab, GitPython等实现.

整个代码/服务托管的运维方案这样就转起来了.

 

发表评论

电子邮件地址不会被公开。 必填项已用*标注