<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" > <channel> <title>Cloud | Mobabel</title> <atom:link href="http://www.mobabel.net/category/cloud/feed/?lang=en_uspage-27page-3page-15page-3page-7" rel="self" type="application/rss+xml" /> <link>http://www.mobabel.net</link> <description>Just One Pure ITer</description> <lastBuildDate>Wed, 09 Nov 2022 14:51:52 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod> <sy:updateFrequency> 1 </sy:updateFrequency> <generator>https://wordpress.org/?v=6.6.2</generator> <item> <title>Increase the building heap-size for Jenkins jobs</title> <link>http://www.mobabel.net/increase-the-building-heap-size-for-jenkins-jobs/</link> <comments>http://www.mobabel.net/increase-the-building-heap-size-for-jenkins-jobs/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Tue, 01 Mar 2022 17:54:30 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Jenkins]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=7364</guid> <description><![CDATA[<p>In Jenkins configuration the following configs have been set: Maven Project Configuration->Global MAVEN_OPTS [crayon-6750e0c48e0d0180158263/] Global properties->Environment variables [crayon-6750e0c48e0d6595854563/] We have a Jenkins job to build a huge Java project, and it has often such...</p> The post <a href="http://www.mobabel.net/increase-the-building-heap-size-for-jenkins-jobs/">Increase the building heap-size for Jenkins jobs</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p>In Jenkins configuration the following configs have been set:</p> <p>Maven Project Configuration->Global MAVEN_OPTS</p><pre class="crayon-plain-tag">-Xmx8192m -XX:MaxPermSize=1024m</pre><p>Global properties->Environment variables</p><pre class="crayon-plain-tag">Name: MAVEN_OPTS Value: -Xmx8192m -XX:MaxPermSize=1024m -Djava.io.tmpdir=/clusters/shared_workspace/tmp</pre><p><span id="more-7364"></span></p> <p>We have a Jenkins job to build a huge Java project, and it has often such OutOfMemoryError during the building process.</p><pre class="crayon-plain-tag"><span class="pipeline-node-56">Exception in thread "Thread-177" java.lang.OutOfMemoryError: GC overhead limit exceeded at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:98) at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:84) at com.codahale.metrics.Histogram.update(Histogram.java:41) at com.codahale.metrics.Timer.update(Timer.java:185) at com.codahale.metrics.Timer.update(Timer.java:89) at io.micrometer.core.instrument.dropwizard.DropwizardTimer.recordNonNegative(DropwizardTimer.java:43) at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:171) at io.micrometer.core.instrument.AbstractTimer.recordValueWithExpectedInterval(AbstractTimer.java:130) at io.micrometer.core.instrument.AbstractTimer.lambda$initPauseDetector$1(AbstractTimer.java:116) at io.micrometer.core.instrument.AbstractTimer$$Lambda$384/1481847479.handlePauseEvent(Unknown Source) at org.LatencyUtils.PauseDetector$PauseDetectorThread.run(PauseDetector.java:107)</span></pre><p>The heap memory logs looks like this:</p><pre class="crayon-plain-tag"><span class="pipeline-node-56">processors=16, physical.memory.total=62.7G, physical.memory.free=10.6G, swap.space.total=4.0G, swap.space.free=3.8G, heap.memory.used=864.7M, heap.memory.free=103.8M, heap.memory.total=968.5M, heap.memory.max=968.5M, heap.memory.used/total=89.17%, heap.memory.used/max=89.17%, minor.gc.count=269, minor.gc.time=3852ms, major.gc.count=418, major.gc.time=506094ms,</span></pre><p>The running Jenkins was built with a docker cluster, one master with two workers.</p> <h2 id='1-1-set-docker-start-java-opts' id="boomdevs_1" >1. 1. Set docker start JAVA_OPTS</h2> <p>The first try is to set the initialized Java options when starting the container:</p><pre class="crayon-plain-tag">master: image: jenkins/jenkins:lts-jdk11 user: '88396' ports: - "8080" build: context: . dockerfile: Dockerfile-jenkins-master environment: - "JAVA_OPTS=-Djava.awt.headless=true -Xmx12g -Xms12g -Djava.net.preferIPv4Stack=true -Duser.home=/jenkins_home -Duser.timezone=Europe/Berlin -Dorg.apache.commons.jelly.tags.fmt.timeZone=Europe/Berlin"</pre><p>The VM has 64GB of memory and it should allow to run three 12GB memory containers and will not face memory issues. After testing, the situation gets better, but the occurrence of failed building jobs is changing from often to not so often.</p> <p>Then I try to set an even larger Xmx from 12g to 16g. The situation gets worse, the Jenkins container was killed by the system sometimes. VM allocates up to <em>25</em>% of your <em>RAM</em> per JVM running on your machine, similar to the docker environment. It seems that 16GB crosses the red line.</p> <p> </p> <h2 id='2-2-set-maven-memory-limits' id="boomdevs_2" >2. 2. Set Maven Memory Limits</h2> <h3 id='2-1-2-1-set-in-project' id="boomdevs_3" >2-1. 2.1 Set in Project</h3> <p>Following this <a href="http://tutorials.jenkov.com/maven/maven-memory-limits.html">http://tutorials.jenkov.com/maven/maven-memory-limits.html</a></p> <p>I add a jvm.config in my project like the following structure:</p> <ul> <li>myproject <ul> <li>.mvn <ul> <li>jvm.config</li> </ul> </li> </ul> </li> </ul> <p>Inside the jvm.config file you need the following settings to be able to control the memory limits of Maven during a Maven build:</p><pre class="crayon-plain-tag">-Xmx6144m -Xms2048m</pre><p>It did change the JVM option when the job is running, but it will be overwritten by the Jenkins configuration.</p> <p><a href="http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_17-27-04.png"><img fetchpriority="high" decoding="async" class="alignnone size-full wp-image-7365" src="http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_17-27-04.png" alt="" width="1278" height="217" srcset="http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_17-27-04.png 1278w, http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_17-27-04-300x51.png 300w, http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_17-27-04-1024x174.png 1024w, http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_17-27-04-768x130.png 768w" sizes="(max-width: 1278px) 100vw, 1278px" /></a></p> <p>This could help you to build projects locally with larger memory, but not helpful in Jenkins.</p> <h3 id='2-2-2-2-set-in-maven-s-mvn-sh' id="boomdevs_4" >2-2. 2.2 Set in Maven’s mvn.sh</h3> <p>Reference: <a href="https://techtractall.wordpress.com/2017/02/22/how-to-increase-java-heap-space-on-maven-and-ant/">https://techtractall.wordpress.com/2017/02/22/how-to-increase-java-heap-space-on-maven-and-ant/</a></p> <p>Tested, it does not help.</p> <p> </p> <h2 id='3-3-set-project-pom-maven-compiler' id="boomdevs_5" >3. 3. Set Project POM maven-compiler</h2> <p>After several testing I noticed that in logs, the heap memory never changes, it keeps with less than 1GB.</p><pre class="crayon-plain-tag">heap.memory.used=864.7M, heap.memory.free=103.8M, heap.memory.total=968.5M, heap.memory.max=968.5M, heap.memory.used/total=89.17%, heap.memory.used/max=89.17%</pre><p>How can increase the heap memory size when compiling the project on Jenkins?</p> <p>Ultimately I found this:<a href="https://stackoverflow.com/questions/12498738/maven-out-of-memory-build-failure"> https://stackoverflow.com/questions/12498738/maven-out-of-memory-build-failure</a></p><pre class="crayon-plain-tag"><plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <fork>true</fork> <meminitial>1024m</meminitial> <maxmem>2024m</maxmem> </configuration> </plugin></pre><p>After testing, you guess what? The heap size has been changed!</p><pre class="crayon-plain-tag">processors=16, physical.memory.total=62.6G, physical.memory.free=1.6G, swap.space.total=4.0G, swap.space.free=1.3G, heap.memory.used=<span style="color: #ff0000;"><strong>1.5G</strong></span>, heap.memory.free=<span style="color: #ff0000;"><strong>4.3G</strong></span>, heap.memory.total=<strong><span style="color: #ff0000;">5.7G</span></strong>, heap.memory.max=13.9G, heap.memory.used/total=25.42%, heap.memory.used/max=10.45%,</pre><p>There is no more OutOfMemoryError issue!</p> <p>More detail: <a href="https://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-with-memory-enhancements.html">https://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-with-memory-enhancements.html</a></p> <p> </p> <h2 id='4-4-how-to-set-jvm-for-jenkins-node-slave' id="boomdevs_6" >4. 4. How to set JVM for Jenkins Node (Slave)</h2> <p>If you face the OutOfMemoryError issue on the node, add this property in node configuration.</p> <p><a href="http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_11-02-23.png"><img decoding="async" class="alignnone size-full wp-image-7366" src="http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_11-02-23.png" alt="" width="1010" height="487" srcset="http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_11-02-23.png 1010w, http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_11-02-23-300x145.png 300w, http://www.mobabel.net/wp-content/uploads/2022/03/01/Snipaste_2022-03-01_11-02-23-768x370.png 768w" sizes="(max-width: 1010px) 100vw, 1010px" /></a></p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p>The post <a href="http://www.mobabel.net/increase-the-building-heap-size-for-jenkins-jobs/">Increase the building heap-size for Jenkins jobs</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/increase-the-building-heap-size-for-jenkins-jobs/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Jenkins pipeline to create and trigger jobs dynamically inside a loop</title> <link>http://www.mobabel.net/jenkins-pipeline-to-create-and-trigger-jobs-dynamically-inside-a-loop/</link> <comments>http://www.mobabel.net/jenkins-pipeline-to-create-and-trigger-jobs-dynamically-inside-a-loop/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Fri, 25 Feb 2022 23:22:55 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Jenkins]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=7357</guid> <description><![CDATA[<p>There is a requirement from the customer, to build a pipeline, which will be executed to call 100 remote entry points. Of course, we have a classic approach to do this: [crayon-6750e0c48e53a963971140/] Is there...</p> The post <a href="http://www.mobabel.net/jenkins-pipeline-to-create-and-trigger-jobs-dynamically-inside-a-loop/">Jenkins pipeline to create and trigger jobs dynamically inside a loop</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p>There is a requirement from the customer, to build a pipeline, which will be executed to call 100 remote entry points. Of course, we have a classic approach to do this:</p><pre class="crayon-plain-tag">stage('aa_11'){ steps{ build job: 'Tasks/pipeline_to_call_remote_api', wait: true, parameters: [ string(name: 'PARAM1', value: 'aa'), string(name: 'PARAM2', value: '11') ] } } stage('aa_22'){ steps{ build job: 'Tasks/pipeline_to_call_remote_api', wait: true, parameters: [ string(name: 'PARAM1', value: 'aa'), string(name: 'PARAM2', value: '22') ] } } .......</pre><p><span id="more-7357"></span></p> <p>Is there any other approach to create stages dynamically and execute them sequentially inside a loop? Yes.</p><pre class="crayon-plain-tag">stage('Init data'){ steps{ script { list = ["aa_11", "aa_22", "aa_33", ......."cc_11", "dd_22"] } } post { cleanup { cleanWs() } } } stage('Dynamic Stages') { steps { script { for(int i=0; i < list.size(); i++) { def t = list[i].split('_')[0] def p = list[i].split('_')[1] stage("Sync ${list[i]}"){ echo "Sync ${list[i]}" //steps{ //DO NOT use steps here //Even stage fails here, but pipeline will continue the execution. catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') { build job: 'Tasks/pipeline_to_call_remote_api', wait: true, parameters: [ string(name: 'PARAM1', value: t), string(name: 'PARAM2', value: p) ] } //} } } } } post { cleanup { cleanWs() } } }</pre><p>Use catchError to let the pipeline continue building even some stages fail.</p><pre class="crayon-plain-tag">catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {}</pre><p><a href="http://www.mobabel.net/wp-content/uploads/2022/02/25/Snipaste_2022-02-25_23-18-30.png"><img decoding="async" class="alignnone size-full wp-image-7358" src="http://www.mobabel.net/wp-content/uploads/2022/02/25/Snipaste_2022-02-25_23-18-30.png" alt="" width="1148" height="271" srcset="http://www.mobabel.net/wp-content/uploads/2022/02/25/Snipaste_2022-02-25_23-18-30.png 1148w, http://www.mobabel.net/wp-content/uploads/2022/02/25/Snipaste_2022-02-25_23-18-30-300x71.png 300w, http://www.mobabel.net/wp-content/uploads/2022/02/25/Snipaste_2022-02-25_23-18-30-1024x242.png 1024w, http://www.mobabel.net/wp-content/uploads/2022/02/25/Snipaste_2022-02-25_23-18-30-768x181.png 768w" sizes="(max-width: 1148px) 100vw, 1148px" /></a></p> <p>Bingo!</p> <p> </p> <p> </p> <p> </p> <p>Reference:</p> <p><a href="https://stackoverflow.com/questions/67482808/jenkins-pipeline-trigger-jobs-sequentially-inside-a-loop">https://stackoverflow.com/questions/67482808/jenkins-pipeline-trigger-jobs-sequentially-inside-a-loop</a></p> <p><a href="https://stackoverflow.com/questions/55723060/is-there-a-way-to-use-propagate-false-in-a-jenkinsfile-with-declarative-syntax">https://stackoverflow.com/questions/55723060/is-there-a-way-to-use-propagate-false-in-a-jenkinsfile-with-declarative-syntax</a></p> <p> </p> <p> </p>The post <a href="http://www.mobabel.net/jenkins-pipeline-to-create-and-trigger-jobs-dynamically-inside-a-loop/">Jenkins pipeline to create and trigger jobs dynamically inside a loop</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/jenkins-pipeline-to-create-and-trigger-jobs-dynamically-inside-a-loop/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>[转]为什么不建议把数据库部署在Docker容器内?</title> <link>http://www.mobabel.net/%e8%bd%ac%e4%b8%ba%e4%bb%80%e4%b9%88%e4%b8%8d%e5%bb%ba%e8%ae%ae%e6%8a%8a%e6%95%b0%e6%8d%ae%e5%ba%93%e9%83%a8%e7%bd%b2%e5%9c%a8docker%e5%ae%b9%e5%99%a8%e5%86%85%ef%bc%9f/</link> <comments>http://www.mobabel.net/%e8%bd%ac%e4%b8%ba%e4%bb%80%e4%b9%88%e4%b8%8d%e5%bb%ba%e8%ae%ae%e6%8a%8a%e6%95%b0%e6%8d%ae%e5%ba%93%e9%83%a8%e7%bd%b2%e5%9c%a8docker%e5%ae%b9%e5%99%a8%e5%86%85%ef%bc%9f/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Tue, 04 May 2021 22:56:43 +0000</pubDate> <category><![CDATA[Docker]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6756</guid> <description><![CDATA[<p>近2年Docker非常的火热,各位开发者恨不得把所有的应用、软件都部署在Docker容器中,但是您确定也要把数据库也部署的容器中吗? 这个问题不是子虚乌有,因为在网上能够找到很多各种操作手册和视频教程,小编整理了一些数据库不适合容器化的原因供大家参考,同时也希望大家在使用时能够谨慎一点。 目前为止将数据库容器化是非常不合理的,但是容器化的优点相信各位开发者都尝到了甜头,希望随着技术的发展能够更加完美的解决方案出现。 1. Docker不适合部署数据库的7大原因 2. 1、数据安全问题 不要将数据储存在容器中,这也是 Docker 官方容器使用技巧中的一条。容器随时可以停止、或者删除。当容器被rm掉,容器里的数据将会丢失。为了避免数据丢失,用户可以使用数据卷挂载来存储数据。但是容器的 Volumes 设计是围绕 Union FS 镜像层提供持久存储,数据安全缺乏保证。如果容器突然崩溃,数据库未正常关闭,可能会损坏数据。另外,容器里共享数据卷组,对物理机硬件损伤也比较大。 即使你要把 Docker 数据放在主机来存储 ,它依然不能保证不丢数据。Docker volumes 的设计围绕 Union FS 镜像层提供持久存储,但它仍然缺乏保证。 使用当前的存储驱动程序,Docker 仍然存在不可靠的风险。如果容器崩溃并数据库未正确关闭,则可能会损坏数据。 3. 2、性能问题 大家都知道,MySQL 属于关系型数据库,对IO要求较高。当一台物理机跑多个时,IO就会累加,导致IO瓶颈,大大降低 MySQL 的读写性能。 在一次Docker应用的十大难点专场上,某国有银行的一位架构师也曾提出过:“数据库的性能瓶颈一般出现在IO上面,如果按 Docker...</p> The post <a href="http://www.mobabel.net/%e8%bd%ac%e4%b8%ba%e4%bb%80%e4%b9%88%e4%b8%8d%e5%bb%ba%e8%ae%ae%e6%8a%8a%e6%95%b0%e6%8d%ae%e5%ba%93%e9%83%a8%e7%bd%b2%e5%9c%a8docker%e5%ae%b9%e5%99%a8%e5%86%85%ef%bc%9f/">[转]为什么不建议把数据库部署在Docker容器内?</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p>近2年Docker非常的火热,各位开发者恨不得把所有的应用、软件都部署在Docker容器中,但是您确定也要把数据库也部署的容器中吗?</p> <p>这个问题不是子虚乌有,因为在网上能够找到很多各种操作手册和视频教程,小编整理了一些数据库不适合容器化的原因供大家参考,同时也希望大家在使用时能够谨慎一点。</p> <p>目前为止将数据库容器化是非常不合理的,但是容器化的优点相信各位开发者都尝到了甜头,希望随着技术的发展能够更加完美的解决方案出现。</p> <p><span id="more-6756"></span></p> <h1 id='1-docker不适合部署数据库的7大原因' id="boomdevs_1" >1. Docker不适合部署数据库的7大原因</h1> <h1 id='2-1-数据安全问题' id="boomdevs_2" >2. 1、数据安全问题</h1> <p>不要将数据储存在容器中,这也是 Docker 官方容器使用技巧中的一条。容器随时可以停止、或者删除。当容器被rm掉,容器里的数据将会丢失。为了避免数据丢失,用户可以使用数据卷挂载来存储数据。但是容器的 Volumes 设计是围绕 Union FS 镜像层提供持久存储,数据安全缺乏保证。如果容器突然崩溃,数据库未正常关闭,可能会损坏数据。另外,容器里共享数据卷组,对物理机硬件损伤也比较大。</p> <p>即使你要把 Docker 数据放在主机来存储 ,它依然不能保证不丢数据。<strong>Docker volumes 的设计围绕 Union FS 镜像层提供持久存储,但它仍然缺乏保证</strong>。</p> <p>使用当前的存储驱动程序,Docker 仍然存在不可靠的风险。如果容器崩溃并数据库未正确关闭,则可能会损坏数据。</p> <h1 id='3-2-性能问题' id="boomdevs_3" >3. 2、性能问题</h1> <p>大家都知道,MySQL 属于关系型数据库,对IO要求较高。<strong>当一台物理机跑多个时,IO就会累加,导致IO瓶颈,大大降低 MySQL 的读写性能</strong>。</p> <p>在一次Docker应用的十大难点专场上,某国有银行的一位架构师也曾提出过:“数据库的性能瓶颈一般出现在IO上面,如果按 Docker 的思路,那么多个docker最终IO请求又会出现在存储上面。现在互联网的数据库多是share nothing的架构,可能这也是不考虑迁移到 Docker 的一个因素吧”。</p> <p><strong>针对性能问题有些同学可能也有相对应的方案来解决:</strong></p> <p>(1)数据库程序与数据分离</p> <p>如果使用Docker 跑 MySQL,数据库程序与数据需要进行分离,将数据存放到共享存储,程序放到容器里。如果容器有异常或 MySQL 服务异常,自动启动一个全新的容器。另外,建议不要把数据存放到宿主机里,宿主机和容器共享卷组,对宿主机损坏的影响比较大。</p> <p>(2)跑轻量级或分布式数据库</p> <p>Docker 里部署轻量级或分布式数据库,Docker 本身就推荐服务挂掉,自动启动新容器,而不是继续重启容器服务。</p> <p>(3)合理布局应用</p> <p><strong>对于IO要求比较高的应用或者服务,将数据库部署在物理机或者KVM中比较合适</strong>。目前TX云的TDSQL和阿里的Oceanbase都是直接部署在物理机器,而非Docker 。</p> <h1 id='4-3-网络问题' id="boomdevs_4" >4. 3、网络问题</h1> <p>要理解 Docker 网络,您必须对网络虚拟化有深入的了解。也必须准备应付好意外情况。你可能需要在没有支持或没有额外工具的情况下,进行 bug 修复。</p> <p>我们知道:数据库需要专用的和持久的吞吐量,以实现更高的负载。我们还知道容器是虚拟机管理程序和主机虚拟机背后的一个隔离层。然而网络对于数据库复制是至关重要的,其中需要主从数据库间 24/7 的稳定连接。未解决的 Docker 网络问题在1.9版本依然没有得到解决。</p> <p>把这些问题放在一起,容器化使数据库容器很难管理。我知道你是一个顶级的工程师,什么问题都可以得到解决。但是,你需要花多少时间解决 Docker 网络问题?将数据库放在专用环境不会更好吗?节省时间来专注于真正重要的业务目标。</p> <h1 id='5-4-状态' id="boomdevs_5" >5. 4、状态</h1> <p>在 Docker 中打包无状态服务是很酷的,可以实现编排容器并解决单点故障问题。但是数据库呢?将数据库放在同一个环境中,它将会是有状态的,并使系统故障的范围更大。下次您的应用程序实例或应用程序崩溃,可能会影响数据库。</p> <p>知识点在 Docker 中水平伸缩只能用于无状态计算服务,而不是数据库。</p> <p>Docker 快速扩展的一个重要特征就是无状态,具有数据状态的都不适合直接放在 Docker 里面,如果 Docker 中安装数据库,存储服务需要单独提供。</p> <p>目前,TX云的TDSQL(金融分布式数据库)和阿里云的Oceanbase(分布式数据库系统)都直接运行中在物理机器上,并非使用便于管理的 Docker 上。</p> <h1 id='6-5-资源隔离' id="boomdevs_6" >6. 5、资源隔离</h1> <p><strong>资源隔离方面,Docker 确实不如虚拟机KVM,<span style="color: #ff6600;">Docker是利用Cgroup实现资源限制的,只能限制资源消耗的最大值,而不能隔绝其他程序占用自己的资源</span></strong>。如果其他应用过渡占用物理机资源,将会影响容器里 MySQL 的读写效率。</p> <p>需要的隔离级别越多,获得的资源开销就越多。相比专用环境而言,容易水平伸缩是Docker的一大优势。然而在 Docker 中水平伸缩只能用于无状态计算服务,数据库并不适用。</p> <p>我们没有看到任何针对数据库的隔离功能,那为什么我们应该把它放在容器中呢?</p> <h1 id='7-6-云平台的不适用性' id="boomdevs_7" >7. 6、云平台的不适用性</h1> <p>大部分人通过共有云开始项目。云简化了虚拟机操作和替换的复杂性,因此不需要在夜间或周末没有人工作时间来测试新的硬件环境。当我们可以迅速启动一个实例的时候,为什么我们需要担心这个实例运行的环境?</p> <p>这就是为什么我们向云提供商支付很多费用的原因。当我们为实例放置数据库容器时,上面说的这些便利性就不存在了。因为数据不匹配,新实例不会与现有的实例兼容,如果要限制实例使用单机服务,应该让 DB 使用非容器化环境,我们仅仅需要为计算服务层保留弹性扩展的能力。</p> <h1 id='8-7-运行数据库的环境需求' id="boomdevs_8" >8. 7、运行数据库的环境需求</h1> <p>常看到 DBMS 容器和其他服务运行在同一主机上。然而这些服务对硬件要求是非常不同的。</p> <p>数据库(特别是关系型数据库)对 IO 的要求较高。一般数据库引擎为了避免并发资源竞争而使用专用环境。如果将你的数据库放在容器中,那么将浪费你的项目的资源。因为你需要为该实例配置大量额外的资源。在公有云,当你需要 34G 内存时,你启动的实例却必须开 64G 内存。在实践中,这些资源并未完全使用。</p> <p>怎么解决?您可以分层设计,并使用固定资源来启动不同层次的多个实例。水平伸缩总是比垂直伸缩更好。</p> <h1 id='9-总结' id="boomdevs_9" >9. 总结</h1> <p>针对上面问题是不是说数据库一定不要部署在容器里吗?</p> <p>答案是:并不是</p> <p>我们可以把数据丢失不敏感的业务(搜索、埋点)就可以数据化,利用数据库分片来来增加实例数,从而增加吞吐量。</p> <p>docker适合跑轻量级或分布式数据库,当docker服务挂掉,会自动启动新容器,而不是继续重启容器服务。</p> <p>数据库利用中间件和容器化系统能够自动伸缩、容灾、切换、自带多个节点,也是可以进行容器化的。</p> <p> </p> <p> </p> <p><a href="https://mp.weixin.qq.com/s/fghaFoPesJ2iA5Hp_ARwSA">为什么不建议把数据库部署在Docker容器内?</a></p>The post <a href="http://www.mobabel.net/%e8%bd%ac%e4%b8%ba%e4%bb%80%e4%b9%88%e4%b8%8d%e5%bb%ba%e8%ae%ae%e6%8a%8a%e6%95%b0%e6%8d%ae%e5%ba%93%e9%83%a8%e7%bd%b2%e5%9c%a8docker%e5%ae%b9%e5%99%a8%e5%86%85%ef%bc%9f/">[转]为什么不建议把数据库部署在Docker容器内?</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/%e8%bd%ac%e4%b8%ba%e4%bb%80%e4%b9%88%e4%b8%8d%e5%bb%ba%e8%ae%ae%e6%8a%8a%e6%95%b0%e6%8d%ae%e5%ba%93%e9%83%a8%e7%bd%b2%e5%9c%a8docker%e5%ae%b9%e5%99%a8%e5%86%85%ef%bc%9f/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>[汇总]DevOps经验</title> <link>http://www.mobabel.net/%e6%b1%87%e6%80%bbdevops%e7%bb%8f%e9%aa%8c/</link> <comments>http://www.mobabel.net/%e6%b1%87%e6%80%bbdevops%e7%bb%8f%e9%aa%8c/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Mon, 03 May 2021 00:43:49 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[DevOPS]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6720</guid> <description><![CDATA[<p>简单脚本监控SSL证书,并到期提醒续签 更高效的SSL证书监控方案 别再用kill -9关闭程序了 通过Vagrant+Hyper-v来构建开发环境 以微博核心业务为例,解读如何仅用1台服务器支持百万DAU DevOps 已死,平台工程才是未来 放弃 Ceph,Salesforce 使用 Apache BookKeeper 在云中实现最强存储   Java Buildpack Memory Calculator 给go项目打最小docker镜像,足足降低99%    </p> The post <a href="http://www.mobabel.net/%e6%b1%87%e6%80%bbdevops%e7%bb%8f%e9%aa%8c/">[汇总]DevOps经验</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p id="activity-name" class="rich_media_title"><a href="https://mp.weixin.qq.com/s/IGJHZXv-XGunjj9Fle7IHQ">简单脚本监控SSL证书,并到期提醒续签</a></p> <p id="activity-name" class="rich_media_title"><a href="https://mp.weixin.qq.com/s/Lp0mlhyIuXLTJPrB7HLdLA">更高效的SSL证书监控方案</a></p> <p id="activity-name" class="rich_media_title"><a href="https://mp.weixin.qq.com/s/s47WIVUntQAhcvCoYuuHZw">别再用kill -9关闭程序了</a></p> <p><a href="https://mp.weixin.qq.com/s/g4PuwGQb920jv5mqowj0IA">通过Vagrant+Hyper-v来构建开发环境</a></p> <p><a href="https://mp.weixin.qq.com/s/oZ3SyleuMDf7Cf_jIK6_Ug">以微博核心业务为例,解读如何仅用1台服务器支持百万DAU</a></p> <p><a href="https://mp.weixin.qq.com/s/P_XdxR0trQA7huXurgI2SQ">DevOps 已死,平台工程才是未来</a></p> <p id="activity-name" class="rich_media_title"><a href="https://mp.weixin.qq.com/s/3nlt91AN_UpntzhjcG6vuQ">放弃 Ceph,Salesforce 使用 Apache BookKeeper 在云中实现最强存储</a></p> <p> </p> <p dir="auto"><a href="https://github.com/cloudfoundry/java-buildpack-memory-calculator">Java Buildpack Memory Calculator</a></p> <p id="activity-name" class="rich_media_title "><a href="https://mp.weixin.qq.com/s/ojNTRA_9kc23OV3c-83AYQ">给go项目打最小docker镜像,足足降低99%</a></p> <p id="activity-name" class="rich_media_title "> <p> </p> <p> </p>The post <a href="http://www.mobabel.net/%e6%b1%87%e6%80%bbdevops%e7%bb%8f%e9%aa%8c/">[汇总]DevOps经验</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/%e6%b1%87%e6%80%bbdevops%e7%bb%8f%e9%aa%8c/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Memory/CPU limit in Docker Swarm</title> <link>http://www.mobabel.net/memory-cpu-limit-in-docker-swarm/</link> <comments>http://www.mobabel.net/memory-cpu-limit-in-docker-swarm/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Sun, 18 Apr 2021 13:03:47 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Docker]]></category> <category><![CDATA[Docker Swarm]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6708</guid> <description><![CDATA[<p>Running 1,000 Containers in Docker Swarm How to Limit Resources for Docker Swarm Services https://docs.docker.com/config/containers/resource_constraints/ What does --oom-kill-disable do for a Docker container? Unable to use oom-kill-disable configuration Docker Container Resource Management: CPU, RAM...</p> The post <a href="http://www.mobabel.net/memory-cpu-limit-in-docker-swarm/">Memory/CPU limit in Docker Swarm</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p class="display-2 font-weight-medium mb-4_5 mw-7"><a href="https://www.cloudbees.com/blog/running-1000-containers-in-docker-swarm/">Running 1,000 Containers in Docker Swarm</a></p> <p id="f37d" class="do dp dq dr b ds dt du dv dw dx dy dz ea eb ec ed ee ef eg eh ei ej ek el em en"><a href="https://betterprogramming.pub/limit-resources-for-swarm-services-249dcebed833">How to Limit Resources for Docker Swarm Services</a></p> <p><a href="https://docs.docker.com/config/containers/resource_constraints/">https://docs.docker.com/config/containers/resource_constraints/</a></p> <p class="fs-headline1 ow-break-word mb8 grid--cell fl1"><a class="question-hyperlink" href="https://stackoverflow.com/questions/48618431/what-does-oom-kill-disable-do-for-a-docker-container">What does <code>--oom-kill-disable</code> do for a Docker container?</a></p> <p data-topic-id="60487"><a class="fancy-title" href="https://forums.docker.com/t/unable-to-use-oom-kill-disable-configuration/60487" data-ember-action="" data-ember-action-22="22">Unable to use oom-kill-disable configuration</a></p> <p class="article-title"><a href="https://dzone.com/articles/docker-container-resource-management-cpu-ram-and-i">Docker Container Resource Management: CPU, RAM and IO, Part 1</a></p> <p class="article-title J-articleTitle"><a href="https://cloud.tencent.com/developer/article/1426760">Docker容器CPU、memory资源限制</a></p> <p id="让jvm感知docker的参数"><a href="https://daocloud-labs.github.io/DX-DMP-Public-Docs/spec/jvm-docker.html">让JVM感知Docker的参数</a></p> <p class="postTitle"><a id="cb_post_title_url" class="postTitle2 vertical-middle" href="https://www.cnblogs.com/xiaoqi/p/container-jvm.html">容器环境的JVM内存设置最佳实践</a></p> <p><a href="https://hal.archives-ouvertes.fr/hal-02424007/document">Highlighting the Container Memory Consolidation Problems in Linux</a></p> <p><a href="https://developers.redhat.com/blog/2017/03/14/java-inside-docker/">Java inside docker: What you must know to not FAIL </a></p> <p> </p> <p id="jvm-oom文件储存方案"><a href="https://daocloud-labs.github.io/DX-DMP-Public-Docs/spec/JVM-OOM%E6%96%87%E4%BB%B6%E5%82%A8%E5%AD%98%E6%96%B9%E6%A1%88.html">JVM OOM文件储存方案</a></p> <p> </p>The post <a href="http://www.mobabel.net/memory-cpu-limit-in-docker-swarm/">Memory/CPU limit in Docker Swarm</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/memory-cpu-limit-in-docker-swarm/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Getting data from docker MySQL database with Jenkins pipeline and send Email as attachment</title> <link>http://www.mobabel.net/getting-data-from-docker-mysql-database-with-jenkins-pipeline-and-send-email-as-attachment/</link> <comments>http://www.mobabel.net/getting-data-from-docker-mysql-database-with-jenkins-pipeline-and-send-email-as-attachment/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Mon, 15 Mar 2021 17:00:41 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Jenkins]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6692</guid> <description><![CDATA[<p>In this post, a simple daily data report for one microservice will be created using the Jenkins pipeline. Prerequisites: Publish Over SSH Mattermost Notification Plugin Groovy Postbuild Jenkins 1. Background This is a special...</p> The post <a href="http://www.mobabel.net/getting-data-from-docker-mysql-database-with-jenkins-pipeline-and-send-email-as-attachment/">Getting data from docker MySQL database with Jenkins pipeline and send Email as attachment</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p>In this post, a simple daily data report for one microservice will be created using the Jenkins pipeline.</p> <p id="136c" class="gd ge bo gf b gg gh gi gj gk gl gm gn go gp gq gr gs gt gu gv cg ee" data-selectable-paragraph="">Prerequisites:</p> <ul class=""> <li data-selectable-paragraph=""> <p class="title"><a href="https://plugins.jenkins.io/publish-over-ssh/">Publish Over SSH</a></p> </li> <li data-selectable-paragraph=""> <p class="title"><a class="bh fr gz ha hb hc" href="https://www.jenkins.io/doc/pipeline/steps/mattermost/" target="_blank" rel="noopener nofollow noreferrer">Mattermost Notification Plugin</a></p> </li> <li id="8190" class="gd ge bo gf b gg hd gi gj he gl gm hf go gp hg gr gs hh gu gv gw gx gy ee" data-selectable-paragraph=""><a href="https://plugins.jenkins.io/groovy-postbuild/">Groovy Postbuild</a></li> <li id="30e0" class="gd ge bo gf b gg hd gi gj he gl gm hf go gp hg gr gs hh gu gv gw gx gy ee" data-selectable-paragraph="">Jenkins</li> </ul> <p><span id="more-6692"></span></p> <h2 id='1-background' id="boomdevs_1" >1. Background</h2> <p>This is a special request from the developer team.</p> <p>We have a web service, which is running in the remote docker cluster.</p> <p> </p> <h2 id='2-1-configure-mattermost-config' id="boomdevs_2" >2. 1 Configure Mattermost Config</h2> <h2 id='3-2-create-job' id="boomdevs_3" >3. 2 Create Job</h2> <h2 id='4-3-add-build-trigger' id="boomdevs_4" >4. 3 Add Build Trigger</h2> <p>For the upper three steps, please check this article: <a title="Permalink to Creating a web service health check with Jenkins pipeline and Mattermost notification" href="http://www.mobabel.net/creating-a-web-service-health-check-with-jenkins-pipelines-and-mattermost-notification/" rel="bookmark">Creating a web service health check with Jenkins pipeline and Mattermost notification</a></p> <h2 id='5-4-publish-over-ssh-config' id="boomdevs_5" >5. 4 Publish Over SSH Config</h2> <p><a href="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15.png"><img loading="lazy" decoding="async" class="alignnone size-full wp-image-6596" src="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15.png" alt="" width="1074" height="346" srcset="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15.png 1074w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15-300x97.png 300w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15-1024x330.png 1024w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15-768x247.png 768w" sizes="(max-width: 1074px) 100vw, 1074px" /></a></p> <p> </p> <p><a href="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14.png"><img loading="lazy" decoding="async" class="alignnone size-full wp-image-6597" src="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14.png" alt="" width="1418" height="666" srcset="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14.png 1418w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-300x141.png 300w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-1024x481.png 1024w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-768x361.png 768w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-520x245.png 520w" sizes="(max-width: 1418px) 100vw, 1418px" /></a></p> <p> </p> <h2 id='6-full-script' id="boomdevs_6" >6. Full script:</h2> <p></p><pre class="crayon-plain-tag">pipeline { agent any environment { stageName = "Daily Data Report" matterMostChannel = "#jenkins-job-debug" mysqlContainer = "a33bdcebf6e4" execOutput = "" datePeriod = "" } stages { stage('Load part data') { steps { script { //ae-clip timezone -1 // the day before yesterday 23:00::00 ---- yesterday 23:00::00 def now = new Date() def dateMinusOne = now - 1 dayMinusOne = dateMinusOne.format("yyyy-MM-dd") def dayStart = dayMinusOne + " 00:00:00" def dayEnd = dayMinusOne + " 23:59:59" datePeriod = dayStart + "--" + dayEnd stageName = stageName + " [" + datePeriod + "]" echo "mysql/date: ${mysqlContainer} ${datePeriod}" echo "BEGINBEGINBEGIN" def cmd = """ cd /local/home/mobabel-dev/ucp_bundle && eval "\$(<env.sh)" && cd /local/home/mobabel-dev/docker-stack container_string=`docker ps | grep service-mysql-name` ; container_name=`echo "\$container_string" | cut -c1-12` ; exec_output=`docker exec -i \$container_name mysql -uroot <<< "use databasename;\ select * from my_data where created_date >= '${dayStart}' and created_date <= '${dayEnd}';"` ; if [ -z "\$exec_output" ] then echo "token""==No_data==" else echo "token""BEGINBEGINBEGIN\n""\$exec_output" echo "token""ENDENDEND\n" fi """ sshPublisher( continueOnError: false, failOnError: true, publishers: [ sshPublisherDesc( configName: 'MyServerClient', transfers: [ sshTransfer( execCommand: ''' cd /local/home/mobabel-dev/ucp_bundle && eval "$(<env.sh)" && cd /local/home/mobabel-dev/docker-stack ''' ), sshTransfer( remoteDirectory: 'docker-stack', removePrefix: '', execCommand: cmd ) ], verbose: true ) ] ) } } } stage('CheckLog') { steps { script { def list = manager.build.logFile.readLines() def BeginCount = list.count {it.startsWith("token") && it.contains("BEGINBEGINBEGIN")} def EndCount = list.count {it.startsWith("token") && it.contains("ENDENDEND")} echo("BeginCount: ${BeginCount}/EndCount: ${EndCount}") if (BeginCount == 1 && EndCount == 1){ // enable "staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.io.File" // in https://jenkins.server/scriptApproval/ def logContent = manager.build.logFile.text def startIndex = logContent.indexOf('tokenBEGINBEGINBEGIN') + 'tokenBEGINBEGINBEGIN'.length() def endIndex = logContent.indexOf('tokenENDENDEND') def rawOutput = logContent.substring(startIndex, endIndex) //execOutput = rawOutput.replace("\n", "<br/>\n") execOutput = 'Please see attachment.' writeFile file: "dailylog.txt", text: rawOutput } } } } } // Finishing Up Actions post { success { script { notifySucc(stageName, matterMostChannel, execOutput) } } failure { script { notifyError(stageName, matterMostChannel) } } unstable { script { notifyUnstable(stageName, matterMostChannel) } } fixed { script { notifyFixed(stageName, matterMostChannel) } } } } def notifySucc(String stageName = '', String matterMostChannel = '', String execOutput = '') { def message = "$stageName" mattermostSend channel: matterMostChannel, color: "good", message: message emailext ( subject: message, to: "your_mail@mail.com", body: """<p>$stageName':</p> <p>$execOutput</p>""", attachmentsPattern: 'dailylog.txt' ) } def notifyError(String stageName = '', String matterMostChannel = '') { def message = ":rage: $stageName failed, please (<${env.BUILD_URL}| check detail for line id>). | ${env.BUILD_NUMBER} " echo(message) mattermostSend channel: matterMostChannel, color: "danger", message: message emailext ( subject: message, to: "your_mail@mail.com;", body: """<p>$stageName Failed: '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p> <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>""" ) } def notifyUnstable(String stageName = '', String matterMostChannel = '') { def message = ":warning: $stageName is aborted. | ${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)" echo(message) mattermostSend channel: matterMostChannel, color: "warning", message: message } def notifyFixed(String stageName = '', String matterMostChannel = '') { def message = ":white_check_mark: $stageName is returning to health. | ${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)" echo(message) mattermostSend channel: matterMostChannel, color: "good", message: message }</pre><p>It will capture part of the Jenkins log output and save it into one text file under the Jenkins workshop. Then send an email with this text file as an attachment.</p> <p> </p> <p> </p> <p>Reference:</p> <blockquote class="wp-embedded-content" data-secret="bDYm0zVG0V"><p><a href="https://fraaargh.wordpress.com/2018/06/20/get-a-jobs-console-logfile-from-a-jenkins-pipeline/">Get a job’s console logfile from a Jenkins pipeline</a></p></blockquote> <p><iframe loading="lazy" class="wp-embedded-content" sandbox="allow-scripts" security="restricted" title="“Get a job’s console logfile from a Jenkins pipeline” — Some thoughts about things" src="https://fraaargh.wordpress.com/2018/06/20/get-a-jobs-console-logfile-from-a-jenkins-pipeline/embed/#?secret=ijsvk4fRHI#?secret=bDYm0zVG0V" data-secret="bDYm0zVG0V" width="500" height="282" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe></p> <p>https://medium.com/@gustavo.guss/jenkins-send-email-with-attachment-cec1e052583a</p> <p> </p>The post <a href="http://www.mobabel.net/getting-data-from-docker-mysql-database-with-jenkins-pipeline-and-send-email-as-attachment/">Getting data from docker MySQL database with Jenkins pipeline and send Email as attachment</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/getting-data-from-docker-mysql-database-with-jenkins-pipeline-and-send-email-as-attachment/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Use multiple JDKs in one Jenkinsfile</title> <link>http://www.mobabel.net/use-multiple-jdks-in-one-jenkinsfile/</link> <comments>http://www.mobabel.net/use-multiple-jdks-in-one-jenkinsfile/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Mon, 01 Mar 2021 15:52:14 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Jenkins]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6687</guid> <description><![CDATA[<p>As usual, the Jenkins pipeline will only use the default JDK to execute the task. How to use multiple JDKs in one Jenkinsfile for different stages? First, define the multiple JDKs in Jenkins “Global...</p> The post <a href="http://www.mobabel.net/use-multiple-jdks-in-one-jenkinsfile/">Use multiple JDKs in one Jenkinsfile</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p>As usual, the Jenkins pipeline will only use the default JDK to execute the task. How to use multiple JDKs in one Jenkinsfile for different stages?</p> <p>First, define the multiple JDKs in Jenkins “Global Tool Configuration”.</p> <p><a href="http://www.mobabel.net/wp-content/uploads/2021/03/01/Snipaste_2021-03-01_15-29-25.png"><img loading="lazy" decoding="async" class="alignnone size-full wp-image-6688" src="http://www.mobabel.net/wp-content/uploads/2021/03/01/Snipaste_2021-03-01_15-29-25.png" alt="" width="1298" height="737" srcset="http://www.mobabel.net/wp-content/uploads/2021/03/01/Snipaste_2021-03-01_15-29-25.png 1298w, http://www.mobabel.net/wp-content/uploads/2021/03/01/Snipaste_2021-03-01_15-29-25-300x170.png 300w, http://www.mobabel.net/wp-content/uploads/2021/03/01/Snipaste_2021-03-01_15-29-25-1024x581.png 1024w, http://www.mobabel.net/wp-content/uploads/2021/03/01/Snipaste_2021-03-01_15-29-25-768x436.png 768w" sizes="(max-width: 1298px) 100vw, 1298px" /></a></p> <p>Now we have for example JDK11 and JDK8(used as default JDK)</p> <p>And here is one sample pipeline file.</p><pre class="crayon-plain-tag">#!/usr/bin/env groovy node { jdk = tool name: 'JDK11' env.JAVA_HOME = "${jdk}" echo "jdk installation path is: ${jdk}" withMaven() { // use pipeline-maven-plugin to config the maven stage('checkout') { checkout scm } stage('check java') { sh "$JAVA_HOME/bin/java -version" } stage('clean') { sh "mvn clean" } stage('backend tests') { sh "mvn test -Pdev" } stage('build docker') { sh "mvn dockerfile:build" } stage('public docker') { sh "mvn dockerfile:push" } stage('stage use default JDK8') { jdk = tool name: 'JDK8' env.JAVA_HOME = "${jdk}" sh "mvn special-stage" } } }</pre><p>This is the solution.</p> <p> </p> <p> </p> <p> </p>The post <a href="http://www.mobabel.net/use-multiple-jdks-in-one-jenkinsfile/">Use multiple JDKs in one Jenkinsfile</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/use-multiple-jdks-in-one-jenkinsfile/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Creating server storage check with Jenkins pipeline and Mattermost notification</title> <link>http://www.mobabel.net/creating-server-storage-check-with-jenkins-pipeline-and-mattermost-notification/</link> <comments>http://www.mobabel.net/creating-server-storage-check-with-jenkins-pipeline-and-mattermost-notification/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Fri, 31 Jul 2020 16:17:43 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Jenkins]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6619</guid> <description><![CDATA[<p>      [crayon-6750e0c48ec08749300305/]  </p> The post <a href="http://www.mobabel.net/creating-server-storage-check-with-jenkins-pipeline-and-mattermost-notification/">Creating server storage check with Jenkins pipeline and Mattermost notification</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p> </p> <p><span id="more-6619"></span></p> <p> </p> <p> </p><pre class="crayon-plain-tag">pipeline { agent any environment { stageName = "Jenkins Health Check" mattermostChannel = "#mattermost-channel" diskUsagePercent = 0 diskUsagePercentMax = 75 } stages { stage('Check Disk Space Usage') { steps { script { // /dev/mapper/vg0-fs0vol 1.0T 576G 449G 57% def cmd = """ check_string=`df -h /dev/mapper/vg0-fs0vol` ; check_result=`"\${check_string}" |& tail -1` ; IFS=' ' read -ra check_array <<< "\$check_result" ; percent=` echo "\${check_array[4]:0:2}" ` ; percent=`echo \$percent | sed "s/%/ /g"` ; percent=`echo -e \$percent | sed -e 's/[[:space:]]*\$//'` ; echo "Disk usage: \$percent%"; echo "token""==\$percent==" """ sshPublisher( continueOnError: false, failOnError: true, publishers: [ sshPublisherDesc( configName: 'jenkins.server.com', transfers: [ sshTransfer( execCommand: ''' cd /local/home/jenkins/ ''' ), sshTransfer( remoteDirectory: '', removePrefix: '', sourceFiles: '*/**', execCommand: cmd ) ], verbose: true ) ] ) } } } stage('CheckLog') { steps { script { def list = manager.build.logFile.readLines() for (i = 1; i <100; i++) { def JobCount = list.count {it.startsWith("token") && it.contains("token==${i}==")} echo("JobCount ${JobCount} at ${i}") if (JobCount == 1){ diskUsagePercent = i def maxInt = Integer.parseInt(diskUsagePercentMax) if(diskUsagePercent >= maxInt){ error("token==${diskUsagePercent}==") } } } } } } } // Finishing Up Actions post { success { script { //notifySucc(stageName, diskUsagePercent) echo(stageName) } } failure { script { notifyError(stageName, diskUsagePercent) } } unstable { script { notifyUnstable(stageName, diskUsagePercent) } } fixed { script { notifyFixed(stageName, diskUsagePercent) } } } } def notifySucc(String stageName = '', Integer diskUsagePercent = 0) { def message = ":beach_umbrella: $stageName is OK. Disk usage: $diskUsagePercent% " echo(message) mattermostSend channel: mattermostChannel, color: "good", message: message } def notifyError(String stageName = '', Integer diskUsagePercent = 0) { def message message = ":hammer_and_wrench: $stageName is warning. Disk usage: $diskUsagePercent%. :rocket: @hui.li GO GO GO to clean :rocket:." echo(message) mattermostSend channel: mattermostChannel, color: "danger", message: message emailext ( subject: message, to: "user@test.com;", body: """<p>$stageName Need Cleanup: '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p> <p>Disk space usage warning, please check &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>""" ) } def notifyUnstable(String stageName = '', Integer diskUsagePercent = 0) { def message = ":warning: $stageName is aborted." echo(message) mattermostSend channel: mattermostChannel, color: "warning", message: message } def notifyFixed(String stageName = '', Integer diskUsagePercent = 0) { def message = ":sun_with_face: $stageName is back, Jenkins has been cleaned up. Disk usage: $diskUsagePercent%" echo(message) mattermostSend channel: mattermostChannel, color: "good", message: message }</pre><p> </p>The post <a href="http://www.mobabel.net/creating-server-storage-check-with-jenkins-pipeline-and-mattermost-notification/">Creating server storage check with Jenkins pipeline and Mattermost notification</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/creating-server-storage-check-with-jenkins-pipeline-and-mattermost-notification/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Docker Swarm monitoring with Prometheus and Grafana</title> <link>http://www.mobabel.net/docker-swarm-monitoring-with-prometheus-and-grafana/</link> <comments>http://www.mobabel.net/docker-swarm-monitoring-with-prometheus-and-grafana/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Fri, 31 Jul 2020 14:53:36 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Docker Swarm]]></category> <category><![CDATA[Grafana]]></category> <category><![CDATA[Prometheus]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6617</guid> <description><![CDATA[<p>Swarmprom is a starter kit for Docker Swarm monitoring with [Prometheus](https://prometheus.io/), [Grafana](http://grafana.org/), [cAdvisor](https://github.com/google/cadvisor), [Node Exporter](https://github.com/prometheus/node_exporter), [Alert Manager](https://github.com/prometheus/alertmanager) and [Unsee](https://github.com/cloudflare/unsee). https://github.com/stefanprodan/swarmprom.git</p> The post <a href="http://www.mobabel.net/docker-swarm-monitoring-with-prometheus-and-grafana/">Docker Swarm monitoring with Prometheus and Grafana</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<div> <div>Swarmprom is a starter kit for Docker Swarm monitoring with [Prometheus](https://prometheus.io/),</div> <div>[Grafana](http://grafana.org/),</div> <div>[cAdvisor](https://github.com/google/cadvisor),</div> <div>[Node Exporter](https://github.com/prometheus/node_exporter),</div> <div>[Alert Manager](https://github.com/prometheus/alertmanager)</div> <div>and [Unsee](https://github.com/cloudflare/unsee).</div> </div> <div></div> <div>https://github.com/stefanprodan/swarmprom.git</div> <div id="gtx-trans" style="position: absolute; left: 457px; top: 33px;"> <div class="gtx-trans-icon"></div> </div>The post <a href="http://www.mobabel.net/docker-swarm-monitoring-with-prometheus-and-grafana/">Docker Swarm monitoring with Prometheus and Grafana</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/docker-swarm-monitoring-with-prometheus-and-grafana/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Creating a remote docker MySQL database record validation check with Jenkins pipeline and Mattermost notification</title> <link>http://www.mobabel.net/creating-a-remote-docker-mysql-database-record-validation-check-with-jenkins-pipeline-and-mattermost-notification/</link> <comments>http://www.mobabel.net/creating-a-remote-docker-mysql-database-record-validation-check-with-jenkins-pipeline-and-mattermost-notification/#respond</comments> <dc:creator><![CDATA[leelight]]></dc:creator> <pubDate>Sat, 25 Jul 2020 11:42:16 +0000</pubDate> <category><![CDATA[DevOps Practice]]></category> <category><![CDATA[Jenkins]]></category> <guid isPermaLink="false">http://www.mobabel.net/?p=6592</guid> <description><![CDATA[<p>In this post, a simple health checker for one microservice will be created using the Jenkins pipeline. Prerequisites: Publish Over SSH Mattermost Notification Plugin Groovy Postbuild Jenkins 1. Background This is a special request...</p> The post <a href="http://www.mobabel.net/creating-a-remote-docker-mysql-database-record-validation-check-with-jenkins-pipeline-and-mattermost-notification/">Creating a remote docker MySQL database record validation check with Jenkins pipeline and Mattermost notification</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></description> <content:encoded><![CDATA[<p>In this post, a simple health checker for one microservice will be created using the Jenkins pipeline.</p> <p id="136c" class="gd ge bo gf b gg gh gi gj gk gl gm gn go gp gq gr gs gt gu gv cg ee" data-selectable-paragraph="">Prerequisites:</p> <ul class=""> <li data-selectable-paragraph=""> <p class="title"><a href="https://plugins.jenkins.io/publish-over-ssh/">Publish Over SSH</a></p> </li> <li data-selectable-paragraph=""> <p class="title"><a class="bh fr gz ha hb hc" href="https://www.jenkins.io/doc/pipeline/steps/mattermost/" target="_blank" rel="noopener nofollow noreferrer">Mattermost Notification Plugin</a></p> </li> <li id="8190" class="gd ge bo gf b gg hd gi gj he gl gm hf go gp hg gr gs hh gu gv gw gx gy ee" data-selectable-paragraph=""><a href="https://plugins.jenkins.io/groovy-postbuild/">Groovy Postbuild</a></li> <li id="30e0" class="gd ge bo gf b gg hd gi gj he gl gm hf go gp hg gr gs hh gu gv gw gx gy ee" data-selectable-paragraph="">Jenkins</li> </ul> <p><span id="more-6592"></span></p> <h2 id='1-background' id="boomdevs_1" >1. Background</h2> <p>This is a special request from the developer team.</p> <p>We have a web service, which is running in the remote docker cluster.</p> <p> </p> <h2 id='2-1-configure-mattermost-config' id="boomdevs_2" >2. 1 Configure Mattermost Config</h2> <h2 id='3-2-create-job' id="boomdevs_3" >3. 2 Create Job</h2> <h2 id='4-3-add-build-trigger' id="boomdevs_4" >4. 3 Add Build Trigger</h2> <p>For the upper three steps, please check this article: <a title="Permalink to Creating a web service health check with Jenkins pipeline and Mattermost notification" href="http://www.mobabel.net/creating-a-web-service-health-check-with-jenkins-pipelines-and-mattermost-notification/" rel="bookmark">Creating a web service health check with Jenkins pipeline and Mattermost notification</a></p> <h2 id='5-4-publish-over-ssh-config' id="boomdevs_5" >5. 4 Publish Over SSH Config</h2> <p> </p> <p> </p> <p> </p> <p> </p> <p><a href="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15.png"><img loading="lazy" decoding="async" class="alignnone size-full wp-image-6596" src="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15.png" alt="" width="1074" height="346" srcset="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15.png 1074w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15-300x97.png 300w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15-1024x330.png 1024w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-21-15-768x247.png 768w" sizes="(max-width: 1074px) 100vw, 1074px" /></a></p> <p> </p> <p><a href="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14.png"><img loading="lazy" decoding="async" class="alignnone size-full wp-image-6597" src="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14.png" alt="" width="1418" height="666" srcset="http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14.png 1418w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-300x141.png 300w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-1024x481.png 1024w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-768x361.png 768w, http://www.mobabel.net/wp-content/uploads/2020/07/25/Snipaste_2020-07-24_17-09-14-520x245.png 520w" sizes="(max-width: 1418px) 100vw, 1418px" /></a></p> <p> </p> <h2 id='6-full-script' id="boomdevs_6" >6. Full script:</h2> <p></p><pre class="crayon-plain-tag">pipeline { agent any environment { stageName = "PSwarm CC-BHP ppc (Duplicated Lots Checking)" ppcMysqlContainer = "a33bdcebf6e4" today = "" httpStatus = "" httpResponse = "" } stages { stage('Check double lots today') { steps { script { def now = new Date() today = now.format("yyyy-MM-dd") echo "mysql/date: ${ppcMysqlContainer} ${today}" def cmd = """ cd /local/home/procon-dev/ucp_bundle && eval "\$(<env.sh)" && cd /local/home/procon-dev/docker-stack //search the service name container_string=`docker ps | grep service-name-mysql` ; //get the first return 12 chars from output as the container id container_name=`echo "\$container_string" | cut -c1-12` ; //execute the SQL validate statement products_output=`docker exec -i \$container_name mysql -uroot -pPassword <<< "use ppc;select line_id from lot_assignment where product_status='IN_PRODUCTION' and date(plan_date) = '${today}' group by line_id having count(*) = 2;"` ; echo "\$products_output"; if [ -z "\$products_output" ] then echo "token""==No_duplicated_products==" else echo "token""==Found_duplicated_products==: \$products_output" fi """ sshPublisher( continueOnError: false, failOnError: true, publishers: [ sshPublisherDesc( configName: 'MyLocalClient', transfers: [ sshTransfer( execCommand: ''' cd /local/home/procon-dev/ucp_bundle && eval "$(<env.sh)" && cd /local/home/procon-dev/docker-stack ''' ), sshTransfer( remoteDirectory: 'docker-stack', removePrefix: '', sourceFiles: '*/**', execCommand: cmd ) ], verbose: true ) ] ) } } } stage('CheckLog') { steps { script { //https://plugins.jenkins.io/groovy-postbuild/ def list = manager.build.logFile.readLines() def JobCount = list.count {it.startsWith("token") && it.contains("token==Found_duplicated_products==")} echo("JobCount ${JobCount}") if (JobCount == 1){ error("Found_duplicated_products") } } } } } // Finishing Up Actions post { success { script { notifySucc(stageName) } } failure { script { notifyError(stageName) } } unstable { mattermostSend channel: "#p-cc-bhp-ppc-app", color: "warning", message: ":warning: $stageName is aborted. | ${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)" } fixed { mattermostSend channel: "#p-cc-bhp-ppc-app", color: "good", message: ":white_check_mark: $stageName is returning to health. | ${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)" } } } def notifySucc(String stageName = '') { def message = ":smiley: $stageName is working. | ${env.BUILD_NUMBER} (<${env.BUILD_URL}|Detail>)" echo(message) mattermostSend channel: "#p-cc-bhp-ppc-app", color: "good", message: message } def notifyError(String stageName = '') { def message message = ":rage: $stageName found duplicated products in production!!, please (<${env.BUILD_URL}| check detail>). | ${env.BUILD_NUMBER} " echo(message) mattermostSend channel: "#p-cc-bhp-ppc-app", color: "danger", message: message //https://www.jenkins.io/blog/2016/07/18/pipeline-notifications/ emailext ( subject: message, to: "email@test.com;", body: """<p>$stageName Failed: '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p> <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>""" //,recipientProviders: [[$class: 'DevelopersRecipientProvider']] ) }</pre><p> </p> <p> </p> <p> </p> <p>Reference:</p> <p><a href="https://www.jenkins.io/doc/pipeline/steps/publish-over-ssh/">https://www.jenkins.io/doc/pipeline/steps/publish-over-ssh/</a></p> <p class="grid--cell fs-headline1 fl1 ow-break-word mb8"><a class="question-hyperlink" href="https://stackoverflow.com/questions/32082205/groovy-postbuild-in-jenkins-parsing-the-log-for-strings-and-counting-them">Groovy Postbuild in Jenkins, parsing the log for strings and counting them</a></p> <p class="post-title"><a href="https://blog.kazge.com/devops/2018/04/25/Jenkins-pipeline-create-testing/" data-flip="title">Jenkins Pipeline Create Testing</a></p>The post <a href="http://www.mobabel.net/creating-a-remote-docker-mysql-database-record-validation-check-with-jenkins-pipeline-and-mattermost-notification/">Creating a remote docker MySQL database record validation check with Jenkins pipeline and Mattermost notification</a> first appeared on <a href="http://www.mobabel.net">Mobabel</a>.]]></content:encoded> <wfw:commentRss>http://www.mobabel.net/creating-a-remote-docker-mysql-database-record-validation-check-with-jenkins-pipeline-and-mattermost-notification/feed/</wfw:commentRss> <slash:comments>0</slash:comments> </item> </channel> </rss> <!-- Performance optimized by W3 Total Cache. Learn more: https://www.boldgrid.com/w3-total-cache/ Page Caching using Disk: Enhanced (Requested URI contains query) Served from: www.mobabel.net @ 2024-12-04 23:07:49 by W3 Total Cache -->