Kubernetes容器集群管理

云原生应用的理念

kubernetes的介绍

kubernetes是谷歌在2014年6月的一个开源集群项目,使用go语言开发,因为除了ks后只有8个字母又被称为k8s。

  • 官方网站:www.kubernetes.io

k8s的主要作用

  • 自动化部署

  • 扩展

  • 容器的管理

  • 提供资源的调度

  • 服务的管理发现

  • 扩容和监控

kubernetes的主要功能:

  • 数据卷:pod中容器之间共享数据,可以使用数据卷

  • 应用程序健康检查:容器内服务可能进程堵塞无法处理请求,可以设置监控检查策略保证应用健康

  • 复制应用程序实例:控制器维护着pod副本数

  • 弹性伸缩:根据设定的指标自动缩放pod副本数

  • 服务发现:使用环境变量或者DNS服务插件保证容器中程序发现pod入口访问地址

  • 负载均衡:一组pod副本分配一个私有的集群ip地址,负载均衡转发请求到后端容器,在集群内部其他pod可以通过这个cluster IP访问应用

  • 滚动更新:更新服务中不中断,一次更新一个pod,而不是删除整个服务(类试灰度发布)

  • 服务编排:通过文件的描述部署服务,使得应用程序部署变得高效

  • 资源监控:node节点组件集成Cadvisor资源收集工具,可以通过heapster汇总整个集群节点的资源数据,然后保存在influxdb时序数据库中,再由ganfana显示

  • 提供认证授权:支持角色访问控制(Rbac)认证授权策略

  • kubernetes提供一整套的解决方案不要提供其他的工具,使用自己集成的一套系统提供给用户使用

简单的Kubernetes架构

Kubernetes整体架构

系统架构及其组件功能

组件 名称 功能
Master kube-apiserver API,集群的统一入口,为HTTP API提供接口服务,所有对象资源的增删改查和监听操作都要交给API-Server处理后再提交给Etcd存储
Master kube-controller-manager 处理集群中常规后台任务,一个资源对应一个控制器,而Controller-Manager就是负责管理这些控制器的
Master kube-scheduler 根据调度算法为新床I安的Pod选择一个Node节点
Node Kubelet Kubelet是Master在Node节点上Agent,管理本机运行容器的生命周期,比如创建容器、Pod挂载数据卷、下载Secret、获取容器和节点状态等工作。Kubelet将每一个Pod转换成一组容器
Node kube-proxy 在Node节点上实现Pod网络代理,维护网络规则和四层负载均衡工作
Node docker/rocket/rkt 运行容器
第三方服务 etcd 分布式键值存储系统,用于保持集群状态,比如Pod、Service等对象

基本对象概念

对象名称 功能作用
Pod Pod是最小部署的单元,一个Pod有一个或者多个容器组成,Pod中容器共享存储和网络,在同一台Docker主机上运行
Service Service一个应用服务的抽象,它定义了Pod逻辑集合和访问这个Pod集合的策略。Service代理Pod集合对外表现是为一个访问的入口,分配一个集群的IP地址,来自这个IP地址的请求将会负载均衡转发后端Pod中的容器。Service通过Lable Selector选择一组Pod提供服务
Volume 数据卷,共享Pod中容器使用的数据
Namespace 命名空间将对象逻辑上分配到不同的Namespece,可以是不同的项目、用户等区分管理,并且设定控制的策略,从而实现多组用户。命名空间也称为虚拟集群
Lable 标签用于区分对象(Pod、Service),键、值对存在,每个对象可以有多个标签,通过标签关联对象

基于基本对象更高层次抽象封装

基本名称 功能作用
ReplicaSet 下一代Replication Controller 确保任何给定时间指定的Pod副本数量,并且提供声明形式等更新功能。RC与RS唯一的区别就是lable selector支持不同,RS支持新的基本集合的标签,RC仅支持基本等式的标签
Deployment Deployment是一个更高层次的API对象,它管理ReplicaSets和Pod,并且提供声明形式等更新功能。官方建议使用Deployment管理ReplicaSets,而不是直接使用ReplicaSets,这就意味着可能永远不需要直接操作ReplicaSet对象
statefulSet statefulSet适合持久性的应用程序,有唯一的网络标识符(ip),持久存储,有序的部署、扩展、删除和滚动更新
DaemonSet DaemonSet确保所有(或一些)节点运行同一个Pod。当节点加入了K8S集群中,Pod会被调度到该节点上运行,当节点从集群中移除时,DaemonSet的Pod会被删除,删除DaemonSet会清理它所有创建的Pod
Job 一次任务,运行完成后Pod销毁,不再重新启动新容器,还可以任务定时运行

Pod

Pod是在Kubernetes集群中运行部署应用或者服务的最小单元,它是可以支持多容器。Pod的设计理念是支持多个容器在一个Pod中共享网络地址和文件系统,可以通过进程间通信和文件共享这种简单高效的方式组合完成服务。
Pod对多容器支持是K8S最基础的设计理念,比如当运行一个操作系统发行版的软件仓库的时候,一个Nginx’容器用来发布软件,另外一个容器专门用来从源仓库做同步,这两个容器的镜像不太可能是一个团队开发的,但是他们一块工作才能提供一个微服务,这样的情况下,不同团队各自开发构建自己的容器镜像,在部署的时候组合成一个整体微服务对外提供服务。

  • Pod是Kubernetes集群中所有业务类型的基础,可以看作运行在K8s集群中一个个小机器人。不同类型的业务就需要不同类型的小机器人去执行任务

  • 目前Kubernetes中业务主要分为长期服务型(Long-running),批处理型(batch),节点后台支持型(node-daemon)和有状态应用型(stateful application)

  • 分别对应控制器为:Deployment、Job、DaemonSet、PetSet

POD创建的整个过程

Pod的生命周期

Pod的生命

一般来说,Pod不会消失,直到人为的销毁它们,这个操作过程是人为或控制器,这个规则除了成功和失败的phase还有超时(由maste决定)的Pod将过期被销毁

常用的三种控制器

  • 使用Job运行预期会终止的Pod,批量计算适用。Job仅适用的策略为:OnFailureNever的Pod

  • 期望不会被终止的Pod使用ReplicationControllerReplicaSetDeployment,ReplicationController仅适用于具有restartPolicy为Always的Pod

  • 提供特定于机器的系统服务,使用DaemonSet为每一台机器运行一个Pod

  • 所有的这三种类型的控制器都包含一个PodTemplate,正确的姿势是用控制器来创建Pod不要手动创建,因为手动创建的Pod在机器故障情况下没办法自动复原,控制器可以

  • 当节点死亡或者集群的其他部分断开连接,则K8s将应用一个策略丢失节点上所有的Pod的phase设置为Faild

Pod详解

Pod是由多个容器组成,Pod中的容器共享IP地址和端口号,相互之间可以通过localhost来发现对方,可以通过进程间来通信,Pod中的容器也有访问共享Volume的权限,这些Volume会被定义为Pod的一部分并且挂载到应用容器的文件系统中。

  • < 1 > kubectl提交创建请求,可以通过API Server的Restful API,也可以使用kubctl命令工具,支持的数据类型包括JSON和YAML

  • < 2 > kube-apiserver处理用户请求,存储Pod数据到etcd

  • < 3 > kube-scheduler通过API Server查看未绑定的Pod,尝试为Pod分配主机

    • < 1 > 过滤主机(调度预选):调度器是一组规则过滤掉不符合要求的主机,比如Pod指定了所需要的资源量,那么可以用资源比Pod需要的资源量少的主机会被过滤掉

    • < 2 > 主机打分(调度优选):对第一步选出的符合要求的主机进行打分,在主机打分阶段,调度器会考虑一些整体优化策略,比如把一个容器一个Replication Controller的副本分布到不同的主机上,使用最低负载的主机

  • < 4 > kube-scheduler选择主机:选择打分最高的主机,进行binding操作,这个操作的本质是通过kube-apiserver修改Pod字段,结果存储到etcd中

  • < 5 > kubelet根据调度的结果执行Pod创建操作:绑定成功后,pod.spec.nodeName 拥有值了,运行在每一个工作节点上kubelet也会定期与etcd同步pod信息(属于自己的Node)

  • < 6 > docker接受到kubelet下发的命令,启动相应的容器,至此,一个Pod启动完成

常见的配置写法参数

Pod
apiVersiom,kind,metadata,spec,status(只是读)

spec:
    containers
    nodeSelector
    nodeName
    restartPolicy:
        always,Nerver,OnFailure

    container:
        name
        tnage
        inagePullPolicy:Always、Nerver、IfNotPresent
        ports:
            name
            containerPort
        livenessProbe
        readinessProbe
        lifecycle

    ExecAction: exec
    TCPSocketAction: tcpSocket
    HTTPGetAction: httpGet

Pod设计的动机

  • (1)容器介于操作系统应用之间,容器的推荐使用方式是每一个容器运行一个进程。
    • 外部掌控多个容器的组合和生命周期redhat和docker公司的控制权限的争斗

    • 单容器多进程的使用案例有阿里云的Pouch项目

  • (2)对外Pod作为一个独立的部署单位,支持横向扩展和复制,共生(协同调度),命运共同体(终结),协助复制,资源共享,依赖管理。

  • (3)对内Pod内容互相协助

    • pod中的应用必须协调端口的占用,每一个Pod都有一个唯一的IP地址,和物理机中的其他的Pod都是处在一个扁平的网络空间中,它们之间可以直接的联通。

    • Pod中应用容器的hostname被设置成Pod的名字

    • Pod中的应用容器可以共享Volume,volume能够保证Pod重启的时候用户的数据不会被丢失。

通常的情况下不会在同一个Pod中运行多个应用的实例

为什么不直接在一个容器中运行多个应用程序?

  • 1.透明 -> 让Pod中的容器对基础设施可见,以便基础设施能够为这些容器提供服务,例如进程管理和资源的监控。使用户得到便利

  • 2.解耦软件的依赖 -> 每一个容器都可以进行版本管理,独立的编译和发布,未来kubernetes会支持单个容器在线升级

  • 3.使用方便 -> 用户不必运行自己的进程管理器,不用担心错误信号传播

  • 4.效率

Pod的非持久性

容器的状态

状态 描述
Pending(挂起) Pod已经被K8s系统接受,但是有一个或者多个容器的镜像尚未创建,等待时间包括调度Pod的时间和通过网络下载镜像的时间
Running(运行) Pod已经绑定到一个节点上,Pod中所有的容器都已经被创建,至少有一个容器正在运行中,或者正处于启动或重启状态
Successed(成功) Pod中所有容器都已经被成功的干掉,并且不会重启
Faild(失败) Pod中所有的容器都已经终止了,并且至少有一个容器是因为失败终止,容器以非零状态退出或者被系统强制终止
Unkonwn(未知) 因为某一些原因无法取得Pod的状态,通常是因为与Pod所在的主机通讯失败

导致Pod死亡的情况

  • 调度失败

  • 节点故障

  • 缺少资源

  • 节点维护

  • 用户主动杀死Pod

init容器

Pod能够具有多个容器,应用运行在容器里面,但是它也可能有一个或者多个优先应用容器启动的init容器。

  • init容器是一种专用的容器,在应用程序启动之前运行

  • init容器总是运行到成功完成为止

  • 每一个init容器都必须在下一个init容器启动之前成功完成

如果Pod的init容器失败,Kubernetes会不断的重启这个Pod,直到init容器成功为止,然而如果Pod对应的restartPolicyNever,则不会启动。若想指定的容器为init,在PodSpec中添加initContainer字段

init的不同点

init容器支持和应用容器的全部字段和特性,包括资源限制、数据卷和安全设置,但是init容器对资源的请求和权限的处理稍微有些不同,并且init容器不支持Readines Probe,因为init容器必须在Pod就绪之前就运行结束

init使用场景

init容器具有与应用程序容器分离的单独镜像,所以它们的启动相关的代码具有优势

  • 1.可以包含并且运行实用的工具,但是出于安全的考虑,是不建议在应用程序镜像中包含这些实用工具

  • 2.可以包含使用工具和定制化代码来安装,但是不能出现在应用程序镜像中

  • 3.init使用Linux Namespace,所以相对的应用程序容器具有不同的文件系统视图,所以,具有能够访问Secret的权限,而应用程序不可以

  • 4.必须在应用程序容器启动之前运行完成而应用程序容器是并行运行的,所以init容器能够提供一种简单的阻塞或延迟应用容器的启动方式,直到满意条件

  • 5.总结:

    • 等待一个Service创建完成运行类似:for i in {1..100};do sleep1;if dig myservice;then exit 0;fi;exit 1
    • 在启动应用容器之前等待一段时间,类似sleep60这种命令

容器的探针

探针是由于kubelet对容器执行的定期诊断,要执行相应的诊断,kubelet调用由容器实现的Handler

探针的探查方式

  • ExecAction:在容器内执行指定的命令,如果命令退出时候返回0则诊断成功

  • TcpSockerAction:对指定的端口上的容器的IP地址进行TCP检查,如果端口打开,则诊断是成功

  • HttpGetAction:对指定的端口和路径上的容器的IP地址执行HttpGet请求,如果响应的状态返回值>=200<=400,则诊断成功

  • 探针返回结果:成功、失败、未知

探针引发的外部动作

  • livenessProbe:指示容器是否正在运行,如果存活则探针失败kubectl会杀死容器,并且容器将会受到重启策略的影响,如果容器不提供存活探针,则默认状态为Success

主动触发健康检查,删除健康检查文件,执行container重启策略。

ExecAction

apiVersion: v1
kind: Pod
metadata:
  name: liveness-exec-pod
  namespace: default
spec:
  containers:
  - name: liveness-exec-container
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ["/bin/sh","-c","touch /tmp/health;sleep 20;rm -rf /tmp/health;sleep 3600"]
    livenessProbe:
      exec:
        command: ["test","-e","/tmp/health"]
      initialDelaySeconds: 1
      periodSeconds: 3

HttpGetAction

apiVersion: v1
kind: Pod
metadata:
  name: liveness-httpget-pod
  namespace: default
spec:
  containers:
  - name: liveness-httpget-container
    image: ikubernetes/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        port: http
        path: /index.html
      initialDelaySeconds: 1
      periodSeconds: 3
kubectl describe pod liveness-httpget-pod

kubectl exec -it liveness-httpget-pod -- /bin/sh -c "rm -rf /usr/share/nginx/html/index.html"   #删除index.html触发

  • readinessProbe:指示容器是否准备好服务请求,如果就绪探针失败,端点控制器将从与Pod匹配的所有Service的端点中删除该Pod的IP地址,初始化延迟之前的就绪状态默认为Failure,如果容器不提供就绪探针,则默认状态为Success
apiVersion: v1
kind: Pod
metadata:
    name: readiness-httpget-pod
    namespace: default
spec:
    containers:
    - name: readiness-httpget-container
      image: ikubernetes/myapp:v1
      imagePullPolicy: IfNotPresent
      ports:
      - name: http
        containerPort: 80
      readinessProbe:
        httpGet:
            port: http
            path: /index.html
        initialDelaySeconds: 1
        periodSeconds: 3

探针存在的意义

当我们的Pod在正常的服务中运行时候,pod新建的Pod会被关联到service中但是无法知道Pod是否已经正常的启动了
此时此刻去关联到我们的服务,执行一下步骤必定报错,因为一个Pod在启动时候会先启动Init容器后启动Container
此时就需要探针的判断,当container初始化完成的时间自动关联到service,未就绪或者失败则不关联service。

存活和就绪探针的使用场景

  • 1.容器中的进程能够在遇见问题或不健康的情况下自行崩溃,则不一定需要存活探针,kubelet将会根据Pod中的restartPolicy自动执行正确的操作

  • 2.如果期望容器在探测失败的时候杀死并且重新启动,那么请指定一个存活的探针,并且指定restartPoilcy为Always或OnFailure

  • 3.如果需求是仅仅在探测成功后才开始向Pod发送流量,需指定就绪探针,就绪探针可能和存活的探针相同,但是spec中的就绪探针的存在意味着Pod将会在没有接受到任何流量的情况下启动,并且探测成功才开始接受流量

  • 4.如果需要容器自行维护,指定一个就绪探针,可以检查与存活探针不同的端点

  • 5.如果想Pod被删除时候能够排除请求,则不一定需要使用就绪探针,在删除Pod时,Pod会自动将自身放置为未完成状态,无论就绪探针是否存在,当等待Pod中容器Stop,Pod任然处于未完成状态

Pod的重启与状态

容器重启的策略

  • 五分钟为上线的指数退避延迟(10、20、40)秒重启,并且在成功执行十分钟后重置
Pod.spec.restart Policy字段 行为:适用于Pod中所有的失败的容器
ALways(默认) exitCode=任何数字,执行重启操作
OnFailure exitCode!=0,执行重启操作
Never exitCode=任何数字,不重启

Pod状态=容器状态之和?

Pod的终止

  • 发送终止信息 -> 宽限期内等待 -> 超时强制停止 -> 结束

使用场景

  • 例如:一个Pod做数学运算,计算结束后退出状态变成Succeeded,这时候如果使用Always重启容器就没有任何意义

  • 如果关心容器退出后的上下文环境,如日志文件和目录就需要设置问Nerver,因为一旦容器重启创建,内容就会丢失被回收了

Pod的设计原理

  • Pod -> 只是一个逻辑上的概念

  • 容器是一种比虚拟机更加轻量级的隔离方式,主要通过namespace和cgroup技术进行资源的隔离,namespace用于负责看起来隔离,cgroup用于负责用起来隔离。

  • Kubernetes真正处理的,是宿主机系统上的Linux容器的Namespace和Cgroups,Pod的边界和隔离环境并不是真实存在的。

所以容器的一个Pod中的容器A和容器B之间的其实是贡献一了一组网络和volume但是对于有状态的容器就有一个启动先后的问题,这个时候就不是对等的关系,变成了拓扑关系了。为了解决这一系列的有状态的容器问题提出了infra容器

  • k8s.gcr.io/pause -> 汇编编写永远暂停状态的Container大小100-200K

  • Pod创建第一个容器就是infra容器,它得到了Network Namespace,用户容器就可以加入到Infra容器中去,去/proc下也可以观察到容器AB均指向一个地址

  • 当在一个Pod中构建两个容器共用一个volume,两容器间的数据是共用可以相互修改的

特征

  • Pod这种超亲密的关系,是容器设计和核心思想,当用户想在一个容器中运行多个功能的且不相关的应用,应该优先思考为Pod中的多个容器

  • 需求 -> javatomcat -> web

    • 评估 -> 1.构建一个镜像文件,采用分布式文件系统挂载到容器中使用 -> 维护一套分布式存储难度大
    • 评估 -> 2.每一次更新都是构建一次新的镜像,run镜像 ->维护很麻烦重复工作多,可以使用jenkins构建
    • 评估 -> 3.war包和tomcat分别做成镜像,然后在一个Pod中组合在一起

举例子

apiVersion: v1
kind: Pod
metadata:
    name: javaweb
spec:
    initContainers:
    - image: geektime/sample:v2
      name: war
      command: ["cp", "/sample.war", "/app"]
      volumeMounts:
    - mountPath: /app
      name: app-volume
    containers:
    - image: geektime/tomcat:7.0
      name: tomcat
      command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
    volumeMounts:
    - mountPath: /root/apache-tomcat-7.0.42-v2/webapps
      name: app-volume
    ports:
    - containerPort: 8080
      hostPort: 8001
    volumes:
    - name: app-volume
    emptyDir: {}
  • Pod中initcontainer容器启动顺序比containers快,只有initcontainer启动后才会启动其他的容器

  • init容器启动后将要使用的war包拷贝到/app共享给即将运行的tomcat运行

  • 充分的利用容器的组合实现了解耦关系,这种模式被定义为sidecar -> 在一个容器中启动一个辅助容器,完成一些独立主进程之外的工作

  • 例如:日志的收集 -> init容器去读取container容器产生的日志,然后传输或者转发给其他的Pod处理日志

Pod的控制器

Pod的控制器使用的是内嵌Pod模板,用于实现代替人工去管理中间层并且帮助人管理Pod始终处于所定义所期望的状态

  • cd kubernetes/pkg/controller/ && ls -d * #当下就是k8s的全部控制器

  • 全部遵循K8s中的通用编排模式,控制编排(control loop)

控制器模型的实现

以一个Nginx的Pod来举例子

apiVersion: apps/v1
kind: Deployment
metadata:
    name: nginx-deployment
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 2
    template:
        metadata:
            labels:
                app: nginx
        spec:
            containers:
            - name: nginx
            image: nginx:1.7.9
            ports:
            - containerPort: 80
  • Deployment控制器从ETCD中获取所有携带了nginx标签的Pod统计 -> 实际状态

  • Deployment对象的Replicas字段的值 -> 期望的状态

  • 控制器将两个的状态进行比较,然后根据结果确定是我们需要创建还是销毁的的Pod

  • 总结kubernetes的编排逻辑实际上是在对比阶段完成的,整个过程叫做同步循环/调谐循环(Sync Loop/Reconcile Loop)

Deployment

Deployment是什么?

Deployment -> ReplicaSet -> Pod 支持滚动更新回滚,声明式配置的逻辑

  • 管理无状态的应用的控制器

DaemonSet

DaemonSet确保全部或部分的Node上运行一个Pod副本,当有Node加入集群的时候,也会为它们都增加一个Pod。当节点被移除的时候Pod会被回收,删除DaemonSet将会删除它创建的所有的Pod

  • 这个Pod运行在Kubernetes集群里面的每一个节点上

  • 每一个节点只有一个这样的实例

  • 当有新的节点加入的kubernetes集群时候这个Pod也会同样被创建出来,节点删除后也会被销毁

ReplicaSet、RS

Replica Set

核心作为:等待用户创建指定数量的副本数,并且确保一直满足用户期望的Pod副本的数量的状态,支持滚动更新,支持扩缩容机制。被称为新一代的ReplicationController。

  • Kubernetes集群中保证Pod高可用的API对象

通关监控运行中的Pod来保证集群中运行指定数目的Pod副本。
指定的数目可以是多个也可以是1个,少于指定数目,RS就会启动运行新的Pod副本,及时在指定数目为1的情况下,通过RS运行Pod也比直接运行Pod更加明智,因为RS也可以发挥它高可用的能力,保证永远有至少一个Pod在运行。RS适用于长期伺服型的业务类型,比如提供高可用的Web服务。

ReplicaSet的升级

statefulSet

kubernetes从1.3开始支持PetSet功能 -> 1.5后支持StatefulSet
StatefulSet三个组件:
headless service(无头服务)
StatefulSet(控制器)
volumeClaimTemplate(存储卷申请模板)

  • 1.稳定且唯一的网络标识符

  • 2.稳定且持久的存储方式

  • 3.有序优雅平滑的扩展方式

  • 4.有序平滑的终止和删除。串行启动串行关闭

  • 5.有序的滚动更新,新更新从后更新主 -> 高版本兼容低版本特性

  • statefulSet名称 -> pod(name).service(name).namespace(name).svc.cluster.local

第三方资源

  • TPR:Third Party Resource (1.2+)-> 1.7 停止

  • CDR:Custom Defined Resource,1.8支持 -> 至今

HPA

AutoScaler作为k8s上的能够进行弹性伸缩的统称,其中最为突出的就是HPA(Horizontal Pod Autoscaler)
CA(Cluster Autoscaler)当集群的node节点资源利用率接近上线则可以通过调用云计算平台接口,快速进行node节点级别的扩容,当利用率下降则可以自动移除。
VPA(Vertical Pod AutoScaler)垂直伸缩增加pod的利用率,弹性的增加和减少POD的CPU和内存大小目前还是不成熟
AR(Addon Resizer)可以看成一个简化版的POD垂直伸缩工具,在正常的pod放入K8s中时候会限制资源的大小,过大或者过小都会出现资源不匹配的情况,AR将他以Sidecar的方式注入容器中一旦POD的资源不足或者过低,会根据一段时间的Pod实际运行状态来调整资源限制的上下限制。

  • HPA-V1 :v1版的HPA只能支持核心指标CPU/内存利用率等

  • HPA-V2 :v2版本支持用户自定义的指标和系统核心指标进行弹性伸缩

集群的资源管理

容器实现资源管理的实质

linux Cgroups(Linux Control Group):限制一个进程组能够使用的资源上线,包括CPU内存磁盘网络带宽。在容器中每一个进程都是创建在宿主机的进程上,为了使得容器里面的PID=1,使用的namespace障眼法让容器自己感觉到自己是最大,但是在宿主机的进程组中又是与其他的的进程并行存在的,这就导致容器进程可以占用所有的宿主机的资源,所以Cgroup的出现就是限制容器进程能够调用的宿主机资源。

  • Cgroups能够对进程进行优先级设置、审计、已经进程的挂起恢复等

cd /sys/fs/cgroup/cpu/  #系统的CPU核心处
mkdir container
ls

cat /sys/fs/cgroup/cpu/container/cpu.cfs_quota_us
-1      #默认不限制CPU_period默认是100ms(100000us)
echo 20000>/sys/fs/cgroup/cpu/container/cpu.cfs_quota_us
20000   #设置后说明每100ms最多使用20ms,也就是一个CPU的20%的资源了
while :;do :;done & #进程占用100%CPU
echo PID进程号 >/sys/fs/cgroup/cpu/container/tasks
top #查看占用的资源20%受到了Cgroup的限制
#Cgroups这就是对容器的资源限制方式

当然除了CPU子系统之外,Cgroup的每一项子系统都有其独有的资源限制能力

  • blkio,为块设备设定指定的I/O限制,用于磁盘设备

  • cpuset,为进程分配单独的CPU核和对应的内存节点

  • memory,为设置进程的内存使用限制

Cgroup的实质

它就是一个子系统目录上加上了一组资源限制文件的组合体,对于Linux中的Docker容器,只是在每一个子系统中为每一个容器创建一个新的受控组,然后在启动容器之前配置相应的限制资源的信息将PID指定到对应的tasks文件中就可以

结论

  • 容器只是一种特殊的进程

  • 由于容器是一个进程,意味着一个容器没有办法去同时运用两个不同的应用,常规解决方式为system或supervisor代替本身作为容器的启动

  • docker就是启动了namespace和通过Cgroup对容器的资源进行的管控

  • 当然Cgroup存在一定的缺陷 -> Cgroup对资源的限制能力有不完善的地方,特别是对/proc文件系统 -> 在容器中运行的top命令实际是查看宿主机的资源情况而不是容器本身所使用的资源情况

    • 原因:/porc文件系统不知道用户通过了Cgroup对容器做了什么限制,两则之间没有进行过沟通活动
  • namespace作为隔离手段,cgroup作文容器的限制,rootfs作为容器的文件系统

创建资源的方法

  • APIserver只接受Json格式的资源定义

  • yaml格式提供配置清单,apiserver可以自动将其转换为JSON格式,然后提交

常用的资源配置清单

  • 自主式Pod资源

  • 资源的清单格式

    • 一级字段:apiVersion、kind、metadata(name,namespace,labels,annotations…)、spec、status
  • 获取清单: kubectl api-versions

  • apiVersion: group/version

  • 获取参数项目的方式:kubectl explain pod.<object>.<object>

pod.spec.containers<obect> 定义解释 参数或Key
name<sting> 自定义名称 none
image<string> 镜像名称 none
imagePullPolicy<string> 获取镜像的方式 Always(总是去下载,dockerhub下载镜像),Nerver(不下载,使用本地),IfNotPresent(如果本地不存在则去下载)
port<objcet> 端口监听但是不是端口的暴露 none
protocol<string> 连接的方式默认为TCP none

总体的框架结构

apiVsersion(group/version)
kind:资源类别
...
...
metadata:元数据
    name
    namespace
    labels
    annotations
    ...
    每一个资源引用PATH
    /api/group(群组)/version(版本)/namespace/NAMESPACE/TYPE/NAME

spec:期望的状态(disired state)

status:当前状态(由K8s集群维护,不能自定义)

实战构建自定义Pod

  • 一个Pod运行多个Container
apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
  namespace: default
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
  - name: busybox
    image: busybox:latest
    command:
    - "/bin/sh"
    - "-c"
    - "sleep 5"

labels

标签是在kube集群中辨识Pod关系的方法,也是K8s中最为有特色的方案

  • 标签匹配度资源检查来进行识别调度

  • 1.一个资源对象可以使用多个标签

  • 2.同一个标签可以放多个对象之上

标签的命名

为什么需要标签

  • 拥有标签后node在添加资源时候让资源对节点有倾向性

  • 定义Pod时候有内建字段modeSelector<map(string)string>

  • 更加便于集群内的操作和管理

声明式API和编程范式

独有特点

  • 声明式:提交一个定义好的API对象 -> 声明 -> 期望的状态

  • 声明式API允许存在多个client来写,以PATCH的方式对API对象进行热修改,不需要关系原始的YAML内容

  • 基于以上两种特性kubernetes的项目能够在外界不干预的情况下调节实际状态期望状态

  • 声明式API -> kubernetes项目编排的核心

  • kubectl apply -> 声明式API

当你在本地编写一个yaml的deployment文件需要让它在kubernetes中跑起来会有两种创建的方式:kubectl create/apply

  • create/apply有什么区别?

当创建一个新的文件时候两者的使用效果一致,但是在修改一个yaml后重新执行,实际上create是创建一个新的对象,而apply是对原来对象中的API进行更新

  • replace和apply区别

replace的执行过程是使用新的YAML文件中的API对象,替换原来的API对象,而apply则执行一个原来的API对象的Pathch操作

Service

  • ipvs是最新的K8s推荐的调度方式,与iptables类似,ipvs基于netfilter 的 hook 功能,但使用哈希表作为底层数据结构并在内核空间中工作。这意味着ipvs可以更快地重定向流量,并且在同步代理规则时具有更好的性能。此外,ipvs为负载均衡算法提供了更多选项

  • 当使用ipvs规则的时候kube-proxy会自动去检测ipvs是否开启,如果没有则会采用自动降级的策略为iptables

  • Server是4层调度器

4层调度有缺陷:
只是工作在OS模型的第四层,如果用户访问的是HTTPS请求的时候,如果跑的Pod中的服务是HTTP想升级为-> HTTPS,这个Https则需要配置Service,但是Service是4层无法调度,则需要去7层的后端服务器上配置Https服务

内部的Pod容器要实现Https方案

service模式

  • userspace、iptables、ipvs

service工作类型

  • ClusterIP:只允许在集群的内部进行访问,最多到达集群的边界

  • NodePort: NodeIP:client -> NodeIP:NodePort-> ClusterIP:Service Port-> PodIP:ContainerPort

  • Loadbalance

  • ExtermdName

  • No ClusterIP:Headless Service

    • ServiceName -> PodIP

服务的发布类型

对于少部分的某些应用希望通过外部的IP地址暴露service服务,k8s允许指定一个service的类型来实现默认clusterIP

  • ClusterIP:通过集群的内部IP来暴露服务,选择该值,服务只能在集群内部可以访问到,这也是默认的ServiceType

  • NodePort:通过每一个Node上的IP和静态的端口暴露服务NodePort服务会路由到ClusterIP服务,若不存在自动创建,通过请求NodeIP:NodePort实现访问,可以在集群外访问集群内服务

  • LoadBalancer:使用负载均衡,可以向外部暴露服务,外部的负载均衡器可以路由到NodePort服务和ClusterIP服务

  • Externalname:通过返回CNAMEvalue,可以将服务映射到它的字段没有任何的创建被依赖,k8s1.7以上的kube-dns支持

使用清单创建service资源

apiVersion: v1
kind: Service
metadata:
    name: redis
    namespace: default
spec:
    selector:
        app: redis
        role: logstor
    clusterIP: 10.99.99.99
    type: ClusterIP
    ports:
    - port: 6379
      targetPort: 6397
  • 当每一个服务创建重启完成的时候都会动态在kube-dns中动态的添加一个地址记录

使用清单创建的service测试

K8s网络调度方式

K8s网络模型设计理念

  • 所有容器不使用Nat就可以互相通信(这和docker默认的实现方式的不同的)

  • 所有节点跟容器之间不适用Nat就可以互相通信

  • 容器自己看到的地址,和其他人访问自己使用的地址应该是相同的

相同的Pod与不同的Pod

  • docker的网络和设计模型

Ingress

Kubernetes中,Services和Pods的IP地址仅能用于集群网络内的通信,所有的网络流量都无法穿透边界路由器(Edge Router)进行集群内外通信。Ingress就是一组规则,它能够把集群外部的访问请求嫁接至集群内部的Services,从而完成流量穿透。它其实是一种将请求基于host或path路由至后端服务的方式,能够将一组后端服务集中于一个访问端点(entrypoint)暴露于集群外。

  • Ingress Controller是一个守护进程,部署为Kubernetes Pod,它监视apiserver的/ingresses端点以更新Ingress资源。它的工作是满足对Ingress的要求

  • https://kubernetes.github.io/ingress-nginx/development/#quick-start

应用场景

  • 当k8s中pod需要进行HTTPS服务有不想麻烦的进行调度,则ingress是你最好的选择

k8s的网络方案

kubernetes网络通信:

  • (1)容器间通信:同一个Pod内的多个容器间的通信,lo本地回环

  • (2)Pod通信:必须从一个POD_A_IP <-> POD_B_IP 直达,POD与POD直接通信不能经过任何的地址转发

  • (3)Pod与Service通信:POD_IP <-> Cluster_IP (IPVS无法取代iptables,NAT转发就无法实现)

  • (4)Service与集群外部客户端的通信

K8s的网络实现使用网络的插件来实现 –> CNI ,k8s本身不提供网络解决方案,全全交给第三方来实现来解决以上的四种问题

cni如何实现网络调度的

kubelet启动时通过指定–network-plugin=cni来使用CNI插件;通过–cni-conf-dir指定CNI对应的配置文件目录(默认为/etc/cni/net.d),并使用该文件中的CNI配置去设置每个pod网络;通过–cni-bin-dir指定插件对应的可执行文件目录(默认为/opt/cni/bin)

  • kubelet 先创建pause容器生成network namespace

  • 调用网络CNI driver

  • CNI driver 根据配置调用具体的cni 插件

  • cni 插件给pause 容器配置网络

  • pod 中其他的容器都使用 pause 容器的网络

常见的CNI

  • flannel

  • calico

  • canel

  • kube-router

解决方案

  • 虚拟网桥:以纯软件的方式实现虚拟的网络网卡,接入网桥去。保证每一个容器都有自己独立的网卡使得主机组件有网络接口网络(叠加的方式)。

  • 多路复用(MacVLan):为每一个虚拟接口配置独立的MAC地址使一个物理网卡能够承受多个容器使用,使用物理网卡并且用物理网卡的MacVlan机制进行跨节点之间通信

  • 硬件交换:网卡支持硬件的交换,使用SR-IOV的方式一个网卡支持单根I/O虚拟化,创建虚拟设备的一种高性能方式,在硬件级别虚拟多块网卡让每个容器使用一个网卡。

K8s使用网络插件的位置

  • kubelet -> /etc/cni/net.d -> 直接把配置文件丢进去就可以使用,由网络插件实现网络配置使用

CNI的实现网络调用

常见的K8s网络方案

Flannel

Flannel是CoreOS团队中针对Kubernetes设计的一个网络规划服务,简单说它功能可以让集群中的不同节点主机创建的Docker都具有,整个集群中唯一的虚拟IP地址。
在默认的docker配置中,每一个节点上的docker服务会分别负责所在的节点容器的IP分配。这样导致的问题是不同的点上容器可能获得相同的内外IP地址,并且使得这些容器之间能够通过IP地址相互找到,也就是ping通。
flannel的设计目的就是为集中的所有及诶单重新规划IP地址的使用规则,从而使得不同的节点上的容器能够获得同属于一个内网且不重复的IP地址,并且使得属于不同节点上的容器能够直接通过内网的IP通信。
flannel实质上是一种->覆盖网络(overlaynetwork),也就是将TCP数据包装在另一个网络包里面进行路由转发和通信,目前已经支持Udp、vxlan、host-gw、aws-vpc、gcc和alloc路由等数据的转发方式,默认的节点间数据通信的方式的Udp转发。

Flannel后端实现

  • Vxlan -> 主流的跨主机方案

Vxlan(Virtual Extensible Lan)虚拟可扩展局域网,是Linux内核本身支持的一种网络虚拟化技术,在内核态实现封装和解封装,构建出覆盖网络。
Vxlan的设计思想是:在现有的三层网路欧尚,覆盖一层虚拟的、由内核Vxlan模块负责维护的二层网络,让连接在这个Vxlan二层网络上的主机(分布在不同的机房)之间形成一个局域网可以自由的通信

  • 基于vxlan技术的网络通信性能上有缺陷性能比较低

  • 但是可以独立管理一个网络,与物理网络之间彼此是不相互干扰

  • host-gw

Host Gateway(主机网关):例如同时有两个node,每个节点上都需要运行Pod且都有一个专属网段,将主机上的所在的网络接口作为网关来使用从而让Pod之间配置IP地址通过网关之间来相互通信

  • 该方式没有产生多余的系统性能开销,除了本地路由之外最大性能的使用网络性能力

  • host-gw的性能相较calico来说略高一筹,但是缺陷是要求各个节点必须工作在三层的网络中不能夸网段,即同一个网段中

一种两全其美的解决方案

原POD所在的节点与目标Pod所在的节点在同一个三层网络中之间使用host-gw,但是如果两个节点不在同一个三层网络中那就自动的将网络降级使用Vxlan模式去使用叠加的网络通信,叠加网络支持不在同一个网段中通信

  • VxLan的多种模式:原生vxlan、Driectouting直接路由

  • UDP -> 最早支持,性能最差 -> Flannel进行UDP封装和解封都在用户状态完成用户态和内核态的切换消耗过多的资源

  • 数据从源容器中出发后,经过所在的主机的docker-0虚拟网卡转发到这是一个P2P的虚拟网卡,flanneld服务监听在网卡的另外一端(flannel通过Etcd服务维护了一张节点间的路由表)

  • 源主机的flanneld服务将原来的数据内容UDP封装后根据自己的路由表投递给目的的节点的flanneld服务,数据到达以后被解包,然后直接进入目的节点的flanneld-0的虚拟网卡,然后被转发到目的主机的docker-0虚拟网卡

  • 最后就像本机容器通信一下的由docker-0路由到达目标容器,这样整个数据包的传递就结束了

flannel的配置参数

  • 简单的定义并且应用到集群中注意生产环境不要这么干会导致所有pod无法通行在一开始构建就需要设置好

  • 下载flannel官方的配置文件修改net-conf.json,而后重新创建flannel文件并且查看路由表变化ip route show

[root@k8s-master kube]# kubectl get configmap -n kube-system
[root@k8s-master kube]# kubectl get configmap flannel-configmap -o yaml -n kube-system
metadata:
    annotations:
        ..... 看字段
````



```json
net-conf.json
    {
        "Network": "10.244.0.0/16",
        "Bacend": {
            "Type": "vxlan",
            "Directrouting": true
        }
    }
  • Network:Flannel使用的CIDR格式的网络地址用于为POD配置网络功能(为全局8/16子网掩码,给每一个节点划分子网)
例子:flannel划分网段
10.244.0.0/16
    master:10.244.1.0/24
    node01:10.244.2.0/24
    ...
    node255:10.244.255.0/24
    最大划分255个网段意味着最多255个节点
10.0.0.0/8
    10.0.0.0/24
    ...
    10.255.255.0/24
    大约有6万多个node可以使用
  • SubnetLen:把Network切分为子网提供给各个节点使用的时候,使用多长的掩码进行切分默认为24位

  • SubnetMin:用于分配给节点使用的子网起始的第一个子网从多少开始分配,可以预留部分网络不使用

  • Backend: 使用flannel的什么工作模式(VxLAN/VxLAN-Driectouting、host-gw、udp)

flannel怎么知道节点和容器网段的对应关系?

通过所有的flannel进程都会去访问etcd的K/V存储,查看表的对应关系,自动的录入flannel的配置文件中

节点上容器的网段怎么保证不会冲突?

flannel介入容器的网络IP地址分配的整个流程,保证数据的一致性和IP不会冲突
flannel会主动去etcd的路由表中创建一个没有被使用的网段,将这个网段加入到docker-0网卡中
flannel会动态的维护路由表,在每一次数据的访问的时候会缓存信息

RS知识保证了支撑服务的微服务Pod的数量,但是没有解决如何访问这些服务的问题,一个Pod只是一个运行服务的实例,随时可能在一个节点上停止,在另外一个节点以一个新的IP启动一个新的Pod。因此不能以确定的IP和端口号提供整个服务,需要稳定地提供服务需要服务发现和负载均衡的能力。服务发现完成的工作,是针对客户端访问的服务,找到对应的后端服务实例,在K8S集群中,客户端需要访问的服务就是Service对象。每一个Service会对应一个集群内部有效的虚拟IP,集群内部通过虚拟IP访问一个服务。

Calico

1.Calico是一个纯3层的数据中心网络方案,而且无缝集成像OpenStack这种Iaas云架构,能够提供可控的VM、容器、裸机之间的IP通信
2.Calico不使用重叠网络比如flannel和libnetwork重叠网络驱动,它是一个纯3层的方案,使用虚拟路由代替虚拟交换,每一台虚拟路由通过BGP协议传播可达信息(路由)到剩余的数据中心去。
3.Calico在每一个计算节点利用Linux Kernel实现了一个搞笑的VRouter来负责数据转发,而每一个VRouter通过BGP协议负责把自己运行的workload的路由信息向整个Calico网络内传播,在小规模的部署情况下可以直接互联,大规模的部署下可以通过制定的BGP route reflector来解决
4.Calico节点组网可以直接利用数据中心的网络结构(L2或L3),不需要额外的NAT,隧道或者Overlay Network
5.Calico基于iptables还提供了丰富的灵活网络Policy,保证通过各个节点上的ACLs来提供Workload的多租户隔离、安全组以及其他可达性限制功能

  • Felix、Calico Agent,跑在每一台需要运行的Workload的节点上,主要负责配置路由及其ACL等信息来确保Endpoint的联通状态

  • etcd,分布式键值存储,主要负责无网络元数据一致性,确保Calico网络状态的准确性

  • BGP Client(BIRD),主要负责把Felix写入Kernel的路由信息分别分发到当前的Calico网络,确保Workload间的通讯有效性‘

  • BGP Route Reflector(BIRD),大规模的部署时候使用,摒弃所有的节点互联的mesh模式,通过一个或者多个BGP route Reflector来完成集中式的路由分发

Calico运行模式

  • Calico-BPG-Peer实际上将集群里面所有的节点都当做边界的路由来处理,浑然一体形成一个自己维护的联通网络,相互之间通过BGP协议交换路由规则,默认情况下使用的是(Node-to-Node Mesh)模式

  • 在这种模式下每台宿主机上的BGP Client都是需要和所有的其他的节点进行通讯就换路由信息,但是这个模式一般只支持100节点以内

大规模集群中使用的模式

  • Route Reflector:在集群规模比较大的情况下,Calico建议在网络中指定一个或多个BGP Speaker作为Router Reflection,RR与所有的BGP Speaker建立BGP连接。每个BGP Speaker只需要与RR交换路由信息,就可以得到全网路由信息。该模式下Calico会指定几个特定的节点,负责和所有的节点之间建立BGP连接学习全局的路由规则,其余几点和特定节点之间交换信息即可。

  • Global BGP Peers:Global Peer是一个BGP Speaker,需要手动在calico中创建,所有的node都会与Global peer建立BGP连接。关闭了全互联模式后,再将RR作为Global Peers添加到calico中,calico网络就切换到了RR模式,可以支撑容纳更多的node

# 开启的方式
apiVersion: v1
kind: bgpPeer
metadata:
  peerIP: 192.20.30.40
  scope: global
spec:
  asNumber: 64567
  • Per-Node BGP Peers:在一些的特定的网络拓扑中,需要为节点指定特定的peer,node Peer就是手动创建的BGP Speaker,只有指定的node会与其建立连接
apiVersion: v1
kind: bgpPeer
metadata:
  peerIP: aa:bb::ff
  scope: node
  node: node1
spec:
  asNumber: 64514

Calico的PIPI模式

当两个Container之间不处于同一个子网,无法通过二层网络之间到达的时候就需要使用Calico的PIPI模式

  • 当在是哟金Calico-PIPI模式的时候,集群的网络性能因为额外的封包和接包工作导致性能的降低,对于性能的损耗和flannel的VXLAN模型性能差不多

网络策略的实现

  • calico即实现网络策略又实现网络功能

  • calico实现网络策略,flannle实现网络功能

  • calico手动部署只提供网络策略

名称 作用
Egress 表示我们的POD作为客户端访问目标对象(出站),自己作为源地址(自己的端口和地址可以预测)->目标地址(无法预测)
ingress 表示Pod作为客户端访问服务端(入站),能够限制自己接受的端口和目标的地址
podselector 确定无论是出站还是入站网络策略都应用在哪一些POD上
policyTypes 当egress/ingress同时存在的时候可以设定某一时刻谁生效或者同时生效
  • calico官方地址:https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/flannel

三层和隧道的异同

相同之处是都实现了跨主机容器的三层互通,而且都是通过对目的 MAC 地址的操作来实现的;不同之处是三层通过配置下一条主机的路由规则来实现互通,隧道则是通过通过在 IP 包外再封装一层 MAC 包头来实现。
三层的优点:少了封包和解包的过程,性能肯定是更高的。
三层的缺点:需要自己想办法维护路由规则。
隧道的优点:简单,原因是大部分工作都是由 Linux 内核的模块实现了,应用层面工作量较少。
隧道的缺点:主要的问题就是性能低。

k8s的存储方式

在容器时代的存储实质

在容器时代容器是一个快速可被迭代的对象,容器的持续存储是使用volume来实现的

  • 只读层是原始的容器镜像文件不可修改

  • init层为了让容器拥有独立的与虚拟机相识的能力特殊开设的一个容器的预设层

  • 可读写层存放在容器构建打包后真实的程序运行的可读写的镜像层

  • 所有这一切组合成一个rootfs文件系统提供一个完整可运行的容器镜像

  • 在宿主机的内核namespace和cgroup技术下对容器进行隔离资源的管控

查看K8s支持的存储方式

kubectl explain pod.spec.volumes

传统的存储方式

  • NAS、SAN

分布式存储

  • glusterfs、rbd、cephfs

云存储

  • 阿里腾讯弹性块存储、EBS、Azure Disk

emptyDir调试

EmptyDir代表一个临时目录,它共享一个pod的生命周期。Pod 中的容器可以读取和写入 emptyDir 卷中的相同文件,尽管该卷可以挂载到每个容器中的相同或不同路径上。当出于任何原因从节点中删除 Pod 时,emptyDir 中的数据将被永久删除。

实用场景

  • 暂存空间,例如用于基于磁盘的合并排序

  • 用作长时间计算崩溃恢复时的检查点

  • Web服务器容器提供数据时,保存内容管理器容器提取的文件

落地应用

  • 一个Pod中定义两个container,一个提供web一个生成页面

  • 创建EmptyDir共享一个存储空间,调试查看在myapp上修改/data下的文件,在busybox上是否能访问到

  • 验证EmptyDir

静态手动模式

apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
  namespace: default
  labels:
    app: myapp
    tier: frontend
  annotations:
    k8s.master/created-by: "cluster admin"
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    volumeMounts:
    - name: html
      mountPath: /data/web/html
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: html
      mountPath: /data/
    command:
    - "/bin/sh"
    - "-c"
    - "echo $(date) >> /data/index.html"
  volumes:
  - name: html
    emptyDir: {}
kubectl create -f pod-volume.yaml
kubectl get pod -w -o wide
kubectl describe pod pod-demo

一个创建一个接受

apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
  namespace: default
  labels:
    app: myapp
    tier: frontend
  annotations:
    k8s.master/created-by: "cluster admin"
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: html
      mountPath: /data/
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date) >> /data/index.html; sleep 2; done"]
  volumes:
  - name: html
    emptyDir: {}

HostPath

HostPath表示主机上已有的文件或目录它直接暴露在容器中。这通常用于系统代理或其他特权可以看到主机机器。大多数容器都不需要这个

  • 在K8S-node1和2上同时创建出 -> /data/pod/volume/index.html 写区别的数据

  • 创建我们的列表清单,curl访问它目标IP,查看返回的结果

  • 验证主机上的文件是否暴露给了容器使用

apiVersion: v1
kind: Pod
metadata:
    name: pod-volume-hostpath
    namespace: default
spec:
    containers:
    - name: myapp-volume
      image: ikubernetes/myapp:v1
      volumeMounts:
      - name: html
        mountPath: /usr/share/nginx/html
    volumes:
    - name: html
      hostPath:
        path: /data/pod/volume
        type: DirectoryOrCreate

nfs共享存储

yum install -y nfs-utils    #所有的节点
mkdir /data/volume
vim /etc/exports
    /data/volume 192.168.42.0/16(rw,no_root_squash)
[root@k8s-node1 ~]# mount -t nfs k8s-master:/data/volume /mnt
[root@k8s-node2 ~]# mount -t nfs k8s-master:/data/volume /mnt

创建NFS的清单

apiVersion: v1
kind: Pod
metadata:
    name: pod-volume-nfs
    namespace: default
spec:
    containers:
    - name: myapp-nfs
      image: ikubernetes/myapp:v1
      volumeMounts:
      - name: html
        mountPath: /usr/share/nginx/html
    volumes:
    - name: html
      nfs:
        path: /data/volume
        server: k8s-master

PV与PVC

StorageClass

在常规的服务器运维时代,每一次修改程序的配置文件的时候都需要重启一次服务,如果在容器时代每更新的一次就需要重新构建一个容器镜像,这样会很麻烦所以在kubernetes时代,为了终结这一切开发了ConfigMap将配置文件提出来当启动时候可以作为存储卷也可以成为环境变量注入即将启动的容器中去使用。

  • 应付在多种情况下配置文件从镜像中解耦,增强应用的可移植性,以及应用的可复用性也就是一系列配置文件的集合

  • 整个ConfigMap中存储的是一组一组的键值对(Key/Value)

  • 属于名称空间级别的资源属于标准的Kubernetes系统资源

配置容器化应用方式:

  • 1.自定义命令行参数

    command 不变
    args:[]

  • 2.将配置文件直接打包到镜像中去:耦合度过于紧密

  • 3.环境变量

要想实现环境变量来配置容器的前提条件有2种:
一:Cloud Native应用本身就支持通过过去环境变量来加载配置文件,传统意义上是不支持的都是去读取配置文件
二:写entrypoint(预处理脚本)脚本传递变量用sed替换配置文件也可以实现

  • 4.存储卷:配置文件挂载在启动文件的目录下,启动时候先去挂载配合文件来实现
kubectl explain pods.spec.containers.env    #查看变量的编写方式
kubectl explain pods.spec.containers.env.valueFrom

Projected Volume

投射数据卷,该volume是kubernetes v1.11之后的最新特性,K8S的数据卷中有几种特殊的volume,他们存在不是为了存放容器里面产生的数据也不是用来作为宿主机与容器的数据交换,他们的目的是:为容器提供预定义好的数据

Kubernetes支持的Projected Volume

  • (1) Secret

  • (2) ConfigMap

  • (3) Downward API

  • (4) ServiceAccountToken

Secret

针对于敏感数据中的加密和加密,secret诞生是帮你把 Pod 想要访问的加密数据,存放到 Etcd 中。然后,你就可以通过在 Pod 的容器里挂载 Volume 的方式,访问到这些 Secret里保存的信息了,主要还是解决了秘钥下发的问题

  • 但是及实际上只是通过改变编码的方式忽悠小白而且,明白的人通过编码转换就可以看见的伪加密方式

  • 将数据写入到容器的env环境变量中可以便于启动的时候去读取环境变量的信息

  • 在真正的生产环境中,需要在Kubernetes中开启Secret的加密插件增强数据的安全性

分类 备注说明
kubernetes.io/service-account-token 用来访问K8S API,K8S自动创建,并且自动挂载到Pod的目录中
Opaque Base64编码格式的Secret,用于存储密码,秘钥
kubernestes.io/dockerconfigjson 用于存储私有的docker registry的认证信息,用于镜像下载

相关分类

  • docker-registry:docker认证信息则需要使用这种方式。一般针对k8s中pod在生成时候本地查找资源需要tongg私有仓库时候有账户认证是使用

  • generic:通用的保存密码数据等

kubectl create secret <名称> <–from-file=文件>/<–from-literal=key=value> #创建的方式

kubectl describe secret 名称      #查看当前的secret的状态,显示的信息只有字节长度不是明文
kubectl get secret 名称 -o yaml   #即可查看到详细的信息明文,一定意义上加密但是实际的值只是base64加密的
echo "加密密码" | base64 -d     #即可解码显示真实的信息,安全性上不是很好但是比env多一个障眼法

实例

kubectl create secret generic redis-passwd --from-literal=redis-password=syaycaz    #创建secret

  • 实际上是一种伪加密的方式一下方式可见

vim pod-secret-1.yaml   #创建POD第一种注入方式

apiVersion: v1
kind: Pod
metadata:
    name: pod-secret-1
    namespace: default
    labels:
        app: myapp
        tier: frontend
    annotations:
        ddy.com/created-by: "cluster ddy"
spec:
    containers:
    - name: myapp
      image: ikubernetes/myapp:v1
      ports:
      - name: http
        containerPort: 8080
      env:
      - name: redis-passwd
        valueFrom:
            secretKeyRef:
                name: redis-passwd
                key: redis-password

kubectl apply -f vim pod-secret-1.yaml  #创建POD
kubectl exec pod-secret-1 -- printenv   #查看是否注入到Pod的环境变量中

  • tls:针对私钥和证书
apiVersion: v1
kind: Secret
metadata: 
    name: mysecret
type: Opaque
data:
    password:NNxcalkdjq123SD
    username:XXPark
ConfigMap

在常规的服务器运维时代,每一次修改程序的配置文件的时候都需要重启一次服务,如果在容器时代每更新的一次就需要重新构建一个容器镜像,这样会很麻烦所以在kubernetes时代,为了终结这一切开发了ConfigMap将配置文件提出来当启动时候可以作为存储卷也可以成为环境变量注入即将启动的容器中去使用。

  • 应付在多种情况下配置文件从镜像中解耦,增强应用的可移植性,以及应用的可复用性也就是一系列配置文件的集合

  • 整个ConfigMap中存储的是一组一组的键值对(Key/Value)

  • 属于名称空间级别的资源属于标准的Kubernetes系统资源

配置容器化应用方式:

  • 1.自定义命令行参数

    command 不变
    args:[]

  • 2.将配置文件直接打包到镜像中去:耦合度过于紧密

  • 3.环境变量

要想实现环境变量来配置容器的前提条件有2种:
一:Cloud Native应用本身就支持通过过去环境变量来加载配置文件,传统意义上是不支持的都是去读取配置文件
二:写entrypoint(预处理脚本)脚本传递变量用sed替换配置文件也可以实现

  • 4.存储卷:配置文件挂载在启动文件的目录下,启动时候先去挂载配合文件来实现
kubectl explain pods.spec.containers.env    #查看变量的编写方式
kubectl explain pods.spec.containers.env.valueFrom

使用ConfigMap

  • 方式一:直接将ConfigMap作为一个存储卷,加载应用的时候首先去挂载它

  • 方式二:通过环境变量的方式注入到Pod中在程序运行的时候自动去读取环境变量并且引用到程序中去

#简单的创建方式
kubectl create configmap <config-name> --from-literal=key=value ....依次类推    #给定名称和参数即可
kubectl describe cm <config-name>   #创建后就直接可以被Pod调用了

#创建一个配置文件
vim ~/configmap/html.conf
server {
    server_name myapp.kubernetes.com;
    listen 80;
    root /data/web/html/;
}
kubectl create configmap nginx-html --form-file=~/configmap/html.conf   #根据配置文件创建configmap
kubectl get cm nginx-html -o yaml   #以yaml格式去查看配置文件
kubectl describe cm nginx-html  #常规的查看方式

注入到Pod

  • 当使用环境变量更新的时候在线编辑configmap文件时候Pod中显示也是修改的,但是进入Pod查看环境变量是没有修改的,因为Pod启动获取的是注入时的环境变量

  • 当时用ConfigMap挂载时候可以对Pod的配置文件实现真的意义上的实时更新

#使用ENV环境变量的方式例子:
vim pod-configmap.yaml

apiVersion: v1
kind: Pod
metadata:
    name: pod-map-1
    namespace: default
    labels: 
        app: myapp
        tier: frontend
    annotations:
        test.com/create-by:"cluster now"
spec:
    containers:
    - name: myapp
      image: ikubernetes/mapp:v1
      ports:
      - name: http
        containerPort: 80
      env:
      - name: NGINX_SERVER_PORT #变量不能使用-会自动转成_
        valueFrom:
            configMapKeyRef:
                name: nginx-html
                key: nginx_port
      - name: NGINX_SERVER_PORT
        valueFrom:
             configMapKeyRef:
                name: nginx-html
                key: server_name

#查看是否生效
kubectl create pod-configmap.yaml   #创建CM
kubectl get pods    #查看Pod是否启动
kubectl exec -it pod-map-1 -- /bin/sh   #进入Pod查看容器环境变量是否读取生效
/ # printenv    #打印环境变量
  • 使用Volume指定存储卷类型指定挂载的位置使用ConfigMap
apiVersion: v1
kind: Pod
metadata:
    name: pod-map-2
    namespace: default
    labels: 
        app: myapp
        tier: frontend
    annotations:
        test.com/create-by:"cluster now"
spec:
    containers:
    - name: myapp
      image: ikubernetes/mapp:v1
      ports:
      - name: http
        containerPort: 80
      volumeMounts:
      - name: nginx-conf
        mountPath: /etc/nginx/config.d/
        readOnly: true
    volumes:
      name: nginx-config
      connfigMap:
          name: nginx-config

#查看是否生效
kubectl create pod-configmap-2.yaml #创建CM
kubectl get pods    #查看Pod是否启动
kubectl exec -it pod-map-2 -- /bin/sh   #进入Pod查看容器环境变量是否读取生效
/ # cd /etc/nginx/conf.d/
/ # ls
server_name nginx_port  #挂载到Pod上的两个ConfigMap的内容

Downward API

作用让Pod里面的容器能够直接的获取到这个Pod API对象本身的信息,而Downward API Volume声明要暴露的Pod的metadata.labes信息给容器

  • Downward API能获取到的信息,一定是Pod里面的容器进程启动之前定下来的信息,如果是启动后的建议使用sidecar辅助容器

k8s-api请求

API server是整个请求进入的网关接口,请求访问中请求用于实现身份识别授权用于实现认证检查,准入控制进一步补充了认证。可以用多个插件组合进行实现一般而言只在创建删除修改代理时候作为补充使用

  • API Server: subject –> action –> object
    • 认证:Token,TLS,User/Password

    • 授权:RBAC = role + rolebinding + ClusterRole + ClusterRolebinding

    • Subject = user + group + ServiceAccount

    • Obejct = Resource Group + Resource + Non-Resource Url

    • ation: get/list/watch/patch/delete/deletecollection/….

  • 允许同一个群组上的多版本并存

  • pod -> proxy -> api

kubectl proxy --port=8080  #开启监听到本地8080端口会占据当前终端

curl http://localhost:8080/api/v1/namespaces   返回序列化的json结果

kubectl get deploy -n kube-system

curl http://localhost:8080/apis/apps/v1/namespaces/kube-system/deployments/
  • 通常Client -> request -> API server

  • request: username(uid) + group + extra + Resource + Subresource + Namespace + API group #一个K8s-api请求应该包含的信息

  • Request path -> http://IP:监听端口/apis/apps/v1/namespaces/default/deployment/deploy名称 #具体对每一个URL的表示对象执行操作(数据对象的证删改查)

  • HTTP request verb(对象的请求操作): get post put delete

  • API requests verb(具体操作): get list(列出) create update patch(打补丁) watch(资源监视) proxy(代理) delete deletecollection(删除一组)

    • Resource:资源

    • Subresource:子资源

    • Namespace:名称空间

    • API group:api群组

kubernetes集群认证时候有两类账号

  • userAccount:用户账号现实中用户使用的账号

  • serviceAccount:服务账号,pod应用账号托管运行在K8S集群上时候想访问当前的k8s集群时需要用到的用户信息

  • kubectl create serviceaccount admin #创建一个名为admin的k8s集群内部账户

创建一个使用sa的实例

[root@k8s-master ~]# vim pod-sa-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-sa-demo
  namespace: default
  labels:
    app: myapp
    tier: frontend
  annotations:
    k8s.master/created-by: cluster admin
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
  serviceAccountName: admin

[root@k8s-master ~]# kubectl apply -f pod-sa-demo.yaml

多集群中的角色

1.创建私钥证书

自定义认证当前APIserver自定义的签署的CA证书并且创建在一个新的用户账号上,通过这个用户账号连接上服务器

[root@k8s-master pki]# openssl genrsa -out ddy.key 2048

2.生成证书签署请求

openssl req -new -key ddy.key -out ddy.csr -subj "/CN=ddy"  #CN就是你的用户账号

3.CA证书生成

[root@k8s-master pki]# openssl x509 -req -in ddy.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ddy.crt -days 1000
Signature ok
subject=/CN=ddy
Getting CA Private Key

4.查看当前证书状态

[root@k8s-master pki]# openssl x509 -in ddy.crt -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            d4:39:e5:fb:aa:8c:ef:52
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Dec 29 10:08:13 2018 GMT
            Not After : Sep 24 10:08:13 2021 GMT
        Subject: CN=ddy
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c3:28:9c:b5:7c:2b:39:1f:b9:f3:74:cb:18:ff:
                    62:df:b3:c9:d6:98:5a:3d:ea:a0:28:aa:b4:c9:a7:
                    f5:5c:62:f6:8e:06:04:16:8d:c4:23:7a:51:01:28:
                    05:8c:36:b9:29:bf:85:75:31:fe:ca:dc:9a:e0:4d:
                    38:3c:1b:5f:71:41:ab:89:e7:9b:50:73:1c:ab:d9:
                    f6:4c:b5:ac:95:9f:9b:86:f7:1a:08:ad:31:ce:9c:
                    e6:24:19:a5:c4:d6:33:52:7b:20:35:b6:4a:c9:d9:
                    b3:aa:5e:bf:db:78:c7:aa:c8:59:24:60:34:7b:71:
                    dd:8b:34:8a:eb:e7:bd:e5:a7:c7:b9:ee:ac:5f:a4:
                    dc:d8:cf:ab:6c:8e:cf:a4:04:ed:f3:8a:62:40:0a:
                    f6:85:20:61:ed:3a:1e:5b:98:40:13:51:e1:62:45:
                    75:58:c0:84:43:ae:f7:ae:9c:07:6d:a8:31:04:a3:
                    b2:3a:85:98:05:40:9d:b8:d4:a9:ed:80:0e:e5:ae:
                    83:b5:ba:0e:eb:d9:f4:aa:64:59:44:44:c6:77:b6:
                    78:02:cd:3c:11:94:a8:8f:e9:55:ac:14:32:c8:28:
                    2b:d5:38:70:d9:3e:a7:0b:17:c7:0a:72:64:c5:73:
                    44:cb:c4:aa:dc:75:08:19:09:21:e8:5a:c4:cc:0d:
                    33:6f
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         62:43:35:8c:ea:05:8e:1a:d4:dc:7b:f6:4a:0f:66:da:16:00:
         3a:21:5a:da:e3:a0:23:1e:4b:0e:85:51:88:45:42:8c:20:e8:
         88:70:16:f3:b1:9a:09:d6:25:11:e3:83:9d:9d:70:eb:56:49:
         e9:bc:b8:12:f4:cf:a2:a4:74:d6:98:67:94:84:62:7b:28:e3:
         72:92:b4:dc:cf:c0:bf:19:99:a5:27:65:78:8a:3d:94:52:18:
         d6:41:8f:8f:be:db:af:03:9b:37:14:45:c2:6a:36:93:ee:1e:
         1e:9f:03:97:a8:18:85:d9:81:75:fb:e7:3b:f6:9d:7c:fc:62:
         83:75:a4:7f:63:07:1a:42:81:b4:70:af:8c:aa:6d:34:df:47:
         28:d3:40:df:f9:6b:b8:f5:06:10:bd:f5:d7:ab:59:91:ad:84:
         41:5a:d3:65:ed:02:3f:0c:72:54:80:c0:20:83:1b:61:67:e6:
         39:0c:ce:29:fd:37:a5:6d:03:f9:3a:de:b6:15:78:49:46:fd:
         d0:41:6d:ff:d4:6b:bc:00:4f:d9:57:12:40:b8:0a:4b:69:cc:
         21:67:a5:4e:10:22:8d:84:eb:a3:d9:9f:b0:6d:85:8e:06:c9:
         3b:8f:19:99:89:45:b3:9a:3f:e3:f2:a1:bf:45:48:6f:82:1a:
         93:9d:df:65

5.设置用户

[root@k8s-master pki]# kubectl config  set-credentials ddyadmin --client-certificate=./ddy.crt --client-key=./ddy.key --embed-certs=true
User "ddyadmin" set.

6.切换新用户上下文

[root@k8s-master pki]# kubectl config set-context ddy@kubernetes --cluster=kubernetes --user=ddy
Context "ddy@kubernetes" created.
[root@k8s-master pki]# kubectl config view
  • 当前目录下的.kube目录里面 -> kubectl config set-cluster -h -> –kubeconfig 指定环境的路径否则使用当前用户Home下的文件
kubectl config set-cluster mycluster --kubeconfig=/tmp/test.conf --server="https://172.20.0.70:6443" --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true

k8s中的 RBAC

RBAC(Role-base AC),基于角色的权限访问控制

  • kubectl config view #查看当前的RBAC用户

  • (1) Role = Operations + Objects

  • (2) Rolebinding = user –扮演–> role

  • (3) clusterrole:当在同一个集群的名称空间下需要做授权时可以使用,

  • (4) clusterrolebinding

RBAC使用场景分析

创建一个RBAC-role

kubecatl create role pods-reader --verb=get,list,watch --resource=pods --dry-run -o yaml >role-demo.yaml

创建一个RBAC-rolebinding

kubectl create rolebinding read-pods --role=pods-reader --user=ddy

切换用户上下文

[root@k8s-master pki]# kubectl config set-context ddy@kubernetes --cluster=kubernetes --user=ddy
kubectl get pods    #可以成功查看,其他不在写入范围内的无法实现

创建一个RBAC-clusterrole

  • kubectl apply -f cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-reader
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch

新建一个用户方便验证

[root@k8s-master ~]# useradd k8s
[root@k8s-master ~]# cp -rp .kube/ /home/k8s/
[root@k8s-master ~]# chown -R k8s:k8s /home/k8s/
[root@k8s-master ~]# su k8s

创建一个RBAC-clusterrolebinding

  • kubectl apply -f clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: clusterrole-read-pods
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: ddy

rolebinding到clusterrole

  • 使用的是rolebinding的namespce,导致clusterrole也会降级

  • 只允许在default名称空间下有查看权限

  • user -> rolebinding -> clusterrole

kubectl create rolebinding ddy-read-pods --clusterrole=cluster-reader --user=ddy --dry-run -o yaml >rolebinding-clusterrole-demo.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ddy-read-pods
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: ddy

kubernetes-dashboard

默认为官方提供的UI配置界面实现实时查看动态的修改

认证方式

  • kuber配置文件

  • 令牌Token

创建令牌认证

常规的方式:
1.保证每一个Node节点上有kubernetes-dashboard的镜像,或者指定部署在某一个节点上
2.原生的权限
部署:
kubectl delete -f    https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml    #创建dashboard
++++++++++++++dashboard创建的资源+++++++++++++
secret 
serviceaccount 
role.rbac.authorization.k8s.io 
rolebinding.rbac.authorization.k8s.io
deployment
service 
+++++++++++++++++++++++++++++++++++++++++++++
sz -r ~/.kube/conf  #原生自带部署好kube-dashboard后下载登入配置
kubectl patch svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}' -n kube-system #给dashboard创建一个svc的暴露端口随机即可也可自己定义

配置认证错误:Not enough data to create auth info structure.

  • 原因ServiceAccount账号没有权限,.kube/config中的账号是user而非service导致的

  • 删除之间创建的dashboard

kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
  • 为dashboard创建专用的私钥
cd /etc/kubernetes/pki
(umask 077; openssl genrsa -out dashboard.key 2048) #创建证书
openssl req -new -key dashboard.key -out dashboard.csr -subj "/O=zpx/CN=dashboard"  #生成证书签署请求,依靠当前API Service的签证。CN当需要和域名绑定时候填写域名
openssl x509 -req -in dashboard.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out dashboard.crt -days 365  #使用系统上的ca.key dashboard.key给签署证书
kubectl create secret generic dashboard-cert -n kube-system --from-file=dashboard.crt=./dashboard.crt --from-file=dashboard.key=./dashboard.key     #创建sceret使用tls证书以及私钥
kubectl create serviceaccount dashboard-admin -n kube-system    #serviceaccount创建

serviceaccount绑定在clusteradmin角色上

kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin  #cluster-admin ->  clusterrole -> clusterrolebinding 绑定

获取Token

kubectl describe  secret dashboard-admin-token-kl6j6 -n kube-system

使用令牌登入

创建Kubeconfig认证

创建只拥有默认名称空间权限用户

[root@k8s-master .kube]# kubectl create serviceaccount default-ns-admin -n default
serviceaccount/default-ns-admin created

创建集群角色

[root@k8s-master .kube]# kubectl create rolebinding default-ns-admin --clusterrole=admin --serviceaccount=default:default-ns-admin
rolebinding.rbac.authorization.k8s.io/default-ns-admin created
  • Metadara

标示API对象,每一个对象都至少有3个元数据:namespace、name、uuid,除此之外还有各种各样的标签labels用来标示和匹配不同的对象,例如用户可以用标签env来标示区分不同的服务部署环境,分别用env=dev、env=testing、env=production来标识开发、测试生产中的不同的服务

  • Spec

描述了用户期望Kubernetes集群中的分布式系统达到了理想状态(Desiredstate),例如用户可以通过复制控制器Replication Controller设置期望的Pod副本数量为3

  • Status

系统实际当前达到的状态(Status),例如系统当前实际的Pod副本数位2个,那么复制控制器当前的程序就是自动启动后的新Pod,争取达到副本数为3个

K8S分类

类比 名称
资源对象 Pod、ReplicaSet、ReplicationController、Deployment、StatefulSet、DaemonSet、Job、CronJob
配置对象 Node、Namespace、Service、Secret、ConfigMap、Ingress、label、ThirdPartyResource、ServiceAccount
存储对象 Volume、Persistent Volume
策略对象 SecurityContext、ResourceQuota、LimitRange

K8s分层架构

  • 生态系统:在接口层之上的庞大容器集群管理调度的生态系统,可以划分为两个方面
    • Kubernetes外部:日志、监控、配置管理、CI/CD、Workflow、FaaS、OTS应用、Chatops

    • kubernetes内部:CRI、CNI、CVI、镜像仓库、Cloud Provider、集群自身的管理和配置

  • 接口层:kubectl命令行工具,客户端SDK以及集群

  • 管理层:系统度量(入基础设施、容器和网络的度量),自动化(自动化扩容、动态Provision)以及策略管理(RBAC、Quota、PSP、NetworkPolicy)

  • 应用层:部署(无状态的应用、有状态的应用、批处理任务、集群应用),路由(服务的发现、DNS的解析)

  • 核心层:Kubernetes最为核心的功能,对外提供API构建高层的应用,对内提供插件式应用执行环境

k8s的高级调度方式

了解最新详情请移步官方源码:
https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/algorithm/predicates/predicates.go

  • 预选(Predicate):从所有的节点当中排除,完全不能符合pod使用的基本条件的Node节点

  • 优选(Priority):将通过了预选的所有节点的属性通过k8s自身提供的数据优选算法进行打分制度进行排序,得分最高的为最优节点

  • 选定(Select):如果通过优选后有相同的分数的节点则随机选择一个作为最终结果返回

调度器预选策略

  • CheckNodeCondition:检查节点是否的磁盘网络的是否正常能否允许调度Pod允许

  • GeneralPredicates(通用预选策略):

    • HostName:去检查对象是否定义pod.spec.hostname,如果定义则与欲调度到的Node的名称相比较是否匹配

    • PodFristPorts:对象如果定义pod.spec.containers.ports.hostPort,是否与欲匹配的Node符合如果Node上被占用端口则不适合,依次去适配

    • MatchNodeSelector:对象如果定义pods.spec.nodeSelector,查看是否有节点的标签适配Pod中定义标签选择

    • PodFristResources:欲调度到的节点是否满足Pod的最低资源消耗,满足则调度

  • NoDiskConflict:是否存在磁盘冲突,pod对象需要使用的存储卷在欲调度的节点上是否可用

  • PodToleratesNodeTaints:检查对象的pod.spec.tolerations,能够容忍的污点是否完全包含节点上的污点

  • PodToleratesNodeNo ExecuteTaints:如果节点定义了污点属性,就会去检查pod上拥有的污点是否能够忍受节点上拥护该定义污点的,能容忍则调度。如果后续node上定义该Pod无法忍受的污点则Pod也会被驱逐调度到其他能满足的节点

  • CheckServiceAffinity:根据当前pod对象所属的service已经有的其他service对象,将含有多个service的Pod对象尽量的调度到同一个节点上运行保障性能最优(默认不启用)

  • MaxEBSVolumeContPred:如果使用亚马逊EBS存储卷,检查欲调度的节点上是否已经达到最大的挂载39个限度

污点和容忍度

什么是污点呢?
即定义在节点上的键值属性的一类数据,可以让当前节点拒绝将Pod调度到自己上运行,但是如果欲创建的Pod对象具有对污点容忍度则,还是可以调度到有污点的节点上的(简称走后门).
Taints的设置机制给了控制节点一个主动权是驱逐pod还是接纳pod,从而能够自由的掌控能够双向的控制,当一个节点上打上了污点后pod能不能调度到节点需要pod上的容忍度

查看污点

类似标识在节点上的key/value值为了描述节点的特性的一组数据,每一个对应的污点由3个数据组成

kubectl describe node 节点名称

Taints and Effect理论

污点的效用Effect

类似标识在节点上的key/value值为了描述节点的特性的一组数据,每一个对应的污点由3个数据组成

键值 参数 描述
key none key为字符串最大不超过253字节以字母或数字开头,可以包含字母、数字、连字符、点和下划线
value none 该值是任何字符串,最多63个字符。值必须以字母或数字开头,可以包含字母、数字、连字符、点和下划线。
effect Noschedule/PreferNoschedule/NoExecute 不调度/最好不调度/驱逐
  • 污点容忍度通过对节点添加污点的信息来控制Pod对象的调度结果,赋予节点能够主动控制Pod对象调度的权限

对于节点上的单个污点对应的容忍度

  • operattor
    • Equal 等值 (key/value/effect三种必须一致才容忍)
    • Exists 存在 (key/effect2中一致即可容忍)

对于节点上的多个污点对应的容忍度

  • 调度的POD需要具有匹配污点的的公差的容忍度

  • 其余没有匹配的污点对pod对的指示作用

  • 存在任意一个Effect为NoSchedule的污点时,Pod将会无法被调度上节点

  • 不存在Effect为NoSchedule的污点,但是存在至少一个为PreferNoSchedule时K8s会尽量的避免Pod被调度到这个节点

  • 但凡有一个Effect为NoExecute的污点,pod对象将会无法调度至此节点,而已经调度完成的则会被驱逐

    • 不能容忍污点的Pod将会立即被驱逐
    • 可以容忍的但是没有指定tolerationSeconds的Pod可一直运行相应的节点
    • 能容忍且指定了tolerationSeconds的Pod可以在对应的节点运行指定的时长时间

add and remove Taints

Adding a Toleration to a pod

# Use Equal operator
tolerations:              #定义pod的容忍度以一个列表表示
- key: "keyname"          #指定key值
  operator: "Equal"       #等值容忍
  value: "valuename"      #指定对应的值
  effect: "NoExecute"     #指定效用
  tolerationSeconds: 3600 #容忍时长

---
# Use Exist operator
tolerations:              #定义pod的容忍度以一个列表表示
- key: "keyname"          #指定key值
  operator: "Exists"      #存在型容忍
  effect: "NoExecute"     #指定效用
  tolerationSeconds: 3600 #容忍时长

查看容忍度

kube-proxy集群组件以pod的形式运行在每个节点上,其中也包括主节点。因为以pod形式运行的主节点组件同时也需要访问Kubernetes服务。为了确保kube-proxy pod也能够运行在主节点上,该pod需要添加相应的污点容忍度。该pod整体包含了7个污点容忍度。

深入理解helm

架构图

K8s的设计优缺点

优点

  • 1.容错性:保证K8s系统稳定性和安全性的基础

  • 2.易扩展性:保证K8s对变更更加友好,可以快速的迭代增加新的功能基础

    • API分版本,API可以自由扩展(CRD)

    • 插件化,调度器,容器运行时候,存储均可以扩展

  • 3.声名式(Declarative)的而不是命令式(Imperative)

声名式操作在分布式系统中的好处就是稳定性,不怕丢失操作或者多次运行
如设置副本数为3的操作运行多次也还是相同的结果,而给予副本数加的操作就不是声名式,运行多次结果就报错了

缺点

  • 配置中心化:所有的状态都保存在中心的etcd上,而非分布式存储,性能上有一定的制约作用

  • 单体调度:调度一致性好而吞吐量低

  • 不同集群调度程序架构。灰色表示集群机器,圆圈对应于任务,s表示调度程序i

k8s的高可用架构

不可用场景分析:
1.etcd

3个node,一个死亡
apiserver -> 请求转移
etcd -> 3 -> raft算法(存活>2则提供服务,可读写)
3 -> 1
5 -> 2
7 -> 3
一旦出超出最大的容错数量,etcd只可读不可写

2.scheduler/CM

本质上也就是争抢锁的一个过程,谁抢到了锁谁就是master

  • 1.死一个Node 非主 -> 毫无影响

  • 2.死一个Node 为主 -> 重新调用算法选举新节点

3.kubelete(3个Node)

  • 当system -> 进程死掉了主动的拉起来

  • 当一个Node死亡 -> 控制面板还是可用

  • 当二个Node死亡 -> 控制面不可用

k8s的部署

常见的部署方法

  • 手动撸逐行代码一个个敲击(复杂消耗时间)

  • minikube快速部署(本地单节点)

  • kubeadm快速部署(本地多节点)

  • 自动化运维工具快速部署


常见的K8S安装方式 安装前的准备 适用的范围及其特点 高可用 安装复杂的程度
kubeadm kubectl/kubelet 不对接节点,自己准备安装的环境,容器被集成到其他的工具中 HA 高难
kops kubectl 对接AWS、GCE/Vmware;帮助管理虚拟机 HA
minnikube kubectl 单机对接VM
rancher rancher 墙内加速,跨云能力 不清楚
手工安装 11+组件 任意集群 HA 地狱

minikube快速部署

  • 什么是minikube?

Minikube是一款让本地运行Kubernetes变得简单的工具。 Minikube在您的笔记本电脑的虚拟机内部运行单节点Kubernetes集群,以供试图尝试Kubernetes或与其一起开发的用户使用。(单节点的k8s)

git地址:

https://github.com/kubernetes/minikube

Minikube使用vm虚拟化安装

1.docker安装

curl -fsSL  get.docker.com  -o  get-docker.sh;sh get-docker.sh –mirror  Aliyun;systemctl restart docker;systemctl enable docker

2.简单部署方式

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

3.kubectl部署

因为墙的问题所以使用其他的镜像拉下来。国内大型的厂商或者Git

wget http://mirrors.ustc.edu.cn/kubernetes/apt/pool/kubectl_1.9.1-00_amd64_bdfb1ad90e0f02a7ae614502079a87ed99209679bdedf0c62873564c186c9f99.deb
dpkg-deb -x kubectl_1.9.1-00_amd64_bdfb1ad90e0f02a7ae614502079a87ed99209679bdedf0c62873564c186c9f99.deb

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1 #选择对应的版本下载

3.ubantu中建立虚拟机环境

vim env

export MINIKUBE_WANTUPDATENOTIFICATION=false
export MINIKUBE_WANTREPORTERRORPROMPT=false
export MINIKUBE_HOME=$HOME
export CHANGE_MINIKUBE_NONE_USER=true
mkdir -p $HOME/.kube
touch $HOME/.kube/config

export KUBECONFIG=$HOME/.kube/config
sudo -E ./minikube start --vm-driver=none

# this for loop waits until kubectl can access the api server that Minikube has created
for i in {1..150}; do # timeout for 5 minutes
   ./kubectl get po &> /dev/null
   if [ $? -ne 1 ]; then
      break
  fi
  sleep 2
done

4.启动minikube

minikube start

Minikube使用容器安装

1.配置清华大学的镜像源

https://mirrors.tuna.tsinghua.edu.cn/help/ubuntu/

2.配置docker-ce国内源

https://mirrors.tuna.tsinghua.edu.cn/help/docker-ce/    #清华大学的源
http://www.runoob.com/docker/ubuntu-docker-install.html     #针对个版本的安装方式

3.安装起服务

systemctl restart docker

4.minikube安装

curl -Lo minikube http://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v0.28.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

5.Kubernetes安装

wget https://storage.googleapis.com/kubernetes-release/release/v1.9.4/bin/linux/amd64/kubectl&&chmod +x kubectl&&sudo mv kubectl /usr/bin&&kubectl version
  • github-aliyun-image : https://github.com/kubernetes/minikube/releases/tag/v1.3.0

6.启动k8s服务

minikube start --vm-driver=none #参数不使用虚拟机使用docker
minikube start  #上一个不行用这个

7.当前的系统组件

8.启动控制面板

sudo minikube dashboard
访问:http://IP:30000

第一个web程序

创建类似docker

[root@minikube ~]# kubectl run hello-minikube --image=nginx --port=80
deployment "hello-minikube" created
[root@minikube ~]# kubectl descible pod #查看创建的过程

kubectl使用小技巧

使用自动补全命令
(kubectl completion bash) >>/etc/bashrc

kubectl上下文配置

  • 显示合并后的kubeconfig配置

  • 显示当前的上下文
kubectl config current-context
  • 设置默认的上下文为name
kubectl config use-context <names>
  • 使用指定的用户名和namespace设置上下文
kubectl config set-context gce --user --namespace && kubectl config use-context gce 

kubectl创建对象

  • 创建资源
kubectl create -f ./xxxxxxx.yaml
  • 使用多个文件创建资源
kubectl create -f ./xx1.yaml ./xx2.yamle
  • 使用目录下的所有清单文件来创建资源
kubectl create -f ./dir
  • 使用curl创建资源
kubectl create -f https://xxx/xx/
  • run一个实例
kubectl run xxx --image=xxx

kubectl显示和查询资源

kubectl get services    # 列出所有namespace中的所有service
kubectl get pods --all-namespaces   # 列出所有namespace中的所有pod
kubectl get pod -o wide     # 列出所有pod并且显示详细信息
kubectl get deployment my-xxx   # 列出指定的deployment
kubectl get pods --include-uninitialized    # 列出该namespace中所有的pod包
  • 显示指定的节点的所有负载信息类似TOP
kubectl describe node <node-name>
  • 显示指定pod的所有负载信息
kubectl describe pod <pod-name>
  • 排序列出service的信息
kubectl get services --sort-by=.metadata.name
  • 生成简单的配置清单模板的两种方式
kubectl create <类型> <名称> -o yaml --dry-run  #生成一个没有多余项目的配置清单表方便编写创建
kubectl get <类型> <名称> -o -yaml --export #导出的包含所有的选项比较麻烦

资源统计类型命令

  • 根据重启的次数列出pod
kubectl get pods --sort-by='.status.containerStatus[0].restartCount'
  • 获取所有具有app=cassandra的pod中的version标签
kubectl get pods --selector=app=cassandra rc -o jsonpath='{.items[*].metadata.labels.version}'
  • 获取所有节点的ExternalIP
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'

jq命令需要手动安装

  • 列出属于某一个PC的Pod的名字(jq-命令用于转化复杂的jsonpath,详情:https://stedolan.github.io/jq)
sel=${$(kubectl get rc <rc-name> --output=json|jq -j '.spec.selector|to_entries|.[]|"(.key)=(.value),"')%?}
echo $(kubectl get pods --selector=$sel --output=jsonpath={.items..metadata.name})
  • 查看哪一些节点已经就绪
JSONPATH='{range.items[*]}{@.metadata.name}:{range@.status.conditions[*]}{@.type}={@.status};{end}{end}' && kubectl get nodes -o jsonpath="$JSONPATH"|grep "Ready=True"

简单的集群的伸缩

kubectl与运行中的Pod交互

kubectl logs <pod-name>     #dump输出pod的日志(stdout)
kubectl logs <pod-name> -c <my-container>   #dump输出pod中容器的日志(stdout,pod中有多个容器的情况下使用)
kubectl logs -f <pod-name>  #流式输出pod的日志(stdout)
kubectl logs -f <pod-name> -c <my-container>    #流式输出pod中容器的日志(stdout、pod中有多个容器的情况下使用)
kubectl run -i --tty busybox --images=busybox --sh  #交互式shell的方式运行Pod
kubectl attach <pod-name> -i    #连接到运行中的容器
kubectl port-forward <pod-name> pod端口:本地端口  #转发pod中的端口到本地的端口
kubectl exec <pod-name> -- <commond>    #在已经存在的容器中执行命令(Pod中有一个容器情况)
kubectl exec <pod-name> -c <my-container> -- <commond> #在一个Pod中存在多个容器的情况
kubectl top pod <pod-name> --container  #显示指定的pod和容器的指标度量

kubeadm部署

初始化步骤

先决前提:

  • 1.基于主机的通信:/etc/host

  • 2.时间同步

  • 3.关闭firewalld和iptables

  • 4.启动docker17.3以下的版本

安装和配置

  • 1.etcd cluster,仅仅Master节点

  • 2.flamnel,集群的所有节点

  • 3.配置K8s的master,仅仅Master节点

    • 启动服务:kube-apiserver.kube-scheduler,kube-contriller-manage
  • 4.配置K8s的各节点:
    • 启动服务:kube-proxy,kubelet

配置YUM或者手动上传包

懒人必备

docker-17版本安装:
wget -P /etc/yum.repos.d/ http://k8s-cfssl-ca-1252464731.file.myqcloud.com/docker-ce.repo 
k8s安装:
wget -P /etc/yum.repos.d/ http://k8s-cfssl-ca-1252464731.file.myqcloud.com/kubernetes.repo && wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg && rpm --import rpm-package-key.gpg

动手党

  • 阿里云k8s:https://mirrors.aliyun.com/kubernetes/yum/repos/

  • 清华大学镜像:https://mirrors.tuna.tsinghua.edu.cn/help/AOSP/

1.基础环境的配置

Master节点设置:
设置docker代理加速:
sed -i '/[Service]/aEnvironment="HTTPS_PROXY=http://www.ik8s.io:10080"' /usr/lib/systemd/system/docker.service
sed -i '/[Service]/aEnvironment="NO_PROXY=127.0.0.0/8,172.20.0.0/16"' /usr/lib/systemd/system/docker.service
systemctl daemon-reload
systemctl restart docker
docker info

针对无法过墙的大佬使用一下懒人方式即可解决一切问题:
wget https://k8s-cfssl-ca-1252464731.cos.ap-guangzhou.myqcloud.com/pull_k8s_images.sh && chmod +x pull_k8s_images.sh && sh -x pull_k8s_images.sh

yum -y install kubeadm kubelet kubectl
k8s不建议开启:
swapoff -a  #关闭swap分区
如果特殊需求要开启swap则:
1.echo "KUBELET_EXTRA_ARGS="--fail-swap-on=false"" >> /etc/systemd/kubelet

2.在初始化kubeadm时候需要使用--ingore-preflight-errors=swap
初始化master节点信息:
kubeadm init --kubernetes-version=v1.13.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12  #要开启则加参数

3.构建成功会提示你需要做的步骤
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.部署创建将要使用的网络方式
wget https://k8s-cfssl-ca-1252464731.cos.ap-guangzhou.myqcloud.com/kube-flannel.yaml
kubectl apply -f kube-flannel.yaml

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Node-1和node-2节点
安装docker-17、kubelet -> 启动 -> swap -> 加入集群 -> master查看组件 -> 状态是否为redy -> 简单集群结束
  • kubeadm token create –print-join-command #查看master加入到集群的命令
#设置kubectl补全命令
vim /etc/profile
source <(kubectl completion bash)
source /etc/profile 

集群部署成功

确认规则是否开启

查看集群组件是否成功安装


集群初始化成功的标志

节点加入集群

最终状态

记录一次踩坑记录

  • docker-ce-18

  • kubernetes组件版本-1.13.1

  • 关闭防火墙开启转发无ipvs

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

kubeadm init --kubernetes-version=v1.13.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12  #发现coredns一直无法启动很蹊跷

journalctl -f -u kubelet.service

解决方案

  • https://github.com/kubernetes/kubernetes/issues/48798
vim /etc/sysconfig/network-scripts/ifcfg-eth0
NM_CONTROLLED=yes
systemctl restart network

vim /etc/cni/net.d/10-flannel.conflist
{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
  • 下载CNi:https://github.com/containernetworking/plugins/releases

  • cd /etc/cni/net.d/ 解压 -> 重启kubelet -> scp -r 到每一个节点

  • node节点事前准备好镜像文件

  • jurnalctl -f -u kubelet.service #有任何报错时刻去看日志

手动部署K8S

  • CA认证

  • k8s-DNS

  • k8s-node-3

  • k8s-etcd

基础的环境配置

# 1.永久关闭修改/etc/sysconfig/selinux文件设置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

# 2.永久关闭 注释/etc/fstab文件里swap相关的行,1.8以后都需要关闭swap
swapoff -a

# 3.开启forward
# Docker从1.13版本开始调整了默认的防火墙规则禁用了iptables filter表中FOWARD链这样会引起Kubernetes集群中跨Node的Pod无法通信
iptables -P FORWARD ACCEPT

# 4.配置转发相关参数,否则可能会出错
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system

# 5.加载ipvs相关内核模块,如果重新开机,需要重新加载
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe ip_vs_lc
modprobe ip_vs_wlc
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_dh
modprobe ip_vs_fo
modprobe ip_vs_nq
modprobe ip_vs_sed
modprobe ip_vs_ftp
modprobe nf_conntrack_ipv4
lsmod | grep ip_vs


# 6.设置host
cat >>/etc/hosts<<EOF
k8s-master IP
k8s-node1  IP
k8s-node2  IP
EOF

# 7.设置docker的yum源,推荐使用国内的清华大学或者阿里的源
curl -fsSL "https://get.docker.com/" | bash -s -- --mirror Aliyun

# 8.在K8s-v1.0后性能远远高于iptables
yum install ipvsadm -y

安装CFSSL

wget http://k8s-cfssl-ca-1252464731.file.myqcloud.com/cfssl.tgz
mkdir ~/cfssl && tar -xf cfssl.tgz -C ~/cfssl && rm -rf cfssl.tgz
cd ~/ssl && mv * /usr/bin

配置CA-Master

cat >ca-config.json<<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

配置etcd集群的CA

  • 官方文档地址:https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md#dns-discovery
# 1.写入相应的配置信息
cat >etcd-ca-csr.json<<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

# 2.生成 etcd root ca的配置信息
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca

cat >etcd-csr.json<<EOF
{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "192.168.42.140",
      "192.168.42.141",
      "192.168.42.142"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "etcd",
            "OU": "Etcd Security"
        }
    ]
}
EOF

# 3.生成 etcd ca
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

mkdir -pv /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl
ls /etc/etcd/ssl/etcd*.pem

# 4.分发到所有的etcd的节点
cd /etc/etcd && tar cvzf etcd-ssl.tgz ssl
for i in {192.168.42.141,192.168.42.142};do scp ./etcd-ssl.tgz $i:~/;ssh $i "mkdir -pv /etc/etcd && tar xf etcd-ssl.tgz -C /etc/etcd && ls -l /etc/etcd/ssl";done  #for循环{IP,IP}里面填写自己的IP地址

`配置启动脚本

export ETCD_NAME=$(hostname)
export INTERNAL_IP=$(hostname -i | awk '{print $NF}')
export ECTD_CLUSTER='k8s-master=https://192.168.42.140:2380,k8s-node1=https://192.168.42.141:2380,k8s-node2=https://192.168.42.142:2380'
mkdir -pv /data/etcd

cat > /etc/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/data/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/etcd/bin/etcd 
  --name ${ETCD_NAME} 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem 
  --peer-cert-file=/etc/etcd/ssl/etcd.pem 
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem 
  --trusted-ca-file=/etc/etcd/ssl/etcd-ca.pem 
  --peer-trusted-ca-file=/etc/etcd/ssl/etcd-ca.pem 
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 
  --listen-peer-urls https://${INTERNAL_IP}:2380 
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 
  --advertise-client-urls https://${INTERNAL_IP}:2379 
  --initial-cluster-token my-etcd-token 
  --initial-cluster $ECTD_CLUSTER 
  --initial-cluster-state new 
  --data-dir=/data/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# 启动并设置开机启动
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

安装部署etcd

mkdir -pv /server/software/k8s
cd /server/software/k8s
wget http://k8s-cfssl-ca-1252464731.file.myqcloud.com/etcd-v3.3.9-linux-amd64.tar.gz
tar xf etcd-v3.3.9-linux-amd64.tar.gz
mv etcd-v3.3.9-linux-amd64 /usr/local/etcd-v3.3.9
ln -sv /usr/local/etcd-v3.3.9 /usr/local/etcd
cd /usr/local/etcd && mkdir bin && mv etcd etcdctl bin
/usr/local/etcd/bin/etcd --version

部署步骤

  • 1.获取kubernetes二进制包

  • 2.运行master组件

  • 3.运行Node组件

  • 4.查看集群状态

  • 5.启动一个示例

机器准备

  • k8s版本为1.8
主机名 IP 操作系统 安装的功能组件
k8s-master 193.112.171.100 Ubuntu16.04/centos etcd、kube-apiservice、kube-controller-manager、kube-scheduler
k8s-node01 193.112.171.179 Ubuntu16.04/centos kubelet、kube-proxy、docker
k8s-node2 193.112.172.69 Ubuntu16.04/centos kubelet、kube-proxy、docker

下载1.8版本二进制包地址

https://github.com/kubernetes/kubernetes/releases

下载master包

http://k8s-master-18-1252464731.cosgz.myqcloud.com/master.zip

master节点的部署

环境基础配置

mkdir /opt/kubernetes/{bin,cfg}
chmod +x *
mv bube* /opt/kubernetes/bin/

安装etcd

yum -y install etcd tree
vim /etc/defaults/etcd
    ETCD_DATA_DIR="/var/lib/etcd/default"
    ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
    ETCD_ADVERTISE_CLIENT_URLS=HTTP://0.0.0.0:2379
systemctl restart etcd
netstat -lnput

apiserver配置文件

#!/bin/bash

MASTER_ADDRESS=${1:-"10.135.167.207"} #指定master地址
ETCD_SERVERS=${2:-"http://10.135.167.207:2379"} #etcd的监听端口
SERVICE_CLUSTER_IP_RANGE=${3:-"10.10.10.0/24"} #监控集群的网段
ADMISSION_CONTROL=${4:-""}

cat <<EOF >/opt/kubernetes/cfg/kube-apiserver
# 启用日志标准错误 
KUBE_LOGTOSTDERR="--logtostderr=true"

# 日志级别
KUBE_LOG_LEVEL="--v=4"

# Etcd服务地址
KUBE_ETCD_SERVERS="--etcd-servers=${ETCD_SERVERS}"

# 指定Etcd版本,默认etcd3
KUBE_ETCD_VERSION="--storage-backend=etcd2"

# API服务监听地址
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# API服务监听端口
KUBE_API_PORT="--insecure-port=8080"

# Kubelet端口
NODE_PORT="--kubelet-port=10250"

# 对集群中成员提供API服务地址
KUBE_ADVERTISE_ADDR="--advertise-address=${MASTER_ADDRESS}"

# 允许容器请求特权模式,默认false
KUBE_ALLOW_PRIV="--allow-privileged=false"

# 集群分配的IP范围
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=${SERVICE_CLUSTER_IP_RANGE}"

# 控制资源进入集群, 默认AlwaysAdmit
# Comma-delimited list of:
#   LimitRanger, AlwaysDeny, SecurityContextDeny, NamespaceExists,
#   NamespaceLifecycle, NamespaceAutoProvision, AlwaysAdmit,
#   ServiceAccount, DefaultStorageClass, DefaultTolerationSeconds, ResourceQuota
KUBE_ADMISSION_CONTROL="--admission-control=${ADMISSION_CONTROL}"
EOF

KUBE_APISERVER_OPTS="   ${KUBE_LOGTOSTDERR}         
                        ${KUBE_LOG_LEVEL}           
                        ${KUBE_ETCD_SERVERS}        
                        ${KUBE_ETCD_VERSION}        
                        ${KUBE_API_ADDRESS}         
                        ${KUBE_API_PORT}            
                        ${NODE_PORT}                
                        ${KUBE_ADVERTISE_ADDR}      
                        ${KUBE_ALLOW_PRIV}          
                        ${KUBE_SERVICE_ADDRESSES}"

cat <<EOF >/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver ${KUBE_APISERVER_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver

启动apiserver

[root@master ~]# bash apiserver.sh 10.135.167.207 http://10.135.167.207:2379
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
[root@master ~]# journalctl -u kube-apiserver   #-u指定程序名称,查看运行日志

scheduler配置文件

#!/bin/bash

MASTER_ADDRESS=${1:-"10.135.167.207"} #修改地址

cat <<EOF >/opt/kubernetes/cfg/kube-scheduler
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=4"
KUBE_MASTER="--master=${MASTER_ADDRESS}:8080"
KUBE_LEADER_ELECT="--leader-elect"
# 其他参数
KUBE_SCHEDULER_ARGS=""

EOF

KUBE_SCHEDULER_OPTS="   ${KUBE_LOGTOSTDERR}     
                        ${KUBE_LOG_LEVEL}       
                        ${KUBE_MASTER}          
                        ${KUBE_LEADER_ELECT}    
                        ${KUBE_SCHEDULER_ARGS}"

cat <<EOF >/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler ${KUBE_SCHEDULER_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler


启动scheduler

[root@master ~]# ps -ef |grep scheduler
[root@master ~]# journalctl -u scheduler

controller-manager配置文件

#!/bin/bash

MASTER_ADDRESS=${1:-"10.135.167.207"} #修改为管理节点的地址

cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=4"
KUBE_MASTER="--master=${MASTER_ADDRESS}:8080"

# 在执行主循环之前,先选举一个leader。高可用性运行组件时启用此功能,默认true 
KUBE_LEADER_ELECT="--leader-elect"
EOF

KUBE_CONTROLLER_MANAGER_OPTS="  ${KUBE_LOGTOSTDERR} 
                                ${KUBE_LOG_LEVEL}   
                                ${KUBE_MASTER}      
                                ${KUBE_LEADER_ELECT}"

cat <<EOF >/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager ${KUBE_CONTROLLER_MANAGER_OPTS}
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager

启动管理节点

[root@master ~]# ./controller-manager.sh 10.135.167.207

查看节点是否启动

node节点的部署

mkdir -p /opt/kubernetes/{bin,cfg}
mv bukbelet kube-proxy /opt/kubernetes/bin/
swapoff -a #关闭缓冲区
#!/bin/bash

MASTER_ADDRESS=${1:-"10.135.167.207"} #管理节点地址
NODE_ADDRESS=${2:-"10.186.10.195"} #自己在管理节点上显示的地址
DNS_SERVER_IP=${3:-"10.135.167.207"} #DNS地址
DNS_DOMAIN=${4:-"cluster.local"}
KUBECONFIG_DIR=${KUBECONFIG_DIR:-/opt/kubernetes/cfg}

cat <<EOF > "${KUBECONFIG_DIR}/kubelet.kubeconfig"
apiVersion: v1
kind: Config
clusters:
  - cluster:
      server: http://${MASTER_ADDRESS}:8080/
    name: local
contexts:
  - context:
      cluster: local
    name: local
current-context: local
EOF

cat <<EOF >/opt/kubernetes/cfg/kubelet
# 启用日志标准错误 
KUBE_LOGTOSTDERR="--logtostderr=true"

# 日志级别 
KUBE_LOG_LEVEL="--v=4"

# Kubelet服务IP地址 
NODE_ADDRESS="--address=${NODE_ADDRESS}"

# Kubelet服务端口。弃用
# NODE_PORT="--port=10250"

# 自定义节点名称
NODE_HOSTNAME="--hostname-override=${NODE_ADDRESS}"

# kubeconfig路径,指定连接API服务器
KUBELET_KUBECONFIG="--kubeconfig=${KUBECONFIG_DIR}/kubelet.kubeconfig"

# 允许容器请求特权模式,默认false
KUBE_ALLOW_PRIV="--allow-privileged=false"

# DNS信息
KUBELET_DNS_IP="--cluster-dns=${DNS_SERVER_IP}"

启动

Kubernetes容器集群管理》有861个想法

  1. My brother recommended I would possibly like this
    website. He was once entirely right. This put up actually made my day.
    You cann’t believe just how a lot time I had spent for this information! Thank you!

  2. Singing worsһip sοngs is good however that?s not the one method to ᴡoгship.?
    Daddy said, possibly to make Lɑгry stoⲣ sіngіng.
    ?There are many ways to worship.

  3. Hello! I just wanted to ask if you ever have any trouble with hackers?
    My last blog (wordpress) was hacked and I ended up losing many months of hard work
    due to no backup. Do you have any solutions to protect against
    hackers?

  4. Hey would you mind letting me know which web host you’re working with?
    I’ve loaded your blog in 3 completely different internet browsers and I
    must say this blog loads a lot faster then most.
    Can you recommend a good internet hosting
    provider at a reasonable price? Thank you, I appreciate it!

  5. Woah! I’m really enjoying the template/theme of
    this website. It’s simple, yet effective. A lot of times it’s tough
    to get that “perfect balance” between superb usability and visual appeal.
    I must say you have done a great job with this. Also, the blog
    loads extremely fast for me on Internet explorer.
    Exceptional Blog!

  6. I am really inspired along with your writing talents and also with the layout to your blog.
    Is that this a paid subject or did you modify it yourself?

    Anyway keep up the excellent quality writing, it’s rare to peer a great weblog like this one nowadays..

  7. Howdy, i read your blog occasionally and i own a similar one and i
    was just curious if you get a lot of spam remarks?
    If so how do you stop it, any plugin or anything you can suggest?
    I get so much lately it’s driving me mad so
    any support is very much appreciated.

  8. I’m pretty pleased to find this website. I need to to thank
    you for ones time for this particularly fantastic read!!
    I definitely enjoyed every bit of it and i also have
    you book-marked to look at new stuff on your blog.

  9. An impressive share! I have just forwarded this onto a co-worker
    who was doing a little research on this. And he actually bought me lunch simply because I found it for him…
    lol. So allow me to reword this…. Thank YOU for the meal!!
    But yeah, thanks for spending the time to talk about this topic here on your internet site.

  10. Hiya! Quick question that’s totally off topic. Do you know how to make your site mobile friendly?
    My web site looks weird when viewing from my iphone. I’m trying to find a theme
    or plugin that might be able to fix this problem. If you have any recommendations,
    please share. Thanks!

  11. you’re in reality a just right webmaster. The site loading velocity is incredible.

    It sort of feels that you’re doing any distinctive trick.
    Also, The contents are masterwork. you have performed a
    excellent task on this topic!

  12. hi!,I love your writing very so much! percentage we keep in touch extra approximately your article on AOL? I require a specialist on this house to unravel my problem. May be that’s you! Having a look ahead to see you.

  13. Great post. I was checking constantly this weblog and I’m impressed!
    Extremely useful info particularly the closing section 🙂 I care for such info a lot.
    I used to be looking for this particular information for
    a long time. Thank you and best of luck.

  14. With havin so much content and articles do you ever run into any issues of plagorism or copyright violation?
    My blog has a lot of completely unique content I’ve either written myself or
    outsourced but it appears a lot of it is popping it up all over the web without my agreement.

    Do you know any ways to help prevent content from being ripped off?
    I’d genuinely appreciate it.

  15. Do you mind if I quote a few of your posts as long as I provide credit and sources back to
    your weblog? My blog site is in the very same niche as yours and my visitors would genuinely
    benefit from some of the information you provide here. Please let me
    know if this alright with you. Appreciate it!

  16. Good day! This is my first comment here so I just wanted to give a
    quick shout out and tell you I truly enjoy reading through your articles.
    Can you recommend any other blogs/websites/forums that go over the same topics?
    Many thanks!

  17. Hello I am so grateful I found your website, I really found you by mistake, while I was researching on Bing for something else, Nonetheless I am here now and would just like to say cheers for a tremendous post and a all round
    entertaining blog (I also love the theme/design), I don’t have time to read it all at the moment but I have book-marked it and also added
    in your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the
    excellent work.

  18. Do you have a spam problem on this site; I also
    am a blogger, and I was wanting to know your situation; we have
    created some nice practices and we are looking to swap techniques with
    others, please shoot me an email if interested.

  19. I’ve been exploring for a bit for any high-quality articles or weblog posts in this kind of house .
    Exploring in Yahoo I finally stumbled upon this web site.
    Reading this info So i’m satisfied to express that I’ve a very good uncanny feeling I discovered
    exactly what I needed. I most unquestionably will make certain to do not forget this website
    and provides it a glance regularly.

  20. Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my
    newest twitter updates. I’ve been looking for a plug-in like this for quite some time
    and was hoping maybe you would have some experience with
    something like this. Please let me know if you
    run into anything. I truly enjoy reading your blog and I look forward to your new updates.

  21. Hey I know this is off topic but I was wondering if you knew
    of any widgets I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for quite some time and was hoping
    maybe you would have some experience with something like this.
    Please let me know if you run into anything. I truly enjoy reading your blog
    and I look forward to your new updates.

    1. First you have to have your own VPS server, then you have to have your own LNMP environment.Then build a wordpress and start your writing

  22. What i don’t realize is in fact how you’re not actually a lot more well-preferred than you might be now.
    You are very intelligent. You recognize thus considerably in the case of
    this subject, made me in my view imagine it from numerous
    varied angles. Its like women and men don’t seem to be fascinated until it is
    one thing to do with Woman gaga! Your own stuffs outstanding.
    All the time maintain it up!

  23. Hmm it seems like your site ate my first comment (it was super long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog.
    I too am an aspiring blog blogger but I’m still new to the whole thing.
    Do you have any suggestions for first-time blog writers?
    I’d definitely appreciate it.

  24. Pingback: Homepage
  25. Hello! I just wanted to ask if you ever have any issues with
    hackers? My last blog (wordpress) was hacked and I ended up losing months of
    hard work due to no backup. Do you have any methods to prevent hackers?

    1. Web page reinforcement or set more complex password prohibit some of the less commonly used port, set firewall rules, I hope to help you

  26. Greetings, I do believe your blog might be having internet browser compatibility problems.
    When I look at your website in Safari, it looks fine however, if opening in Internet Explorer,
    it’s got some overlapping issues. I just wanted to give you a quick heads up!
    Aside from that, fantastic website!

  27. Great work! That is the kind of information that are supposed to be shared across
    the web. Disgrace on Google for not positioning this post upper!
    Come on over and consult with my web site . Thanks =)

  28. Great blog here! Also your web site loads up very fast!
    What web host are you using? Can I get your affiliate link to your host?
    I wish my site loaded up as quickly as yours lol

  29. Greetings from Colorado! I’m bored at work so I decided to check out your site
    on my iphone during lunch break. I enjoy the info
    you present here and can’t wait to take a look when I get home.
    I’m surprised at how fast your blog loaded on my mobile ..

    I’m not even using WIFI, just 3G .. Anyways, amazing site!

  30. Can I simply just say what a relief to find somebody who genuinely understands what they are
    talking about on the internet. You definitely know how to bring an issue to light and make it important.
    A lot more people have to check this out and understand this
    side of your story. I was surprised that you are not more popular because you surely have the
    gift.

  31. This design is wicked! You certainly know how to keep a reader
    entertained. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!)
    Wonderful job. I really loved what you had to say, and more than that,
    how you presented it. Too cool!

  32. tingkah-laku fiil memakai jasa seo ini , sebab jasa seo ini jasa pendusta
    iming iming di situs nya hanya bujukan dan tak mampu di pertanggung jawabkan, seluruhnya dilakukan OLEH NANDO DAN amat tak cendekiawan

  33. Thanks, I’ve just been searching for informatiօn about this
    topic for a while аnd yours is the greatest I’ve founjd out tkll
    now. Howeѵer, what concerning the ƅottom line?
    Are you sure concerning the souгce?

  34. I do accept as tгue with all of the ideas you һave
    presented for your post. Tһey arе very convincing and сann ceгtainly work.
    Nonetheless, tһe posts aare tooo brief foor newbies. Could you please
    lengthen them a bit from subsequent time? Thanks for the post.

  35. Somebody essentially assist to make severely articles I might state. That is the first time I frequented your website page and thus far? I amazed with the research you made to make this particular post amazing. Fantastic task!

  36. I’ll immediately seize your rss feed as I can not in finding your email subscription link or e-newsletter service.

    Do you have any? Please let me recognize in order that I could subscribe.

    Thanks.

  37. Thanks for any other informative website. The
    place else could I get that type of information written in such a perfect way?

    I have a undertaking that I am simply now working on, and I’ve been on the look out for such info.

  38. Thanks for one’s marvelous posting! I quite enjoyed reading it,
    you will be a great author. I will make sure to bookmark your
    blog and will often come back later in life. I want to encourage one to continue your
    great writing, have a nice holiday weekend!

  39. I used to be recommended this blog by my cousin. I’m not positive
    whether this publish is written by him as nobody else understand such unique approximately
    my problem. You are incredible! Thank you!

  40. Hmm it seems like your blog ate my first comment (it was super long)
    so I guess I’ll just sum it up what I had written and say, I’m thoroughly
    enjoying your blog. I too am an aspiring blog writer but
    I’m still new to everything. Do you have any tips and hints for beginner
    blog writers? I’d certainly appreciate it.

    1. First of all, you have to ask what you are good at, you can use it as a record of life notes, you can, most need, you must have a site and start writing your story

  41. Does your blog have a contact page? I’m having problems locating it but, I’d like
    to send you an email. I’ve got some recommendations for your blog
    you might be interested in hearing. Either way, great website and I
    look forward to seeing it expand over time.

  42. Hello, I believe your site might be having web browser compatibility issues.
    Whenever I take a look at your web site in Safari, it looks fine however, when opening in I.E., it’s
    got some overlapping issues. I just wanted to give you a quick heads up!
    Aside from that, fantastic website!

  43. I was wondering if you ever thought of changing the layout of your website?
    Its very well written; I love what youve got to say.
    But maybe you could a little more in the way of content so
    people could connect with it better. Youve got an awful lot of text for only having 1
    or 2 images. Maybe you could space it out better?

  44. Hello, Neat post. There is a problem together with your web
    site in internet explorer, may test this?
    IE nonetheless is the market leader and a big section of people will leave out your wonderful writing due
    to this problem.

  45. I’m extremely impressed along with your writing abilities and also with the structure on your weblog.
    Is this a paid subject matter or did you customize it your self?
    Either way keep up the nice high quality writing, it’s uncommon to see a nice
    blog like this one these days..

  46. I was curious if you ever considered changing the page layout of your blog?

    Its very well written; I love what youve got to say.
    But maybe you could a little more in the way of content so people could
    connect with it better. Youve got an awful lot of text for only having 1 or two
    images. Maybe you could space it out better?

  47. I believe everything said was actually very reasonable.
    However, think on this, what if you wrote a catchier post title?

    I ain’t saying your content isn’t solid., however suppose you added
    something that grabbed folk’s attention? I mean Kubernetes容器集群管理 – 董的博客 is a little vanilla.
    You might glance at Yahoo’s front page and see how they create news titles to get people to click.
    You might try adding a video or a picture or two to get readers interested about
    what you’ve written. Just my opinion, it would bring your website
    a little livelier.
    라인티비
    라인티비
    스포츠중계
    스포츠중계
    스포츠중계

  48. Its like you read my mind! You seem to know so much about this, like you wrote the book in it or
    something. I think that you could do with a few pics to drive the message home a little
    bit, but other than that, this is wonderful blog.
    A great read. I will definitely be back.

  49. I do agree with all the ideas you have offered
    on your post. They are really convincing and will certainly work.

    Nonetheless, the posts are too brief for starters.

    Could you please prolong them a bit from next time?
    Thanks for the post.

  50. Have you ever thought about including a little bit more than just your articles?
    I mean, what you say is valuable and everything. But just imagine
    if you added some great pictures or video clips to give your posts more, “pop”!
    Your content is excellent but with images and video clips, this blog could definitely be one of the very best in its field.
    Fantastic blog!

  51. of course like your web site but you need to take
    a look at the spelling on several of your posts. Several of them are rife with spelling issues and
    I find it very troublesome to tell the truth on the other hand I’ll definitely
    come back again.

  52. I know thi website provides quality bassed content and other materіal, is tһere
    any other website which presents thesee kinds of information in qualitʏ?

  53. After looking at a handful of the blog posts on your website, I truly like your technique of
    writing a blog. I book-marked it to my bookmark webpage list and will be checking back soon.
    Please check out my website too and tell me what you think.

  54. Howdy, I believe your website could possibly be having internet browser compatibility problems.
    Whenever I look at your site in Safari, it looks fine but
    when opening in I.E., it’s got some overlapping issues.

    I just wanted to provide you with a quick heads up!
    Other than that, great site!

  55. Someone necessarily lend a hand to make severely posts I’d state.
    That is the very first time I frequented your web
    page and up to now? I amazed with the analysis you made to make this particular put up incredible.

    Great task!

  56. Pretty component ᧐f content. I jᥙst stumbled սpon уߋur web site аnd inn accession capital tօ asser that I get іn fact loved account үouг blog posts.

    Аnyway I’ll be subscribing on yoᥙr feeds ɑnd even I achievement you access
    persistently fɑst.

  57. Hey There. I found your blog using msn. This is a really well written article.
    I’ll be sure to bookmark it and come back to read more of your useful information. Thanks for the post.
    I’ll certainly comeback.

  58. I have read several just right stuff here.
    Definitely worth bookmarking for revisiting. I surprise how much effort you set to create this kind of excellent informative site.

  59. Hi I am so excited I found your blog, I really found
    you by mistake, while I was browsing on Aol for something else, Anyways I am here now and would just like to say many thanks for a fantastic post and a all round interesting blog
    (I also love the theme/design), I don’t have time to
    read it all at the minute but I have saved
    it and also added your RSS feeds, so when I have time I will
    be back to read a great deal more, Please do keep up the awesome job.

  60. Its like you read my mind! Yoou seem to know so much aout this, like you wrote the book in it or something.
    I think that you could do with some pics to drive the message home a little bit, but other than that, this is wonderful blog.
    An excellent read. I’ll certainly be back.

  61. Do you have a spam issue on this site; I also am a blogger,
    and I was curious about your situation; many of us have
    created some nice practices and we are looking to swap techniques with others, be sure to shoot
    me an email if interested.

  62. I rarely drop comments, however i did a few searching and wound up here
    Kubernetes容器集群管理 – 董的博客. And I
    do hve 2 questions for you if it’s allright. Is it just me or
    does it give thhe impression luke a few of the remarks look as if they are
    left by brain dead folks? 😛 And, if you are writing at addiional online social sites, I’d like to follow anything fresh you have to post.
    Could you make a list of every one of your public sites
    like our twitter feed, Facebook page or linkedin profile?

  63. I know this if off topic but I’m looking into starting my own blog and was curious what all
    is required to get set up? I’m assuming having a blog like yours would cost a pretty penny?
    I’m not very internet smart so I’m not 100% positive.
    Any tips or advice would be greatly appreciated. Appreciate it

  64. Hey would you mind sharing which blog platform you’re using?
    I’m planning to start my own blog soon but I’m having a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different then most blogs and I’m looking for something completely unique.
    P.S My apologies for being off-topic but I had to ask!

  65. When I originally commented I clicked the “Notify me when new comments are added”
    checkbox and now each time a comment is added I get four emails with the
    same comment. Is there any way you can remove people from
    that service? Thank you!

  66. Yes, even though everything that listing down, you still ought to sit and compose an entire response, the
    same way you’d write any essay. This will present you with sufficient time and employ to brainstorm and be sure what you really are currently talking about is applicable and what
    you would like to show in. If you say because continuously, the only
    thing your reader will likely be mindful of happens because – it’ll stifle your argument and it’s also towards the top of their list of items you should avoid within your academic
    work.

  67. Howdy! I could have sworn I’ve been to this site before but after checking
    through some of the post I realized it’s new to me.

    Anyhow, I’m definitely delighted I found it and I’ll be book-marking and checking back often!

  68. I think this is one of the most significant info for me.

    And i am glad reading your article. But wanna remark on few general things,
    The site style is great, the articles is really nice : D. Good job, cheers

  69. Wonderful beat ! I would like to apprentice while
    you amend your website, how could i subscribe for a blog web site?
    Thhe acccount aided me a acceptable deal. I had been a little bit acquainted of
    this your broadcast offered bright clear idea

  70. What i do not understood is if truth be told how you’re now not actually
    a lot more well-favored than you might be right now. You are very intelligent.
    You know thus considerably on the subject of
    this topic, produced me personally believe it from a lot
    of varied angles. Its like women and men aren’t involved
    except it’s one thing to accomplish with Woman gaga! Your individual stuffs
    nice. All the time maintain it up!

  71. Have you ever considered writing an ebook or guest
    authoring on other sites? I have a blog based on the same
    information you discuss and would love to have you share some stories/information. I know
    my visitors would appreciate your work. If you’re even remotely interested, feel free to
    shoot me an e mail.

  72. Hey There. I discovered your blog using msn. That is a very neatly written article.
    I will make sure to bookmark it and return to read extra of your helpful information.
    Thanks for the post. I’ll definitely comeback.

  73. Great items from you, man. I have be aware your stuff previous to and you’re simply too
    excellent. I actually like what you’ve bought here, certainly like
    what you are saying and the way during which you assert it.
    You’re making it enjoyable and you continue to take care of to keep it
    sensible. I can not wait to learn much more
    from you. That is actually a terrific site.

  74. Please let me know if you’re looking for a article writer for your blog.
    You have some really great articles and I think
    I would be a good asset. If you ever want to take some of the load off, I’d love to write some material for your blog
    in exchange for a link back to mine. Please send me an email if interested.
    Thanks!

  75. Greate post. Keep posting such kind of information on your site.
    Im really impressed by your blog.
    Hi there, You’ve done an incredible job. I will certainly digg it and individually
    suggest to my friends. I’m confident they’ll be benefited from this web
    site.

  76. I love your blog.. very nice colors & theme. Did you create this website yourself or did you
    hire someone to do it for you? Plz answer back as I’m looking to design my own blog and would like to find out where u got this from.
    thanks a lot

  77. It’s perfect time to make some plans for the future and it’s time to
    be happy. I’ve read this submit and if I may just I wish to counsel you few fascinating issues or advice.
    Maybe you could write subsequent articles relating to this article.

    I desire to read even more things approximately it!

  78. Appreciating the commitment you put into your site and in depth information you
    present. It’s good to come across a blog every once in a while that
    isn’t the same outdated rehashed information. Great read!
    I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.

  79. Hello there! Quick question that’s completely off
    topic. Do you know how to make your site mobile friendly?

    My website looks weird when viewing from my iphone 4. I’m trying to find a template or plugin that
    might be able to correct this issue. If you have any recommendations, please share.
    Thank you!

  80. I loved as much as you’ll receive carried
    out right here. The sketch is tasteful, your authored subject matter
    stylish. nonetheless, you command get bought an shakiness
    over that you wish be delivering the following. unwell unquestionably
    come further formerly again as exactly the same nearly
    very often inside case you shield this hike.

  81. What’s Taking place i am new to this, I stumbled upon this I’ve discovered It absolutely useful and it
    has aided me out loads. I hope to contribute & assist other
    customers like its helped me. Great job.

  82. Hey there, I think your site might be having browser compatibility issues.
    When I look at your blog site in Ie, it looks fine but when opening in Internet Explorer, it has
    some overlapping. I just wanted to give you a quick heads up!

    Other then that, terrific blog!

  83. I’m not that much of a internet reader to be honest but your blogs really nice, keep it
    up! I’ll go ahead and bookmark your site to come back down the road.
    All the best

  84. We are a group of volunteers and opening a new scheme in our community.

    Your website provided us with helpful information to work on. You have done an impressive
    task and our whole group will probably be thankful to
    you.

  85. I would like to take the opportunity of saying thanks to
    you for your professional direction I have always enjoyed viewing your site.
    I will be looking forward to the particular commencement of my university
    research and the whole groundwork would never have been complete without visiting your
    blog. If I might be of any help to others, I’d personally be ready to help by way of what I have
    discovered from here.

  86. Excellent site you have here but I was curious about if you knew
    of any forums that cover the same topics discussed
    in this article? I’d really love to be a part of online community where I
    can get suggestions from other knowledgeable individuals that share the same interest.
    If you have any recommendations, please let me know.

    Many thanks!

  87. With havin so much written content do you ever run into any
    problems of plagorism or copyright violation? My
    website has a lot of completely unique content I’ve either
    authored myself or outsourced but it seems a lot of it
    is popping it up all over the web without my permission. Do you know any methods to help protect against content from being
    stolen? I’d genuinely appreciate it.

  88. I was very pleased to find this web-site.I wanted to thanks for your time for
    this wonderful read!! I definitely enjoying every little bit of it and I have you bookmarked to
    check out new stuff you blog post.

  89. Good day I am so glad I found your blog page, I really found you by mistake, while
    I was searching on Digg for something else, Anyhow I am here now and would just like to say many
    thanks for a incredible post and a all round thrilling blog (I also love the theme/design), I
    don’t have time to look over it all at the minute but
    I have bookmarked it and also added in your RSS feeds, so when I have time I will be back to read a
    great deal more, Please do keep up the great job.

  90. I have been surfing online more than 3 hours today, yet I never found any interesting article
    like yours. It is pretty worth enough for me. In my view,
    if all site owners and bloggers made good content as you did, the net
    will be much more useful than ever before.

  91. That is very attention-grabbing, You’re an excessively professional blogger.

    I’ve joined your feed and look ahead to in search of more of your fantastic post.
    Additionally, I’ve shared your site in my social networks

  92. Hi, i think that i saw you visited my website thus i came to “return the favor”.I am attempting to find things to improve
    my website!I suppose its ok to use a few of your ideas!!

  93. I want to move out don’t make enough. What can I do?
    I’m 20 years old in Vegas and haven’t gone to college but just working a low pay job, I want to move out but I already have my car insurance to pay for so wouldn’t be able to afford place for myself.

    Even without car insurance I wouldn’t be able to afford it because of my pay.

    Are there jobs here I can find that… show more Make people pay for watching you eat, rubbing your feat, or satisfying
    there weird fetishes. Don t think I m weird
    its a real thing that people will pay you for.
    or you know, DON T MOVE OUT.

  94. Today, I went to the beachfront with my children. I found a sea shell and gave it
    to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She placed the shell to her ear
    and screamed. There was a hermit crab inside and
    it pinched her ear. She never wants to go back!
    LoL I know this is totally off topic but I had to tell someone!

  95. I have to thank you for the efforts you have put in penning this
    website. I am hoping to view the same high-grade content from you later on as well.

    In truth, your creative writing abilities has inspired me
    to get my own blog now 😉

  96. Excellent weblog right here! Also your site rather a lot up very fast!

    What host are you the usage of? Can I am getting your affiliate link on your
    host? I want my site loaded up as fast as yours lol

  97. Right here is the right blog for anyone who would like to understand this topic.
    You realize a whole lot its almost hard to argue with
    you (not that I actually will need to…HaHa). You certainly
    put a fresh spin on a topic that has been discussed for many years.
    Excellent stuff, just excellent!

  98. excellent submit, very informative. I wonder why the
    other specialists of this sector don’t notice this.
    You must proceed your writing. I am sure, you’ve a great readers’ base
    already!

  99. This work reveals a sort of poetic mood and everyone
    would simply be attracted by it. If this is a question of yours
    too, then you definitely should learn concerning the best ways to procure such things.
    Matisse also had become the king with the Fauvism and was famous inside the art circle.

  100. Pretty nice post. I just stumbled upon your weblog
    and wanted to say that I’ve truly enjoyed surfing
    around your blog posts. In any case I’ll be subscribing to your rss feed and I hope you write again soon!

  101. Discover how to watch free movies online without download or enrollment, also watch movies without the annoying advertising on putlocker47.
    com easily on the web, just watch this video to learn:

  102. I’m no longer certain the place you are getting your info,
    but good topic. I needs to spend some time learning much more
    or figuring out more. Thank you for wonderful info I used to
    be looking for this info for my mission.

  103. Generally I do not learn post on blogs, but I would like to
    say that this write-up very forced me to check out and do so!

    Your writing style has been amazed me. Thanks, very nice post.

  104. Hello there! This blog post couldn’t be written much better!
    Going through this post reminds me of my previous
    roommate! He constantly kept talking about this. I will send this information to him.

    Fairly certain he’s going to have a good read.
    Thank you for sharing!

  105. Wow, marvelous weblog layout! How lengthy have you been running a blog for?

    you make running a blog glance easy. The total glance of your website is excellent, as
    neatly as the content material!

  106. Why is there a double standard with interracial friends?
    Ok at a school with a majority of white people, a white guy with mainly black friends is deemed to be cool among his mostly white peers, but if they circumstances were flipped and it was an all black school and a black guy with
    mainly white friends he is considered weird, There is so much more I want to know as…
    show more Sure

  107. I am really loving the theme/design of your site.
    Do you ever run into any internet browser compatibility issues?

    A small number of my blog readers have complained about
    my blog not working correctly in Explorer but looks great in Safari.
    Do you have any suggestions to help fix this issue?
    istanbul escort
    şirinevler escort
    taksim escort
    mecidiyeköy escort
    şişli escort
    istanbul escort
    kartal escort
    pendik escort
    tuzla escort
    kurtköy escort

  108. Hey! This is my 1st comment here so I just wanted to give a quick shout out and say I truly
    enjoy reading through your blog posts. Can you recommend any
    other blogs/websites/forums that deal with the same subjects?
    Appreciate it!

  109. Howdy just wanted to give you a brief heads up and let you know a few
    of the images aren’t loading correctly. I’m not sure why but I think its
    a linking issue. I’ve tried it in two different internet browsers and both show the
    same outcome.

  110. We’re a group of volunteers and starting a new scheme in our community.
    Your web site provided us with valuable information to work on. You have done an impressive job and
    our whole community will be grateful to you.

  111. Good – I should certainly pronounce, impressed with
    your website. I had no trouble navigating through
    all tabs as well as related information ended up being truly simple
    to do to access. I recently found what I hoped for before you know
    it in the least. Quite unusual. Is likely to appreciate it for those who add forums or something,
    web site theme . a tones way for your customer to
    communicate. Excellent task.

  112. (3^2x+3 – 28)(3^x-1 + 9) = 0? ((3^2)x + 3 – 28)(3^x – 1
    + 9) = 0 (9x – 25)(3^x + 8) = 0 9x – 25 = 0 or 3^x + 8 = 0 9x = 25 or 3^x = -8
    x = 25/9 or x = (ln(8) + ipin) / ln(3), for any odd number n

  113. Just want to say your article is as astounding.
    The clarity in your post is just nice and i can assume you are an expert
    on this subject. Fine with your permission let
    me to grab your feed to keep up to date with forthcoming post.
    Thanks a million and please continue the gratifying work.

  114. Thanks for one’s marvelous posting! I truly enjoyed reading
    it, you could be a great author.I will be sure to bookmark your blog and will eventually come back very soon. I want to encourage continue your great writing, have a nice morning!

  115. Attractive element of content. I simply stumbled upon your site and in accession capital to
    claim that I acquire in fact loved account your weblog posts.
    Any way I will be subscribing for your feeds and even I achievement
    you access constantly rapidly.

  116. Just wish to say your article is as astonishing.
    The clearness in your post is just great and i can assume
    you are an expert on this subject. Fine with your permission let me
    to grab your feed to keep updated with forthcoming
    post. Thanks a million and please carry on the enjoyable work.

  117. Wonderful goods from you, man. I’ve understand your stuff previous to and you are just extremely excellent.
    I really like what you have acquired here, really
    like what you are stating and the way in which
    you say it. You make it entertaining and you still take care of to keep it smart.
    I can not wait to read far more from you. This is really
    a wonderful website.

  118. This is very interesting, You are a very skilled blogger.

    I have joined your rss feed and look forward to
    seeking more of your excellent post. Also, I have shared your
    web site in my social networks!

  119. This is the right blog for everyone who wishes to find out about this topic.
    You understand a whole lot its almost hard to argue with you (not that I actually would want to…HaHa).
    You certainly put a fresh spin on a topic that’s
    been written about for decades. Wonderful stuff,
    just excellent!

  120. Very good blog! Do you have any hints for aspiring writers?
    I’m planning to start my own website soon but I’m a little
    lost on everything. Would you advise starting with a free platform
    like WordPress or go for a paid option? There are
    so many choices out there that I’m completely overwhelmed ..

    Any ideas? Appreciate it!

  121. Hi there very cool website!! Man .. Beautiful ..

    Superb .. I will bookmark your web site and take the feeds additionally?

    I’m happy to search out a lot of useful info here in the put up, we’d like
    develop extra strategies in this regard, thank
    you for sharing. . . . . .

  122. I tend not to leave a comment, but I looked at a bunch of comments on this
    page Kubernetes容器集群管理 – 董的博客. I actually do have a couple of questions for you if it’s allright.
    Could it be only me or do a few of the responses
    look like they are coming from brain dead people? 😛 And, if
    you are posting at additional online social sites, I’d like to
    follow anything new you have to post. Would you list of every one
    of all your public pages like your linkedin profile, Facebook page or twitter
    feed?

  123. Hi, I do believe this is an excellent site.
    I stumbledupon it 😉 I’m going to return once again since i have book-marked it.
    Money and freedom is the greatest way to change, may
    you be rich and continue to guide other people.

  124. I like the helpful information you provide in your articles.

    I will bookmark your weblog and check again here frequently.

    I am quite sure I will learn many new stuff right here!
    Good luck for the next!

  125. What nationality am I? My grqndparents came to England fight in ww1 and
    got a british passport (as you might know Pakistan was
    once part of India and India helped fight in ww1) Then my parents came england (born in Pakistan) in 1990 who
    then had me in 2001 then I movied to Australia at the age of 9 and then movied to Canada at age 14…
    show more You are English unless you have changed your nationality to Canadian.

  126. Hey I am so glad I found your blog page, I really found you by error,
    while I was searching on Yahoo for something else, Anyhow I am here now and would just like to say thank
    you for a tremendous post and a all round entertaining blog
    (I also love the theme/design), I don’t have time to look over
    it all at the moment but I have bookmarked
    it and also added your RSS feeds, so when I have time I will be
    back to read a lot more, Please do keep up the excellent
    work.

  127. hey there and thank you for your info – I have definitely picked up anything new from
    right here. I did however expertise some technical issues
    using this website, as I experienced to reload the site lots of times previous to I could get it to
    load correctly. I had been wondering if your hosting is OK?

    Not that I am complaining, but slow loading instances times will often affect your placement
    in google and can damage your high quality score if ads and marketing with Adwords.
    Well I’m adding this RSS to my e-mail and could look out for a lot more of your respective fascinating content.
    Ensure that you update this again very soon.

  128. I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site.!!!!

  129. I’ve been browsing on-line more than three hours nowadays, but
    I by no means found any interesting article like yours.
    It is pretty worth sufficient for me. In my view, if all web owners and bloggers made
    good content as you probably did, the web will likely be a lot
    more useful than ever before.

  130. Needed to create you a very small observation so as to give
    many thanks over again considering the gorgeous knowledge you have discussed
    here. It’s so extremely open-handed of you to make freely precisely what a few
    people could possibly have advertised for an electronic book to get some money on their own,
    most notably since you might well have done it if you desired.
    The strategies additionally worked to become easy way to
    comprehend most people have a similar dream just as my own to understand somewhat more
    concerning this problem. I am certain there are some more pleasurable
    situations up front for many who examine your blog.

  131. Have you ever considered about adding a little bit more than just your articles?

    I mean, what you say is important and all. However imagine if you added some
    great visuals or videos to give your posts more, “pop”!

    Your content is excellent but with pics and videos, this site could definitely be
    one of the very best in its field. Awesome
    blog!

  132. I loved as much as you will receive carried out right here.
    The sketch is attractive, your authored material stylish.
    nonetheless, you command get got an shakiness over that you wish be
    delivering the following. unwell unquestionably come more
    formerly again as exactly the same nearly very often inside case you shield
    this increase.

  133. Simply want to say your article is as amazing.
    The clarity to your submit is simply cool and i can suppose you are a professional in this subject.

    Well with your permission let me to grasp your
    RSS feed to stay updated with approaching post. Thanks 1,000,
    000 and please carry on the enjoyable work.

  134. Greetings! I know this is somewhat off topic but I was wondering which blog platform are you using for this site?
    I’m getting sick and tired of WordPress because I’ve had problems with hackers and
    I’m looking at options for another platform. I would be fantastic
    if you could point me in the direction of a good platform.

  135. Great beat ! I wish to apprentice even as you amend
    your website, how can i subscribe for a weblog web site?
    The account helped me a acceptable deal. I were tiny bit
    familiar of this your broadcast provided vibrant clear
    idea

  136. I am extremely impressed together with your writing skills
    and also with the layout for your weblog. Is that this a paid subject matter or did you
    modify it yourself? Either way keep up the nice quality writing, it is rare to look a
    nice weblog like this one nowadays..

  137. Hi there! Would you mind if I share your blog with my myspace group?
    There’s a lot of people that I think would
    really enjoy your content. Please let me know. Many thanks

  138. Great blog! Do you have any recommendations for aspiring
    writers? I’m planning to start my own website soon but I’m a little lost on everything.
    Would you propose starting with a free platform like WordPress or go for
    a paid option? There are so many choices out there
    that I’m completely confused .. Any tips? Many thanks!

  139. I think this is among the most vital information for me.
    And i’m glad reading your article. But want to remark
    on some general things, The site style is great, the articles is really
    nice : D. Good job, cheers

  140. Hello, i read your blog from time to time and i own a similar onne and i was just wondering if you get a lot of spam comments?
    If so how do you stop it, any plugin or anything you can advise?
    I get sso much lately it’s driving me insane so any support is very much appreciated.

  141. I believe that is one of the most significant information for me.
    And i’m happy reading your article. But want to remark on some normal issues, The
    web site style is great, the articles is actually nice
    D. Good process, cheers

  142. You actually make it seem so easy with your presentation but I find this topic to be really something
    which I think I would never understand. It seems too
    complicated and very broad for me. I’m looking forward for your next post, I’ll try to get the hang of it!

  143. Nice post. I learn something new and challenging on websites I stumbleupon every day.
    It’s always useful to read content from other authors and use a little something from other websites.

  144. Greetings from Idaho! I’m bored to death at work so I decided
    to check out your website on my iphone during lunch break.
    I enjoy the knowledge you provide here and can’t wait to take a
    look when I get home. I’m shocked at how
    fast your blog loaded on my mobile .. I’m not even using WIFI,
    just 3G .. Anyways, good blog!

  145. Magnificent beat ! I would like to apprentice while you amend your web site, how could i subscribe for a blog website?
    The account helped me a acceptable deal. I had been tiny bit
    acquainted of this your broadcast provided bright clear idea

  146. This is very attention-grabbing, You’re an overly professional blogger.

    I have joined your rss feed and sit up for in the hunt for more
    of your excellent post. Additionally, I have shared your web site in my social networks

  147. I have been exploring for a little bit for any high-quality articles
    or blog posts on this kind of area . Exploring in Yahoo I ultimately stumbled upon this website.
    Reading this info So i am happy to convey that I’ve a very just right uncanny feeling
    I came upon exactly what I needed. I most indubitably will make
    certain to do not forget this site and give it a glance on a continuing basis.

  148. I’ve been exploring for a bit for any high-quality articles
    or weblog posts on this sort of house . Exploring in Yahoo I finally stumbled upon this site.
    Reading this info So i’m glad to convey that I’ve an incredibly just right uncanny feeling I found out exactly
    what I needed. I such a lot certainly will make sure to do not disregard this site and provides it a
    look regularly.

  149. You’re so interesting! I don’t believe I have read through a single thing like this before.
    So good to find another person with unique thoughts on this topic.

    Seriously.. thanks for starting this up. This site is one thing that’s needed on the
    web, someone with a little originality!

  150. Right here is the perfect web site for anybody who wishes
    to find out about this topic. You realize so much its almost tough to argue with
    you (not that I really will need to…HaHa). You definitely put a brand new spin on a
    topic which has been discussed for a long time. Excellent stuff, just excellent!

  151. Thanks for your whole labor on this web site. My aunt loves engaging in research and it is obvious why. All of us notice all relating to the powerful means you render efficient things through the web blog and therefore strongly encourage participation from website visitors about this area of interest so our daughter has always been studying a great deal. Enjoy the rest of the new year. You are always conducting a stunning job.

  152. Someone necessarily assist to make seriously posts I might state.
    That is the first time I frequented your website
    page and up to now? I surprised with the analysis you
    made to make this actual publish incredible.

    Fantastic activity!

  153. Have you ever thought about adding a little bit more than just your articles?
    I mean, what you say is important and all. However imagine if you added some great images or videos to give your posts more, “pop”!

    Your content is excellent but with pics and video clips, this blog could undeniably be one of the very best in its niche.

    Good blog!

  154. Hiya very cool website!! Man .. Beautiful .. Wonderful ..
    I’ll bookmark your web site and take the feeds also? I am satisfied to seek out so many helpful information here within the post, we want develop extra techniques in this regard, thank you for
    sharing. . . . . .

  155. It is appropriate time to make a few plans for the long run and it’s time to be happy.
    I have read this submit and if I may I wish to recommend yoou few interesting things or suggestions.
    Perhaps you can write subsequent articles regarding this article.

    I want tto read more issues about it!

  156. Simply want to say your article is as astounding. The clearness
    in your post is just excellent and i could assume you are an expert on this subject.
    Well with your permission let me to grab your RSS feed to keep
    updated with forthcoming post. Thanks a million and please keep up the enjoyable work.

  157. Thanks , I have just been looking for info approximately this topic for a long time and yours is the best I have found out so far.
    But, what in regards to the conclusion? Are you certain in regards to the
    source?

  158. Hello! I’ve been following your website for a long time now
    and finally got the bravery to go ahead and give you a shout
    out from Porter Tx! Just wanted to tell
    you keep up the great job!

  159. I really love your website.. Pleasant colors & theme. Did you
    make this website yourself? Please reply back as I’m trying to create my
    very own blog and would like to learn where you got
    this from or just what the theme is called. Thanks!

  160. Hi there, I discovered your website by the use of Google whilst searching for
    a comparable matter, your website got here up, it seems good.
    I’ve bookmarked it in my google bookmarks.
    Hello there, simply changed into alert to your weblog thru Google,
    and located that it’s truly informative. I am going to watch out for
    brussels. I will appreciate when you proceed this in future.
    Lots of other folks might be benefited from your writing.
    Cheers!

  161. I have been browsing online more than 2 hours today, yet I never found any interesting article like
    yours. It’s pretty worth enough for me. In my opinion, if all site owners and bloggers
    made good content as you did, the web will be much more
    useful than ever before.

  162. I’m impressed, I have to admit. Seldom do I encounter a blog
    that’s both educative and amusing, and let me tell you,
    you’ve hit the nail on the head. The problem is something that too few
    men and women are speaking intelligently about. I am very happy I found this during my hunt for something concerning this.

  163. I am really enjoying the theme/design of your site.

    Do you ever run into any browser compatibility issues?
    A number of my blog readers have complained about my website
    not operating correctly in Explorer but looks great in Firefox.
    Do you have any suggestions to help fix this issue?

  164. Hi there! I know this is somewhat off topic but I was wondering if you knew where I
    could find a captcha plugin for my comment form?
    I’m using the same blog platform as yours and I’m having problems finding one?
    Thanks a lot!

  165. Terrific work! That is the kind of information that
    should be shared around the internet. Shame
    on the search engines for no longer positioning this post higher!
    Come on over and consult with my website . Thanks =)

  166. Please let me know if you’re looking for a author for your weblog.
    You have some really good articles and I feel I would be a good asset.

    If you ever want to take some of the load off, I’d really like
    to write some content for your blog in exchange for a link back to mine.
    Please shoot me an e-mail if interested. Cheers!

  167. Great blog! Do you have any recommendations for axpiring writers?
    I’m planning to start my own blog soon but I’m a little lost on everything.

    Would you advise starting with a free platform likme WordPress or go for a paid option? There are so many choces out there that I’m totally overwhelmed ..
    Any suggestions? Appreciate it!

  168. Greetings from Colorado! I’m bored to death at work so I decided to browse your blog on my iphone during lunch
    break. I enjoy the knowledge you provide here and can’t wait to take a
    look when I get home. I’m amazed at how fast your
    blog loaded on my mobile .. I’m not even using WIFI, just 3G ..
    Anyhow, awesome blog!

  169. fantastic put up, very informative. I ponder why the other specialists of this sector do not notice this.
    You should proceed your writing. I’m confident, you’ve a great readers’ base already!

  170. Aw, this was an extremely nice post. Taking
    a few minutes and actual effort to create a really good article…
    but what can I say… I put things off a lot and don’t seem to get
    anything done.

  171. Have you ever thought about publishing an ebook or guest authoring on other sites?
    I have a blog centered on the same topics you discuss and would really like to have you share some stories/information. I know my viewers would appreciate your work.
    If you are even remotely interested, feel free to shoot
    me an e mail.

  172. Thanks on your marvelous posting! I quite enjoyed reading it, you might
    be a great author. I will be sure to bookmark your blog and definitely will come back someday.
    I want to encourage continue your great work, have a nice
    holiday weekend!

  173. Hello my friend! I wish to say that this post is amazing, nice written and come with approximately all important infos.

    I would like to look extra posts like this
    .

  174. Thank you for some other informative website. Where
    else may just I get that type of information written in such an ideal manner?

    I have a project that I’m simply now working on, and I’ve been on the look out for such information.

  175. Great – I should definitely pronounce, impressed with your website.
    I had no trouble navigating through all tabs and related info ended up being truly simple to do to access.
    I recently found what I hoped for before you know it in the
    least. Reasonably unusual. Is likely to appreciate it for those who add forums or anything,
    site theme . a tones way for your client to communicate.
    Nice task.

  176. Wonderful blog! Do you have any hints for aspiring writers?
    I’m planning to start my own website soon but I’m a little lost
    on everything. Would you advise starting with a free platform
    like WordPress or go for a paid option? There are so many choices out there that I’m
    completely confused .. Any suggestions? Thanks a
    lot!

  177. Hello, i believe that i saw you visited my website so i got here to go back the
    desire?.I am attempting to in finding issues
    to enhance my web site!I suppose its adequate to make use of some of
    your ideas!!

  178. Hey! I realize this is somewhat off-topic but I had to ask.
    Does building a well-established blog such as yours require a lot of work?
    I’m brand new to running a blog however I do write in my journal every day.
    I’d like to start a blog so I will be able to share my experience and feelings
    online. Please let me know if you have any ideas or tips for brand new aspiring bloggers.

    Appreciate it!

  179. Have you ever thought about publishing an ebook or guest
    authoring on other sites? I have a blog based on the same subjects you
    discuss and would love to have you share some stories/information. I
    know my visitors would appreciate your work. If you are even remotely interested, feel free to shoot me an e mail.

  180. Someone necessarily help to make critically articles I would state.

    That is the very first time I frequented your website page
    and up to now? I surprised with the analysis you made to make this particular put up extraordinary.
    Magnificent activity!

  181. Hi there! This post could not be written much better!
    Looking through this article reminds me of my previous roommate!

    He continually kept talking about this. I will forward this
    information to him. Fairly certain he will have a
    very good read. I appreciate you for sharing!

  182. Hello There. I discovered your weblog the use of msn. This is a very smartly
    written article. I’ll make sure to bookmark it and return to learn more of your helpful information. Thank
    you for the post. I will certainly return.

  183. wonderful pointѕ altogether, you just gɑineɗ a logo new reader.
    What would you sᥙɡgest about your publish that you simply maⅾe some
    days ago? Any sure?

  184. I would like to use the chance of saying thanks
    to you for thhe professional assistance I have enjoyed viewing your site.
    I’m looking forward to tthe actual commencement of
    my university research and the whole planning would never have been complete without coming over
    to your site. If I might be of any help to
    others, I might be thankful to help by what I have gained from here.

  185. Wow that was οdd. I just wrote an extremely long comment but after I сlicked suƅmіt
    my comment diɗn’t appeаr. Grrrr… well I’m not writing
    all that over again. Anyway, just wanted to say superb blog!

  186. I think what youu wrote mqde a lot of sense. But, consider this, what if you wrote
    a catchier post title? I ain’t suggesting your information isn’t good.,
    but what if you added a post title that makes people want more?

    Imean Kubernetes容器集群管理 – 董的博客 is a little plain. Yoou should peek at Yahoo’s home page and watch how they create post headlines to grab people to oopen the links.
    You might add a video or a picture or two to grab readers excited about what you’ve got to say.
    In my opinion, it could bring your posts a
    little bit more interesting.

  187. I’m not that much of a online reader too be honest but your sites reawlly nice, keep it up!
    I’ll goo ahead and bookmark your site to come back doen the road.
    All the best

  188. I feel that is among the most vital information for me.
    And i’m glad studying your article. But should remark on some common things, The website style is perfect, the
    articles is actually excellent : D. Good process, cheers

  189. What’s up to every body, it’s my first go to see of this
    webpage; this web site includes awesome and actually excellent stuff for readers.

  190. It’s really a great and helpful piece of info. I’m happy that
    you just shared this helpful info with us. Please keep us up to date like this.
    Thanks for sharing.

  191. Heⅼlo! I understand this is somеwhat off-topic but I neeԁed to
    ask. Doеs operating a well-establiѕhed weЬsite such as yⲟurs take a large amount of work?
    I am brand new to writing a blog but I do write in my journal ᧐n a daily basiѕ.
    I’d liкe to start a blog so I will be able to share my own experience and views online.
    Please let me know іf yoս havе any recommendations ߋr tips for
    brand new aspiring bloggeгs. Thankyou!

  192. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You definitely know what youre talking about, why throw away your
    intelligence on just posting videos to your blog when you could be giving us something enlightening to read?

  193. helⅼo!,I like your writing so much! рroportion we communicate extra about yоur post on AOᒪ?
    I require a specialist in this space to unravel my problem.
    Maybe that’s you! Taking a look ahead to рeer you.

  194. Heya i’m for the primary time һere. I came across this
    board and I find It truⅼy helpful & it helpеd me
    out much. I am hoping to provіdе something back and aid others like you helped me.

  195. We’re a group of volunteers and starting a new scheme
    in our community. Your site offered us with valuable information to work on. You have done an impressive job and our whole community will
    be thankful to you.

  196. What’s Going down i’m new to this, I stumbled upon this I have found It absolutely useful and it has helped me out loads.
    I am hoping to contribute & aid different users like its helped me.
    Great job.

  197. Great beat ! I wish to apprentice at the same time as you amend your website, how could i subscribe
    for a blog web site? The account helped me a applicable deal.
    I had been a little bit acquainted of this your broadcast provided vivid clear
    idea

  198. Hey tis is somewhat of off topic but I was wondering if blogs usse WYSIWYG editors or if you
    have to manually code with HTML. I’m starting a blog soon but have no coding
    knowledge so I wanted tto get advice from someone with experience.
    Any help would bbe enormously appreciated!