磁盘空间不足导致zookeeper宕机

楔子

某服务需要使用Zookeeper作为配置管理和服务注册与发现,今天突然发现服务不可用,病症就是无法连接zk,服务不断的重试,报如下的错误。(zk版本为3.4.10 单机部署)
在这里插入图片描述

定位

使用df -Th查看文件的使用情况 , 发现磁盘空间满了,使用du -sh *查看当前目录下各个文件及目录占用空间大小 ,发现有个日志文件占据了195G, oh my god!
在这里插入图片描述
果断删除该文件。然后重启zk,依然不能重启,查看zk得日志,发现如下的报错信息:

ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial.

意思就是由于磁盘空间满了导致Zookeeper的最后一次事务性操作不完整导致无法重启。则么办?

先查看zk的配置文件,查看dataLogDir的目录,如下:
在这里插入图片描述
进入到该目录的version-2目录下,把大小为0(异常的)日志全部删除掉,文中已删除,然后就可重启成功,但是最后一个事务操作可能会破坏或者丢失。
在这里插入图片描述
结果如下所示:
在这里插入图片描述

发布了418 篇原创文章 · 获赞 738 · 访问量 125万+
展开阅读全文

调用dubbo服务,服务端zk内存溢出 求助

12-02

遇到了这样一个问题,定时任务调用dubbo做操作,调用完成后更新定时任务状态。 很突然的出现了一个问题,定时任务调用dubbo服务后就不向下执行了,但是查看服务端记录,调用的方法是正常执行完的。而且定时任务和服务端都没有报错…………求助,可能是什么问题? 分析日志后发现服务端 会报一个错误 14:54:09,510 [ERROR] [org.I0Itec.zkclient.ZkEventThread] :77 - Error handling event ZkEvent[New session event sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@7ac2e933] java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringBuilder.toString(StringBuilder.java:407) at java.util.AbstractCollection.toString(AbstractCollection.java:464) at java.lang.String.valueOf(String.java:2994) at java.lang.StringBuilder.append(StringBuilder.java:131) at com.alibaba.dubbo.registry.support.FailbackRegistry.recover(FailbackRegistry.java:282) at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.access$000(ZookeeperRegistry.java:45) at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry$1.stateChanged(ZookeeperRegistry.java:76) at com.alibaba.dubbo.remoting.zookeeper.support.AbstractZookeeperClient.stateChanged(AbstractZookeeperClient.java:86) at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.access$300(ZkclientZookeeperClient.java:17) at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1.handleNewSession(ZkclientZookeeperClient.java:36) at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) 求助是什么原因 问答

什么情况会造成dubbo链接中断?

12-12

一个手动调用的dubbo服务,调用后,服务端正常执行完了。但是客户端没有接受到任何返回信息。 检查日志后 服务端: 2018-12-12 02:51:26 [org.I0Itec.zkclient.ZkClient]-[DEBUG] Received event: WatchedEvent state:Disconnected type:None path:null 2018-12-12 02:51:26 [org.I0Itec.zkclient.ZkEventThread]-[DEBUG] New event: ZkEvent[State changed to Disconnected sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@28ce7d23] 2018-12-12 02:51:26 [org.I0Itec.zkclient.ZkClient]-[DEBUG] Leaving process event 2018-12-12 02:51:26 [org.I0Itec.zkclient.ZkEventThread]-[DEBUG] Delivering event #1 ZkEvent[State changed to Disconnected sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@28ce7d23] 2018-12-12 02:51:26 [org.I0Itec.zkclient.ZkEventThread]-[DEBUG] Delivering event #1 done 2018-12-12 02:51:46 [org.I0Itec.zkclient.ZkClient]-[DEBUG] Received event: WatchedEvent state:Expired type:None path:null 2018-12-12 02:51:46 [org.I0Itec.zkclient.ZkEventThread]-[DEBUG] New event: ZkEvent[State changed to Expired sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@28ce7d23] 2018-12-12 02:51:46 [org.I0Itec.zkclient.ZkConnection]-[DEBUG] Closing ZooKeeper connected to 10.1.10.91:2181,10.1.10.92:2181,10.1.10.93:2181 2018-12-12 02:51:46 [org.I0Itec.zkclient.ZkEventThread]-[DEBUG] Delivering event #2 ZkEvent[State changed to Expired sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@28ce7d23] 2018-12-12 02:51:46 [org.I0Itec.zkclient.ZkEventThread]-[DEBUG] Delivering event #2 done 2018-12-12 02:51:46 [org.apache.zookeeper.ZooKeeper]-[DEBUG] Close called on already closed client zk日志: 2018-12-12 02:51:46,630 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.1.10.47:47124 2018-12-12 02:51:46,637 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /10.1.10.47:47124; will be dropped if server is in r-o mode 2018-12-12 02:51:46,638 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@935] - Client attempting to renew session 0x365ee3f0f34002c at /10.1.10.47:47124 2018-12-12 02:51:46,638 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x365ee3f0f34002c 2018-12-12 02:51:46,639 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@680] - Invalid session 0x365ee3f0f34002c for client /10.1.10.47:47124, probably expired 2018-12-12 02:51:46,640 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /10.1.10.47:47124 which had sessionid 0x365ee3f0f34002c 2018-12-12 02:52:23,235 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.1.10.47:47284 2018-12-12 02:52:23,236 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /10.1.10.47:47284; will be dropped if server is in r-o mode 2018-12-12 02:52:23,238 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /10.1.10.47:47284 2018-12-12 02:52:23,244 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x165ee402a7d003d with negotiated timeout 30000 for client /10.1.10.47:47284 ------- 服务端的方法是一个多线程方法,执行期间没有任何报错,想请教大神们一下,什么情况下才会造成dubbo链接的中断?通常这种问题要怎么调整? ------ zkserver端日志 2018-12-12 02:51:46,630 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.1.10.47:47124 2018-12-12 02:51:46,637 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /10.1.10.47:47124; will be dropped if server is in r-o mode 2018-12-12 02:51:46,638 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@935] - Client attempting to renew session 0x365ee3f0f34002c at /10.1.10.47:47124 2018-12-12 02:51:46,638 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x365ee3f0f34002c 2018-12-12 02:51:46,639 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@680] - Invalid session 0x365ee3f0f34002c for client /10.1.10.47:47124, probably expired 2018-12-12 02:51:46,640 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /10.1.10.47:47124 which had sessionid 0x365ee3f0f34002c 问答

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 技术黑板 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览