日志报错问题追踪 about mdrill HOT 10 CLOSED

2013-09-26 13:31:54 HigoJoinInvert [INFO] ##fqlist.size()##1
2013-09-26 13:31:54 HigoJoinInvert [INFO] ##joinright##7
2013-09-26 13:31:54 TryLockFile [ERROR] trylock
java.io.IOException: Resource deadlock avoided
at sun.nio.ch.FileChannelImpl.lock0(Native Method)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:845)
at java.nio.channels.FileChannel.lock(FileChannel.java:860)
at com.alimama.mdrill.utils.TryLockFile.trylock(TryLockFile.java:33)
at com.alimama.mdrill.buffer.TryLock.tryLock(TryLock.java:41)
at org.apache.solr.request.uninverted.UnInvertedField.(UnInvertedField.java:77)
at org.apache.solr.request.uninverted.UnInvertedField.getUnInvertedField(UnInvertedField.java:702)
at org.apache.solr.request.uninverted.UnInvertedField.getUnInvertedField(UnInvertedField.java:684)
at org.apache.solr.request.mdrill.MdrillPorcessUtils$UnvertFields.(MdrillPorcessUtils.java:294)
at org.apache.solr.request.join.HigoJoinInvert.open(HigoJoinInvert.java:68)
at org.apache.solr.request.mdrill.MdrillGroupBy.execute(MdrillGroupBy.java:241)
at org.apache.solr.request.mdrill.MdrillGroupBy.getCross(MdrillGroupBy.java:115)
at org.apache.lucene.index.SegmentReader.invertScan(SegmentReader.java:559)
at org.apache.lucene.index.DirectoryReader.invertScan(DirectoryReader.java:591)
at org.apache.lucene.index.FilterIndexReader.invertScan(FilterIndexReader.java:317)
at org.apache.solr.request.mdrill.FacetComponent.getByGroupby(FacetComponent.java:115)
at org.apache.solr.request.mdrill.FacetComponent.process(FacetComponent.java:88)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:101)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1510)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:264)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2013-09-26 13:31:54 SolrCore [INFO] ####UnInverted#### begin
2013-09-26 13:31:54 BigReUsedBuffer [INFO] ####BigByteBuffer### calloc f

from mdrill.

from mdrill.

心跳超时
2013-09-26 13:40:32 SyncProcesses [INFO] Shutting down and clearing state for id e0e49c91-e95d-4720-9eb4-d472e03d84a8;State:timedOut;HeartbeattopologyId:adhoc-1-1380018903, timeSecs:1380174001, port:6705, taskIds:[55]

from mdrill.

}
            // kill those in allocated that are dead or disallowed;
            if (hbstate.getState() != State.valid) {
                StringBuilder sb = new StringBuilder();
                sb.append("Shutting down and clearing state for id ");
                sb.append(workerid);
                sb.append(";State:");
                sb.append(hbstate.getState());
                sb.append(";Heartbeat");
                sb.append(hbstate.getHeartbeat());
                LOG.info(sb);

                try {
                    shutWorker(conf, supervisorId, workerid,
                            workerThreadPids);
                } catch (IOException e) {
                    String errMsg = "Failed to shutdown worker workId:"
                            + workerid + ",supervisorId: " + supervisorId
                            + ",workerThreadPids:" + workerThreadPids;
                    LOG.error(errMsg, e);
                }
            }
        }

from mdrill.

30秒 没有心跳，意味着超时
supervisor.worker.timeout.secs: 30

from mdrill.

1.现象 adhoc4和adhoc8 全部的worker被杀掉，并重启，导致重启期间用户查询adhoc前端报错
2.重启的原因：worker的心跳超时（worker的心跳实际上是向本地硬盘写入一个序列化的文件）
3.期间task线程的zookeeper心跳正常

可能原因
1.异常期间，正在进行比较大的复杂的查询，但是task心跳正常，排除
2.期间硬盘繁忙，写入进程锁住（这个还不能确定）

from mdrill.

目前临时解决办法， 增加心跳超时时间间隔

from mdrill.

2013-09-26 10:40:21 SegmentReader [ERROR] ##getpos##p4p_e_gmv_direct_amtfileNum#-1
java.lang.NullPointerException
at org.apache.lucene.index.SegmentReader.getQuickPos(SegmentReader.java:502)
at org.apache.solr.request.uninverted.UnInvertedField.uninvert(UnInvertedField.java:466)
at org.apache.solr.request.uninverted.UnInvertedField.(UnInvertedField.java:88)
at org.apache.solr.request.uninverted.UnInvertedField.getUnInvertedField(UnInvertedField.java:720)
at org.apache.solr.request.mdrill.MdrillPorcessUtils$UnvertFields.(MdrillPorcessUtils.java:324)
at org.apache.solr.request.mdrill.MdrillGroupBy.execute(MdrillGroupBy.java:230)
at org.apache.solr.request.mdrill.MdrillGroupBy.getCross(MdrillGroupBy.java:115)
at org.apache.lucene.index.SegmentReader.invertScan(SegmentReader.java:559)
at org.apache.lucene.index.DirectoryReader.invertScan(DirectoryReader.java:591)
at org.apache.lucene.index.FilterIndexReader.invertScan(FilterIndexReader.java:317)
at org.apache.solr.request.mdrill.FacetComponent.getByGroupby(FacetComponent.java:115)
at org.apache.solr.request.mdrill.FacetComponent.process(FacetComponent.java:88)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:101)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1510)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:264)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2013-09-26 10:40:21 SegmentReader [INFO] ##getCount##p4p_e_gmv_direct_amtfileNum#-1#count#0

此错误重现，要重视

from mdrill.

2013-09-26 10:40:21 SegmentReader [ERROR] ##getpos##p4p_e_gmv_direct_amtfileNum#-1
原因已经查明，是该表不存在该字段，故会报错
添加对不存在字段的判断
r_rpt_cps_adhoc_payment->�p4p_e_gmv_direct_amt� ->gmv_cnt

发现是监控程序，在读这俩字段

from mdrill.

死锁的原因已经找到，经过观察日志 只有当使用join的时候才发生死锁，
发现open方法中含有synchronized 这个是造成死锁的主要原因

public synchronized void open(SolrQueryRequest req) throws IOException, ParseException
{
    this.search=HigoJoinUtils.getSearch(req, this.tableName);
    this.fields = req.getParams().getParams(HigoJoinUtils.getFields(this.tableName));
    List<Query> fqlist=HigoJoinUtils.getFilterQuery(req, this.tableName);
    LOG.info("##fqlist.size()##"+fqlist.size());
    this.docset=this.search.get().getDocSet(fqlist);
    LOG.info("##joinright##"+this.docset.size());
    String fieldLeft=req.getParams().get(HigoJoinUtils.getLeftField(this.tableName));
    String fieldRigth=req.getParams().get(HigoJoinUtils.getRightField(this.tableName));
    this.ufsRight=new UnvertFields(fields, this.search.get());

from mdrill.