宜宾网站建设公司

spark怎么读取hbase表-创新互联

这篇文章主要介绍了spark怎么读取hbase表的相关知识，内容详细易懂，操作简单快捷，具有一定借鉴价值，相信大家阅读完这篇spark怎么读取hbase表文章都会有所收获，下面我们一起来看看吧。

创新互联服务项目包括临武网站建设、临武网站制作、临武网页制作以及临武网络营销策划等。多年来，我们专注于互联网行业，利用自身积累的技术优势、行业经验、深度合作伙伴关系等，向广大中小型企业、政府机构等提供互联网行业的解决方案，临武网站推广取得了明显的社会效益与经济效益。目前，我们服务的客户以成都为中心已经辐射到临武省份的部分城市，未来相信会继续扩大服务区域并继续获得客户的支持与信任！

一.场景:
    spark通过phoenix 读取hbase表，其实说白了先要去Zookeeper建立connection。

二.代码:

  1. val zkUrl = "192.168.100.39,192.168.100.40,192.168.100.41:2181"

  2. val formatStr = "org.apache.phoenix.spark"

  3. val oms_orderinfoDF = spark.read.format(formatStr)

  4.   .options(Map("table" -> "oms_orderinfo", "zkUrl" -> zkUrl))

  5.   .load


三.查看SparkJob日志:

  1. 17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop40/192.168.100.40:2181. Will not attempt to authenticate using SASL (unknown error)

  2. 17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.100.48:35952, server: hadoop40/192.168.100.40:2181

  3. 17/10/24 03:25:25 WARN zookeeper.ClientCnxn: Session 0x0 for server hadoop40/192.168.100.40:2181, unexpected error, closing socket connection and attempting reconnect

  4. java.io.IOException: Connection reset by peer

  5.     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

  6.     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

  7.     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)

  8.     at sun.nio.ch.IOUtil.read(IOUtil.java:192)

  9.     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)

  10.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)

  11.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)

  12.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

  13. 17/10/24 03:25:25 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice1/user/hdfs/.sparkStaging/application_1507703377455_4854

  14. 17/10/24 03:25:25 INFO util.ShutdownHookManager: Shutdown hook called


四.查看Zookeeper日志:

  1. 2017-10-24 03:25:22,498 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

  2. 2017-10-24 03:25:25,588 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

  3. 2017-10-24 03:25:25,819 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

  4. 2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

  5. EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee09897d, likely client has closed socket

  6.     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)

  7.     at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

  8.     at java.lang.Thread.run(Thread.java:745)

  9. 2017-10-24 03:25:26,092 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47084 which had sessionid 0x15ed091ee09897d

  10. 2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

  11. EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee098981, likely client has closed socket

  12.     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)

  13.     at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

  14.     at java.lang.Thread.run(Thread.java:745)

  15. 2017-10-24 03:25:26,093 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47304 which had sessionid 0x15ed091ee098981

  16. 2017-10-24 03:25:26,093 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception


五.解决方法:
undefinedmaxClientCnxns调大到1000，重启zookeeper服务生效配置即可。
spark怎么读取hbase表

关于“spark怎么读取hbase表”这篇文章的内容就介绍到这里，感谢各位的阅读！相信大家对“spark怎么读取hbase表”知识都有一定的了解，大家如果还想学习更多知识，欢迎关注创新互联-成都网站建设公司行业资讯频道。


当前文章：spark怎么读取hbase表-创新互联
分享URL：http://ybzwz.com/article/dceojc.html

其他资讯