数据采集工具的安装和使用
数据采集是大数据分析全流程的重要环节,典型的数据采集工具包括ETL工具、日志采集工具、数据迁移工具等。
为寿县等地区用户提供了全套网页设计制作服务,及寿县网站建设行业解决方案。主营业务为成都网站制作、成都网站设计、寿县网站设计,以传统方式定制建设网站,并提供域名空间备案等一条龙服务,秉承以专业、用心的态度为用户提供真诚的服务。我们深信只要达到每一位用户的要求,就会得到认可,从而选择与我们长期合作。这样,我们也可以走得更远!
Flume是一个高可用的、高可靠的、分布式的海量日志采集、聚合和传输的系统。
1.安装Flume
下载:http://www.apache.org/dist/flume/
hadoop@dblab:/usr/local$ sudo wgethttp://www.apache.org/dist/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
hadoop@dblab:/usr/local$ sudo tar -zxvf apache-flume-1.7.0-bin.tar.gz
hadoop@dblab:/usr/local$ sudo mv apache-flume-1.7.0-bin ./flume
2.配置环境变量
hadoop@dblab:/usr/local$ sudo vim ~/.bashrc
export FLUME_HOME=/usr/local/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export JAVA_HOME=/usr/lib/jvm/default-java
export PATH=$PATH:$HIVE_HOME/bin
hadoop@dblab:/usr/local$ source ~/.bashrc
hadoop@dblab:/usr/local/flume/conf$ mv flume-env.sh.template flume-env.sh
hadoop@dblab:/usr/local/flume/conf$ sudo vim flume-env.sh
#在flume-env.sh文件开头加入如下语句:
export JAVA_HOME=/usr/lib/jvm/default-java
3.启动Flume
hadoop@dblab:/usr/local/flume$ cd /usr/local/flume
hadoop@dblab:/usr/local/flume$ ./bin/flume-ng version
错误: 找不到或无法加载主类 org.apache.flume.tools.GetJavaProperty
Flume 1.7.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
From source with checksum 0d21b3ffdc55a07e1d08875872c00523
hadoop@dblab:/usr/local/flume$ cd /usr/local/hbase/conf
hadoop@dblab:/usr/local/hbase/conf$ sudo vim hbase-env.sh
#export HBASE_CLASSPATH=/usr/local/hadoop/conf #注释该行,即解决上述问题
hadoop@dblab:/usr/local/flume$ ./bin/flume-ng version
Flume 1.7.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
From source with checksum 0d21b3ffdc55a07e1d08875872c00523
分享题目:数据采集工具的安装和使用
分享网址:http://ybzwz.com/article/gepedi.html