flume收集日志的简单配置

我就是我 2022-05-28 10:53 335阅读 0赞

flume收集日志的形式  
flume的三大组件source和channel，sink的分类和属性文档  
[flume三大组件文档][flume]

**通过使用avro将多台机器的日志写入到一台机器，使用场景集群中的日志集中写到某一台，然后由某一台集中输入，减少每台机器的IO压力**  
角色：  
collect收集者  
client 发送者  
**avro-collect.properties**

# in this case called 'a1'
    
    a1.sources = s1
    a1.channels = c1
    a1.sinks = k1
    
    # For each one of the sources, the type is defined
    a1.sources.s1.type = avro
    a1.sources.s1.bind = hadoop.madman.com
    a1.sources.s1.port = 45454
    # The channel can be defined as follows.
    a1.sources.s1.channels = c1
    a1.sinks.k1.channel = c1
    
    # Each sink's type must be defined
    a1.sinks.k1.type = hdfs
    a1.sinks.k1.hdfs.path = /flume/event/avro
    a1.sinks.k1.hdfs.filePrefix = hive-log
    a1.sinks.k1.hdfs.rollSize = 10240
    a1.sinks.k1.hdfs.rollInterval = 0
    a1.sinks.k1.hdfs.rollCount = 0
    a1.sinks.k1.hdfs.useLocalTimeStamp = true
    #Specify the channel the sink should use
    
    # Each channel's type is defined.
    a1.channels.c1.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 1000

**avro-client.properties**

# in this case called 'a1'
    
    a1.sources = s1
    a1.channels = c1
    a1.sinks = k1
    
    # For each one of the sources, the type is defined
    a1.sources.s1.type = exec
    a1.sources.s1.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    # The channel can be defined as follows.
    a1.sources.s1.channels = c1
    a1.sinks.k1.channel = c1
    
    # Each sink's type must be defined
    a1.sinks.k1.type = avro
    a1.sinks.k1.hostname = hadoop.madman.com
    a1.sinks.k1.port = 45454
    #Specify the channel the sink should use
    
    # Each channel's type is defined.
    a1.channels.c1.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    a1.channels.c1.capacity = 100
    a1.channels.c1.transactionCapacity = 100

启动顺序  
先启动collect收集者，免得丢失数据，然后再启动客户端。

bin/flume-ng agent --conf conf/ --name a1 --conf-file  conf/avro-client.properties -Dflume.root.logger=INFO,console
    
    bin/flume-ng agent --conf conf/ --name a1 --conf-file  conf/avro-collect.properties -Dflume.root.logger=INFO,console

--------------------

**flume配置多个channel对应多个sink，写到不同的地方**  
**flume-sink.properties**

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per a1, 
    # in this case called 'a1'
    
    a1.sources = s1
    #配置两个channels 
    a1.channels = c1 c2
    #配置两个sinks
    a1.sinks = k1 k2
    
    # For each one of the sources, the type is defined
    a1.sources.s1.type = exec
    a1.sources.s1.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    # The channel can be defined as follows.
    a1.sources.s1.channels = c1 c2
    a1.sinks.k1.channel = c1
    a1.sinks.k2.channel = c2
    
    # Each sink's type must be defined
    a1.sinks.k1.type = hdfs
    a1.sinks.k1.hdfs.path = /flume/event/hdfs1
    a1.sinks.k1.hdfs.filePrefix = hive-log1
    
    # Each sink's type must be defined
    a1.sinks.k2.type = hdfs
    a1.sinks.k2.hdfs.path = /flume/event/hdfs2
    a1.sinks.k2.hdfs.filePrefix = hive-log2
    #Specify the channel the sink should use
    
    # Each channel's type is defined.
    a1.channels.c1.type = memory
    a1.channels.c2.type = memory
    
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 1000
    
    a1.channels.c2.capacity = 1000
    a1.channels.c2.transactionCapacity = 1000

启动命令

bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/taildir.properties -Dflume.root.logger=INFO,console
    --name a1这个名字需要跟配置文件里面的名字一致

--------------------

#### flume-file-conf.properties ####

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per agent, 
    # in this case called 'agent'
    
    agent.sources = seqGenSrc
    agent.channels = memoryChannel
    agent.sinks = loggerSink
    
    # For each one of the sources, the type is defined
    agent.sources.seqGenSrc.type = exec
    agent.sources.seqGenSrc.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    
    # The channel can be defined as follows.
    agent.sources.seqGenSrc.channels = memoryChannel
    
    # Each sink's type must be defined
    agent.sinks.loggerSink.type = logger
    
    #Specify the channel the sink should use
    agent.sinks.loggerSink.channel = memoryChannel
    
    # Each channel's type is defined.
    agent.channels.memoryChannel.type = file
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    agent.channels.memoryChannel.capacity = 100
    agent.channels.memoryChannel.transactionCapacity = 100
    agent.channels.memoryChannel.checkpointDir = /opt/cdhmoduels/apache-flume-1.5.0-cdh5.3.6-bin/file/check
    agent.channels.memoryChannel.dataDirs = /opt/cdhmoduels/apache-flume-1.5.0-cdh5.3.6-bin/file/data

启动命令：

bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-file-conf.properties -Dflume.root.logger=INFO,console

#### flume-hdfs-mem-conf.properties ####

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per agent, 
    # in this case called 'agent'
    
    agent.sources = seqGenSrc
    agent.channels = memoryChannel
    agent.sinks = loggerSink
    
    # For each one of the sources, the type is defined
    agent.sources.seqGenSrc.type = exec
    agent.sources.seqGenSrc.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    
    # The channel can be defined as follows.
    agent.sources.seqGenSrc.channels = memoryChannel
    
    # Each sink's type must be defined
    agent.sinks.loggerSink.type = logger
    agent.sinks.loggerSink.type = hdfs
    agent.sinks.loggerSink.hdfs.path = /flume/event/hdfs
    agent.sinks.loggerSink.hdfs.filePrefix = hive-log
    #Specify the channel the sink should use
    agent.sinks.loggerSink.channel = memoryChannel
    
    # Each channel's type is defined.
    agent.channels.memoryChannel.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    agent.channels.memoryChannel.capacity = 1000
    agent.channels.memoryChannel.transactionCapacity = 1000

启动命令

bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-hdfs-mem-conf.properties -Dflume.root.logger=INFO,console

#### flume-hdfs-size-mem-conf.properties ####

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per agent, 
    # in this case called 'agent'
    
    agent.sources = seqGenSrc
    agent.channels = memoryChannel
    agent.sinks = loggerSink
    
    # For each one of the sources, the type is defined
    agent.sources.seqGenSrc.type = exec
    agent.sources.seqGenSrc.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    
    # The channel can be defined as follows.
    agent.sources.seqGenSrc.channels = memoryChannel
    
    # Each sink's type must be defined
    agent.sinks.loggerSink.type = logger
    agent.sinks.loggerSink.type = hdfs
    agent.sinks.loggerSink.hdfs.path = /flume/event/hdfssize
    agent.sinks.loggerSink.hdfs.filePrefix = hive-log
    agent.sinks.loggerSink.hdfs.rollSize = 10240
    agent.sinks.loggerSink.hdfs.rollInterval = 0
    agent.sinks.loggerSink.hdfs.rollCount = 0
    #Specify the channel the sink should use
    agent.sinks.loggerSink.channel = memoryChannel
    
    # Each channel's type is defined.	
    agent.channels.memoryChannel.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    agent.channels.memoryChannel.capacity = 1000
    agent.channels.memoryChannel.transactionCapacity = 1000

启动命令

bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-hdfs-size-mem-conf.properties -Dflume.root.logger=INFO,console

#### flume-hdfs-size-part-mem-conf.properties ####

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per agent, 
    # in this case called 'agent'
    
    agent.sources = seqGenSrc
    agent.channels = memoryChannel
    agent.sinks = loggerSink
    
    # For each one of the sources, the type is defined
    agent.sources.seqGenSrc.type = exec
    agent.sources.seqGenSrc.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    
    # The channel can be defined as follows.
    agent.sources.seqGenSrc.channels = memoryChannel
    
    # Each sink's type must be defined
    agent.sinks.loggerSink.type = logger
    agent.sinks.loggerSink.type = hdfs
    agent.sinks.loggerSink.hdfs.path = /flume/event/hdfspart
    agent.sinks.loggerSink.hdfs.path = /flume/event/part/date=%Y-%m-%d
    agent.sinks.loggerSink.hdfs.useLocalTimeStamp = true
    agent.sinks.loggerSink.hdfs.filePrefix = hive-log
    agent.sinks.loggerSink.hdfs.rollSize = 10240
    agent.sinks.loggerSink.hdfs.rollInterval = 0
    agent.sinks.loggerSink.hdfs.rollCount = 0
    #Specify the channel the sink should use
    agent.sinks.loggerSink.channel = memoryChannel
    
    # Each channel's type is defined.	
    agent.channels.memoryChannel.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    agent.channels.memoryChannel.capacity = 1000
    agent.channels.memoryChannel.transactionCapacity = 1000

启动命令

bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-hdfs-size-part-mem-conf.properties -Dflume.root.logger=INFO,console

#### flume-mem-conf.properties ####

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per agent, 
    # in this case called 'agent'
    
    agent.sources = seqGenSrc
    agent.channels = memoryChannel
    agent.sinks = loggerSink
    
    # For each one of the sources, the type is defined
    agent.sources.seqGenSrc.type = exec
    agent.sources.seqGenSrc.command = tail -f /opt/cdhmoduels/hive-0.13.1-cdh5.3.6/log/hive.log
    
    # The channel can be defined as follows.
    agent.sources.seqGenSrc.channels = memoryChannel
    
    # Each sink's type must be defined
    agent.sinks.loggerSink.type = logger
    
    #Specify the channel the sink should use
    agent.sinks.loggerSink.channel = memoryChannel
    
    # Each channel's type is defined.
    agent.channels.memoryChannel.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    agent.channels.memoryChannel.capacity = 100
    agent.channels.memoryChannel.transactionCapacity = 100

启动命令

bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-mem-conf.properties -Dflume.root.logger=INFO,console

#### flume-hdfs-dir-mem-conf.properties ####

监控文件夹  
agent.sources.seqGenSrc.spoolDir = /opt/cdhmoduels/apache-flume-1.5.0-cdh5.3.6-bin/file/spooling

# Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #  http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
    
    # The configuration file needs to define the sources, 
    # the channels and the sinks.
    # Sources, channels and sinks are defined per agent, 
    # in this case called 'agent'
    
    agent.sources = seqGenSrc
    agent.channels = memoryChannel
    agent.sinks = loggerSink
    
    # For each one of the sources, the type is defined
    agent.sources.seqGenSrc.type = spooldir
    agent.sources.seqGenSrc.spoolDir = /opt/cdhmoduels/apache-flume-1.5.0-cdh5.3.6-bin/file/spooling
    
    # The channel can be defined as follows.
    agent.sources.seqGenSrc.channels = memoryChannel
    
    # Each sink's type must be defined
    agent.sinks.loggerSink.type = logger
    agent.sinks.loggerSink.type = hdfs
    agent.sinks.loggerSink.hdfs.path = /flume/event/hdfsdir
    agent.sinks.loggerSink.hdfs.filePrefix = hive-log
    agent.sinks.loggerSink.hdfs.rollSize = 10240
    agent.sinks.loggerSink.hdfs.rollInterval = 0
    agent.sinks.loggerSink.hdfs.rollCount = 0
    #Specify the channel the sink should use
    agent.sinks.loggerSink.channel = memoryChannel
    
    # Each channel's type is defined.	
    agent.channels.memoryChannel.type = memory
    
    # Other config values specific to each type of channel(sink or source)
    # can be defined as well
    # In this case, it specifies the capacity of the memory channel
    agent.channels.memoryChannel.capacity = 1000
    agent.channels.memoryChannel.transactionCapacity = 1000

启动

bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-hdfs-dir-mem-conf.properties -Dflume.root.logger=INFO,console

[flume]: http://flume.apache.org/FlumeUserGuide.html