gobblin.util.logs

Class LogCopier

java.lang.Object

com.google.common.util.concurrent.AbstractScheduledService

gobblin.util.logs.LogCopier

All Implemented Interfaces:

Service

public class LogCopier
extends AbstractScheduledService

A utility class that periodically reads log files in a source log file directory for changes since the last reads and appends the changes to destination log files with the same names as the source log files in a destination log directory. The source and destination log files can be on different FileSystems.

This class extends the AbstractScheduledService so it can be used with a ServiceManager that manages the lifecycle of a LogCopier.

This class is intended to be used in the following pattern:

     
       LogCopier.Builder logCopierBuilder = LogCopier.newBuilder();
       LogCopier logCopier = logCopierBuilder
           .useSrcFileSystem(FileSystem.getLocal(new Configuration()))
           .useDestFileSystem(FileSystem.get(URI.create(destFsUri), new Configuration()))
           .readFrom(new Path(srcLogDir))
           .writeTo(new Path(destLogDir))
           .useInitialDelay(60)
           .useCopyInterval(60)
           .useMaxMinutesPerLogFile(60)
           .useMaxBytesPerLogFile(1024 * 1024)
           .useTimeUnit(TimeUnit.SECONDS)
           .build();

       ServiceManager serviceManager = new ServiceManager(Lists.newArrayList(logCopier));
       serviceManager.startAsync();

       // ...
       serviceManager.stopAsync().awaitStopped(60, TimeUnit.SECONDS);
     
   

Checkout the Javadoc of LogCopier.Builder to see the available options for customization.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LogCopier.Builder A builder class for LogCopier.

Nested classes/interfaces inherited from class com.google.common.util.concurrent.AbstractScheduledService

AbstractScheduledService.CustomScheduler, AbstractScheduledService.Scheduler

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

Service.Listener, Service.State

Method Summary

Modifier and Type	Method and Description
static LogCopier.Builder	newBuilder() Get a new LogCopier.Builder instance for building a LogCopier.
protected void	runOneIteration()
protected AbstractScheduledService.Scheduler	scheduler()
protected void	shutDown()

Methods inherited from class com.google.common.util.concurrent.AbstractScheduledService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, start, startAndWait, startAsync, startUp, state, stop, stopAndWait, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

shutDown

protected void shutDown()
                 throws Exception

Overrides:

shutDown in class AbstractScheduledService

Throws:

Exception

runOneIteration

protected void runOneIteration()
                        throws IOException

Specified by:

runOneIteration in class AbstractScheduledService

Throws:

IOException

scheduler

protected AbstractScheduledService.Scheduler scheduler()

Specified by:

scheduler in class AbstractScheduledService

newBuilder

public static LogCopier.Builder newBuilder()

Get a new LogCopier.Builder instance for building a LogCopier.

Returns:

a new LogCopier.Builder instance