gobblin.util

Class FileListUtils

java.lang.Object

gobblin.util.FileListUtils

public class FileListUtils
extends Object

Utility class for listing files on a FileSystem.

See Also:

FileSystem

Field Summary

Fields

Modifier and Type	Field and Description
static Comparator<FileStatus>	LATEST_MOD_TIME_ORDER
static PathFilter	NO_OP_PATH_FILTER

Constructor Summary

Constructors

Constructor and Description
FileListUtils()

Method Summary

Modifier and Type	Method and Description
static List<FileStatus>	listFilesRecursively(FileSystem fs, Iterable<Path> paths)
static List<FileStatus>	listFilesRecursively(FileSystem fs, Path path)
static List<FileStatus>	listFilesRecursively(FileSystem fs, Path path, PathFilter fileFilter) Helper method to list out all files under a specified path.
static List<FileStatus>	listFilesRecursively(FileSystem fs, Path path, PathFilter fileFilter, boolean applyFilterToDirectories) Helper method to list out all files under a specified path.
static List<FileStatus>	listMostNestedPathRecursively(FileSystem fs, Iterable<Path> paths)
static List<FileStatus>	listMostNestedPathRecursively(FileSystem fs, Path path) Method to list out all files, or directory if no file exists, under a specified path.
static List<FileStatus>	listMostNestedPathRecursively(FileSystem fs, Path path, PathFilter fileFilter) Method to list out all files, or directory if no file exists, under a specified path.
static List<FileStatus>	listPathsRecursively(FileSystem fs, Path path, PathFilter fileFilter) Helper method to list out all paths under a specified path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LATEST_MOD_TIME_ORDER

public static final Comparator<FileStatus> LATEST_MOD_TIME_ORDER

NO_OP_PATH_FILTER

public static final PathFilter NO_OP_PATH_FILTER

Constructor Detail

FileListUtils

public FileListUtils()

Method Detail

listFilesRecursively

public static List<FileStatus> listFilesRecursively(FileSystem fs,
                                                    Path path)
                                             throws IOException

Throws:

IOException

listFilesRecursively

public static List<FileStatus> listFilesRecursively(FileSystem fs,
                                                    Iterable<Path> paths)
                                             throws IOException

Throws:

IOException

listFilesRecursively

public static List<FileStatus> listFilesRecursively(FileSystem fs,
                                                    Path path,
                                                    PathFilter fileFilter)
                                             throws IOException

Helper method to list out all files under a specified path. The specified PathFilter is treated as a file filter, that is it is only applied to file Paths.

Throws:

IOException

listFilesRecursively

public static List<FileStatus> listFilesRecursively(FileSystem fs,
                                                    Path path,
                                                    PathFilter fileFilter,
                                                    boolean applyFilterToDirectories)
                                             throws IOException

Helper method to list out all files under a specified path. If applyFilterToDirectories is false, the supplied PathFilter will only be applied to files.

Throws:

IOException

listMostNestedPathRecursively

public static List<FileStatus> listMostNestedPathRecursively(FileSystem fs,
                                                             Path path)
                                                      throws IOException

Method to list out all files, or directory if no file exists, under a specified path.

Throws:

IOException

listMostNestedPathRecursively

public static List<FileStatus> listMostNestedPathRecursively(FileSystem fs,
                                                             Iterable<Path> paths)
                                                      throws IOException

Throws:

IOException

listMostNestedPathRecursively

public static List<FileStatus> listMostNestedPathRecursively(FileSystem fs,
                                                             Path path,
                                                             PathFilter fileFilter)
                                                      throws IOException

Method to list out all files, or directory if no file exists, under a specified path. The specified PathFilter is treated as a file filter, that is it is only applied to file Paths.

Throws:

IOException

listPathsRecursively

public static List<FileStatus> listPathsRecursively(FileSystem fs,
                                                    Path path,
                                                    PathFilter fileFilter)
                                             throws IOException

Helper method to list out all paths under a specified path. If the FileSystem is unable to list the contents of a relevant directory, will log an error and skip.

Throws:

IOException