Distributed file systems that underlie big data processing frameworks often employ data replication to guard against node failures. A common practice in these file systems is to couple replication with primary copy writes. Unfortunately, under real workloads, the replication traffic ends up contending arbitrarily for limited disk bandwidth against each other and with time sensitive data flows. As a result, task execution times are significantly increased and cluster-wide resource utilization becomes inefficient.
We present FC-FS(Flow controlled file system), a departure from conventional design that allows decoupling of replication from primary copy writes and performs job-aware flow control on file system traffic. Just as you would not backup your computer when it is busy, FC-FS pauses replication when computers in a cluster are busy.
FC-FS employs established flow control concepts such as credit-based flow control and weighted fair queuing in novel ways to allow the realization of a variety of policy objectives, including speeding up executing tasks, rapidly allocating under-utilized disk bandwidth to replication writes, reducing the time to reach failure resilience for important jobs, and progressively achieving increasing levels of failure resilience.
We have augmented HDFS to realize FC-FS. Extensive evaluation on a 30-node Hadoop cluster shows that FC-FS can halve the duration of a job’s write phase while reducing average job runtime by up to 20%.
