Is there a way to diagnose unexpected flushes of pagecache in RHEL 5?

Environment

• Red Hat Enterprise Linux (RHEL) 5

Issue

• My system is building up pagecache during backups or other operations and then it is suddenly being flushed, causing performance problems.
• What is causing the kernel to suddenly flush page cache?
• My cluster node is experiencing sudden cache flushes which leads to increased CPU usage by glock_workqueue

Resolution

Install SystemTap and then run the following script, capturing the output:

    #!/usr/bin/env stap
    probe begin {
            printf("Beginning drop_caches watch at %s\n", ctime(gettimeofday_s()));
    }
    probe end {
            printf("Ending drop_caches watch at    %s\n", ctime(gettimeofday_s()));
    }
    probe kernel.function("drop_slab") {
            printf("slab dropped by %s at %s\n", execname(), ctime(gettimeofday_s()));
    }
    probe kernel.function("drop_pagecache") {
            printf("pagecache dropped by %s at %s\n", execname(), ctime(gettimeofday_s()));
    }
    probe kernel.function("try_to_free_pages") {
    printf("Entered page reclamation, time %s\n", ctime(gettimeofday_s()));
    print_backtrace();
    printf("\n");
    }

NOTE: This is a diagnostic script provided to capture specific events. It may not determine every source of cache flushing, nor is it guaranteed to be effective in all environments or kernel releases. Please contact Red Hat Global Support Services for further assistance in diagnosing problems such as these.

Run the above script until one of the cache flushes in question occurs, and then review the information printed by the script at the time. It may indicate a manual cache flush using vm.drop_caches, or an attempt to free pages and the relevant backtrace in the kernel that triggered it.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.