How can I run strace forever
The following script will create a ring of N files (N provided as an argument) with the names /tmp/cycle-strace-PID-N. PID (provided as an argument) is the the process ID of a process and N ranges from 1-N. Each file is the output of running strace over the PID for S seconds, with S provided as an argument. At the end of each run of strace a test is conducted to determine if the script should continue or be terminated.
CYCLE-STRACE.SH
#!/bin/bash
# cycle-strace.sh begins on the previous line
# Save this script as cycle-starce.sh
# Allocate read write execute permissions: chmod +rwx cycle-strace.sh
# Help available with: ./cycle-strace.sh -h
VERSION=3
USAGE=$(cat <<-EOM
Usage: cycle-strace.sh [-f FILES] [-s SECONDS] [-t test-script ][-h] [-a ARGS] PID
This script will run strace over the process PID for [-s SECONDS]
writing the output to a /tmp/cycle-strace-PID-N file.
After [-s SECONDS] the strace will terminate and restart writting to file
/tmp/cycle-strace-PID-(N+1 % [-f FILES]).
-f FILES (default: 10)
Number of files to be collected. Indexcies at the end of the file name will
range from 0 to FILES - 1
-s SECONDS (default: 60)
The number of seconds to run a single instance of strace
-t test-script (default: no script, continue until manually killed)
The name/path to a script that returns either "true" or "false". If false
the loop will terminate. If no script is provided you will need to manually
kill the looping process -- control-C will not work. Killing the looping
process will not kill the strace process that is currently running you will
need to kill it separately or just let the execute time out.
-h
Displays this help message
-a ARGS (default: Tttvxfs 1024 or if yy is supported: Tttyyvxfs 1024)
Strace arguments, if including a space it must be enclosed in quotes.
Do not include a leading dash character.
PID
PID of the process to run strace over -- this is not an optional argument
EOM
)
## defaults
FILES=10
TIMEOUT=60
TEST="none"
if [ $( which strace 2>/dev/null | wc -l) -eq 0 ]
then echo "Do not seem to be able to find strace."
echo "Please be sure it is installed."
echo "You can \"yum install strace\" to install it."
exit
fi
if [ $(man strace | grep -c "\\-yy") -eq 1 ]
then ARGS="Tttyyvxfs 1024"
else ARGS="Tttvxfs 1024"
fi
## option parsing
while getopts ":s:f:t:a:h" OPT
do
case "$OPT" in
"f")
# Make sure number of files is a positive integer
if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] \
2>/dev/null
then FILES=$OPTARG
else echo "ERROR: $OPTARG is not a value number of FILES"
exit 1
fi
;;
"s")
# Make sure number of seconds is a positive integer
if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] \
2>/dev/null
then TIMEOUT=$OPTARG
else echo "ERROR: $OPTARG is not a value number of SECONDS"
exit 1
fi
;;
"t")
TEST=$OPTARG
;;
"h")
echo "$USAGE"
exit 0
;;
"a")
ARGS=$OPTARG
;;
":")
echo "ERROR: -$OPTARG requires an argument, run cycle-strace.sh -h"
exit 1
;;
"?")
echo "ERROR: -$OPTARG is not a valid option, run cycle-strace.sh -h"
exit 1
;;
esac
done
shift $((OPTIND-1))
if [ "X$@X" == "XX" ]
then echo "ERROR: PID argument not provided, PID is not optional"
exit 1
else PID=$@
fi 2>/dev/null
if ! [ "$PID" -eq "$PID" ]
then echo "$PID is not a valid PID"
exit
fi
rm -f /tmp/cycle-strace-"$PID"-*
# print out the argument values that will be used
echo "$0" -a "$ARGS" -s "$TIMEOUT" -f "$FILES" -t "$TEST" "$PID"
while true
do
for n in $(seq "$FILES")
do
if [ $(ps -e | awk '{print $1}' | grep -c "^$PID$") -eq 0 ]
then echo "Process with PID $PID has terminated, exiting at" \
$(date +"%F %T.%N%:z (%Z)")
exit
fi
timeout "${TIMEOUT}"s strace -"$ARGS" -o /tmp/cycle-strace-"$PID"-"$n" -p "$PID"
echo $(date +"%F %T.%N%:z (%Z)") "$0" time out \
>> /tmp/cycle-strace-"$PID"-"$n"
if [ "$TEST" == "none" ]
then echo To end this looping data collection run type: kill -9 $$
elif [ $($TEST) == "false" ]
then echo "Test script has returned false, exiting at" \
$(date +"%F %T.%N%:z (%Z)")
exit
fi
done
done
#
# cycle-strace.sh ends here
EXAMPLE TEST SCRIPTS
Test if something is listening to a given port and return "false" when there is no longer a process listening to the port.
#!/bin/sh
#
# Takes 1 argument which is a port number. Return true only if it is found that something is listening
# on that port number
if [ $# -ne 1 ]
then echo false
exit
fi
if ! [ "$1" -eq "$1" ]
then echo false
exit
fi 2>/dev/null
if [ $(ss -ntl src :"$1" | grep -c LISTEN) -gt 0 ]
then echo true
else echo false
fi
example usage
# ./cycle-strace.sh -s 5 -t "./test.sh 21" 1690
./cycle-strace.sh -a Tttyyvxfs 1024 -s 5 -f 10 -t ./test.sh 21 1690
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
Test script has returned false, exiting at 2018-05-04 14:18:00.128139605-07:00 (MST)
test2.sh -- send 5 pings to the IP address indicated in the first argument, return "false" if the reported packet loss percentage is greater than the percentage indicated by the second argument. To report "false" if there is any packet loss the second argument should be 0.
#!/bin/sh
#
# Takes 2 arguments. The first is an IP address and the second is a percentage.
# Sends 5 pings to the IP address in argument 1, 0.2 seconds apart. Returns
# true if packet loss is less than or equal to the percentage in argument 2
if [ $# -ne 2 ]
then echo false
exit
fi
if [[ ! "$1" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
then echo false
exit
fi
if ! [ "$2" -eq "$2" ]
then echo false
exit
fi 2>/dev/null
ping -q -i 0.2 -c 5 "$1" | tail -2 | head -1 | \
awk '{ for (x=1; x<=NF;x++) if ($x=="loss," && $(x-1) == "packet" ) print $(x-2)}' | \
tr -d "%" | awk -v percent="$2" '{if ($1 > percent) print "false"; else print "true"}'
example usage
# ./cycle-strace.sh -s 5 -t "./test2.sh 192.168.1.1 50" 1690
./cycle-strace.sh -a Tttyyvxfs 1024 -s 5 -f 10 -t ./test2.sh 192.168.1.1 50 1690
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
Test script has returned false, exiting at 2018-05-04 14:21:26.957025358-07:00 (MST)
test3.sh looks for a string in a file and if the number of occurrences of that string have gone up then return "false". This is a little tricky since each invocation of the test is independent there is no memory. To create a memory the test is run once before the cycle-strace script is called to record the number of occurrences in a file. The file name is passed as the first argument to the test script. If you used a fixed name only one instance of the script can be run at a time. The example creates a random-ish string so that multiple instances can be run. All arguments after the second are considered as part of the trigger string do not enclose them in quotes.
# cat test3.sh
#!/bin/sh
#
# Return true if the number of lines in a file that match a string have gone up
if [ $# -lt 3 ]
then echo false
exit
fi
if ! [ -e "$2" ]
then echo false
exit
fi
MATCH=""
for x in $(seq 3 $#)
do MATCH=$(echo $MATCH "${!x}")
done
if ! [ -e /tmp/"$1"-first ]
then grep -c "$MATCH" "$2" > /tmp/"$1"-first
exit
fi
grep -c "$MATCH" "$2" > /tmp/"$1"-new
if [ $(cat /tmp/"$1"-new) -gt $(cat /tmp/"$1"-first) ]
then rm -f /tmp/"$1"-*X
echo false
else echo true
fi
example usage
# X=$(date|md5sum|cut -c 1-16); ./test3.sh $X /var/log/messages mount failed; ./cycle-strace.sh -s 5 -t "./test3.sh $X /var/log/messages mount failed" 1690
./cycle-strace.sh -a Tttyyvxfs 1024 -s 5 -f 10 -t ./test3.sh c93c9693be1d7de3 /var/log/messages mount failed 1690
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
strace: Process 1690 attached
strace: Process 1690 detached
Test script has returned false, exiting at 2018-05-04 15:12:25.394737967-07:00 (MST)