Piping shell commands – Mike's home page

I did some looking at my shell history for the past year or so and made some casual observations, here:

and wanted to single out a number of commands and pipes that might be interesting to someone. What follows is a handful of commands from my shell history, from the last year or so.

First though in case you don’t know, a “filter” is a unix/linux command that reads from standard input and writes to standard output (stdin, stdout). There are many of them. You can feed the output from one to the next with the pipe char “|”…

commas is a nifty command (which I wrote) to add commas to any numbers in your stdout: https://github.com/mikemakuch/commas You know how when you do an “ls -l” on a large file, it’s a little tedious to see how many digits are in a long string of integers:


$ ls -l testdatafile.1000M 
-rw-rw-r-- 1 mkm staff 1048576000 Apr 24  2015 testdatafile.1000M

So just add commas to make it more easily readable;


$ ls -l testdatafile.1000M | commas
-rw-rw-r-- 1 mkm staff 1,048,576,000 Apr 24  2,015 testdatafile.1000M

mysqllocalhost is a bash function wrapper for mysql:


$ type mysqllocalhost
mysqllocalhost is a function
mysqllocalhost () 
{ 
    set -x;
    /usr/local/bin/mysql -u<user> -p<pass> -DmyDbName $* 2>&1;
    set +x
}

This reminds me of another nifty trick I do with mysql: (So, the function has my uid and pass coded into it (I know, slap my wrist, but I keep the perms locked down on everything). And adding the “$*” at the end allows me to issue all the sql from the bash shell e.g. If you’re not aware of it you can have mysql read from stdin with <.


$ mysqllocalhost < myquery.sql

And you can have it read from the command line with 3, <‘s which is true of any command that is a well behaved “filter” in the unix shell universe.


$ mysqllocalhost <<< 'select * from my_table;'

Either way we get the sql command history into the shell command history, where it belongs, instead of only inside the mysql shell.

Another way to pipe sql into mysqllocalhost alias:


cat invites.sql | mysqllocalhost | sort | uniq | grep sherman

output sql into a csv


mysqllocalhost   <<< "select *  from phpuxer where user_image is not null and email is not null limit 4 fields terminated by ','"

I run fetchmail 4 times a day to grab a copy of all my incoming emails and archive them locally. I guess I was checking that I had it enabled since I might have it turned off on occasion


crontab -l | grep fetch

Apparently I was working on some restul api that returns json. “json.tool” does a pretty print, it’s just an alias to “python -mjson.tool”.


curl  http://localhost:10010/query/misc | json.tool

Here I was probably examing disk usage in some dir(s) that are getting large


du -ks * | commas
du -ksc * | sort -n
du -ksc Dropbox.bak/ | commas
du -ks IE*  | commas

find gets used a lot


find  . -type f|wc                              # count files in a dir hierarchy
find . -name "*eml" | wc                        # count files ending in "eml"
find . -name "*jpg" -o -name "*png" | head -1

find all occurencs of a file named “counts” and append the contents to all.counts.txt


find . -name counts -exec cat {} \; | sort >> all.counts.txt

find directories containing the string “ssl”


find . -type d | grep -i ssl

find directories ending with the string “avi”


find . -type f | grep  "avi$"

and now remove them


find . -type f | grep  "avi$" | xargs rm

“for x in <glob>” is one of my gotos for renaming files etc:, tho sometimes I’ll just do it in emacs/dired.


for x in *mp4; do f=`echo $x | sed 's/^/xyz/'`; mv "$x" "$f"; done

Here I was looking for certain patches in a git repo


git branch | grep 515
git log | egrep 'GRX-197|GRX-199|GRX-318|GRX-319|GRX-326|GRX-479|GRX-494'
git log | grep HOTFIX

Here I was grabbing all the email addresses out of a spam folder. Each ‘from’ header is of the form: “From: Firstname Lastname <addr@domain.com>”


grep -i "^from:" *eml | sed 's/.*<// ; s/>//' >> domains

osx has the pbcopy and pbpaste which copy to/from clipboard which is nice. I think linux has it now too.


pbpaste  | json.tool
pbpaste  | json.tool |grep descrip

This will strip any extraneous non-ascii chars out of the clipboard – sometimes when I copy stuff off a web page I get a bunch of non-ascii gorp I don’t want:


pbpaste  | pbcopy

Looks like chrome ran away on me. “ps” shows the process status on a Unix/Linux system


ps -ef|grep -i chrome | cut -c1-99 | killps

and killps is a function I have:


$ type killps
killps is a function
killps () 
{ 
    kopt=$1;
    while read line; do
        pid=`echo $line | awk '{print $2}'`;
        pids="$pids $pid";
    done;
    echo -n "Kill $pids?";
    read ans < /dev/tty;
    if [[ "y" != "$ans" ]]; then
        exit;
    fi;
    com="kill $kopt $pids";
    $com
}

I use rsync frequently; for copying directories recursively from one host to another. If you’re not familiar with it all you need to startout is this


rsync -av <source> <dest>

where <source> is the dir you want to copy to <dest>. One of source/dest can be remote like this


rsync -av /path/to/my/dir remoteHost:/path/to/remote/place

The trailing slash is critical in the source/dest. Above will copy the source “dir” to the “place” folder on remoteHost. If you instead issue;


rsync -av /path/to/my/dir/ remoteHost:/path/to/remote/place

Then it’ll copy the contents of source “dir” to the “place” folder on remoteHost.

copying pics from my android phone


rsync -nav -e 'ssh -p 2222'  192.168.1.11:/storage/extSdCard/DCIM/Camera/ camera 
rsync -navu -e 'ssh -p 2222'  192.168.1.11:/storage/extSdCard/DCIM/Camera/ .

Looking for env vars. “set” dumps all the env vars in your shell.


set|grep -i node
set|grep NODE
set|grep EMAIL

I’m on a mac. launchctl is how you start/stop daemons on osx.


sudo launchctl list | grep elastic

I use locate/updatedb to keep all files indexed. “updatedb” is run as a crontask and indexes all files. “locate” searches the index for you.


sudo locate -i ".com" | egrep -iv "common|comp" | tee dot.com
sudo locate -i ".com" | egrep -iv "mail|common|comp" | tee dot.com

What we need is more cowbell! Well, here’s more on the shell instead Mastering the shell