Howto : Automate Your Tasks With BASH Scripting

[darrenm]

The main thing that draws me to Linux is the way it gives you ability to truly harness the power of your system. You can become the director of your computer and compose beautiful symphonies of automation just by understanding a little about Bash and Bash scripting.

I'm in no way a Linux guru, I've been using it for years but I've never had time to properly get into software development with Linux so even if you're new to the world of Ubuntu/Linux this guide should make sense and help you automate your tasks giving you more time to enjoy the system.
This will cover a few things including:

• Simple bash scripting

• Simple ImageMagick conversion

• Simple Expect scripting

Don't let the word scripting scare you off, it really is simple and - once you understand it you will realise how your computer can do any mundane task for you quickly and efficiently.

Our scenario:
Random J. Newb loves his cars, he loves to fiddle with the engine and take pictures as he goes along to upload to his car forum where he can describe how difficult the front cam sprocket was to remove. He has a digital camera that he plugs into his computer running Ubuntu 6.06 Dapper. He wants to keep his photos at full size (1600x1200) but to display on the web he wants to resize them to 800x600 and compress them as JPEG files so he can upload them to his webspace via FTP. He can then toddle along to his www.ladarivaownersclub.com forum and insert the images with [img] tags.
Currently he has to open the photos using the GIMP, resize each one individually and save each one with JPEG compression to a location on his computer. Then he has to open gFTP, browse to the correct local and remote directory and send the files over.
If it has been a long day on the car this can take a long time to do and he wonders if there can be a quicker way...

Fear not Randy, this isn't Windows(tm)

Lets get started with the basics. When the camera is connected it is automatically mounted in /media/usbdisk by Ubuntu.

Lets create a file to make our script in. You don't really need to but it doesn't hurt to know what you're going to do.

sudo touch /bin/picwhizz

this 'touches' the file into existance.

OK Heres where the meat starts.

cd /media/usbdisk
ls
somedodgysettingsfile.ini PICT2270.JPG  PICT2271.JPG  PICT2272.JPG  PICT2273.JPG  PICT2274.JPG  PICT2275.JPG  PICT2276.JPG  PICT2278.JPG  PICT2279.JPG

So we know all the files we are going to want end in .JPG (not .jpg, it matters in our world)

ls *JPG
PICT2270.JPG  PICT2271.JPG  PICT2272.JPG  PICT2273.JPG  PICT2274.JPG  PICT2275.JPG  PICT2276.JPG  PICT2278.JPG  PICT2279.JPG

Gives us only the pictures. Aha. We now know how to get a list of the correct files, thats all we need to start the script.

sudo gedit /bin/picwhizz

Lets tell the script to use bash to run. Not required but makes you feel a bit nerdy.

#!/bin/bash

The she-bang (pound-bang / hash-bang) tells bash to er, use bash to run the file.

MY_HOME=`echo ~`
PICS_TEMP=$MY_HOME/picstemp
FOR_WEB=$PICS_TEMP/forweb

This is where we set up some constants. We need something concrete to stand on before using the results in the upcoming tests. You can see we use command substitution for MY_HOME to assign it the result of "echo ~" This just prints out the home directory of the current user.

[ ! -d "$PICS_TEMP" ] && mkdir $PICS_TEMP
[ ! -d "$FOR_WEB" ] && mkdir $FOR_WEB

Wow. Deep end? Nah, lets just see what its doing. It's testing to see if the directory ~/picstemp exists. The "-d" tells it to look for a directory and report true if its there. But wait... The exclamation mark reverses this and makes it report true if the directory does not exist. The "&&" means "do the following if true."

So in essence the statement checks for ~/picstemp (the ~ means your home directory) and if its not there, it creates it. Lovely.

After its done that, it does exactly the same for a directory to store the resized pics in called ~/picstemp/forweb

The next part is to move the photos into this directory. Random J. doesnt want to have to delete the photos off his camera himself so we will move them rather than just copy them.

mv /media/usbdisk/*JPG $PICS_TEMP
cd $PICS_TEMP

Simple enough. Moves anything that ends in JPG to the picstemp directory in his home then cd's into ~/picstemp.

Now it gets exciting - For loops . I'll loosely describe the way they work as for ANYVARIABLE in `substitute your command here and send then output back` ; do something on that $ANYVARIABLE ; done Perhaps the following may make sense:

for MYPIC in `ls *JPG` ; do convert $MYPIC -resize 800x600 -quality 75% $FOR_WEB/$MYPIC ; done

Go over your head? Don't worry, its easy. Read through it and decipher what its doing. Its all about substitution...

for MYPIC in `ls *JPG`
literally does ls *JPG and takes the first result. In this case PICT2270.JPG and assigns it to the variable MYPIC. So now the variable MYPIC contains "PICT2270.JPG"

do convert $MYPIC -resize 800x600 -quality 75% forweb/$MYPIC ; done .
To see what this is doing just substitute the value of our variable so its really running convert PICT2270.JPG -resize 800x600 -quality 75% forweb/PICT2270JPG . This runs the imagemagick command convert (A wonderful program), resizes it to 800x600, compresses it to 75% quality and saves the resulting file in the directory forweb/ and uses the same filename to save it.

Now the magic bit. We don't only want 1 file to be converted, we could have just typed in the filename ourselves. The for loop is called a loop because it, well, loops. Once the first iteration succeeds, it then does the whole thing again on the next line it got. Remember the ls JPG first result was PICT2270.JPG? Well now the 2nd line is PICT2271.JPG. Making sense? Getting a feeling of how wonderful Linux/BASH/GNU is?

It runs the convert tool on every file until there are no more and completes. 7 lines of code (1 line that actually does something meaningful) and we have got some nice smaller, compressed photos automagically from the camera (Well, Randy has anyway.)

Now we just need to make the script executable. Easy.

sudo chmod a+x /bin/picwhizz

All thats left is to run it.

picwhizz

Depending on the amount of photos and the speed of the computer it may take a few seconds to a few minutes to convert the images. Its pretty silent but you can test it worked by going into ~/picstemp/ and running

ls -alth

to show you the file sizes then

cd ~/picstemp/forweb
ls -alth

The files should be a lot smaller.

Now to FTP them. Shall we let the computer connect to the FTP server, log in and send the files to the correct directory and then disconnect again all by itself? For this we need a wonderful thing called expect. It does what it says, it expects things. When it gets what it expects it does something and then expects something else. When you tell it what to expect and then what to do when it gets something it expects then the world is your cloister.

I'm off to write it. Is anyone interested in this series? shall I continue?

[ZenobiaBE]

Please do continue, if the next parts are going to be as good as this one, I can't wait

[kabus]

[ -d ! "$PICS_TEMP ] && mkdir ~/picstemp

You got a little typo there, it should be:

[ ! -d "$PICS_TEMP" ] && mkdir ~/picstemp

[kabus]

for MYPIC in `ls *.JPG`

for MYPIC in *JPG

Easier and can handle filenames with spaces in them.

[darrenm]

OK, all tidied up. Teach me to do it while I'm being disturbed at work.

This is the script after you put it together to avoid confusion:

#!/bin/bash

MY_HOME=`echo ~`
PICS_TEMP=$MY_HOME/picstemp
FOR_WEB=$PICS_TEMP/forweb

[ ! -d "$PICS_TEMP" ] && mkdir $PICS_TEMP
[ ! -d "$FOR_WEB" ] && mkdir $FOR_WEB

mv /media/usbdisk/*JPG $PICS_TEMP
cd $PICS_TEMP

for MYPIC in `ls *.JPG` ; do convert $MYPIC -resize 800x600 -quality 75% $FOR_WEB/$MYPIC ; done

[drewsimon]

Yes! Please do continue this series. This is exactly the kind of thing I've been looking for.

[Jvaldezjr]

I want to add my vote for continuing. I'm not a linux pro, but I'm not totally a newb either. Needless to say I find this helpful in learning better linux.

[ghowells]

Yes yes, expect scripting is still somewhat of a mystery to me, please continue oh master of all that is script

[darrenm]

If you read the BASH scripting bit and thought it was pretty cool what you can do with a few tests, operators and variables then this will be the coolest thing since the Fonz's fridge.

Remember Random J. Newb wants to plug his digital camera in to his PC, press one button and a few moments later have them appear on his webspace? Well previously we wrote a script that moves the files from his camera, resizes them to 800x600 and then compresses them 75%. Finally it puts them in a directory called forweb.

To top it off, to really put the icing on the cake we should make the script connect to his ISP's FTP server, upload any files in the forweb directory and then disconnect. Making him a cup of Coffee would be nice too but thats for a later time.

Let's get started. We need a program called expect. Like Ronseal it does what it says on the tin.

sudo apt-get install expect

You should already have it really, a few things depend on it.

Before we integrate it into our earlier script we should really test to see what happens and if it's going to do the job.

cd ~
sudo ln -sf /usr/share/doc/expect/examples/autoexpect /usr/bin/

I think once you sample autoexpect you may be using it a little more so lets put it into somewhere in the path. ln -sf creates a symbolic link from /usr/share/doc/expect/examples/autoexpect to /usr/bin/autoexpect

Now get ready to work nice and tidy because autoexpect will be watching you. It may be worth a test run of logging into the FTP server and transferring files to make sure it works. Here's what I do (names changed to protect the innocent):

darrenm@dazlinux:~/picstemp/forweb$ ftp ftp.c*****ce.org
Connected to ftp.c*****ce.org.
220---------- Welcome to Pure-FTPd [TLS] ----------
220-You are user number 2 of 50 allowed.
220-Local time is now 09:47. Server port: 21.
220 You will be disconnected after 15 minutes of inactivity.
Name (ftp.c*****ce.org:darrenm): w*****
331 User w***** OK. Password required
Password:
230-User w***** has group access to:  ***** 
230 OK. Current restricted directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt off
Interactive mode off.
ftp> cd www/images
250 OK. Current directory is /www/images
ftp> mput *
local: PICT2740.JPG remote: PICT2740.JPG
200 PORT command successful
150 Connecting to port 12017
226-File successfully transferred
226 5.949 seconds (measured here), 17.20 Kbytes per second
...
153630 bytes sent in 6.39 secs (23.5 kB/s)
ftp> quit
221-Goodbye. You uploaded 1285 and downloaded 0 kbytes.
221 Logout.
darrenm@dazlinux:~/picstemp/forweb$

So lets do it. Deep breath...

autoexpect -p

This is where you do your own thing. Type the same commands as you normally would. For me these consist of:

cd ~/picstemp/forweb
ftp ftp.c*****ce.org
(username)
(password)
cd (remote directory)
prompt off
mput *
quit

Type exit to stop autoexpect recording. Test it works by running it

./script.exp

And sit back in amazement as it replays what you just did. If you have different files in the ~/picstemp/forweb directory than before it will FTP those instead as we are transferring with mput *

If all goes well with the autoexpect script we can integrate it into the old picwhizz BASH script we did earlier. Rename it so you know what it is and put it somewhere sensible:

sudo cp script.exp /usr/bin/autoftp.exp

And add it to the bottom of the picwhizz script (also add a cd ~ to give the expect script the same starting location as when you recorded it)

sudo gedit /bin/picwhizz
#!/bin/bash

MY_HOME=`echo ~`
PICS_TEMP=$MY_HOME/picstemp
FOR_WEB=$PICS_TEMP/forweb

[ ! -d "$PICS_TEMP" ] && mkdir $PICS_TEMP
[ ! -d "$FOR_WEB" ] && mkdir $FOR_WEB

mv /media/usbdisk/*JPG $PICS_TEMP
cd $PICS_TEMP

for MYPIC in `ls *.JPG` ; do convert $MYPIC -resize 800x600 -quality 75% $FOR_WEB/$MYPIC ; done

cd ~
/usr/bin/autoftp.exp

All being well, this will now take any files ending in JPG off the camera, move them into a folder in Randys home directory called picstemp, resize them to 800x600, compress them to 75% quality, copy them to a directory called forweb, FTP them to his webspace and then disconnect. Wow.

If we're feeling really flash we can add an icon to the panel which runs picwhizz. So then all he has to do is connect his camera, click the icon on the panel and then open firefox, type in www.ladarivaownersclub.com and get posting!

Anyone have any problems with any of this?

[chk9]

Cool darrenm!

Now I have to go and checkout what other cool things I can do with autoexpect...