shell scripts

[JunkMale]

Any idea what shell scripts are written in?

The web host was kind enough to provide me with a script to help run webalizer to update the webalize.hist file

I wrote a script that scrapes this .hist file for its stats for the web site main page.

All was fine until the site started getting more than 40k a month in visits as I was starting to get cronjob error messages because of some locking a file.

So how does one write a shell script? I found some sites byt they all do the same thing, launch off in to technical jargon assuming that you have prior systems knowledge of what square peg gets hammered in to whar round hole.

The shell script has... #! /bin/bash
cd /home/sites/****.***.**/public_html/stats
/usr/bin/webalizer -D dns.cache -nlocalhost -p -o . /home/sites/****.***.**/logs/****.***.**-access_log*

I would like to add to this sctipt the ability to repeat 20 times maximum attempts to read the dns.cache when theirs a lock error or is their a way to tell webalizer to access the file with a shared lock?

[scragar]

Dude, bash is easy:
#! /bin/bash

# cd = change directory.
cd /home/sites/****.***.**/public_html/stats

# for(I = 1; I <= 20; I++){
for I in $(seq 20); do
## Try the script, if it returns true we break out of the loop
/usr/bin/webalizer -D dns.cache -nlocalhost -p -o . /home/sites/****.***.**/logs/****.***.**-access_log* && break;

## we couldn't lock the file. Wait a second, then try again:
sleep 1s;

#end for loop
done

Oh, and to answer your very first question, shell scripts are written in whatever it says to use as a parser:
#!/bin/bashEffectively means "Run me using the parser bash, which can be found in /bin/ ". If you wanted to you could use PHP, Perl, Python, or almost any language you could think of as a parser(including other scripts, which are handled in the same way).

[JunkMale]

OK, thanks, I just about followed that, why they cant just have a for loop?!

What about making it pause for a one second limit per loop? as the thing currently stands, it will execute immediately one after another? or will it wait until the result of the last execution of the webalizer command?

[scragar]

OK, thanks, I just about followed that, why they cant just have a for loop?!Bash supports C style for loops. I just find typing a few less characters and using seq is easier(especially given the `break` usage).

http://bash-hackers.org/wiki/doku.php/syntax/ccmd/c_for
What about making it pause for a one second limit per loop? as the thing currently stands, it will execute immediately one after another? or will it wait until the result of the last execution of the webalizer command?

Bash will always wait for each command to return success or failure before continuing to the next command, UNLESS the first command is switched to the background.

So the 1second wait isn't really needed, I just included it because I have no idea how long webaliser tries to run for before it encounters the error, if it's a matter of a few seconds then the wait isn't needed, but if the script tries to run for a millisecond, and returns failure the instant it finds the file lock then all the iterations are likely to run before the file lock would be released.

[JunkMale]

Thanks.

[JunkMale]

I have hit a snag and need to have this script run every hour but only at certain times the webalizer needs to run like every 3 hours..

The times are in hours 00,06,09,12,15,18,21

Would I be right in thinking that this would work if thehour has the hour value? and next question would be how do you set thehour to the current hour?


#! /bin/bash

THEHOUR = ???

# THEHOUR will be the current hour of the day.
case THEHOUR in (00 | 06 | 09 | 12 | 15 | 18 | 21 )

# cd = change directory.
cd /home/sites/****.***.**/public_html/stats

# for(I = 1; I <= 20; I++){
for I in $(seq 20); do
## Try the script, if it returns true we break out of the loop
/usr/bin/webalizer -D dns.cache -nlocalhost -p -o . /home/sites/****.***.**/logs/****.***.**-access_log* && break;

## we couldn't lock the file. Wait a second, then try again:
sleep 1s;

#end for loop
done;;
esac



What I need is to check if the hours are any in the list of hours and if so eun the update routine. This will help me free up the rather limited access to cronjobs as I need to run a PHP script as a shell at 9pm and that means I need to modify this shell so the daily image roatate routine can be run automatically.

Or can you point me to a dummies guide? I have been trying to find examples but I havent a clue what to put as using shell scripts as a search term turns up all other scripting that can run as a shell.

[scragar]

Firstly I'd like to point out that cron supports comma seperated lists for time, provided there's no spaces.

0 0,3,6,9,12,15,18,21 * * * your_bash_script.sh

Alternatively you can get the time with date and test it that way:


HOUR=$(date +%l); ## date [options] +FORMAT
## %l = hour 1-12, no leading 0s

IS3HOUR=$(($HOUR % 3));## modulus maths :p

if [ $IS3HOUR -eq 0 ]; then

## loop in here

fi

Personally I'd do it all as a 1 liner :p
[ $(( $(date +%l) % 3)) -ne 0 ] && exit;
## exit the script if the current hour is not an exactly multiple of 3

[JunkMale]

First off, thanks for the rocket science lessons...

I neglected to mention that the host limits each domain to a maximum of 3 crons and they are only settable via the account CP and that they are also limited to what can be set as times.


So if [ $(( $(date +%l) % 3)) -ne 0 ] && exit;
## exit the script if the current hour is not an exactly multiple of 3 is placed as the first line it will end the script if its not a division of 3.

#! /bin/bash

[ $(( $(date +%l) % 3)) -ne 0 ] && exit;
## exit the script if the current hour is not an exactly multiple of 3

# cd = change directory.
cd /home/sites/****.***.**/public_html/stats

# for(I = 1; I <= 20; I++){
for I in $(seq 20); do
## Try the script, if it returns true we break out of the loop
/usr/bin/webalizer -D dns.cache -nlocalhost -p -o . /home/sites/****.***.**/logs/****.***.**-access_log* && break;

## we couldn't lock the file. Wait a second, then try again:
sleep 1s;

#end for loop
done


I guess this is going to take me some time to learn a new trick.

[scragar]

If there's anything I've not explained, or you have a hard time understanding I'd love to try and explain it to you(again?).

Bash is rather hard to explain though when working with more complex code, if you start with bash where most people do(it's similar to batch files in windows, a list of commands to be executed in the shell) it's a lot easier.

For reference:
$( something ) execute something
$(( something )) evaluate something using the maths rules
[ something ] test something and return success(true 0) or failure(false 1)

So some of the parts that might have been confusing explained:
for I in $(seq 20);
++ `seq 20` == 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+ for I in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20;

[ $(( $(date +%l) % 3)) -ne 0 ] && exit;
+++ `date +%l` == " 8"
++ `8%3` == 2
+ [2 -ne 0] == true
+ exit
## && means if the previous exited on success do this

[JunkMale]

I may have to sleep on that and look at it with fresh eyes.

[JunkMale]

[ $(( $(date +%l) % 3)) -ne 0 ] && exit;

Breaking this down, date +%l how does the +%l work? I am lost on the GNU BASH site, can't even find 'date'

Anyway, I guess this is roughly what it does is it gets the datetime stamp from the system, extracts the hour, passes it throught a mod of 3 that is tested to see if it is not equal to 0 then return value. The value returned depends on the hour, if a true is returned then the script is terminated.

I dont suppose you have any pointers on how the following problem.

I wrote a PHP script. I tested it, it worked, I then converted it to a shell running as a PHP file. They have said I had to remove the shebang which I did, put in to the cron job the shebang path followed by the script with the php extention.

I hit the test script button in the admin CP and I get a green light and when I check the result in web space, it has done its job of rotating a picture on a test URL. The aim is to pick at random from an image pool, call the image by a new name that is potd.jpg for example then when its time is up, the image is copied under a new name, has a date and time stamp in the name, potd_0909112100.jpg for example.

The new image is picked at random ind is copied with the new name... you guessed it potd.jpg where the extention of the file is generated automatically.

Now, the web host, as usual the tech dept is dragging its heels and want proof that the script is not working when it is automated as a cron.

So my question is this, can BASH run a PHP script because BASH does not seem to have any issues as the other script works fine. I get the odd error message but the script has still worked and done its job.

Anyway, what can I do to resolve this?

[scragar]

I love your understanding of bash, it come from the same understanding of scripting languages, which I should stress bash is only sort of a scripting language, bash is both a scripting language and a command shell(like cmd.exe for windows(or dos if you want to call it that)).
The call to date is a program called date. It doesn't exist in the bash documentation because it's to a program called date, not a feature of bash called date.
http://unixhelp.ed.ac.uk/CGI/man-cgi?date
If there's ever a line you don't understand look at the first word, and try it in that url in place of date, so if I was to use echo like so:
echo "foo\nbar"
echo -n "foo\nbar"
echo -e "foo\nbar"
echo -en "foo\nbar"You could look up echo and see exactly what it does and why the output is different for all of those lines. Man pages can be a little complex to read, but that's because of the power given to most commands(Do one thing and do it really, really well, as opposed to do loads of things rather badly, so you'll find echo supports so many options and features you never expected that it'll make your eyes water :p), rather than a flaw in the language itself.

As for running a PHP script from bash, there are a good number of ways to do it, if you know the path to the PHP executable:
#!/usr/bin/php
<?php
//.......
Or maybe if you don't know that just trust it's in the $PATH
#!/bin/bash
php $0; exit
<?php
//.........
Either options should work, although I have more faith in the second if you don't know the setup of the box.

I'm assuming in both of those that your wanting to add the .php script directly to the cron, and not, wanting a wrapper. if you want a wrapper you'll have to use the second, and just replace $0; exit with the path to your PHP script, and don't put anything else after it.

[JunkMale]

OK, so + is a pad 0 ?

%l is the hours 1 thru 12 so if %k was subbed, we would have 24 hour version of that test.


Anyway, I now get this in my email and I guess that this is the message I was looking for. X-Powered-By: PHP/5.2.10
Content-type: text/html
The output I guessed was indication that the script executed and I checked and saw that the image had changed. So I guess my moan about the fact that the script was working when I manually executed it and asked why the cron failed sort of got them looking in to it.

I will keep the BASH option in mind and set it up like that if the cron fails again.

Cheers.

[scragar]

The + is part of dates syntax, it uses the + to make the start of the format, since the format can include spaces and such.

%k would give you the hour in 24 hour format, but for the purposes of dividing by 3 the hour format makes no difference, 12 is just easier to work with than 24.

If you do wish to learn more about bash I recommend you download a liveCD for a linux distro or two and run it on your home PC from time to time, take a look at the terminals and play around with a few things(You can't cause any damage unless you choose to mount your hard drives or run commands like `dd if=/dev/random of/dev/sda` which would overwrite your first hard drive(s for sata, d for device, a because it's first) with random data(/dev/random, obviously)).

My personal recommendation for a first run liveCD would be either linux mint( http://www.linuxmint.com/ ) or ubuntu ( http://www.ubuntu.com/ ), both are debian based, which is what you'll find on a lot of servers.

It's also very nice to have a linux liveCD if you want to set bash scripts in future, that way you can avoid people who might want to give you bad commands(watch out for anything involving the keywords sudo(gives root perms), dd(can write to devices, and with random data as an input device, not nice), yes(repeatedly sends 'y' as output, great for some things, but also very bad for people planning to cripple your ram) or with lot's of brackets and braces(likely fork bomb))

[JunkMale]

Yep, have ubuntu. Very new to Linux.

Back to the BASH script.

So if I had the following code
#! /bin/bash

[ $(( $(date +%l) % 3)) -ne 0 ] && exit;
## exit the script if the current hour is not an exactly multiple of 3

# cd = change directory.
cd /home/sites/****.***.**/public_html/stats

# for(I = 1; I <= 20; I++){
for I in $(seq 20); do
## Try the script, if it returns true we break out of the loop
/usr/bin/webalizer -D dns.cache -nlocalhost -p -o . /home/sites/****.***.**/logs/****.***.**-access_log* && break;

## we couldn't lock the file. Wait a second, then try again:
sleep 1s;

#end for loop
done
PHP $0; exit;
<?php

// my script here

?>
would it run the appended PHP script?

[scragar]

Yes it would, although you should note that the bits of bash at the top of the code will get printed as output when run as PHP, so if you log the output of these scripts, or email it to yourself it might be worth making it two files, just to save yourself a bit of scrolling from time to time.

[JunkMale]

Well the output of the PHP script is an empty email which is no bother.
The script doesn't output anything, its what it will be doing that is important.

Any output like errors is emailed to me.

As for PHP file, would it be better if it was a completely sepperate file and called via its path?PHP /home/sites/****.***.**/database.phpis the last line of the BASH script, would this run that PHP script?

[scragar]

Lower case. Write php not PHP otherwise, because linux is case sensitive, it the bash processor won't understand what you're looking for.

[JunkMale]

ok, I shall give it a whirl.

Thx.

[JunkMale]

Ok, I have a couple of things I would like to expand on with this shell script.

The first one is that I would like to copy a particular file and just add the extension .copy to the name.

Would I be right in thinking that this cp /home/sites/****.***.**/public_html/stats/webalizer.hist /home/sites/****.***.**/public_html/stats/webalizer.hist.copy would do the trick?

[scragar]

Ok, I have a couple of things I would like to expand on with this shell script.

The first one is that I would like to copy a particular file and just add the extension .copy to the name.

Would I be right in thinking that this cp /home/sites/****.***.**/public_html/stats/webalizer.hist /home/sites/****.***.**/public_html/stats/webalizer.hist.copy would do the trick?

That would work, but there is a small shorthand that may be used to make your life easier:
cp /home/sites/****.***.**/public_html/stats/webalizer.hist{,.copy}
Whenever {} is used outside of quotes it's expanded by the shell as a list of comma separated values, so:
echo a{1,2}
Will echo out a1 a2As if there were two different arguments echoed out, the first one matching the first value, and the second the second value.
Also, just for future reference if you are copying a file to the same directory it's often a good idea to cd to that directory first, to avoid writing the file path a second time or worrying about commas and/or spaces in the path causing problems.

[JunkMale]

OK, thx. The files being copied are following a cd command and the files exist in the same directory.

[scragar]

OK, just want to further explain what I said earlier though, the commands expand, but you can mix a few things together, for example:
echo "something with spaces in "{a,b,c}" more spaces.{foo,bar}"{1..3}
Produces(Each thing on a new line, although echo wouldn't do that)
something with spaces in a more spaces.{foo,bar}1
something with spaces in a more spaces.{foo,bar}2
something with spaces in a more spaces.{foo,bar}3
something with spaces in b more spaces.{foo,bar}1
something with spaces in b more spaces.{foo,bar}2
something with spaces in b more spaces.{foo,bar}3
something with spaces in c more spaces.{foo,bar}1
something with spaces in c more spaces.{foo,bar}2
something with spaces in c more spaces.{foo,bar}3
As this shows, you can use .. to imply a list(I could have done this in my first reply, rather than using seq, but I find seq easier to explain and it comes with good documentation, as opposed to the expansion, which really doesn't), and {} inside quotes renders normally. You can also use quotes and unquoted sections to take advantage of bash expansion without having to worry about using \ to escape things, just include everything you don't want to risk being expanded in quotes(Just don't put a space outside of the quoted sections, or bash will interpret it as two different arguments to be handled separately).

[JunkMale]

Whoa! Hold up, thats way too much power. Well useful to know. I now have the changes made and the shell script when done calls the PHP script. It at the moment echos "OK" to me in an email when ever the cronjob runs, I also see a .copy of the webalizer history file. So all is going well. I forgot to change the file permissions and got access denied. So thats one for the books if anyone is following this.

You got any thoughts on getting this .copy file body in to a sqlite2 database in BASH? I was looking at http://ss64.com/bash/ to see if I was in the right place and I don't really see anything that is going to be of use or have I gone cross eyed?

Well I am off to test my PHP functions and scripting... When that is working I will be looking to get those processes converted if possible.

[scragar]

The .copy file, is it sql code to be run, or raw text to be saved in a field?

SQL code to be run is easy, first the mysql command you'd need to connect
mysql -h host -P port -u username -p password
Any or all of those may be omitted, depending on the default settings, personally I specify them all just to be sure, but if you don't know something it might be best to leave it off and test it first.
mysql -u root -p ******
The next part is that you need to pass into that command the contents of the file, now you could set up some complex line loop, or read the contents of the file using cat, but the easiest way is to just use one of bashes inbuilt features:
mysql -u root -p ****** <filename

If you want to store the whole file, then the method is very similar, you just need to make a query for the data and protect against anything that might cause problems:
UNSAFECHARS="\\'" ## backslash and single quote
mysql -u root -p ****** <<DOING_SQL_STUFF
INSERT INTO tbl_bob
VALUES('Field 1', 'Field 2',
'$(sed "s/[$UNSAFECHARS]/\\\\\0/g" filename)'
);
DOING_SQL_STUFF


Seriously though, as much as I love bash if you know how to do something in PHP you should do it in PHP, bash relies on a lot of things that might not exist in a given environment, and normally the bash scripts can only be run in cron-jobs or via system calls(unless you have SSH/direct access), making running them manually much harder.


EDIT: sed is short for stream editor, it has many features, here I use a regular expression(Yes, the \s is excessive, but I wanted to use double quotes so I could use the variable for unsafe characters, forgive me).

[JunkMale]

OK, thx. Is worth knowing this stuff can be done. I am however stuck using sqlite2 as thats what the CMS uses and it is nice to have everything under one roof to keep things portable. If it can not be done in BASH, not an issue, it would be nice to have the bulk of the processes as a server shell.

How would you go about including or testing for a resource? It is obvious that the host has the sqlite library installed like the php library?

[scragar]

Testing if something exists isn't that hard, you use the which command, it will check everywhere in the $PATH variable(where executables are normally found) for something matching that name, and return the path and success, or nothing and failure.
if ! which php; then
echo "PHP could not be found." >&2
exit 1
fi
(The >&2 part sends the output to the STDERR output, rather than the STDOUT output, since this is an error message it's the normal way to handle it. I also tell the command to exit with status 1, this is a failure(as is anything non-zero), again, indicating a problem. Normally you would want to replace this error detection with something better, using the mail command to send a notification to yourself perhaps?)