Hi, I'm working on a forum script which does what it's supposed to do, except for one thing. That is, listing the latest reply or new thread at the top of the topic list correctly.
I had help on the code below from a forum which is now closed. The code works, but - this happens:
If this month is say May, then the latest post goes to the top of the list (only because May is at the top).
If someone posts a reply to a thread made in say April or March, then the reply only goes to the top of the section in the list where that particular month is displayed. Which is not what I want it to do.
Does that make sense?
What is needed to make any posting go to the top of the whole list, regardless of the month the original thread began?
I do not know how to do that (I'm not all that good a programmer).
sub printMonthLines {
my($filename) = @_;
$filename =~ s/[\^<>'\$!#;\*\?\&\|\`\/\~\\\(\)\{\}\"\n\r]//go;
$filename =~ s/\.\.//go;
# open file
$thefilenm = "$cgidir/$forumdata/$filename.txt";
if (open(DATA,"$thefilenm")) {
my %records;
while (<DATA>) {
chomp;
my @data = split(/_/);
my $date = join '', (split(/[\/:-]/, $data[5]))[2,1,0,3,4,5];
push @{$records{$date}},$_;
}
close(DATA);
Field 0 = Number of entry (1 to whatever)
Field 1 = Topic
Field 2 = Author
Field 3 = Thread date
Field 4 = Number of replies
Field 5 = Reply date
Field 6 = Name of replier
The reply date is the one the code uses. I think (I'm more an artist/designer than programmer) what is happening is that the code needs somehow to take into consideration the total number of all records in all the files. Insteadof one at a time (which I think is what is happening).
Nedals
05-29-2009, 05:04 PM
Your sample record and your field descriptions do NOT agree.
1_Topic_Title_Author_2_05/28/2009-10:48:47_0_05/28/2009-10:48:47_Replier
I'm guessing; missing '_' before title and '205/28/209' looks strange. Another missing '_' ??
Field 0 = Number of entry (1 to whatever)
Field 1 = Topic
Field 2 = Title
Field 3 = Author
Field 4 = ???
Field 5 = Thread date
Field 6 = Number of replies
Field 7 = Reply date (post date?)
Field 8 = Name of replier
If this is correct, then the date is the thread date not the post date.
edatz
05-29-2009, 05:19 PM
Hi, actually they do (except for the extra character typo - now removed)
There are 6 fields: "Topic Title" is one field, not 2.
The 2_05/28/2009 as you see should be 05/28/2009
I've been trying different methods on this and still got nowhere. Maybe I should post the original without the attempt for the latest to the top. Dunno, is it worth it?
perl_diver
05-29-2009, 05:48 PM
The main problem is the date/times are in human readable format, which are not easily sorted by computers. You should always store a time stamp in epoch seconds format even you don't plan on using it for anything right away. Then in the future when you want to do something like sort the entries by date its simple, all you have to do is sort the epoch seconds instead of jumping through hoops converting human readable dates into sortable numbers or strings.
If the latest reply is the last line of the flat file, all you need to does is move that line to the "top".
edatz
05-29-2009, 11:03 PM
Hi Perl Diver, I realize this. The code is not something I wrote, I'm just trying to make it work differently.
There can be dozens (even scores) of these types of files, plus one for each thread, a monthly list file and total number file. I don't want to get into relational databases because I don't understand them nor know how to make them work.
Thing is these forum scripts can do quite a lot and not end up some massive set of scripts megabytes in size. Originally it was done by a guy named Dan Steinman (years back) as a text only forum that worked a treat until the spam bots hit. I had prettied it up and offered from my old Scriptles site and had many downloads. I just thought I'd update it and realized that the latest posting to the top needed sorting out. As it stand now the latest only goes to the top of each month in the list (so it does work, but only partially)
I could try to figure out a way that the files can handle epoch seconds (all my other scripts use these, often just as an ID). It's just the way these are handled. I could have many read into a hash or array, since they are small and today's computers are just tad faster than they used to be :). I thought it might be easy to sort them by the 5th field and so someone else gave me the code - trouble is that forum is now closed and I can't get back to the guy.
It look like I'll just have to rethink it all :(
perl_diver
05-30-2009, 12:08 AM
Looking at the code you posted in the first post, it looks like it is already putting the latest post at the top of the list. It creates a hash of arrays where the hash keys are the date stamp converted into a sortable string. The hash of arrays is then sorted in descending order by the hash keys, so the most recent post should be at the top of the list.
Nedals
05-30-2009, 03:53 AM
Me thinks :)
my $date = join '', (split(/[\/:-]/, $string))[2,1,0,3,4,5];
This line sets the $date (key) to 'ydmhms', but it should be 'ymdhms'. You were probably getting a somewhat random sort.
So change to:..
my $date = join '', (split(/[\/:-]/, $string))[2,0,1,3,4,5];
That could solve it.
perl_diver
05-30-2009, 04:03 AM
Me thinks :)
my $date = join '', (split(/[\/:-]/, $string))[2,1,0,3,4,5];
This line sets the $date (key) to 'ydmhms', but it should be 'ymdhms'. You were probably getting a somewhat random sort.
So change to:..
my $date = join '', (split(/[\/:-]/, $string))[2,0,1,3,4,5];
That could solve it.
Good observation. He said the script was working so I have a feeling its only sorting posts from within the same month. Hopefully he tries your suggestion and we will see if it helps.
edatz
05-30-2009, 06:32 AM
I did give it a go and got the same result.
Let's say I have two FFDBs
200904.db
-----
2_Two_Bbb_04/30/2009-14:43:49_0_04/30/2009-14:43:49_Bbb
1_One_Aaa_04/22/2009-11:22:15_0_04/22/2009-11:22:15_Aaa
-----
The entire list is:
=====
Four
Three
Two
One
=====
"One" has a reply. The 200904.db changes to this
-----
2_Two_Bbb_04/30/2009-14:43:49_0_04/30/2009-14:43:49_Bbb
1_One_Aaa_04/22/2009-11:22:15_1_05/30/2009-11:04:31_Zzz
-----
The entire list is now:
=====
Four
Three
One
Two
=====
This happens with both "join" lines being tried. In fact I changed the '2,0,1,3,4,5' to '0,1,2,3,4,5' and it stayed the same too.
I looked up on hashes (I am not familiar with these and have never used them) and saw that the example had brackets around the data. At the line
my %records;
There are no curly brackets. So I tried putting some in at various points and still got the same results.
Maybe this code is missing something. But I can't see what. The whole lot is being read in and displayed, so it's not as though the data is beinng lost somewhere. Any ideas? Should there be another "if else" to handle the months themselves? Maybe the foreach osrt is wrong?? Certainly doing my brain in...
I can see where epoch seconds could well solve this (thinking as I type) and so will see if I can implement that somehow. As it's just the "replytime" field I'll probably just need one more field at the end of the record.
edatz
05-30-2009, 08:35 AM
I may have used the wrong term with epoch seconds, what I mean is something like this: 20090530100524 (whatever it's called).
perl_diver
05-30-2009, 02:46 PM
It looks like the script is opening each file in succession, so first it opens the most recent file, 200905.db, it sorts it and displays the results. It then opens 200904.db and does the same thing. So it never even knows that there is a higher date in 200904 than was in 200905 because it is processing each file individually.
What it needs to do is process all the files as a whole before printing any output back to the browser. That way it can find the most recent post in relationship to all the files.
Nedals
05-30-2009, 03:04 PM
sub printMonthLines { ## GREATLY SIMPLIFIED
# Open month file and push post into a hash, keyed by timestamp
my %records;
open(DATA, '200904.db');
while (<DATA>) { push @{$records{$key}},$_; }
close(DATA);
foreach my $key (sort {$b cmp $a} keys %records) {
@sorted = "@{$records{$key}}";
foreach $line (@sorted ) {
next unless $line;
print $linedata; # Print posts in this month's file
}
}
}
The sub prints the posts for sorted records in '200904.db' ONLY. It knows nothing about 200905.db
So if 200904.db contains a month 05 record it does not know how to sort that into the 200905.db data.
Somewhere you likely have a sub that calls the 'printMonthLines() sub' for each of the monthly files.
If you push data into %records for each monthly file and then print the result, that may solve your problem
something like this maybe...
sub get_each_file {
my %records;
loop through files
printMonthLines($filename, \%records);
}
foreach my $key (sort {$b cmp $a} keys %records) {
@sorted = "@{$records{$key}}";
foreach $line (@sorted ) {
next unless $line;
print $linedata; # Print posts
}
}
}
sub printMonthLines { ## SIMPLIFIED
my ($filename, $r_record) = @_;
open(DATA, '200904.db');
while (<DATA>) {
my $key = datastring
push @{$r_record{$key}},$_;
}
close(DATA);
}
I hope that helps a little. :)
edatz
05-30-2009, 04:44 PM
Just got back while you guys were doing the above. I don't get notified when a post is made so have to check manaully.
Okay I've gotten this far.
The records have a field at the end with the "fulldate" in (3 threads, 2 different months) and read in like this to the screen when I tested:
-----
2_What is it?_Fred_02/17/2009-04:19:38_1_05/30/2009-17:47:07_Elmer_20090530174707
1_Now What?_Barney_02/17/2009-03:04:37_1_05/30/2009-10:32:41_Fred_20090530223241
3_Pondlife_Daffy_05/25/2009-17:11:26_1_05/30/2009-03:34:57_Homer_20090530033457
-----
New code:
sub printMonthLines {
$dir = "$cgidir/fadata";
# open the folder
opendir (DIR, ($dir));
@mdbs = grep(/\.db$/,readdir(DIR));
closedir(DIR);
# open the file
foreach $f (@mdbs) {
open(TFL, "$dir/$f") || die("could not open");
@mdb = <TFL>;
close(TFL);
# split it up
foreach $rec (@mdb){
chomp($rec);
($num,$subject,$name,$date,$responses,$replytime,$replyname,$srtdate)=split(/\_/,$rec);
## sort by $srtdate -- ?????
##### Now I need to figure out how to sort it by the last field in each record.
# print it all out ---- why does it print out all the duplicates (11 of them)?
print qq ~
<TR valign="top"><td>
<A HREF="$ENV{'SCRIPT_NAME'}?msg=$num" onFocus="if(this.blur)this.blur()"><b>$subject</b></A>
<div>Posted on: $date</div></td>
<td>$name</td>
<td>$responses</td>
<td>$replytime<br>by $replyname <A HREF="$ENV{'SCRIPT_NAME'}?msg=$num#$responses" onFocus="if(this.blur)this.blur()">$viewpost</a></td></TR>
~;
} ## end forach @mdb
}# end foreach @mdbs
}# end sub
Not sure what to do on sorting by the last field, but as it's a better format it should do the trick now that I'm seeing the contents of both files as a single list.
I'll check out what you've post Nedals
It looks like the script is opening each file in succession......Yeh, that's what I figured too perl_diver. More than likely the problem with the whole thing.
It's late here (London, England) and I didn't get much sleep last nite so will get back to this tomorrow.
Nedals
05-30-2009, 06:07 PM
Format your code a little like this. It's much easier to read.
sub printMonthLines {
my $dir = "$cgidir/fadata";
# open the folder
opendir (DIR, ($dir));
my @mdbs = grep(/\.db$/,readdir(DIR));
closedir(DIR);
# open each file
my %records; # Hash, keyed by $strdate, will contain all the posts for all files processed
for (@mdbs) {
open(TFL, "$dir/$_") || die("could not open");
while (<TFL>) {
chomp;
# 2_What is it?_Fred_02/17/2009-04:19:38_1_05/30/2009-17:47:07_Elmer_20090530174707. Format changed a little??
my @record = split(/\_/);
$records{$record[7]} = \@record;
}
close(TFL);
}
# At this point all the posts are in the %records hash.
# for simplicity, this version will only allow 1 posting each second. A second posting at the same day and time will overwrite the first.
# This may be undesirable. In which case you will need to figure a way to get multiple records for each key but try it this way first.
for (sort {$b cmp $a} keys %records) { ## Sort and print all the posts
my @record = @$_; # de-reference the array
my ($num, $subject, $name, $date, $responses, $replydate, $replyname, $srtdate) = split(/\_/, @record);
print qq ~
<TR valign="top"><td>
<A HREF="$ENV{'SCRIPT_NAME'}?msg=$num" onFocus="if(this.blur)this.blur()"><b>$subject</b></A>
<div>Posted on: $date</div></td>
<td>$name</td>
<td>$responses</td>
<td>$replytime<br>by $replyname <A HREF="$ENV{'SCRIPT_NAME'}?msg=$num#$responses" onFocus="if(this.blur)this.blur()">$viewpost</a></td></TR>
~;
}
}
edatz
05-31-2009, 05:54 AM
I did give it a go, but all the fields came up empty.
I think we are all getting a bit frustrated with this so will call it a day. thanks for all the input you guys gave. I have learned some things, so it has not been fruitless.
It look like I'll just have to rethink it all, as I thought a few replies back. There's a lot more I would have to do to this set of scripts if we had got it working. So a doing it from scratch may actually be better than rewriting old code.
Thanks again - Edatz
Nedals
05-31-2009, 01:29 PM
I did give it a go, but all the fields came up empty.That's because there's an error in my code.
for (sort {$b cmp $a} keys %records) {
my @record = @$_;
my ($num, $subject, $name, $date, $responses, $replydate, $replyname, $srtdate) = split(/\_/, @record);
...
}
should read....
for (sort {$b cmp $a} keys %records) {
my ($num, $subject, $name, $date, $responses, $replydate, $replyname, $srtdate) = @{$records{$_}};
...
}
The data was already split earlier.
I think we are all getting a bit frustrated with this so will call it a day.
No more posts from me unless you want to continue.. :)
edatz
06-02-2009, 04:41 PM
That seems to have done the trick. Well done. When I test it in a single script I get four records showing like I should. Fine.
In my forum, stuff is divided into library files etc. This particular bit of code is used in two different places so it's accessed from a file for common items.
What happens then is that it works, but for some reason I get a bunch of duplicates. So instead of 4 there are 60. 56 are repeats.
Does this mean I have some bit of code somewhere that's active. I've never run across this before. It must be something I've done elsewhere as the single script test works like it should.
Wondering if I have a variable open somewhere that should have single quotes around instead of double ones - or something along those lines
Any Ideas?
perl_diver
06-02-2009, 05:05 PM
The code Nedals posted searches for all *.db files and stores them @mdbs:
my @mdbs = grep(/\.db$/,readdir(DIR));
It then loops through and opens them all and eventually returns the html output. More than likely you are calling the function like you always have but the new code added by Nedals is duplicating that behavior inside the function. Kapish?
edatz
06-03-2009, 04:37 AM
Not being sure what to do I did a search for "stop repeated lines" and found somehting that might work. The whole bit of code was a disaster so I tried just using a little portion of it all:
for (sort {$b cmp $a} keys %records) { ## Sort and print all the posts
my @record = @$_; # de-reference the array
my ($num, $subject, $name, $date, $responses, $replydate, $replyname, $srtdate) = split(/\_/, @record);
# Stop the repeats
$tlns{$_}++;
next if $tlns{$_} > 1;
The "Stop the repeats" bit seems to do the trick and I end up with the correct number of topics. The latest posting goes to top whether it's a new topic or a reply. I'm going to play around and see if I can break it and if that doesn't happen then I'm okay :D
It looks like it's the sort of code that will come in handy, so will go into my "library" for use on other scripts if needed.
Thanks for all your help guys.
perl_diver
06-03-2009, 05:07 PM
That works but just introduces inefficiency because your code is probably opening files unecessarily and reading files more than once. Look for something in your script where the function "printMonthLines" is called. Its probably called in a loop:
foreach (@list) {
printMonthLines($_);
}
Thats the part you need to change.
Nedals
06-03-2009, 07:55 PM
for (sort {$b cmp $a} keys %records) { ## Sort and print all the posts
my @record = @$_; # de-reference the array
my ($num, $subject, $name, $date, $responses, $replydate, $replyname, $srtdate) = split(/\_/, @record);
Be careful. That's the bit that was wrong. See my post above.
edatz
06-04-2009, 01:49 PM
I never thought of that perl_diver. When I do my own sub codes I normally make them do everything that's needed within the sub, so that when it's used the functions are all there (tested in solo scripts to make sure). Or have ones that can be called from my library and used where/when ever.
I did a search through the 2 main scripts and found 2 places in each that call "printMonthLines". One is this - in the default display (the part that was giving the problem):
for ($i = 0; $i < $numnths; $i++) {&printMonthLines ("$curmonthprint[$i]");}
You're right it's in a for loop.
I changed it to this (after a bit of trial and error)
for ($i = 0) {&printMonthLines ("$curmonthprint[$i]");}
And it seems to have solved it - no repeats. I'm even thinking, it may not be needed at all, only the sub call by itself as the subroutine works like it should when done in a test script all by itself (will test that out - does work with just the sub call).
What I've done to the original script is split it up so that I can have multiple forums (orig only had one - period) eventually. Anything that can be used in more than one place is going into a library file (printMonthLines is one of those things).
The other place is in the archive display which works (sort of), but I had not got that far - it's next on the agenda. What we've done here will probably be enough that I can sort out that fairly easily (famous last words...)
Be careful. That's the bit that was wrong
Sorry Nedals, I pasted the wrong bit into this forum, the working bit had your adjustment in.
perl_diver
06-04-2009, 02:29 PM
And it seems to have solved it - no repeats. I'm even thinking, it may not be needed at all, only the sub call by itself as the subroutine works like it should when done in a test script all by itself (will test that out - does work with just the sub call).
Without seeing your entire script its hard to say for sure but if the code Nedals posted earlier is correct in that it searches for all *.db files in the directory:
my @mdbs = grep(/\.db$/,readdir(DIR));
and uses the list @mdbs to open all the files and generate the output, then all you have to do is call the function in the other parts without any looping.
edatz
06-04-2009, 04:45 PM
..... then all you have to do is call the function in the other parts without any looping.
Yes, I've done that and it's working fine.
Thanks again for your input on this. It's been a great help, and I've learned a few things too :)
perl_diver
06-04-2009, 05:12 PM
All credit to Nedals, I just commented on the code he took the time to write.
edatz
06-08-2009, 03:41 PM
Actually it was both of you. Nedals for the code and you for the inspiration about the usage within a loop elsewhere (I would never have thought of that). So it is a big thanks to you both.
I don't know how to PM here. What I'd like to do is give you both credit in the scripts. Eventually it will be downloadable from my site as an alternative to the heavyweight forum scripts. The Original that I did had input from a number of people - their code is still used and credited to them, with links etc. (those that are still around).
There's a fair ways to go here and I will be having a live version on my site to be there as it is developed. Also to test vulnerability etc. I am approaching how scripts are used and called for this, but only live testing will determine what is needed and where the next bits have to go.
I haven't posted a link here as it's not up yet on my server (this had to be sorted first). I can do that when it's live.
perl_diver
06-08-2009, 05:19 PM
No need to mention me as having helped with your script. Thank you though for the consideration. I would like to see the script in action when its ready so come back and post a link or figure out how to PM people (its easy) and send me a PM in the future.
webdeveloper.com
Copyright Internet.com Inc., All Rights Reserved.