Looper's Delight Archive Top (Search)
Date Index
Thread Index
Author Index
Looper's Delight Home
Mailing List Info

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Date Index][Thread Index][Author Index]

Re: problem w/mailing list archive search engine?



At 02:11 PM 3/17/99 -0800, Chris Chovit wrote:
>Having trouble with the mailing list archive search engine.  Tried
>searching on a couple topics, with no matches....then tried testing it by
>searching for "echoplex" and still no matches.......anyone else having
>trouble?
>
>Any ideas, Kim?

sorry, it's broken at the moment. Actually it's been broken for a couple of
weeks, but you're the first to comment. Amusing story, unless you are me 
and
have to deal with it:

My web host provider recently installed this resource monitor that watches
everything and kills any process it doesn't like. Ha, ha, they didn't 
happen
to tell us this. And, oh by the way they are still tuning it and deciding
how the parameters need to be set, so you know, maybe it's killing things
for no good reason. But, heck, how bad could that be?

So, I have a cron job that runs each night at about 4am, when the server
load is very light. A script converts that day's posts to html and adds 
them
to the archive, and then does an incremental indexing to add the keywords 
to
the search engine index. This is a bit processor intensive for just that 
one
period, but not too bad since it's just one day's worth. And then the
regular user searches the rest of the day go very fast since they just 
refer
to the index.  

Now, It happens that a 4am, when nobody is using the system, my little
script had no competition for resources and Linux was happily giving it as
much of the cpu as it wanted. So one night it crossed some arbitrary
threshold, and the monitor promptly killed it. Since it was in the middle 
of
incrementing the index, the whole index file wound up corrupted. So the
search engine was hosed.

So I discovered this and realized I had to do a complete index on the whole
bloody archive, which after 2.5 years of loop babbling is a REALLY big job.
As you might imagine, the monitor REALLY hates that full-on indexing 
process
and kills it too. Which of course, means I've been unable to fix it.

So my provider and I have been having some talks. ;-)  

After a couple of weeks of screwing around with monitor parameters and me
forcing all my scripts to run in low priority "nice" modes, various
statements about "Load Average" and "multi-threading" and "priority
assignment", and mostly waiting around for the sysadmin to actually do
something about it, it still don't work. So finally, after my most recent
"what the fuck is going on?" mail to them, we reached today's point in the
negotiations, where they agree to shut off the monitor for 15 minutes
tonight while I run my big indexing script to fix everything. And hopefully
my usual nightly script is now "nice" enough, and their monitor is tuned
right, and everything will be just roses after that. hopefully...

So check it tomorrow, maybe it'll be working....

kim
________________________________________________________
Kim Flint, MTS                  408-752-9284
ATI Research                    kflint@atitech.com
http://www.atitech.com