<section><title>Monitoring and administration</title>
<para>
Once the Usenet News system is in place and running, the news administrator
is then aided in monitoring the system by various reports generated by it.
Also, he needs to make regular checks in specific directories and
files to ascertain the smooth working of the system.
</para>

<section><title>The <literal>newsdaily</literal> report</title>
<para>
This report is generated by the script <literal>newsdaily</literal> which is 
typically run through <literal>cron</literal>. I shall enumerate some of the 
problems that are reported by it, based on my observations .</para>

<itemizedlist>
<listitem><para><emphasis>bad input batches</emphasis>:  
    This gives a list of articles that have been declared bad after processing
    and hence have not been digested. The reason for this is not given. You
    are expected to check each article and determine the cause. 
</para></listitem>

<listitem><para><emphasis>leading unknown newsgroups by articles</emphasis>:  
    Newsgroup names that do not appear in the <literal>active</literal> file
    but their hierarchy has been subscribed to, would find their names mentioned
    under this heading. Choose to add the name in the active file if you think 
    it is important. For <emphasis>e.g.</emphasis>, you would see this happen
    if you have subscribed to the hierarchy <literal>comp</literal> but the
    <literal>active</literal> does not contain the newsgroup name 
    <literal>comp.lang.java.3d</literal>. You could deny subscription to this 
    particular newsgroup by specifying so in the <literal>sys</literal> file.
    
</para></listitem>

<listitem><para><emphasis>leading unsubscribed newsgroups</emphasis>:
    If the news server receives maximum articles of a particular newsgroup
    hierarchy to which you haven't subscribed, it will appear under this
    heading. You really cannot do much about this except to 
    subscribe to them if they are required.
</para></listitem>

<listitem><para><emphasis>leading sites sending bad headers</emphasis>: 
    This will list your NDNs who
    are sending articles with malformed/insufficient headers. 
</para></listitem>

<listitem><para><emphasis>leading sites sending stale/future/misdated news</emphasis>: 
    This will list your NDNs who are sending you articles that are older than
    the date you have specified for accepting feeds. 
</para></listitem>

<listitem><para>Some of the reports generated by us: 
    We have modified the newsdaily script to include some more statistics. 
    <itemizedlist>
    <listitem><para><emphasis>disk usage</emphasis>: 
	This reports the size in bytes of the <literal>$NEWSARTS</literal>
	area. If you are receiving feeds regularly, you should see this figure
	increasing.
    </para></listitem>

    <listitem><para><emphasis>incoming feed statistics</emphasis>: 
	This reports the number of articles and total bytes recevied from each 
	of your NDNs.
    </para></listitem>

    <listitem><para><emphasis>NNTP traffic report</emphasis>: 
	The output of nestor has also been included in this report which gives
	details of each <literal>nntp</literal> connection and the overall 
	performance of the network connection read from the <literal>newslog
	</literal> file.  To understand the format, read the manpage of 
	<literal>nestor</literal>.
    </para></listitem>
    </itemizedlist>
</para></listitem>

<listitem><para><emphasis>Error reporting from the <literal>errorlog</literal>
file</emphasis>: 
    Reports  errors logged in the <literal>errorlog</literal> file. Usually
    these are  file ownership or file missing problems which can be easily
    handled. 
</para></listitem>
</itemizedlist>
</section>

<section><title>Crisis reports from <literal>newswatch</literal></title>
<para>
Most of the problems reported to me are those with either space shortage or
persistent locks. There are instances when the scripts have created locks files
and have aborted/terminated without removing them. Sometimes they are 
innocuous enough to be deleted but this should be determined after a careful
analysis. They could be an indication of some part of the system not working
correctly. For <emphasis>e.g.</emphasis> I would receive this error message when
sendbatches would abnormally terminate trying to transmit huge togo files. I had
to determine why sendbatches was failing this often.
</para>

<para>
The space shortage issue has to be addressed immediately. You could
delete unwanted articles by running <literal>doexpire</literal> or add more disk
space at the OS level. 
</para>
</section>

<section><title>Disk space</title>
<para>
The <literal>$NEWSBIN</literal> area occupies space that is fixed. Since the
binaries do not grow once installed, you do not have to worry about disk 
shortage here. The areas that take up more space as feeds come in are
<literal>$NEWSCTL</literal> and <literal>$NEWSARTS</literal>. The 
<literal>$NEWSCTL</literal> has log files that keep growing with each feed.
As the articles are digested in huge numbers, the <literal>$NEWSARTS</literal>
area continues to grow. Also, you will need space if you have chosen to archive
articles on expiry.  Allocate a few GB of disk space for
<literal>$NEWSARTS</literal> depending on the number of hierarchies you are
subscribing and the feeds that come in everyday. <literal>$NEWSCTL</literal>
grows to a lesser proportion as compared to <literal>$NEWSARTS</literal>.
Allocate space for this accordingly.
</para>
</section>

<section><title>CPU load and RAM usage</title>
<para>
With modern C-News and NNTPd, there is very little usage of these
system resources for processing news article flow. Key components like
<literal>newsrun</literal> or <literal>sendbatches</literal> do not load
the system much, except for cases where you have a very heavy flow of
compressed outgoing batches and the compression utility is run by
<literal>sendbatches</literal> frequently. <literal>newsrun</literal> is
amazingly efficient in the current C-News release. Even when it takes
half an hour to digest a large consignment of batches, it hardly loads the
CPU of a slow Pentium 200 MHz CPU or consumes much RAM in a 64 MB
system.
</para>

<para>
One thing which does slow down a system is a large bunch of
users connecting using NNTP to browse newsgroups. We do not have
heuristic based figures off-hand to provide a guidance figure for
resource consumption for this, but we have found that the load on the
CPU and RAM for a certain number of active users invoking
<literal>nntpd</literal> is more than with an equal number of
users connecting to the POP3 port of the same system for pulling
out mailboxes. A few hundred active NNTP users can really slow down
a dual-P-III Intel Linux server, for instance. This loading has no
bearing on whether you are using INN or <literal>nntpd</literal>;
both have practically identical implementations for NNTP
<emphasis>reading</emphasis> and differ only in their handling of
feeds.
</para>

<para>Another situation which will slow down your Usenet news server is
when downstream servers connect to you for pulling out NNTP feeds using
the pull method. This has been mentioned before. This can really load
your server's I/O system and CPU.</para>

</section>

<section><title>The <literal>in.coming/bad</literal> directory</title>
<para>
The <literal>in.coming</literal> directory is where the batches/articles reside when you have 
received feeds from your NDN and before processing happens. Checking this
directory regularly to see if there are batches is a good way of determining
that feeds are coming in. The batches and articles have different nomenclature.
Batches, typically, have names like <emphasis>nntp.GxhsDj</emphasis> and 
individual articles are named beginning with digits like <emphasis>0.10022643380.t</emphasis>
</para>

<para>
The <literal>bad</literal> sub-directory under <literal>in.coming</literal> 
holds batches/articles that have encountered errors when they were being
processed by <literal>relaynews</literal>. You will have to look at the 
individual files in this directory to determine the cause . Ideally speaking, 
this directory should be empty.
</para>
</section>

<section><title>Long pending queues in <literal>out.going</literal></title>

<para>TO BE ADDED.</para>

</section>

<section><title>Problems with <literal>nntpxmit</literal> and <literal>nntpsend</literal></title>

<para>TO BE ADDED.</para>

</section>

<section><title>The <literal>junk</literal> and <literal>control</literal> groups</title>
<para>
Control messages are those that have a <literal>newgroup/rmgroup/cancel/checkgroup</literal> in
their subject line. Such messages result in <literal>relaynews</literal> calling
the appropriate script and on execution a message is mailed to the admin about
the action taken. These control messages are stored in the 
<literal>control</literal> directory of <literal>$NEWSARTS</literal>.  For the
propogation of such messages, one must subscribe to the
<literal>control</literal> hierarchy.
</para>

<para>
When your news system determines that a certain article has not been subscribed
by you, it is <quote>junked</quote> i.e. such articles appear in the junk
directory. This
directory plays a key role in transferring articles to your NDNs as they would
subscribe to the junk hierarchy to receive feeds. If you are a leaf node, there
is no reason why articles should pile here. Keep deleting them on a daily 
basis.
</para>

</section>
</section>
