Logs eating disk space

Xibo logs its stats back to the sql database.

It sounds like you've moved part of the database from under MySQL. I'm not sure how you'd go about fixing that. A good place to start might be to stop MySQL and put the file back where you removed it from, then start MySQL up again.

If you're not interested in the stats, then I'd suggest you switch them off as they do generate alot of data.

My Xibo generates a few mb of log each month. I guess you have logging and auditing switched on. You should have those set to off in a production environment, and that is how we ship the system. Check also that audting is switched off against each display record. Again that is the default.

You can then connect to the MySQL console and truncate the log and stat tables safely to free the space used currently.

Cheers

Alex

Thanks for prompt response.

Forgot to say -
1 - error settings on server are audit-off, debug-off and server mode -production. ie as they should be to minimise the log expansion.

2 - Also - all displays are set individually to auditing -off.

(these were before we realised the problem - so in theory should not be contributing to it.)

We renamed the sql datafiles for the log to stop Xibo making them even bigger (and filling disk, causing problems elsewhere), Can rename them back - then try the truncation - any clue on how to do this from MySQL administrator (windows GUI) or if not then using sql via phpmyadmin? (we would welcome more expert advice before messing directly with the database).

We will do this and then monitor situation.

1 - Do stats go in to the log and stats tables or only into the stats database table?
2 - how can we find server version from web interface?
3- and incidental - maybe this needs to be feature request - change display client settings from server web interface?

(our clients show version 2.0.0.0 - yet we think we installed 1.03 - is this right?)
TIA

OK. If they're definitely off then there actually is something wrong that's filling the logs.

Stats go in to the stat table, logs in to log.

I'm not a MySQL guru, but renaming those files was a really bad plan. My suggestion is you seek help from a MySQL forum, or try what I suggested initially.

Once you have the database back and working, select the last few hundred rows and see what it is that's filling the log. Either it'll be audit data from a client where it's been on in the past, or there's an actual underlying problem that the system has been logging in there.

I'm not familiar with the Windows tools, but I think PHPMyAdmin has a section where you can execute a query directly.

You literally want "truncate log; truncate stat;" to clear out those two tables.

The server version is held in the database. I don't think it's visible anywhere from the web interface, but if you go in to the Report Fault routine and just click on the generate troubleshoot.txt button, the version is included in there as part of the database settings dump.

If you are running 1.0.3 then you really should upgrade to 1.0.4 as there is a security vulnerability in 1.0.3. You can't change the client settings from the server at the moment. If I were you I wouldn't be making that change right now anyhow. If it's taken a couple of months to get to 25GB once you truncate those tables we've got ages to find and solve the problem before it causes you problems again.

Alex

I know, I know - renaming the files was a bad idea ;-( -- we were desperate.

- the disk was full and MySQL was falling over - and it also runs our main webserver. (needs to be on same machine so Xibo is externally accessible for management)

OK - we have truncated the stats file - no problem.
... and we have recovered the log file so SQL can 'see' it.

Unfortunately, phpmyadmin shows it as over 2million screens long (30 per screen), and MySQL query GUI crashes when trying to get to the end. (there are over 67million records).

However, there does seem to be a pattern emerging - looking at records around the 500 thousand mark -
they show an error -

<errormsg>Creating default object from empty value</errormsg>
<errornum>2048</errornum>
<errortype>Runtime Notice</errortype>
<scriptname>W:\wwwroot\xibo\3rdparty\nuSoap\nusoap.php</scriptname>
<scriptlinenum>75</scriptlinenum>

consistent from around record 400 forwards to as far as we can go. They appear in batches of 20-50 with identical time (incrementing seconds as they occur)

Various ip addresses are shown corresponding mostly to the display PCs. UserID is zero as are schedule, media, display etc.

Have now truncated the log table and will see what happens next - if error re-occurs and how quickly.

Thanks for your help and support so far.
Andy

<errormsg>Creating default object from empty value</errormsg>
<errornum>2048</errornum>
<errortype>Runtime Notice</errortype>
<scriptname>W:\wwwroot\xibo\3rdparty\nuSoap\nusoap.php</scriptname>
<scriptlinenum>75</scriptlinenum>

That should be about the only message you see if all the logging and auditing is turned off. It's a deprecated call used in the nusoap library the causes it - and we don't want to get in to supporting modified versions of nusoap.

The long term plan is to write a little PHP page (like Moodle has) that you call on a schedule that summarises the stats and deletes old log data. Unfortunately, like alot of things, it just needs someone to sit down and write it!

Cheers

Alex

It is reassuring that this is an 'expected' message - but having cleared the log file, one display machine generated over 7700 log entries within a matter of 12 minutes and was still going until we rebooted it.

The stat file is also increasing again - currently at 1200 records after 15mins again until we rebooted the client.

Things appear much quieter now - a few error records being added sporadically from different display PCs and the stat file seems static.

Will truncate both again, keep watch and see how things are in the morning.

Sorry cannot offer to do the php (lack of relevant skills), but let me know if you need to test it when it does happen, I think we have test server still running somewhere.

Appreciate the software and the prompt support - thanks again.
Andy

It sounds to me then like those are 1.0.2 clients. They did have an issue with spamming the server with large amounts of log data.

I would urge you to upgrade the whole lot to 1.0.4 and turn off the stats on each client while you're there.

Cheers

Alex

By 11pm we have 350,000 records in the log and 23,300 in the stat file.

As far as we know we had switched off the stat on the clients and rebooted them.

Will double check in the morning on the client version as well.

... to be continued ...

We will probably re-install the 1.03.1 client if we find a particularly troublesome display PC - but as far as we can see this shows as version 2.0.0 - is this right?

All the versions show that.

I can't impress upon you enough how important upgrading to 1.0.4 is, especially as you have your Xibo instance exposed to the internet. The upgrade on the server will take less than 5 minutes and say 30 minutes to do your 7 clients.

While you're at the clients, I'd stop them running and empty the local library directory to make sure there's no old log or stats waiting to be sent back.

Alex

--- original message ---
From: "bede" <email address hidden>
Subject: Re: [Question #91501]: Logs eating disk space
Date: 24th November 2009
Time: 11:28:12 pm

Question #91501 on Xibo changed:
https://answers.launchpad.net/xibo/+question/91501

bede posted a new comment:
We will probably re-install the 1.03.1 client if we find a particularly
troublesome display PC - but as far as we can see this shows as version
2.0.0 - is this right?

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Sorry Alex I am a bit confused -

in an earlier post you suggested checking all clients and server were same version and that logs/stats were 'under control' so we could see if we had a problem - and not to upgrade to 1.04 until we had sorted that.

Is your advice now to upgrade server to 1.04 regardless (and clients), now we have the log file back in the database.

TIA
Andy

No worry - we have gone ahead and upgraded the server to 1.04. Now tackling the clients.

The server version number is in the 'version' tale in the database - and can be seen from phpmyadmin.

Trying to upgrade the clients -

running the msi does not appear to be updating the files in the program folder - tried this manually and via GPO.

Do we have to uninstall each time? (did not see this in the manual - sorry if we missed it)

How can we check which client version we have (esp if they all show v2.0.0)? This sounds like a daft question, perhaps we have missed something somewhere.

Is there a datestamp for a file in the client install that we can check for the version? (1.04 .exe seems to show August 2009)

TIA

1.0.4 is only slightly different from 1.0.3 at the client so that might well be normal.

I don't have access to a client to check right now.

Once you've upgraded the server and all the clients, and (with clients all stopped) cleaned the local library directory, restart the webserver and mysql and then fire the clients back up.

the reason that's necessary is I've had on one installation the server go in to some kind of loop which caused the continual logging to continue even with all the clients stopped.

Alex

--- original message ---
From: "bede" <email address hidden>
Subject: Re: [Question #91501]: Logs eating disk space
Date: 25th November 2009
Time: 10:42:47 am

Question #91501 on Xibo changed:
https://answers.launchpad.net/xibo/+question/91501

    Status: Answered => Open

bede is still having a problem:
Trying to upgrade the clients -

running the msi does not appear to be updating the files in the program
folder - tried this manually and via GPO.

Do we have to uninstall each time? (did not see this in the manual -
sorry if we missed it)

How can we check which client version we have (esp if they all show
v2.0.0)? This sounds like a daft question, perhaps we have missed
something somewhere.

Is there a datestamp for a file in the client install that we can check
for the version? (1.04 .exe seems to show August 2009)

TIA

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

For me 1.0.4's modification date is 08 August 2009.

Dan: Can we not have the version string in the player set correctly? Even if it showed 2.0.4?

A

Yes, I am sure we could do so.

Can we target a bug to 1.0.5 for this and ill look into it.

2009/11/25 Alex Harrington <email address hidden>

> Question #91501 on Xibo changed:
> https://answers.launchpad.net/xibo/+question/91501
>
> Alex Harrington proposed the following answer:
> For me 1.0.4's modification date is 08 August 2009.
>
> Dan: Can we not have the version string in the player set correctly?
> Even if it showed 2.0.4?
>
> A
>
> --
> You received this question notification because you are a member of Xibo
> Developers, which is an answer contact for Xibo.
>

Even better if the client version showed in the list of displays on the web management interface (along with stats, refresh and ppt settings).

and better still if the settings could be changed from there as well.

and I know this sounds greedy, but being able to clear the local client library/cache from there would also be good.

All this would save journeys to PCs, slow VNC connections or repetitive file changes via remote explorer connections.

Pretty please. (along with the server version somewhere in the web interface?) (do I need to add these requests to a feature/bug report somewhere?)

In the meantime, we think we found a rogue 1.02 that was pumping the logs and things have settled since upgrading it. We are currently upgrading all other clients and then will do the shutdown/library clearance you recommended.

It looks like 1.03 client has same datestamp of 8th Aug, but upgrading via GPO has worked for that rogue one - excellent, thanks. Useful for future upgrades.

Will hopefully sign this off later today or tomorrow once we have finished the tasks. Things looking much more promising.

... thanks yet again for your prompt help and support.
Andy

Hi Andy

I think you'll find most of those things are covered by existing blueprints, but feel free to read through and create one for any/all of those points.

It's not appropriate to add those to the existing bug report because they're functional changes to the system (and in most cases would require changes to the protocol used between client and server).

If they're logged as blueprints they're considered for inclusion as we revisit each area of the system, so I can't give you a timescale on them but it's certainly all sensible additions so I don't see why they wouldn't be included at a later date - with the possible exception of clearing the library as that requires the client to be stopped for it to happen.

Cheers

Alex

I'm not sure there is much merit in me mentioning this but... in 1.0.5 it
shouldn't be necessary to clear the local library any longer - as everything
in there will be replaced automatically if there is any corruption.

2009/11/25 Alex Harrington <email address hidden>

> Question #91501 on Xibo changed:
> https://answers.launchpad.net/xibo/+question/91501
>
> Alex Harrington proposed the following answer:
> Hi Andy
>
> I think you'll find most of those things are covered by existing
> blueprints, but feel free to read through and create one for any/all of
> those points.
>
> It's not appropriate to add those to the existing bug report because
> they're functional changes to the system (and in most cases would
> require changes to the protocol used between client and server).
>
> If they're logged as blueprints they're considered for inclusion as we
> revisit each area of the system, so I can't give you a timescale on them
> but it's certainly all sensible additions so I don't see why they
> wouldn't be included at a later date - with the possible exception of
> clearing the library as that requires the client to be stopped for it to
> happen.
>
> Cheers
>
> Alex
>
> --
> You received this question notification because you are a member of Xibo
> Developers, which is an answer contact for Xibo.
>

Good news and bad news.

The log file is now increasing much more slowly - 500 records in 15 minutes.

.. but the stat file is increasing more rapidly - 6400 records in 15 mins.

We are doing a triple check that all the settings etc are OK but are reasonably certain that stats are off on clients with 900 refresh interval and that all are now running 1.04.

The log file does not have the same kind of pattern of ip addresses logging lots of data.- a mix, but there are occasional errors where page field is client showing submitstats completed error, and the settings on that display PC are stats off.

Would it be helpful to post some entries from the stat file?

or are we worrying about nothing?

Maybe another feature request is logfile and stat file rotation to preven them getting too big anyway.

TIA
Andy

Hi Andy

You restarted Apache and MySQL as instructed too?

If you go to the Report Fault section on the Management menu. Skip the initial steps and just run the troubleshoot.txt bit and email it to <email address hidden> so we can see it.

Alex

This was solved in an email thread. To update anyone that may also be experiencing this problem.

There is a bug in the 1.0.4 client which will always send stats regardless of the setting. This will be fixed in 1.0.5.

A workaround for anyone having a major problem with this would be to:
 "comment out lines 956-960 of XMDS.php which will stop it writing the stat records back to the database"

Cheers,
Dan