Xibo loading time with file corruption detection

Hi Bartek

You can't have it both ways. How is the client supposed to check if the files are all present and correct if it doesn't check them?

At the moment it just assumes that they will be, but as you say if you're restarting after a crash then there could be work to do which is the whole point of the modifications you've been pushing for.

Who's to say that just because the files were correct last time they were run in a client nothing changed - eg filesystem corruption.

Again I'm not intimate with the codebase but I'd imagine the client will only checksum files that are in the current schedule at boot time so your 20gb layout isn't a reasonable or fair test.

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: [Question #93892]: Xibo loading time with file corruption detection
Date: 13th December 2009
Time: 12:31:50 am

New question #93892 on Xibo:
https://answers.launchpad.net/xibo/+question/93892

Hello again,

As You know i'm testing dev version of Xibo with filecorruption improvements. As You also know ;) in out enviroment we're using a lot of big mpeg files in layouts.
And there is a problem - it's took ages Xibo to start. I can guess that client recalculate md5's of all files in layout on start before playing anything. Probably there is no problem on layout with few flash animations, jpegs etc. But when starting client with a lot of big files it's hangs for minutes with black screen - even without xibo splash screen...

Maybe You should add some option in client for skipping md5's on start for layouts completed earlier - for example when layout was succesfully downloaded once before all files don't need to be recalculated everytime when xibo starts, only when downloading and validating etc? Of course then files could be damaged later, but when choosing between all clients hanging for a while every time when they are started VS. file damaged later in use on one terminal (no in download time!, in effect of disk errors etc.) could be nice.

For example:

client starts -> no changes on server in layouts and layout is already downloaded locally with all files -> skip check md5 if some skipping option is turned on.
client starts -> some changes in layouts used on client made on server ->start playing locally stored version of layout -> in bg download missing files & recalc md5 of files needed in new layouts.

Or something like that ;)

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Hello Alex,

I'm checking on smaller layouts than 20gb - this one loads 30mins and hangs every 5 minutes for some more calculation or something - it's not important becouse its only download layout (but in other way i don't understand those hangs during playback - md5 are checked later again or on start only?)

Problematic ones are usage layout that have 3-5gigs and start for 5-10 minutes - my computers have slow processor, to decode mpegs using nvidia gpus and codec.

Of course you are right about general situation - when client won't check files every time it start it won't be sure that every file is correct.

But I would like assume that there will be no local files problem after downloading. I suppose to have problems only on download time in effect of my network type. In my installation there is very big chance that file going to be broken during install and very smal chance to be destroyed later - separated hdd for content, cached disabled etc. Only xibo writes to that harddrive so chance that something destroys files later is very small. Thats why i'm thinking about separating file checking to two levels - download level and later recheck level with ability of disabling second one.
As written before - in my situation one damaged file(damaged - not incorrecty downloaded) somewhere in the world is better that 70 terminals displaying black screen on start for 5 mins after power on/power restart etc.

So if in the code is easy to separate those two levels it could be nice to have one of them in if(some switch){}; . If no, maybe at least client can display splashscreen instead of empty screen

If you were to start the client with the network cable out I suspect that it would start in a timely fashion - however the first time it successfully got the list of required files from the server it would "hang" trying to MD5 all of those files against what it has cached. You could argue (and Alex would) that it should keep a cache of the required files and MD5 regardless of network presence so it can check immediately on start up, but it doesn't currently do this.

I believe the cause of the behaviour you are seeing is that the entire client operates in a single thread. Therefore when the MD5 is calculated any changes to the display are not actioned.

There are two possibilities I will look into:
1. Build in a delay between file collections which will give the display actions time to "fire up".
2. Interrupt the XMDS calls to action some display actions.

I don't see that either of these will make a difference if you have 20GB of files to MD5.

The only way of fixing this properly is to move the entire XMDS file collection out into its own thread, which it not something we can do in the stable branch of Xibo.

Maybe I have another idea...

The MD5 object that I keep in memory contains the following information:
- The file path
- The MD5
- The date the MD5 was created

When the client checks the MD5 of an existing file it will first check the windows modified date of that file to see if it is has changed since the time it stored its MD5 - if it hasn't changed then it will serve the MD5 in the cache without calculating it.

What if the MD5 object was written out to file when the client shut down, and then read back in again when it starts up.

In the scenario Bartek describes none of the files would have changed (modified dates would still be before the cached date) and therefore none of the files would be re-downloaded. However if one of the files had been altered its modified date would have changed and it would have its MD5 recalculated.

Thoughts?

> Thoughts?

It seems a reasonable compromise.

One of the nice side-benefits of the md5 checking is that if someone is attempting to modify what is being shown on the display by means other than the Xibo server then that would be caught. It's trivial to modify the modification date of a file so you'd loose that.

You also have the potential for FS corruption after a crash or power failure etc and again this wouldn't be detected. Perhaps we need to look at what the client does if you feed it a mangled video?

Alex

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Bartek

If you're able to, could you make your library of videos available for us to use to test with? I'd be interested to see what effect large numbers of large videos have on the python client but I don't have access to that kind of volume of material at the moment.

I'd be happy to download that quantity of data or send you a postal address for a stack of DVDs if you're able.

Alex

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Alex,

Perhaps it would be wise to extend the model I have to also include a freshness date? If I did make the change to persist the MD5 across restarts we would likely never re-calc them.

I could have a 5 day freshness check which would force a rehash..

The large volume of videos would also be useful for my testing process - especially with the changes 1.0.5 will bring to the way videos are handled. Bartek, what would be even better if possible, is to have a backup of your Xibo database (in addition to the media from your server library)... we could then test with exactly the same data.

> I could have a 5 day freshness check which would force a rehash..

You'd still have the black screen at some point though?

I think what you described before is a reasonable compromise, so long as people are aware of those limitations.

Alex

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Problem is that if I persist the Cache across restarts and there is some FS level corruption, as you point out, then it will never be detected (modified date still valid).

Freshness would be good if it was on another thread - otherwise perhaps not.

No problem with sharing our mpegs and layouts, there is no top secret data inside ;)

Right now there is 35GB of content on server hd + sql.

Also no problem with whole server sharing - it's xenserver vm so it can be easily exported to single file if you like. But then you need at least 500gb of free space on such server - this is size of VM's local storage.

I can guess that you'll prefer tar copy of server xibo lib dir + sql dump ?
No problem with sharing over net, there is 40mbit symmetric network connection and no problem with full transfer over night and 10-15 mbit 7.00-18.00

Hi Bartek

Thanks for that. I can pull them down at work (only 10 mbit!) overnight.

I'm geographically close enough to Dan so I can share it with him directly.

Sql + library dir would be ideal.

If you could mail a link over when you have time along with preferred time for me to pull it down I'll do that.

Cheers

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: RE: [Question #93892]: Xibo loading time with file corruptiondetection
Date: 13th December 2009
Time: 4:24:14 pm

Question #93892 on Xibo changed:
https://answers.launchpad.net/xibo/+question/93892

Bartek posted a new comment:
No problem with sharing our mpegs and layouts, there is no top secret
data inside ;)

Right now there is 35GB of content on server hd + sql.

Also no problem with whole server sharing - it's xenserver vm so it can
be easily exported to single file if you like. But then you need at
least 500gb of free space on such server - this is size of VM's local
storage.

I can guess that you'll prefer tar copy of server xibo lib dir + sql dump ?
No problem with sharing over net, there is 40mbit symmetric network connection and no problem with full transfer over night and 10-15 mbit 7.00-18.00

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Hi, i've sent url to your mail address given to me by Mariusz.
Please let me know that you got it (or no ;))

I have implemented this Cache to disk functionality and it seems to be working well (it loads quicker than ever ;-) however I would like to do some more testing before letting you try it.

Once I am happy with it I'll put a link here to a ZIP containing the client files to patch.

Something has come to my attention though - there was a change in XMDS.php from the server that should also be changed in order to test this correctly. The MD5 sent by XMDS was slightly wrong and if you don't also patch that file the Layout will be downloaded each time.

I'll include a link to that file with the client ZIP files.

Cheers,
Dan

There is a ZIP file here: http://dl.dropbox.com/u/353651/Xibo/Xibo%20Client%201-0-5.zip for you to try. Just copy the contents over the top of your Xibo install directory.

There is also the XMDS.php file to drop into your server here: http://dl.dropbox.com/u/353651/Xibo/xmds.php

I urge you to backup your current config first!!!!

Thanks for the testing help.

Hi,
Thans, great info, i'll try in 24h :)

What do You mean config? Server sql and overwrited file or local test client?

xdms can be safely overwrited on production serwer (theoretically ;))? I'm using same server for testing and production ;)

I just meant to backup the files you will be replacing in case anything goes wrong ;-)

In theory you can replace XMDS.php on your live site without any problems - but it would be a good idea to have a copy of your current file just in case I am wrong!

Please note that once you replace your XMDS all the current clients (that are on-line) will start to re-download their layouts every collection interval until you replace their client files.

Maybe it would be better for you to clone your current server installation as a test system?

My system right now is one big test system ;) becouse devel version is more stable than stable 1.0.4 in my case ;)
I'll start testing soon ;)

Hi Bartek

I don't seem to have it.

Could you send it to <email address hidden>

Cheers

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: RE: [Question #93892]: Xibo loading time with file corruptiondetection
Date: 14th December 2009
Time: 1:15:02 pm

Question #93892 on Xibo changed:
https://answers.launchpad.net/xibo/+question/93892

Bartek posted a new comment:
Hi, i've sent url to your mail address given to me by Mariusz.
Please let me know that you got it (or no ;))

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Sent again, have you got it?

Ok, i know where problem is.
You're using gmail as mail provider, and gmail hates my ip and puts all mails directly in trash ;(
Check your spam folder and it should be there...

Dan:

> Please note that once you replace your XMDS all the current clients (that are on-line) will start to re-download their layouts
> every collection interval until you replace their client files.

Redownloading with no visible effect - no problem for me, question is: Will Redownloading generate layout restart and playback start from beginning?
Right now changing layout generates client reload to start playback of new content.
Will old client consider new xdms as good reason to reload playback every 5mins on all displays? ;)

Dan, there is a raport for You about new test version.

At the beginning small info: i've created separatte folder for test xibo version on server in /test subdir, copied there whole Xibo dir, and overwrited xdms. Used the same db that production version. BUT please, i still like to know answer for previous question, it will be important later on version ugrade on production clients :)

On testing used our beloved Whole Library playlist.
Luckily one of my co-workers added today some content to WL, so checkeded two things in point 1:

1. Xibo started correctly, and recalculated md5 od existing files in ~10mins (win7 have fast cpu in oposition to production hardware), next downloaded missing files, and started playback, everything OK.

2. Xibo shutdown and restart without file modification, everything OK, xibo stored file data somewhere ;) and started without problems in 1 sec, playback started.

3.Removed file from the near end of playlist so xibo had a chance to download it before reaching it in playlist. File missing correctly spotted. Playback started, then file was downloaded in background and played correctly when it should be played later. Other files played without rehash correctly.

AND TWO FAILED TESTS:

4. Removed first file in default & already synced playlist. Xibo hanged on black screen, but downloaded missing file. After dowloading still hanged out on blackscreen for ever. After restart played ok. so in general - test FAILED

5. Removed second file in sequence to check what wolud happen when Xibo will reach with playback file that is being downloaded in same time.
So: First file played ok, during that time Xibo downloaded 9Megs of second file. Then started playing second file just how sequence said, downloading was stopped prior to file access conflict. Then played part of file (as you probably now partilally downloaded mpeg can be opened and played in wmp as streaming format. Avi for example cannot becouse its container format and need to be correctly closed etc) and stopped after that. Client hanged with piece of broken frame on screen, downloaded ofcourse not restarted. so test FAILED.

Conclusion:

4 and 5 failed, but also failed on previous versions of xibo with firecorruption detect, probably needs handling a situation when playing at client start layout that is default to client, and some files are missing. Due to fact that there can be more regions time synced with this broken one, there is probably only one good choice: pause whole playback until missing files are completed and display splashscreen (black screen it's a bad idea ;)). But then, what it best option to offline terminals, those should play nothing at all if there is no needed file? Or maybe offline is simply unsupported conception so there is no problem at all, and it doesn't need play any content with lib errors :)

1 and 2 passed great, so your solution for rehashing works great ;)

3 is truly 5 with some lucky it it, so fixing 5 will also make 3 more reliable ;)

Ok. Thanks. I can't check from here but I'll look when I get home in an hour or so.

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: RE: [Question #93892]: Xibo loading time with file corruptiondetection
Date: 14th December 2009
Time: 6:08:29 pm

Question #93892 on Xibo changed:
https://answers.launchpad.net/xibo/+question/93892

Bartek gave more information on the question:
Ok, i know where problem is.
You're using gmail as mail provider, and gmail hates my ip and puts all mails directly in trash ;(
Check your spam folder and it should be there...

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Hi Bartek

Got it - thank you.

Alex

Re-downloading a layout will only cause playback to restart if that is the currently playing layout.

4 and 5 failed because Xibo tried to play those files before it detected that they were not there / bad. It shouldn't have hung though - setting all the work we have done aside for a moment, if someone uploaded a corrupt video into the Xibo server and a client tried to play it then it should fail gracefully and move onto the next file.

I will try to get to the bottom of why that is...

You can kill me but I still can't understant if I can replace my production xdms. So maybe I'll describe source of my problem:

Each client right now plays it's default layout set in panel. There is no sheduled other layouts. Can I replace xdms, or there will be restart every 5mins? You've said that each old client will redownload layout every it's interval:

"Please note that once you replace your XMDS all the current clients (that are on-line) will start to re-download their layouts every collection interval until you replace their client files."

so I undestand it like this:
current clients (that are on-line) will start to re-download their layouts every collection interval -> so default layout also will be redownloaded every 5mins -> so there will be restart every 5 mins?

When i'm playing such broken mpeg in wmp, wmp also hang out and progress pointer stops for ever. It's different situlation that broken avi file that wmp will simply refuse to play. maybe You're detecting wmp control errors and wmp don't report any error becouse waiting for next part of mpeg stream that will never appear, and there was no end marker so it can wait for a very long time ;)

Anyway both of You will probably have a lot of mpegs in next few days, so there will be a chance to reproduce this situation ;)

Hi Bartek

I downloaded about half of it last night but now I'm getting connection refused.

Alex

Congratulations,

You're probably just found bug in debian thttpd package related with log rotating ;)

Restarted :)

> You're probably just found bug in debian thttpd package
> related with log rotating ;)

:)

Thanks

Alex

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Alex:
Am I right that you've already downloaded all files? If yes i would like to shutdown thttpd. If you want download again somewhere else - no problem, but as I said i want to disable service when it won't be needed any more.

Dan:
I've just spotted one additional thing in last test version. I don't know if it is expected behaviour or no:

Player synced with last version of WL layout. Restarted to check that will start fast and normal - everything ok.
Stopped client
Then added some aditional files to WL (WL is default for test player)
Started client
Player started to play locally stored old version of layout (as long as I can guess that was old version). And started downloading in background missing files.
Then i've interrupted download process and shuted down client.
Client started with splascreen and displayed it until downloading of missing files was finished. Then playback started.
So there I don't know if it should keep playing old version until sync finish, or maybe you've planned it like it happened.
I can ever see the reason - you're now deleting old .xlf version when something is changed, and saving it on transfer end, so there is no playback after restart - client acts like first run, downloading files, and starts playback after finish.
In old client version, old version of layout is there until new one is downloaded in 100% - thats probably produce better efect ;)

Hi Bartek

Yes - I've got the files thank you. They are extracting now. By all means close the service off but if you could keep the files handy until they've extracted properly and I know they're OK that would be much appreciated.

Cheers

Alex

> Yes - I've got the files thank you. They are extracting now.
> By all means close the service off but if you could keep the
> files handy until they've extracted properly and I know
> they're OK that would be much appreciated.

All downloaded and extracted thanks.

Alex

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Hi Bartek,

That is the intended behaviour for the client as it stands in the 1.0 series. It detects that the layout was incorrect so deletes it - when you restart you therefore have nothing to play.

Cheers,
Dan

Sorry to hear it, becouse if all previous versions xlf stayed untouched until all media to new version was downloaded and stored locally. So there was a playback of older version, also after restart.
But of course we can forge new layout instead of modifying old one and send it to client, it's not so hard until there is only one region on screen. Only adding 50mpegs again is a little boring ;) But's not my problem anyway :)
I can't complain in this subject, becouse as Alex said some time before, xibo was designed for a little different usage model than ours, and probably its not important unconvinience when using shudeler instead of changing default layout :)

One more info - wmp fix work flawlessly with our playlists, no white screens (or so short that no able to spot them ;)) any more, great :) RSS cooperation not checked, but I can guess that's also ok :)

Anyway, do You need any other tests? If no, feel free to close subject :)

It was actually your previous request that changed the behaviour in this area - if the layout was corrupted then it must be deleted precisely because it will play.

Generally speaking Xibo was designed to show lots of smaller layouts rather than one large one. If you split yours up into a few more sections (say 4?) and then schedule all those on the display for a long time (years maybe) then this problem would be much less noticeable for you... (unless you corrupted all 4 layouts of course ;-)

I am glad the changes to the video have worked out well for you!

The only thing you could do is note your suggestions on the blueprint [https://blueprints.launchpad.net/xibo/+spec/dotnetclient-getfile-improvements]. We can then make sure the Python client will handle downloading media well and also consider future plans for the C# client.

Thanks for all your help!!

On Thu, 17 Dec 2009, Dan Garner wrote:

> It was actually your previous request that changed the behaviour in this
> area - if the layout was corrupted then it must be deleted precisely
> because it will play.

In simple decision process if (file on server == locally stored file )
ofcourse yes, buf there is probably a place for some execption when
upgrading layout file (only for xlfs) from server after modification. But
probably some implications will appear :)
Generally we can leave old layout file without deleting if all media files
of it are present on hd, or delete if any of them missing. But then it's
hard to say why file on server is diferrent - becouse of damage or becouse
of changing layout itself on srv.

It's generally one additional point to think of, and add to conception of
some unofficial stable version fork after 1.0.5.

Maybe You can read https://answers.launchpad.net/xibo/+question/93734 and
say, theoretically only, do You think that idea have any sense? But
ofcourse without rush :)

> Generally speaking Xibo was designed to show lots of smaller layouts
> rather than one large one. If you split yours up into a few more
> sections (say 4?) and then schedule all those on the display for a long
> time (years maybe) then this problem would be much less noticeable for
> you... (unless you corrupted all 4 layouts of course ;-)
>

Right, thats exactly what Alex said some days ago.
I think one big layout sheluded somhere near 2050 ;) should be also fine,
can be safely downloaded in backgroud during playing current layout, and
then changed for default. Am i right?

> I am glad the changes to the video have worked out well for you!
>
> The only thing you could do is note your suggestions on the blueprint
> [https://blueprints.launchpad.net/xibo/+spec/dotnetclient-getfile-
> improvements]. We can then make sure the Python client will handle
> downloading media well and also consider future plans for the C# client.
>

Why not, but i have to know if it has any sense (for example previus
resume version concept was undoneable becouse of exist->valid logic)
So if You'll comment 93734 shortly, i'll add something there after next
rethink :)

> Thanks for all your help!!

Thanks for your hard work :) 1.0.5 should be great ;)

PS. It there some tag that can be typed in mail, to act as 'That solved my
question' button? :)

It would be better for you to schedule smaller layouts... that way if you change one of them (for example to add some new content) the others will continue to play interrupted (whilst the one you changed is deleted and re-downloaded).

I think we need to go back to basics... the goal of this change was to ensure that if media did not download correctly - or it was corrupted in some way that it would be re-downloaded the next time Xibo contacted XMDS, which is what happens...

It is possible that it is a little too ruthless! For example the action of adding a new item to the layout will be that the layout is deleted and only re-downloaded once the media is all available. This would cause the layout to be unavailable until the media is finished transferring (a big problem in your case, where you have 1 big layout).

I think the best thing to do is release 1.0.5 in its current state and I'll comment on the 93724 question now.

And yes, it would be great to have the option of adding some keywords or a tag to the mail reply which marked the question as solved... perhaps you could ask about that on the Launchpad project page.

Cheers!

Thanks Dan Garner, that solved my question.

Hello,

I decided to open this question to ask one more question about Your last test client version,

I'm going to start using this client in my network becouse it's acct quite good (deleting changed xlf is not very important right now), but I've made a simple test:

I've replaced xdms.php with new one in some test location, and pointed OLD client (as it's not possible for me tu update all clients in one time) to new xdms. Also i've reduced refresh time to 60 secs.
And client restart to the beginning every 60 secs. New clients acts ok.

So the question is: Are you going to release this version as part 1.0.5, if there will be no any news additions discuised few days ago? It could be problematic for some people if old clients need to be replaced everywhere. Maybe some backward compatibility is needed if it will be 1.0.5 part.

Generally no problem to me, i'm going to keep both versions for tests. Recompilation of last test branch and replacing all request do xdms.php with xdms-new.php should do the thing.

Hey Bartek,

We always suggest that the server and the client are updated at the same time - because there is no support in XMDS for backwards compatibility. It is something we would like to do in the future (support older clients on the network), but isn't a short term goal.

Usually it doesn't matter so much, but in this case the layout will be detected as invalid with each collection interval.

If you cannot update all your clients at once, I would suggest the following workaround:

Clone your 1.0.4 installation into another folder. Upgrade this clone and point all new clients at it (leaving the old clients pointed at the previous folder). The old clients will continue to get served by the 1.0.4 XMDS and the new ones by the 1.0.5 XMDS.

As I said, we will eventually support backwards compatibility.

Cheers,
Dan

Ok, sounds reasonable if right now there is no officialy backward support.

I think i'll prefer my solution with changing xmds.php name hardcoded it client. For me there is one big advantage in compare to Your method (that I've used in previous tests): There is no way to make a mistake and mix incorrect client and server version, and it's probably quite easy when there is 45 clients in operational state...
Later when all clients will be replaced simple changing xmds-new.php to symlink to new xmds.php would finish a issue.

Anyway, thanks for the answer :)

From other hand, probably some info (and Your workaround) about this issue should be added to relase notes if this test branch will be merged to 1.0.5

Yes - thank you Bartek.

We're well aware that is the behaviour. Dan informed you of that when he gave you the updated xmds.php.

As ever the release notes will tell people the compatibility between various server/client versions.

eg http://wiki.xibo.org.uk/wiki/Release_Notes:1.0.3
Quite clearly says that 1.0.3 clients are required with 1.0.3 server

and
http://wiki.xibo.org.uk/wiki/Release_Notes:1.0.4
Quite clearly says that 1.0.3 or 1.0.4 clients will work with 1.0.4 server.

If it's a problem for people to upgrade their clients then they should hold off upgrading until they are in a position to do so. We won't post "workaround" details on GA release because while it will work, it's not the "right" way to go about things.

We're not in the business of putting short term hacks in to the software to fulfill short term goals. There is versioning support built in to the specification for xmds and in time it will be possible to use that to distinguish between client versions. Unfortunately it's not a priority right now.

Alex

I've got some problems with test version, maybe some ideas why?

Since now i was testing it only on my notebook with win7 and ffmpeg. And everything was ok. Now I wanted to start using it on some production terminals,

So downloaded last 105-test branch, changed splashscreen for different jpg, and replaced all xmds.php apperances with xmds-new.php as desribed up here. Next copied such binary to my notebook and two test production dells prepared to installation somewhere after xmas.
Next, i've clicked Save button on all three clients to recalc XiboClient_xmds_xmds with new -new suffix. Since now everything ok - modified urls appeared on register tab, communication with server ok.
Next removed all xlf's form lib, and CacheManager.xml (if there was one - on notebook) from settings folder, to have similar test enviroment. 300 secs refresh time, same Whole Library playlist.

Started on notebook:

Work as before with your binary and new xmds in separated test folder, splashscreen, md5 recalc, xlf saved, playback, fine.

But on two test dells:

Started, splashscreen, md5 recalc, xlf saved, playback of first mpeg, and... xibo exits somhere near end of first movie :(

So two question again:

1. Any ideas how to diagnose problem on xp dells? There is nothing interesting in logfile.
Maybe problem is related with used codec (only single instance with hardware acceleration possible), etc. System is quite different that test notebook.
I can make tests again (If You're going to say that there is no support for modified sources) without my '-new' source modification, but i don't think that the reason is...

2. CacheManager.xls on notebook is saved only after correct client close by alt+f4. Killing client from taskman or pc reboot -> no file on hdd and after next start checksuming is started from beginning. Even waited for some next collection intervals - no luck ;)
Is it expected behavior? Shoudn't be saved just after all sheduled xlfs correctly downloaded and saved?

Alex:

Yep, youre right again. Probably i've entered whole testing process to deep, and i've fotgot that some users (amost all) probably want's stable software after new version release with clear information, work with A, B, C version, no some fancy compatibyility tricks for last 10 version backward :) Quite ridiculous idea, there is no way to support it later :)

Just checked on binary from http://dl.dropbox.com/u/353651/Xibo/Xibo%20Client%201-0-5.zip. Same effect, crash after first mpeg.

Hi Bartek

Just a thought on modifying the source and the AGPL license Xibo is released under. I appreciate it looks unlikely that any modification is the cause here.

You are of course entitled to modify the source as you see fit, but that makes it really hard for us to support you in any way as we don't know exactly what was changed etc. You are also obligated by the license to make those modifications available to your customers by some reasonable means.

As you know we're a very small dev team and we have very limited resources for development and support. If we spend time on support / documentation development suffers and visa versa.

It would make our lives considerably easier if you pushed your changes in to your own branch in Launchpad so we can see what changed between mainline Xibo and what you're running. It also serves as a bullet-proof way of ensuring compliance with the license as you can then point your customers to your branch in Launchpad so long as that remains a true and honest reflection of the code your running.

Of course if you have other arrangements in place to distribute the sources to your customers then feel free to continue using those.

Best wishes

Alex

Can you try playing the MPEG in Windows Media Player. Do you see the same crash in WMP?

Another thing to check...

After the client crashes is there anything in the Application Event Log (the
windows one).

Alex:

I'm not able to make anyrational/big modifications in source becouse i'm not a programmer. I'm only modyfing some params/variables to fit better in my enviroment. xmds name, sheduler download time limit etc.
Also we don't have any customer in agpl meaning. Our network belong to us in 100%, right now we don't sell any hardware/software to anybody. But if it going to change someday ofcourse sources won't be a secret.

WMP plays files correctly, all previous client versions (1.0.4, 1.0.4 + first file collector) works fine on those computers. All ale identical, and there is 41 pcs working correctly with older clients. Most of them under 1.0.4 + first file collector version.
File checksuming also passed ok, so local files are correct.

Dan:

I would be quite problematic, becouse or windows are stripped from everything that wasn't required to operate properly :) Let's say - wasn't needed since today ;) So there is no Application Event Log ;)
But ofcourse if there is no other way to diagnose it, i'm going to spend some time after xmas trying to reproduce error on some similar windows installation with WEL avaiable.

Some more info that I can share right now:

- same effect with working net and broken net - if there is no recalc on beginning becouse of false xmds patch - crashing as always.

- no error on screen after enabling windows reporting - this one is operational, there is no saving infos in WEL only. But maybe there is some other error crash reporting in .net that could be enabled?

- there is 1-2 secs black screen after first mpeg (probably - becouse is hard to be sure over vnc), so looks that there some problem with loading next mpeg. Second codec's icon appear on taskbar after playbacl of first movie, so loading process is started somehow and then crash appear.

- after reordering playlist (other two movies on begginning) - no change - first movie, black screen, silent exit to windows

- when disabled coreavc and switched to ffmpeg - no change.

And maybe most important - 3 start sequences:

1. When there is no net (fake xmds in setup) and default xlf exist:

playback start just after start (no mds from server, no recalc, ok). there is Options Started and Application Finished events in Xibo log.

2. When there is net and xlf removed:

black screen during recalc on begin, xlf saved correctly, but no single record in log

3. Where there is net and correct xlf exist:

black screen during recalc, xlf no touched, no log again

Hi Bartek

I don't really want to get in to the intricacies of the license but if you're selling a service based on our software then it's my understanding that you must make the source for the app available to your customers. Presumably you charge your venues cash to put the screens in, or they're free by means of subsidy by advertising?

Alex

Last info, probably the worst :/

Same problem on last 1.0 branch with 105-webbrowser merged :( So problems probably caused with some wmp bugfixes...

Alex:

Ok, no problem with sources sharing for me, as I said since now one modified thing is a xmds reference change.
But topic is quite interesting in general, probably well to know later. Right now we prefer to help fixing xibo instead of building own fork.

Now we're operating in some test period. We have lcd is some pubs. We're giving them free hotspot for customers, they gives us a piece of wall to install a lcd. And thats all right now.

Later we're going to sell ads in those locations to others (no pubs owners). They're not buying any system, only paying for displaying their ad on our lcd. So should we share any sources with them (theoreticaly, right now there is no modifications to share) when there is only dispay-add-for-some-time-service sold?

Hi Bartek

I wasn't suggesting you should build a fork, all I was saying is that were you are making modifications it would be really helpful if you could push the changes in to Launchpad so Dan has them to hand when investigating problems - as a general principle.

IANAL but so long as you're not off building a fork then I'm cool on the licensing issue. Your company may wish to consult a lawyer to find out which obligations apply.

Cheers

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: Re: [Question #93892]: Xibo loading time with file corruptiondetection
Date: 24th December 2009
Time: 1:35:39 pm

Question #93892 on Xibo changed:
https://answers.launchpad.net/xibo/+question/93892

    Status: Answered => Open

Bartek is still having a problem:
Alex:

Ok, no problem with sources sharing for me, as I said since now one modified thing is a xmds reference change.
But topic is quite interesting in general, probably well to know later. Right now we prefer to help fixing xibo instead of building own fork.

Now we're operating in some test period. We have lcd is some pubs. We're
giving them free hotspot for customers, they gives us a piece of wall to
install a lcd. And thats all right now.

Later we're going to sell ads in those locations to others (no pubs
owners). They're not buying any system, only paying for displaying their
ad on our lcd. So should we share any sources with them (theoreticaly,
right now there is no modifications to share) when there is only dispay-
add-for-some-time-service sold?

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Regarding the Cache Manager... it is designed so that it will only be written to disk if the client cleanly exits - maybe it would also be advisable to write it to disk after a successful collection interval (if anything has changed since the last interval). I'll look into that.

Regarding the crash... if I could summarise the problem.

On your laptop running with the latest binary I gave you the client is fine. But on your production equipment the same client crashes out to windows silently after the first video is finished playing.

The only thing that has changed (as I remember) in that branch is that if the duration is 0 an event is raised which will kick off the end of a video. Are the durations of your videos all set to 0, or have you timed the videos and set a specific duration for each?

Can you try editing Video.cs at line 46 and replace the method with:

void videoPlayer_VideoPlayerElapsedEvent()
        {
            try
            {
              Hide();
              videoPlayer.Hide();

              // Time is up
              SignalElapsedEvent();
            }
            catch {}
        }

On Thu, 24 Dec 2009, Dan Garner wrote:

> Regarding the crash... if I could summarise the problem.
>
> On your laptop running with the latest binary I gave you the client is
> fine. But on your production equipment the same client crashes out to
> windows silently after the first video is finished playing.
>
Yep.

> The only thing that has changed (as I remember) in that branch is that
> if the duration is 0 an event is raised which will kick off the end of a
> video. Are the durations of your videos all set to 0, or have you timed
> the videos and set a specific duration for each?

All durations set to 0, other still unusable as using 1.0.4 without fixes
in this region.

> Can you try editing Video.cs at line 46 and replace the method with:
>
> void videoPlayer_VideoPlayerElapsedEvent()
> {
> try
> {
> Hide();
> videoPlayer.Hide();
>
> // Time is up
> SignalElapsedEvent();
> }
> catch {}
> }

Only catch {} difference, right?
No luck, nothing changed ;)

PS. Some break is nessesary sometime, i've just realised from Options
Started, Application Finished came from :) :) :)

PS2. If You want check anything in my system (maybe some ideas easier
to check by yourself instead of pack of questions) there is no problem
with sharing one client over vnc.

Ok, one thing changed after catch add-on. White flashes on notebook again ;)

I assume you put the "try" on there too? It probably didn't compile otherwise...

Maybe the only way we will get to the bottom of this is to put Visual Studio on one of your boxes and single step through it. I will be doing some testing over xmas with your media / database on a Revo... we can wait until then if you prefer?

On Thu, 24 Dec 2009, Dan Garner wrote:

>
> Dan Garner posted a new comment:
> I assume you put the "try" on there too? It probably didn't compile
> otherwise...

Yes, 'try' was also there ;)

There is no crash if duration is set to 10 secs (before end), next clip
played correctly

When duration was set to 200 (much more than mpegs duration) crashed just
after end of playback as usual.

It must be caused by some xibo modification - as base windows partition
was unmodified since 2-3 months. Any changes made by operating os are
lost during restart, with exception of xibo folders placed on separate hdd
and junctioned to c:\ filesystem.

> Maybe the only way we will get to the bottom of this is to put Visual
> Studio on one of your boxes and single step through it. I will be doing
> some testing over xmas with your media / database on a Revo... we can
> wait until then if you prefer?

Hmm, i'm not sure if it would install on my stripped windows installation.
Probably some pieces of system going to me missing for such big install
program. I can guess that i've unchecked some debugging subsystem in nLite
when creating instalation cd :)

I can check it after xmas with visual studio express 2005 installation -
moving system to hdd from CF card probably will be needed, as there is ony
512megs cache space in driver used to prevent writes on card...

> --
> You received this question notification because you are a direct
> subscriber of the question.
>

So it looks like the event raised at the end of playback is somehow the culprit.

Is Windows Media Player the same version between Notebook and production?

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: Re: [Question #93892]: Xibo loading time with file corruptiondetection
Date: 24th December 2009
Time: 8:23:02 pm

Question #93892 on Xibo changed:
https://answers.launchpad.net/xibo/+question/93892

Bartek gave more information on the question:
On Thu, 24 Dec 2009, Dan Garner wrote:

>
> Dan Garner posted a new comment:
> I assume you put the "try" on there too? It probably didn't compile
> otherwise...

Yes, 'try' was also there ;)

There is no crash if duration is set to 10 secs (before end), next clip
played correctly

When duration was set to 200 (much more than mpegs duration) crashed just
after end of playback as usual.

It must be caused by some xibo modification - as base windows partition
was unmodified since 2-3 months. Any changes made by operating os are
lost during restart, with exception of xibo folders placed on separate hdd
and junctioned to c:\ filesystem.

> Maybe the only way we will get to the bottom of this is to put Visual
> Studio on one of your boxes and single step through it. I will be doing
> some testing over xmas with your media / database on a Revo... we can
> wait until then if you prefer?

Hmm, i'm not sure if it would install on my stripped windows installation.
Probably some pieces of system going to me missing for such big install
program. I can guess that i've unchecked some debugging subsystem in nLite
when creating instalation cd :)

I can check it after xmas with visual studio express 2005 installation -
moving system to hdd from CF card probably will be needed, as there is ony
512megs cache space in driver used to prevent writes on card...

> --
> You received this question notification because you are a direct
> subscriber of the question.
>

You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

On Thu, 24 Dec 2009, Alex Harrington wrote:

>
> Is Windows Media Player the same version between Notebook and
> production?
>

No,

Tests: Win7 & WMP12 (installed out of a box, probably downgrade not
possible)

Production: XP & WMP11 installed

Hello,

Today i've prepared pc like our production pcs. One difference - ordinary hdd instead of cf card.
VC# 2005 Express + SP1 ENU installed without problems.

Checked on my -new.php xmds modified binardy, still crashing where there is 0 duration inserted,
no change during xmas ;)
There is also copied full server stucture in /test subdir so normal binaries also can be tested.

If you like to do some debugging job let me know. i'll send vnc hostname and password to mail.

Anyway, i've checked what 'Revo' is, and it's look very promising :) With cuda accelerated coreavc probably should work like a charm :)

Hi Bartek,

I've looked at this today and discovered the problem... unfortunately it means rolling back the changes made to improve the White flashing between videos. However, not all is lost... we have put in another workaround which should improve the white flashing - basically we can hold the first frame of the video until it gets de-constructed (a maximum of one second).

I've put the binaries here: http://dl.dropbox.com/u/353651/Xibo/Xibo%20Client%201-0-5%20v2.zip for you to test if you can.

I have a Revo running here now using the files from your database and it all runs smoothly. I must say your videos are nice!

Let me know if you have any problems.

Cheers,
Dan

I've just started new binary on test machine - looks that there is no more crashes, great :) Definitly crash was much bigger problem that white screens ;)

If there is related source branch I can start testing it on bigger extent of clients starting from tomorrow :)

Other thing: When i've checked displays after xmas 50% of them were crashed on older versions. What's strange - there are days (weeks even) when all of them keeps working ok, But when start crashing, there is ~50% of clients crashed. And those clients are not related one to another - different connections (some on dsl, some on 3g), different towns even. Maybe some problems with server connection, but server is installed our main fiber, very stable connection (5mins downtime during last 2 years). So i've no idea about reason :). Probably implementation of some simple cron script to respawn Xibo if there no Xibo in memory (no problem for me as long there is max one crash / 2 weeks) would finally fix problem , but still strange :)

Nice videos? Personally i really hate them :) But maybe it's a effect of watching them for whole day when i'm preparing new clients :)

I've just found a revo 'replicas :)' supplier in China. Lower prices, xp instead of Vista, customizable hdd and ram, no wifi option. Looks promising. I can see a new generation of clients on horizon :) I have strange feeling that those replicas source is same production line than revos, one difference is no acer logo on box :)

Ha ha, I meant that they looked good on the displays... I too got a little tired of watching them after a few hours!

I have just pushed the source that generated those binaries into here: https://code.launchpad.net/~dangarner/xibo/105-client-test. We will be doing more testing on this branch over the coming days and it will then be merged into "lp:xibo/1.0" ready for release.

It is difficult to say what would cause the crashing of the clients - I am sure that 1.0.5 is more stable, it has extra exception handling and better file downloading, as you know. 1.0.5 is essentially the binary I gave you, provided we find no more problems.

If you do find more problems over the next few days, so let us know and we will try to debug them before we code freeze for 1.0.5.

Cheers!

Ok, so i'm going to compile that branch and start test immedialety! I hope there is no more bugs :)

One more question: Are You considering saving cachemanager.xml more often that properly application close?
I don't want to bother You again, but in my situation (and probably some other users) there is no application close stage at all in production usage scheme. Theoretically client run 24/7 or there is no power at all so there is no chance to save anything :)
So I'm wondering if is it important part of testing, or maybe I should focus on other aspects (crashes, video playback, etc)

It is probably a little naive of us to assume that the application will always close correctly (although that was the assumption).

I have changed it so that the CacheManager will get written out after every successful file collection. These changes have been pushed to the same branch as before.

Cheers

19 clients running right now correctly. Po problems at the moment.
Probably there will be 25 of them at tomorrow.

So i'm closing that question for now. In case of some problems I'll definitly report them :)

Thanks again for your help :)