File download interruption

This is _exactly_ what I'm talking about when I'm saying about writing these changes in to stable.

The Python client has full file management that would deal correctly with all these edge cases, but only because it's fundamentally different in it's design.

Perhaps it's time to roll the corruption changes out of stable and just bugfix the existing code.

Alex

--- original message ---
From: "Bartek" <email address hidden>
Subject: [Question #94072]: File download interruption
Date: 14th December 2009
Time: 8:13:12 pm

New question #94072 on Xibo:
https://answers.launchpad.net/xibo/+question/94072

I've made a separate thread for this question to keep those simillar subject separated.

I'm thinking about file download interruption during layout sync. (It's not important in with version: filecorruption detection with md5 and deletion whole file, version with appending to file on hd, etc).

At the beginning client probably scans it's resources and generate some list aka. Those Files Are Missing. That could be a list of all files in layout if there is a new layout queued, files from some place of layout if download was interrupted by shutdown, some random files in effect of deletion of fs corruption when md5 is wrong, etc. But there is probably some sequence of files that client need to download.

As far I can guess right now, clients call some download(some_file); for all of those files, one by one, in sequence. And only once for every one of them.
If someting goes wrong (network disconnection, full disk, some file lock (file clicked on windows, opened accidently in wmp, other) downloaded is stopped and client pass to next file in queue, and never goes back to broken file.

At last at the end of layout download client start layout playback, regardless of download state of downloaded files.
(Or maybe I'm wrong and there is another iteration on files queue?)

Alex, Dan, what do You think about that? Fixing it is a bug of additional functionality targeted to unstable version?

Again, maybe it's a good idea to implement some workaround in stable, and plan some good file queue support in separated thread in unstable?

Simple workaround could look something like this:

Now (probably):
foreach (file in missing_files)
{
download(file); //download returns when there is some problem
}
play_some_layout();

Workaround:
foreach (file in missing_files)
{
while(download(file) != SUCCESS) sleep_some_time_or_move_file_to_queue_end);
}
play_some_layout();

Question is - better to allow situation when playing unfinished files and take risk of hanging up client, or wait as long as it need to download file in 100%

--
You received this question notification because you are a member of Xibo
Developers, which is an answer contact for Xibo.

This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

I'm starting to wonder if you're going to kill me in effect of filecorruption subject :)

I'm undestand your motivations, but I also have mine, and thats why we're discussing about it :) But of course you are co-owner of the project and your opinios are final ;)

That's great that python client would have everything of those, i can't wait to test it, maybe i'll change windows to linux some day, but it's probably take some more months until there will be usable 1.2 version :)
That's why i'm using 1.0.x branch right now and i've found some bugs there since now:

- general movies playing errors.
- rss speed and interaction with others regions
- file download problems
- maybe other, not remeber right now

Where last 'file download problems' its quite complex and still open subject, rest of them was quite easy to fix and already fixed, and that's really great :)

Last thing is making those file-things usable, but there is a lot of small problem inside described globally as 'my files are broken!'

In my opinion this needs fixing to call client stable at all :) If you want build a big network you simply can't control 100 terminal by hand and check that everything was downloaded ok. I probably needs some kind of fixing, as simple as it can be but it should work :)

So maybe becouse problem is more complex your idea is not bad at all. For example you can release 1.0.5 client without all filetransfer corruption stuff, and make a separate branch with code writen by Dan already it this subject, and all of those simple workarounds described later. Its almost ready right now and needs two simple workarounds to be complete.

I'm ready to test such branch deeply - as I said, better for me to have 50 unchecked terminals with probably solved problems, instead of 50 superstable terminals with problem for sure. Becouse in my network those problems are very important and in effect stable client is unstable ;)

If everything will be ok, it's can be merged in 1.0.6 for example :)

Last thing - some people probably would prefer to stand on 1.0.x version for longer time that to python client release date. .NET client in my current enviroment is much easier to manage right now, i need to change a lot of things to start using pyton some day. Some of my terminals are 500km away from and it's quite unsafe to make big modifications remotely. Putting new exe version is quite easy :) Especially that right now i don't need any new features planned and that won't change during next months.

Regards

> And there lies the problem. I run maybe 20 clients and have NEVER had
> any problem because they're either on LAN connections or stable internet
> feeds. My concern is introducing bugs in to the system in the process of
> fixing an issue which has been reported to us by one or two users.
>
Ok, but if that means that other bugs/limitations that didn't happened on
your installation shoudn't be fixed?

I don't think thats going to by only my problem if you like to people
around the world to use xibo. I've also slow but stable connections, that
problem persist there also if somebody will shut down power over there.

> I think you're greatly oversimplifying the problem here. The issue isn't
> the complexity of any particular fix, but the implication that fix has
> on the rest of the system. EG in this case Dan has fixed issues where
> files aren't downloaded correctly, but the client assumes that they are
> always downloaded correctly so when the file collection routine
> completes it goes to play the layout assuming that all the content is
> present and correct. So now to satisfy YOUR requirements, we have to
> introduce yet another raft of changes to support checking each
> individual layout to see if it's complete before the client starts to
> play it.
>

Of course i've simplified it a bit. I know that modifications in one place
could break 5 other, but probably good programmer and good code could
reduce problems, but programer is not a wizard :)

That's not me reported problem at first, and not me saved is as a bug in
launchpad.
I've stared to writing far later that that.
Probably You or Dan agreed that this problem needs fixing in stable.
Now You or Dan can agree on not that's need a some more modifications, or
not. I'm only writing about some other details that i've found later
during devel version Xibo usage.
As I said twice, i'm already happy with current code state, its much better
than nothing. Also I can only guess that main problem in this case is
already sloved, ands need only small modification, and probably in code
added to slove main bug, etc. Again, You don't need agree with me :)

> Don't get my wrong those are all useful features and in the fullness of time should be integrated,
> but it's not how the .net client was designed from the outset and so
> isn't a bug, more a limitation in my view.
>

You call it limitation, I 100% agree with you. Main problem is
probably single thread conception of client. That's why im calling
conception to fix it workarounds, becouse it won't main problem. But i'm
not even know that they can be realized becouse i don't know your code...
Its only sugestions. I'm not saying that you have to fix it :)

> It's not my code, but it normally falls to me to support this lot so I'll defer to Dan's
>better judgement - however I really hope we don't end up releasing a
>client in 1.0.5 that works for you and nobody else.

Just read Dan's answer and i think it's close subject. If something can't
be done, i won't keep saying that it should be fixed :)

Thanks Alex Harrington, that solved my question.