Base64 encoding of video content

Asked by Shankar Jayagopi

I am writing a simple android client for Xibo.
When i receive a Image ( whose size is less than the chunk size 512000), i am able to download the file, decode it and use it.

But, when it comes to a video file that is 3.xMB, I get multiple chunks, with each of them base64 encoded.
When i try to retrieve each of the base64 strings, append them and try to do a full decode, i get Out of memory error.

When i try to decode every chunk and append to a single file, the player says its corrupted.
What is the way to work with such large files ?

thanks
Shankar

Question information

Language:
English Edit question
Status:
Answered
For:
Xibo Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Dan Garner (dangarner) said :
#1

The file chunk returned is just a file stream, so you can append it to a file without manually decoding it. After you have appended each chunk and finished downloading the file, MD5 it and compare that with the MD5 you received from RequiredFiles.

We would be really really interested to see your work in this area - it would be a fantastic addition to the project. Do you plan on open sourcing the work?

Revision history for this message
Shankar Jayagopi (shankar-l) said :
#2

The response i get is base64Binary encoded :(

<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="urn:xmds" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><ns1:GetFileResponse><file xsi:type="xsd:base64Binary">AAAAHGZ0eXBtcDQyAAAAAG1wNDJpc29tY.............................

I get base64 encoded strings for both image and video types. Not sure if there is a configuration somewhere to turn it into raw filechunks.
The workaround I followed is to increase the chunk size to 10MB. Then i am able to decode and save it in one go. That seems to work without any OOM.

We are an early stage startup, and have not thought about open-sourcing this yet.
We will post a lite version of this on android market in a few weeks for people to try out and give feedback.

Revision history for this message
Dan Garner (dangarner) said :
#3

Apologies, I remember now... the response is returned as xsd:base64Binary automatically by PHP's SOAP implementation. I've had a look in both our clients, and we dont explicitly handle it anywhere...

Have you tried pumping the output directly into a file??

Revision history for this message
Shankar Jayagopi (shankar-l) said :
#4

hmm. Not sure why the wsdl file says response is goona be base64 encoded, if its the responsibility of PHP-SOAP - client negotiation.

<message name="GetFileRequest">
<part name="serverKey" type="xsd:string"/>
<part name="hardwareKey" type="xsd:string"/>
<part name="filePath" type="xsd:string"/>
<part name="fileType" type="xsd:string"/>
<part name="chunkOffset" type="xsd:int"/>
<part name="chuckSize" type="xsd:int"/>
<part name="version" type="xsd:string"/>
</message>
<message name="GetFileResponse">
<part name="file" type="xsd:base64Binary"/>
</message>

I tried appending the output of all the chunks without decoding, into a file, but it was not usable.

Revision history for this message
Shankar Jayagopi (shankar-l) said :
#5

I can reproduce this by using a HTTP Client [http://ditchnet.org/httpclient/]

Body : <soapenv:Envelope xmlns:ns0="urn:xmds" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Header></soapenv:Header><soapenv:Body><ns0:GetFile xmlns:ns0="urn:xmds"><string>jabbers</string><string>myhw1</string><string>16.mp4</string><string>media</string><string>0</string><string>51200</string><string>3</string></ns0:GetFile></soapenv:Body></soapenv:Envelope>

Xibo Server URL : http://ec2-176-34-169-57.eu-west-1.compute.amazonaws.com/xibo/xmds.php

Response seen : <?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="urn:xmds" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><ns1:GetFileResponse><file xsi:type="xsd:base64Binary">AAAAHGZ0eXBtcDQyAAAAAG1.....

So, it doesn't seem to be the client that is opting for encode via headers.

Revision history for this message
Alex Harrington (alexharrington) said :
#6

The WSDL specifies that the response will be base64 encoded. It's not negotiated. It's base64 encoded as it's a standard way to transmit binary data in a circumstance where binary content could break escaping - such as email, embedded in XML etc.

It's the responsibility of the SOAP client library to parse the WSDL, understand that the response will be base64 encoded, and decode it before returning the output to you.

SOAPpy on Python copes with this, as does the .net SOAP client, as does AXIS on Java. I would suggest you look at the documentation for whatever SOAP library you're using and/or speak to their developer to see if there's something you need to do to have the library return the correct output,

Alex

Revision history for this message
Shankar Jayagopi (shankar-l) said :
#7

Alex,
I had previously tried to write it in sun-java using Apache-axis, but this was not portable to android-java environment.
Other libraries i found for writing soap clients on android like ksoap2, looked overkill for 3 or so apis that we usually call.
So, i ended up writing a DefaultHTTPClient based implementation, with POST requests for SOAP, In which case it now relies on me to handle the base64 decode logic.

Revision history for this message
Alex Harrington (alexharrington) said :
#8

If decoding works on a single chunk but when you concatenate multiple decoded chunks the file is invalid, my best guess would be that you're either not writing the file in a binary safe fashion, or you're adding whitespace (new lines, spaces etc) after each chunk.

Alex

Revision history for this message
Dan Garner (dangarner) said :
#9

So your going to handle the entire soap implementation yourself? Seems risky and a little pointless... one of the main reasons we chose soap is that there were clients that handled all the complex stuff.

Just like PHPs soap implementation understanding the wsdl and returning the correct encoding for getfile

Can you help with this problem?

Provide an answer of your own, or ask Shankar Jayagopi for more information if necessary.

To post a message you must log in.