file size decreased during transfer

Asked by ralph

get and mget return smaller file sizes - including 0 byte size
using default mode of binary
Files are .pgp (encrypted zip files)

Environment: Ubuntu 16.04
lftp: 4.6.3a

See test example below:
1) Do an ls to show file sizes on FTP server
2) get an encrypted zip file (note message: "file size decreased during transfer"
3) get a simple text file (contents are: helloWorld): no messages - appears to be a successful transfer
4) ls of Unix directory (note file size differences)

The same steps are done using filezilla without error or problems. Need lftp for command line scripting so filezilla is not an option.

lftp:/> ls
-rw-r--r-- 964372 2018-05-19 10:31:45 F1.pgp
-rw-r--r-- 11 2018-05-19 11:09:58 t1.txt

lftp:/> get F1.pgp
get: F1.pgp: file size decreased during transfer

lftp:/> get t1.txt
lftp:/> quit

ubuntu@Dev:~/test4$ ls -l
total 928
-rw-rw-r-- 1 ubuntu ubuntu 950272 May 19 11:55 F1.pgp
-rw-rw-r-- 1 ubuntu ubuntu 0 May 19 11:09 t1.txt

Question information

Language:
English Edit question
Status:
Answered
For:
Ubuntu lftp Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said :
#1

What file system and block size is the client and server side using?

Revision history for this message
ralph (callaway) said :
#2

Client side
- File system: ext4
- Block size: 512

Server side
- Unknown will ask on Monday
- Guessing it is a Microsoft Server running Filezilla for FTP

Not sure why relevant. lftp can't even transfer an 11 byte file correctly - returning 0 bytes with no error message.

Connection is via sftp, standard port 22

Fyi... filezilla and sftp both work correctly with no special settings

I maintain 22 FTP connections for corporate information pulls of information. This is the 23rd that I am setting up with lftp and I've never seen this type behavior before. Internet searches for "file size decreased during transfer" imply a bug that was fixed back in version 4.3 or 4.4

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#3

Because it depends how you are checking size.
If the block size is large then the slack space will be greater. Making the file seem larger.

If you MD5 test the file on the server side, is it the same?

Is there no scope to switch to SFTP? FTP is awful

Revision history for this message
ralph (callaway) said :
#4

Backing up a little.

Lets forget about encrypted or zipped files and focus on basic plain text transfer.

1) MD5 or SHA256 information will obviously be different if the file sizes are different in byte count size

2) I can successfully get the correct file size (and can decrypt and unzip real data files using Filezilla GUI interface and sftp interactively)
--- I would still like to use lftp, but can use sftp if I have to.

3) There is ZERO corruption of data. ONLY truncation. For example data in t1MB.txt file is exactly correct through record 9733. Record 9734 is the last record and is a partial record. There should be 10000 records

4) >>>>>>>>>>>>>>>>>>>>>>> using sftp <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
sshpass -e sftp <email address hidden>
mget t*.txt
>>>>>>>>>>>>>>>>>>>>>>> WORKS PERFECTLY

>>>>>>>>>>>>>>>>>>>>>> using filezilla GUI client <<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>> WORKS PERFECTLY

5) If this was a MD5 or SHA256, we would see corruption and varying data values.
     If this was a charset issue, we would also see corruption or byte count variances
     If this was a block-size issue, I would expect to see some correlation/recognizable pattern of truncated data. I built 10MB, 20MB, 30MB, 40MB, 50MB files based on exactly 1024 character size records and got varying numbers of truncated record of ~74 to 267 records
   Truncation occurs ONLY at the end of the file

6) See following example. Example tested for 5 different file sizes. Small files return 0 bytes, yet report no errors and that files transferred. Larger files truncate data at the end of the file

ubuntu@Dev:~/scratch/test5$ ls -l
total 60184
-rw-rw-r-- 1 ubuntu ubuntu 10100000 May 20 09:21 t10MB.txt
-rw-r--r-- 1 ubuntu ubuntu 1010000 May 20 09:50 t1MB.txt
-rw-rw-r-- 1 ubuntu ubuntu 11 May 20 08:56 t1.txt
-rw-rw-r-- 1 ubuntu ubuntu 22 May 20 08:56 t2.txt
-rw-rw-r-- 1 ubuntu ubuntu 4040 May 20 08:55 t4K.txt
ubuntu@Dev:~/scratch/test5$ wc t*txt
  100000 100000 10100000 ../test5/t10MB.txt
   10000 10000 1010000 ../test5/t1MB.txt
       1 1 11 ../test5/t1.txt
       2 2 22 ../test5/t2.txt
      40 40 4040 ../test5/t4K.txt

ubuntu@Dev:~/scratch/test5$ lftp -u username,password sftp://ftp1.XXXX.com
lftp:/> mput t*txt
lftp:/> ls -l
-rw-r--r-- 11 2018-05-20 08:56:29 t1.txt
-rw-r--r-- 10100000 2018-05-20 09:21:06 t10MB.txt
-rw-r--r-- 1010000 2018-05-20 09:50:20 t1MB.txt
-rw-r--r-- 22 2018-05-20 08:56:07 t2.txt
-rw-r--r-- 4040 2018-05-20 08:55:22 t4K.txt
lftp:/> quit

ubuntu@Dev:~/scratch/test5$ cd ../test6
ubuntu@Dev:~/scratch/test6$ lftp -u username,password sftp://ftp1.XXXX.com
lftp:/> mget t*txt
mget: t10MB.txt: file size decreased during transfer
mget: t1MB.txt: file size decreased during transfer
11075584 bytes transferred in 1 second (7.08 MiB/s)
Transfer of 2 of 5 files failed
lftp:/> quit

ubuntu@Dev:~/scratch/test6$ ls -l
total 60096
-rw-rw-r-- 1 ubuntu ubuntu 10092544 May 20 09:53 t10MB.txt
-rw-rw-r-- 1 ubuntu ubuntu 983040 May 20 09:53 t1MB.txt
-rw-rw-r-- 1 ubuntu ubuntu 0 May 20 08:56 t1.txt
-rw-rw-r-- 1 ubuntu ubuntu 0 May 20 08:56 t2.txt
-rw-rw-r-- 1 ubuntu ubuntu 0 May 20 08:55 t4K.txt
ubuntu@Dev:~/scratch/test6$ wc t*txt
   99926 99927 10092544 t10MB.txt
    9733 9734 983040 t1MB.txt
       0 0 0 t1.txt
       0 0 0 t2.txt
       0 0 0 t4K.txt

Revision history for this message
Manfred Hampl (m-hampl) said :
#5

There is a bug report about such problem, Bug #1713270
but this should already have been corrected in version 4.4.13

Revision history for this message
Marios Nikolaou (marios.nikolaou) said :
#6

There was a fix to upstream lftp when using sftp that can result in truncated files: https://github.com/lavv17/lftp/pull/288

Can you help with this problem?

Provide an answer of your own, or ask ralph for more information if necessary.

To post a message you must log in.