FTP upload speed compared to other clients...

This site is best viewed in a modern browser with JavaScript enabled.

FTP upload speed compared to other clients...

Lubox

I have a really simple script that uploads a file using ftp_put(). This script is about 10 times (!) slower than uploading the same file from the same server to the same server using Filezilla. The PHP script is as far as I can tell doing exactly the same as Filezilla, passive mode, utf-8 opt, etc. We have captured the traffic using Wireshart and the company responsible for the FTP server is currently investigating those, but I thought I'd just check in on this forum. Edit; the question is if anybody has seen this and has some advice

Thanks
Lubox

eval_BadCode_

You can use the "cp" command from ssh+[ba]sh. That would make the most sense and would be exceptionally faster than using FTP to copy files.

Lubox

Using Filezilla a FTP upload is as fast as the internet lines at the other end is (we have exceptional high bandwidth ourselves, so we're almost always maxing the other end). Of course, with PHP as FTP client we're not doing this since it is 10-20 times slower on all versions I've tried. If you have any examples of "cp" command from ssh+[ba]sh on Windows (!), I'll gladly check it out. I've tried the PHP copy(), but that is as slow as ftp_put() (probably using same methods).

Edit: Note that this only applies to uploads (ftp_put()). FTP downloads (ftp_get()) with PHP is as fast as Filezilla.

Thanks

Derokorian

The reason its faster with filezilla is because it can create multiple streams. You may try forking the process, giving each fork a batch of files to process, this will allow the job to be running with multiple down streams and greatly decrease the total transfer time.

eval_BadCode_

Lubox;1241011 wrote:
Using Filezilla a FTP upload is as fast as the internet lines at the other end is (we have exceptional high bandwidth ourselves, so we're almost always maxing the other end). Of course, with PHP as FTP client we're not doing this since it is 10-20 times slower on all versions I've tried. If you have any examples of "cp" command from ssh+[ba]sh on Windows (!), I'll gladly check it out. I've tried the PHP copy(), but that is as slow as ftp_put() (probably using same methods).

Edit: Note that this only applies to uploads (ftp_put()). FTP downloads (ftp_get()) with PHP is as fast as Filezilla.

Thanks

wat...

ok you get putty, log in with your "SFTP" credential (SFTP meaning... ftp over ssh). So ssh will spawn a shell (probably bash or sh), and you can use "cp" the copy command,
cp <src> <dst>
it will be hundreds of times faster than trying to copy crap from the server to your computer and then back to the server again, lol.

Lubox

Derokorian;1241057 wrote:
The reason its faster with filezilla is because it can create multiple streams. You may try forking the process, giving each fork a batch of files to process, this will allow the job to be running with multiple down streams and greatly decrease the total transfer time.

Multiple streams within the ftp-protocol? Is that even possible? Filezilla are able to transfer multiple files, but the protocol doesn't allow multiple threads for one and the same file (if I understood you correctly) as far as I know.
I'm already transferring many files at the same time in the system (40-100 simultaneous uploads depending on receiving server), so all in all the speed is not bad. But it could be much, much faster.

I've also tried to upload using cUrl, but that is equally slow. I will also try with socket operations and see if I manage uploads on that level.

Note that ftp downloads (ftp_get) is as fast as Filezilla, so there's a huge difference in ftp_get and ftp_put.

Could it still be the servers that are using som kind of limit if the connection is not from a known client or something? I'm trying to find servers that I do not see this on, but so far they've all been the same.

To "eval", any script examples would be nice. By the way, why do you think I copy crap from the server to my computer and back to the server?

Thanks
Lubox

Derokorian

Lubox;1241379 wrote:
Multiple streams within the ftp-protocol? Is that even possible? Filezilla are able to transfer multiple files, but the protocol doesn't allow multiple threads for one and the same file (if I understood you correctly) as far as I know.
I'm already transferring many files at the same time in the system (40-100 simultaneous uploads depending on receiving server), so all in all the speed is not bad. But it could be much, much faster.

Ah my misunderstanding. I thought you were uploading a file and moving on to the next when it was complete. So my suggestion was based on that assumption. If you are already doing multiple files at a time then I'm not sure I can give you any pointers to increase the speed. I just know my site back up started finishing much quicker when I spawned multiple processes to handle sets of files (aka pushing multiple files at once) and I thought maybe you were running into the same problem.

Sorry I'm less than useful sometimes.

Lubox

Derokorian;1241385 wrote:
Ah my misunderstanding. I thought you were uploading a file and moving on to the next when it was complete. So my suggestion was based on that assumption. If you are already doing multiple files at a time then I'm not sure I can give you any pointers to increase the speed. I just know my site back up started finishing much quicker when I spawned multiple processes to handle sets of files (aka pushing multiple files at once) and I thought maybe you were running into the same problem.

Sorry I'm less than useful sometimes.

Hehe, no worries, I know I didn't make this clear in the first post, I just thought I'd stick to the real issue, the single transfer. So, I've tried ftp_put(), I've tried copy(), and I've tried the cUrl upload. All 10-20 times slower than the same upload from Filezilla. And it's not like there's a limit reached within PHP. I've tried it on fast servers and it may upload at 2500kB/s, I've tried it on slow servers and it may upload at 500kB/s. And Filezilla uploads 10 times faster on both servers. It just seems like there's a throttle somewhere saying "if PHP {10% of max speed}"

Thanks
Lubox