TupambAdminOrg [2024.03]

How to wget large files from one server to another

https://tupambae.org/display/0ac89072-1667-e961-8377-1b5707689027

How to wget large files from one server to another

#fediAdmins @Friendica Support @Friendica Developers Hi there, a technical question not directly related to friendica itself. Trying to transfer an instanc...

Mar 30 at 3:21pmWeb
Show threadutopiArteMar 30

update:
found the following hints pointing out that php-fpm .ini could be involved:
askubuntu.com/questions/332022…

Changes in that .ini and restarting apache2 and the respective -fpm didn't work out either.

Of course we are talking about management with the console over SSH.

Changes for upload_max_filesize, memory_limit & post_max_size in the php.ini aren't affect

I'm trying to increase the upload_max_filesize etc in order to import bigger files with phpMyAdmin. I have changed the php.ini and restart Apache. I can see the changes on the phpMyAdmin import scr...

Ask Ubuntu
Show threadutopiArteMar 30

To make sure that the settings that rule are the ones I changed, I searched for a comand line command to print the respective values within the console:
php -r "echo ini_get('my_value');"
(the print is a littel bit strange tho, just a number in line before the user handle, like:
8000MrootName@VPShosting:~# )

php -r "echo ini_get('upload_max_filesize');";"
8000M
php -r "echo ini_get('post_max_size');"e');"
8000M
php -r "echo ini_get('max_execution_time');"e');"
0
php -r "echo ini_get('memory_limit');"t');"
-1
php -r "echo ini_get('max_input_time');";"
-1

Show threadutopiArteMar 30
Just managed to pull two files of 1.9gig and 1.6gig with the existing setting and the command as displayed, so the limitiation is above 1.9gig and most likely a wget problem itself (?).
Show threadDaniel AresMar 30
@admin@tupambae.org If you have SSH access to both systems you can just use SFTP. Wget sounds like the wrong tool for this.
Show threadutopiArteMar 30

@daniel

If you have SSH access to both systems you can just use SFTP

Not really.
I do have ftp access to the shared hoster using fileZilla not SSH, so I can use those credentials and SSH with putty over console on the VPS. I was just starting to look into such an option but not sure for the correct command to do so. I'd probably also need to install some ftp package I guess. It's a ubuntu 2022.04 standard setup VPS.

Show threadRainer "friendica" SokollMar 30
@daniel @utopiarte rsync is the answer.
Show threadutopiArteMar 30

@rainer

Read that a lot, but I guess that my access to the shared hosting only with a tool like filezilla isn't enough to use resync.

Also, I'm in the middle of nowhere with internet access thru a cellphone located at a specific 20x10cm spot in the roof where I catch a littel bit of the provider antena that's at a distance of like 12km. So downloading 3.7gig to than upload it with scp is kinda impossible.

@daniel

Show threadRainer "friendica" SokollMar 30
@daniel @utopiarte You said you have ssh access. That should be enough for god's own rsync 😀
And: rsync can resume (-P option), so you would not have to start over after an interrupted transfer.
Show threadutopiArteMar 30

Not sure if I understand what you are saying @rainer.

As for what I saw, on a brief look in conversations on the matter on the web, was that rsync has to be installed on both sides, on both servers. I ony can install it on the VPS, not on the shared hosting where the files is.

@daniel

Show threadRainer "friendica" SokollMar 30
@daniel @utopiarte There is a good chance that rsync is already installed on the server side. You may try it.
Show threadutopiArteMar 30

@rainer @daniel The web say's this is not possible:

It is not possible to use rsync on http URLs.
To use the rsync algorithm the client-side rsync needs to interact with a server-side rsync process. This is done either directly, through ssh or (less common) rsh. HTTP is not an option.
stackoverflow.com/questions/33…

Unless I can use sFTP credentials to contact the shared hosting with rsync this doesn't look like an option.

How can I use rsync to retrieve files from an HTTP URL?

There is a bunch of files here: http://data.pudo.org/flexicadastre/ I want to have a local folder that contains the same files there are in that URL. I've tried: rsync -av http://data.pudo.org/

Stack Overflow
Show threadRainer "friendica" SokollMar 31

@daniel @utopiarte You wrote:

I do have ftp access to the shared hoster using fileZilla not SSH, so I can use those credentials and SSH with putty over console on the VPS.


So I assumed you have ssh access. If you have and rsync is available an both sides, then you really should give rsync a try.

Show threadutopiArteMar 31

@rainer @daniel

I assumed you have ssh access

Nope.
(actually I detailed that in another answer, I guess you didn't get that displayed)

Right now I most likely will try to get that instance running and up to date on the shared hosting, than try to split up the file storage and than move it in parts.
Another option might be looking for a day with a good cellphone reception and do all of it with scp from over here.
Also, investigating how to do this with sFTP over SSH from the VPS is most likely an appropriate option that add's experience and tools to the general setup, as the wget investigation is kinda "ausgereizt" already.

Last but not least there could be the option to ask anyone else who does have (good fiber optics) connections to get the file onto his VPS and than create a resync setup.
👍

Show threadRainer "friendica" SokollMar 31

@daniel @utopiarte

I assumed you have ssh access

Nope.


[…]

investigating how to do this with sFTP over SSH


I am confused. If you do not have ssh access, sftp will fail.

Show threadutopiArteMar 31

@rainer

I am confused. If you do not have ssh access, sftp will fail.

two server

  • server 1
    shared hosting
    hosts the file
    access over sftp
    no access over SSH
    access over https, locating the file in a public folder, access with wget, works at least up to 1.9gig
  • server2
    VPS hosting
    access over SSH
    never intended or tested to access over sFTP till now
    needs/requests the 3.7gig file


Probable sFTP option:
rootName@VPShosting:# someSFTPtool sftpUSERNAME@sftpsSharedhostingID/sharedHostingstoragefolder/file.tar.gz /thisVPSserver/localstorage/file.tar.gz