erikdorthe
ahistorical immaterialistTIL that the scp command does not, by default, compress files it transfers, and that you need to add the -C flag to make it do that, and by doing that I have sped up my transfer speed about 40x, and without this knowledge I have probably wasted literally days of my life waiting for things in the past.
Some may say "why are you using scp rather than rsync" and to that I say, rsync has even more ridiculously complicated flags to learn.
Steve Bennett@jimbob eh, rsync is one of those things where you pretty much always use the same flags. -avzr from memory. Just copy it from the last script you used it in. :)
Beto Dealmeida@SteveBennett @jimbob I always get the trailing slash wrong, and then rsync copies the directory inside the directory...
Andy Norman@beto @SteveBennett @jimbob this is me. I write a new script using sync about once every couple of years. I always end up using trial and error to rediscover whether I need a slash or not. And I still have directories out there with extra levels where I guessed wrong and didn’t realise 😊 #VeryOccasionalLinuxAdmin
Greg@beto @SteveBennett @jimbob I make the source and destination paths look the same near the end of each of them and always with a trailing slash and it seems to always work out.
Gerbrand van Dieyen@SteveBennett @jimbob I thought the r was implied by the a ?
Michael BaconNo shame because for me I would just get compression with
`tar -czf - somedir | ssh somehost tar -xzf -`
Because yanno . . . .
J$
Jerven@MichaelTBacon @jimbob well if you want to use a different compression algorithm that might still be what is needed!
Snosrapkungfu (They/them)@MichaelTBacon @jimbob no v, on dest side?! Blind faith! I am impressed, i lack the patience to believe it's doing anything :)
Nah, that's what the other terminal window is for . . . ;)
(but TBH the destination side really should look something like `ssh somehost bash -c 'cd /actual/target/directory; tar -xzf -'`. I mean you don't want it just dumping in the home dir, right?)
CBO @ le17@MichaelTBacon @jimbob since I discovered it I prefer : tar -czf - somedir | pv | ssh somehost tar -xzf -
Resuna@MichaelTBacon @jimbob Also avoids walking the tree twice and adding roundtrips. But surely "ssh somehost 'cd somedir; tar xfz -'" there's probably a tar option for that that might be the same with GNU and BSD tar but I've been doing it this way for 30-40 years now except when I had to use "cpio -iforgettheoptions".
Matthew "Smiffy" Smith@jimbob Never knew that scp did compression. I would have bzip2 first, or used rsync. -azv
Kevin Boyd (he/him) 🇨🇦@jimbob I usually use rsync -avP ... and that doesn't compress by default, it seems. I'll have to try -avzP next time.
Elias Mårtenson
Pär Björklund@jimbob also scp is included with windows, rsync is not 😀
Tim@penryu @jimbob there is one, windows terminal, it's not included by default https://github.com/microsoft/terminal
dragonfrogOf all the things I would do in a posix terminal, cmd isn't one of them.
@dragonfrog @Paxxi @jimbob ngl all I really want is not to have to download putty as soon as I login
@penryu @Paxxi @jimbob I kept running into Steve Walli at Usenix when he was working on Interix (later to become SFU), and at one I asked him when they were going to get bought by Microsoft and shelved. He said it was unlikely for legal reasons. Next year they had been bought by Microsoft and shelved. I asked him if he was lying when he said that and he asked "was it early or late in the convention?"...
Demand meant they had to bring it back but boy were they passive-aggressive about it.
@jimbob This is 100% valid. And while I do maintain an alias of scp='rsync $some_flags', I do not retain full knowledge of what $some_flags mean.
lertsenem@jimbob Yeah but most of the time you just check the man page examples and copy the most basic one, just like everyone else. :]
synlogic@jimbob plus those extra 2 chars to type are a bridge too far for many
Justin@jimbob it's like when your first learn a lot of Linux commands. I use ps aux due to muscle memory now. I've looked up the meaning of the letters a few times but never remember them. Rsync is avz my default for me now.
@justinz yeah ps aux is in my muscle-memory too!
phryk 🏴@jimbob Definitely true, but I still use it for big transfers because you can "easily" resume a transfer if it fails.
For rsync, compression is -z, tho I just have it in my muscle memory that "rsync" is always followed by "-avzP". :P
That enables compression and tries copying everything as is (think cp -a) while showing detailed progress info and enables resuming the transfer.
Mario Vilas@jimbob rsync needs to die in a fire
nix@jimbob life changing knowledge
VirtualWolf@jimbob I... also had no idea that such a flag existed!
Dag Ågren ↙︎↙︎↙︎@jimbob Good old Unix feature shyness. We can never have an improvement actually be the default! There might be a steam-powered minicomputer out there somewhere that will bust a leak if it ever sees an LZ code.
@WAHa_06x36 @jimbob ofc it can be *made* default
Jeanne Mastodon@jimbob rsync likewise. I was so sure rsync at least used compression by default!
Scott Williams 🐧
ghorwood↙↙↙@jimbob wait. what?
ajft [SEC=continvoucly morged]@jimbob put the worms back in the can where they belong!
Um yeah, I think at one stage i learnt that, but I suspect it was forgotten amongst far too many other options and defaults
Jeff C. 🇺🇦🇬🇱
Hynek Schlawack@jimbob Also scp is actually deprecated which keeps boggling my mind.
Matunos@hynek looks like— in RedHat at least— the scp protocol is deprecated but the command remains, now by default uses the sftp protocol under the hood https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know
⛧TJC⛧@hynek @jimbob what is the source for this? What’s the “official” replacement? I would have assumed even rsync was built on scp
That said, I see no reason to use scp over rsync. Yes , I can never remember all the flags, but that’s what shell aliases are for, since you only need a small handful of combinations to do anything.
Mark Tomczak@mtomczak @hynek @jimbob I agree that rsync can be dangerous. That’s why I heavily use shell aliases / functions (fish) for these. I only ever use a small combo of flags, so once I get those use cases I make functions with pre canned flags and some checks on slashes and whatnot. It’s a one time pain but then I never think about it again.
Dakota 💀@tjc @jimbob Here’s some details: https://lwn.net/Articles/835962/
Not sure at this point if the sftp-based scp that’s teased in the article and IIRC was linked in this thread is more of a stopgap or the new real thing. I personally prefer scp too because it’s just simpler and less dangerous to use.
Quinn Comendant@tjc @hynek @jimbob Scp relies on legacy protocols and lacks proper args sanitation.
New versions of scp have been updated to use sftp internally to avoid the security issues, so maybe safe to use depending on OS, but it’s safer just to always use rsync instead.
Here’s a summary from Redhat with more info and links. https://access.redhat.com/articles/5284081
