Using SCP with problematic character filenames

It’s not often I write about technical problems. More often than not the reason is very simple: Google already knows the answer. It takes little effort to find a solution again, so why write about it? This time it’s a bit different as it has taken me half an hour to properly solve this.

The problem itself is very simple: I’m trying to use scp to copy files across a network, where the local and remote files or folder names have problematic characters. Like spaces, parentheses, etc. The usual solution is simple: just make sure you quote the variables containing those filenames when passing them along. On scp this won’t work though, causing errors like “Badly placed ()’s” on lines that otherwise work. The reason is also simple, but hard to find: the remote path seems to be interpreted twice, so the quoting only gets you past your shell and the actual issuing of the command. The string itself still needs to be escaped though, or scp will have trouble interpreting it, causing those odd errors. It turns out that it’s not too difficult to do this. For example, in bash:

# set up folders
SRC="$PWD/subfolder"
DST=$(printf '%q' "user@host.domain:${SRC:${#HOME}+1}")

# main process
for folder in subfolder1 subfolder2; do
   scp "$SRC/$folder/"*.pgm "$DST/$folder/"
done

The example above copies all files with a pgm extension from the following subpaths of the present directory:

subfolder/subfolder1
subfolder/subfolder2

The files are placed in the equivalent path on the destination host. The remote path is specified relative to the home directory, to allow for remote machines with a different homedir mount point.

Advertisements

2 comments

  1. Many thanks for the useful solution!
    It is necessary to note a couple of important nuances.
    1) We need to use “bash” to execute the script for working the command “printf ‘%q'”:
    #!/usr/local/bin/bash
    2) We cannot put in “scp” settings paths in a single string:
    arch_path=$(printf ‘%q’ “/usr/backups/${arch_name} ./subdir/${dir}/”)
    scp user@10.8.2.124:$arch_path

    See two examples below:

    #!/usr/local/bin/bash

    dir=date +%Y-%m-%d
    arch_name=arch-date "+(%d-%m-%Y)".tbz
    arch_path=$(printf ‘%q’ “/usr/backups/${arch_name} ./subdir/${dir}/”)

    echo “$arch_path”
    scp user@10.8.2.124:$arch_path
    ……………………………………

    /etc/cron.misc/backups_transfering.sh

    /usr/backups/arch-(01-09-2015).tbz\ ./subdir/2015-09-01
    Argument for -c ends in backslash.
    (!)
    We get an error at this point: “tbz\ ./”

    =================================================
    #!/usr/local/bin/bash

    dir=date +%Y-%m-%d
    arch_name=arch-date "+(%d-%m-%Y)".tbz
    arch_path=$(printf ‘%q’ “/usr/backups/${arch_name}”)
    echo “$arch_path”
    scp user@10.8.2.124:$arch_path ./subdir/$dir
    ……………………………………

    /etc/cron.misc/backups_transfering.sh

    /usr/backups/arch-(01-09-2015).tbz\ ./subdir/2015-09-01
    arch-(01-09-2015).tbz 100% 505MB 42.1MB/s 00:12

    • Thanks for stopping by Dmitriy, and for pointing out those important details. For #2, the reason you cannot include both paths in a single string is that if you try to do that, both paths are passed to scp as a single argument, so scp cannot interpret them correctly.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s