Web lists-archives.com

Re: [Samba] Is it possible to clone an NT ACL from one file or dir to a totally different file or dir ?




On 04/05/2018 08:32 PM, Andrew Bartlett wrote:
On Sat, 2018-03-24 at 20:38 -0400, Ken McDonald via samba wrote:
By default, step 4 takes forever to run on large datasets because it
uses named entities. My ultimate plan was to use the numeric
user/group id's in that step instead of named ones so the Winbind cost
is not incurred. Seems for the whole process, the calls to Winbind to
resolve the named entity to it's numeric ID are the reason for the
slowdown. That's why, even when using the normal Windows security tab
or samba-tool, it takes days to update large datasets. I'm exploring
options around that issue.
If this is on the AD DC, then I fixed part of the slowness here a
couple of week ago with

commit d418d0ca33afb41a793a2fff19ca68871aa5e9ef
Author: Andrew Bartlett <abartlet@xxxxxxxxx>
Date:   Wed Mar 21 20:43:10 2018 +1300

     winbindd: Add a cache of the samr and lsa handles for the passdb
domain

Otherwise, I'm not sure folks have looked into this terribly much, you
are already working at a pretty low level.

I hope this helps a little.

Andrew Bartlett

I have found a working solution using parallel processing on multiple cores that greatly reduces the time required to clone permissions. On a user directory with > 1TB data of 100,000s of dirs/files with all kinds of exotic naming styles, the process in the script below completes in about 60 minutes instead of the 5 days it took with either samba-tool or Windows Explorer.

This kind of performance was needed to move a bunch of user home directories over to the Samba file server, and I also have some cron jobs that periodically run to ensure a few shared file directories always have a standard set of permissions. In prior email list replies, I mentioned the perms were not properly cloning because the cloned copy had "extras" in them. This turned was because the top-level POSIX user & group were not the same in the cloned target. While I was correctly cloning POSIX extended ACL & attributes from the source, I did not realize that would not include the top-level permissions entries. Once that setting was added to the process, cloning Samba/POSIX permissions & attributes worked perfectly and ended up with exactly the same permission structure as would have been created by using Windows Explorer.

While I haven't looked at Andrew's code update helping Samba cache winbind lookups, I will mention that the process in the script below does not make use of any Samba/Winbind username/group lookup calls UNLESS the Windows ACL dir/file ownership is being set with a Windows username or group. For my case, all the dirs/files are owned by Administrator (root) and user access rights are delegated through Windows ACL entries, so no winbind lookups were needed when the script runs. If a Windows user/group account is configured for dir/file ownership, there will be slight delays during script execution, but since it's only for dir/file ownership, not actual Windows ACL perms, the delay is relatively tiny.

This process is extremely fast compared to samba-tool and I hope it's helpful to anyone that finds this thread. If you need to configure a standard suite of Windows ACL permissions on a large amount of directories & files this is currently the fastest way I've found to do it.

It's possible "sneak in" pure POSIX ACL entries manually using Linux-side only users/groups and the full cloning still works. In those case, once the POSIX ACL entry is created outside of Samba/Windows, you have to go back into the Windows Explorer Security tab for the perms template source dir/file and clean up some unneeded entries like "Everyone" and "CREATER OWNER." In the end, I was able to clone a source template dir/file with both POSIX/Windows ACL entries from both Linux-side & Samba-winbind sourced usernames/groups!

I'd appreciate any helpful comments on making my script more efficient or better simplified. The one portion I wished could have been compacted is the shell'ing-out of xargs to the separate tiny helper script "setperm.sh". I couldn't get that helper snippet to run as a shell function while still allowing xargs to call it, especially because of the variables being passed. Also, as dir/file names can be complex and contain all kinds of crazy special characters, there were a lot of challenges making the script and command/program calls handle these issues without running into escape'ing problems. I think all of these leaky problems have been corrected but I'd certainly appreciate any feedback there.

By using deep extended attribute cloning, only getfattr is needed because it includes the ACL entries normally configured by using getfacl. This helped ...a lot... in reducing winbind username/group lookup calls. The key line that clones POSIX/Windows ACL entries is this.

getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" | setfattr --restore=-

THE FULL SCRIPTS FOLLOW:

####################################### cloneperms.sh #######################################
#!/bin/bash
# Clone permissions & attributes from a template source to a target destination,
# possibly recursing all subdirectories & files. Also supports parallelism.
#
# USAGE: cloneperms.sh [-r] [-p numcores] template target
#
# -r        : recurse all subdirectories. pointless if template & target are files # -p        : number of cores for parallel processing, defaults to 1. pointless if template & target are files
# template  : source directory or file with permissions to clone
# target    : destination directory or file for cloning perms to
#
# version 1.0

USAGE="Usage: $0 [-r] [-p numcores] template target"

# determine parameter type
test_type()
{
    TYPE_TEST=$1
    if   [ -d "${TYPE_TEST}" ]
    then return 1; # directory
    elif [ -f "${TYPE_TEST}" ]
    then return 2; # file
    else
        echo "${TYPE_TEST} is not found!"; echo $USAGE; exit 1;
    fi
}

# check for options
CORES=1
RECURSE=0
while getopts rp: o
    do    case "$o" in
        r)    RECURSE=1;;
        p)    CORES=$OPTARG
                if [ -z "${CORES##*[!0-9]*}" ] || [ $CORES -le 0 ]
                then
                    echo "numcores must be number greater than 0"; echo $USAGE; exit 1;
                fi;;
        [?])    echo $USAGE; exit 1;;
    esac
done
shift $(($OPTIND - 1))

# enough arguments?
if [ "$#" -ne 2 ]
then
    echo "missing template or target"; echo $USAGE; exit 1;
fi

TEMPLATE=$1
TARGET=$2

# no self targets
if [ $TEMPLATE == $TARGET ]
then
    echo "template & target cannot be the same!"; echo $USAGE; exit 1;
fi

# test template & target types
test_type $TEMPLATE
TEMPLATE_TYPE=$?
test_type $TARGET
TARGET_TYPE=$?

# ensure template, target, recurse, and cores combinations are valid
if [ $TEMPLATE_TYPE == 1 ]
then
    FIND_FLAG="d";
    if [ $TARGET_TYPE == 2 ]
    then
        echo "template cannot be directory and target a file!"; echo $USAGE; exit 1;
    fi
else
    FIND_FLAG="f";
    if [ $TARGET_TYPE == 1 ]
    then
        if [ $RECURSE == 0 ]
        then
            echo "template cannot be a file and target a directory without recurse option"; echo $USAGE; exit 1;
        fi
    else
        if [ $CORES -gt 1 ]
        then
            echo "template & target are files, ignoring numcores option";
            CORES=1;
        fi
        if [ $RECURSE == 1 ]
        then
            echo "template & target are files, ignoring recurse option";
            RECURSE=0;
fi fi fi

TEMPLATE_OWN=$(stat -c %U $TEMPLATE)      # get the template top-level POSIX owner TEMPLATE_GROUP=$(stat -c %G $TEMPLATE)    # get the template top-level POSIX group TEMPLATE_PERMS=$(stat -c %a $TEMPLATE)    # get the template top-level POSIX permissions current_dir=$(dirname $(readlink -f $0))  # script can run from another directory
if [ $RECURSE == 1 ]
    then
        find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES chown $TEMPLATE_OWN;         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES chgrp $TEMPLATE_GROUP;         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES chmod $TEMPLATE_PERMS;         # get the template POSIX extended attributes, which include extended ACL         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES -I {} $current_dir/setperm.sh $TEMPLATE {};
    else
        chown $TEMPLATE_OWN   $TARGET;
        chgrp $TEMPLATE_GROUP $TARGET;
        chmod $TEMPLATE_PERMS $TARGET;
        # get the template POSIX extended attributes, which include extended ACL
        $current_dir/setperm.sh $TEMPLATE $TARGET;
fi
######################################################################################

####################################### setperm.sh #######################################
#!/bin/bash
# Sets a single permissions & attributes entry on a target based on a template source example
#
# There's no parameter checking in this script, so don't run this script directly.
# It's meant to be run from cloneperms.sh.
#
# USAGE: setperm.sh template target
#
# template  : source directory or file with permissions to clone
# target    : destination directory or file for cloning perms to
#
# version 1.0

ENCODED=$(printf '%s\n' "$2" | sed 's|\\|\\\\134|g') # setfattr syntax for target name containing forward slashes getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" | setfattr --restore=- ######################################################################################

I also have a script for recursively removing all extended ACL & attribute entries if anyone is interested. Strangely there is a way to remove the ACL "setfacl -R -b" recursively but I found no way to similarly remove ALL attributes. I made a script to do it regardless of what the attributes are named...

TLDR?

-Ken

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba