Using AFS At Glasgow

Local AFS Cell (/afs/phas.gla.ac.uk)

File System Layout

Beneath /afs/phas.gla.ac.uk are the following directories:

• backup : Snap shot backups for user home directories and other files. The snap shot is done each night at 1am.
• data : Large store area, not backed up.
• group : Group based storage area, backed up.
• project :
• system : Common programs installed system wide.
• user : User home directories are stored under this directory. The initial quota on home directories is 50GB. Nightly snap shot backups and long term backups.

Backups

Overnight backups of user home areas are taken at 1am each night. These backups are user accessible and can be found in /afs/phas.gla.ac.uk/backup/user/_letter_/_username_. Longer term backups are also done - nightly snapshots of user home directories are retained for 7 days except that the snapshots done on a Saturday are retained for two months and the snapshots done on the first Saturday of each month are retained for six months. These longer term backups are not user accessible please contact SystemAdministrators to request a file to be recovered.

Cronjobs

Normal cronjobs can not write to the afs file system and can only read the publicly accessible parts of the file system. Cronjobs which require access to the afs file system can be created using the kcrontab command.

Batch system

The currrent batch system (acess from the machine ppepbs.physics.gla.ac.uk) will not be able to access the local afs cell. A new batch system has been setup which can access the afs file system and which is accessible from any linux desktop machine that has been moved onto the new system.

To use the new system simply use the normal qsub and qstat commands from any PPE linux desktop that has been moved over the the new system.

Home web pages

User's home web pages are served from the directory public_html in their home directory. The web server will use the public_html directory in the afs file system of an account moved over to the afs in preference to the public_html directory in the old account. Until files are moved from the old public_html directory to the new directory a user's web pages will be inaccessible. It is important to move the files under the public_html directory in the old NFS file system to the new public_html directory and not to move the public_html directory itself due to afs file permissions. For example to do this use the command:

mv /home/_username_/public_html/* /afs/phas.gla.ac.uk/user/_letter_/_username_/public_html/

Substituting _letter_ and _username_ as required.

External Kerberos access (linux/mac)

Setting up kerberos on a linux or mac allows for password less login to machines after the initial kerberos ticket has been created.

Edit the file /etc/krb5.conf or for OS X edit /Library/Preferences/edu.mit.Kerberos and add/edit to the realms section:

PHAS.GLA.AC.UK = {
  default_domain = phas.gla.ac.uk
  kdc = kdc2.phas.gla.ac.uk:88
  kdc = kdc1.phas.gla.ac.uk:88
  admin_server = kdc1.phas.gla.ac.uk
 }

Then add/edit to the libdefaults section:

allow_weak_crypto = true
default_realm = PHAS.GLA.AC.UK
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 25h
renew_lifetime = 672h
forwardable = true
proxiable = true

For ssh access edit /etc/ssh/ssh_config on linux machines or /etc/ssh_config on macs (root access is required to edit either file) and ensure the follow are set to yes:

GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
GSSAPIKeyExchange yes

Then try to get an kerberos ticket and log into ppelx:

Changing username as appropriate.

External afs access (linux/mac)

To access the phas.gla.ac.uk afs cell from an external linux machine requires first setting up kerberos access as described above and then installing the openafs client and editing a couple of config files all of which will require root access. Packages exist for all the major linux distributions just use the distributions software installer. If that fails try the openafs web site: https://www.openafs.org . For OS X the packages can be downloaded from http://www.openafs.org/macos.html

After installing the openafs edit the file /etc/krb5.conf or for OS X edit /Library/Preferences/edu.mit.Kerberos and add/edit to the realms section:

Then add/edit to the domain_realm section:

 .phas.gla.ac.uk = PHAS.GLA.AC.UK
 .physics.gla.ac.uk = PHAS.GLA.AC.UK
 .ppe.gla.ac.uk = PHAS.GLA.AC.UK

Or use this example krb5.conf file which also has the kerberos setting for cern and fermi lab.

Then edit the CellServDB file which can usually be found on a linux machine at /usr/vice/etc/CellServDB or /etc/openafs/CellServDB and for OS X is located at /var/db/openafs/etc/CellServDB and add:

>phas.gla.ac.uk         #Univeristy of Glasgow Physics And Astronomy
194.36.1.27                     #afsdb3.phas.gla.ac.uk
194.36.1.19                     #afsdb1.phas.gla.ac.uk
194.36.1.33                     #afsdb2.phas.gla.ac.uk

either at the top or bottom of the file.

Next edit the ThisCell file (/usr/vice/etc/ThisCell or /etc/openafs/ThisCell on linux or /var/db/openafs/etc/ThisCell for OS X) replacing the contents with

phas.gla.ac.uk

Restart the openafs client either by restarting the client service or rebooting the machine) and create a kerberos ticket and get an afs token:

Changing username as appropriate. If both commands are successful then try to access the afs cell:

Hints, Tips And Issues

Common AFS commands

• fs lq : List the size and amount used of the volume of the current working directory.
• fs listacl : List the acl (access control list) for the current working directory.
• fs setacl dir user/group permissions : Add to the acl of a directory.

AFS ACLs (Protecting Data)

AFS file systems use directory based ACLs to determine the access permissions for a given file. The following is taken from the afs documentation (http://docs.openafs.org/AdminGuide/ch15s02.html):

• The permissions on a directory's ACL apply to all of the files in the directory. When you move a file to a different directory, you effectively change the access permissions that apply to it to those on its new directory's ACL. Changing a directory's ACL changes the protection on all the files in it.
• When you create a subdirectory, its initial ACL is created as a copy of its parent directory's ACL. You can then change the subdirectory's ACL independently. However, the parent directory's ACL continues to control access to the subdirectory in the following way: the parent directory's ACL must grant the l (lookup) permission to a user (or a group the user belongs to) in order for the user to access the subdirectory at all.

By default the following directories are created in a user's afs home area:

• private - access only for the user and the system administrator, this is also the default for any new directories.
• public - public (world wide) access for anyone.
• public_html - public (world wide) access for anyone, user web pages belong here.
• public_ppe - readable to the ppe group only.

To see the acl list on a directory use the command fs listacl dir .

To allow a user or group user read access to a directory dir :

fs setacl dir user read

To give write permissions:

fs setacl dir user write

And to remove all permissions for a user or group:

fs setacl dir user none

System administrator access is required to all directories to allow backups to be taken.

There is a limit of around 20 entries that can be added to an acl list. The exact number depends on the size (in characters) of each acl entry. To get around this user created groups can be used and then the group can be added to the acl list. The cern web page http://information-technology.web.cern.ch/services/fe/afs/howto/work-afs-groups has instructions on how to create groups and add people to them.

Recursive setting of ACLs

A helper script to recursively set an ACL is available on all the scientific linux desktop machines. Usage is similar to the standard fs setacl command:

fs-recursive-setacl dir user acl

Accessing the CERN afs cell (/afs/cern.ch)

Getting a kerberos ticket for another kerberos realm using kinit will overwrite the existing kerberos tickets file destroying any local Glasgow ticket. To get around this tickets for different kerberos realms have to be written to different files. A couple of helper scripts have been written to made this straight forward: kinit-cern, klist-cern, kdestroy-cern, ssh-cern, kinit-fnal, klist-fnal, kdestroy-fnal, ssh-fnal.

They are used in the same way as the standard kinit, klist and kdestroy. For example:

$ kinit-cern <lxplus user name>@CERN.CH

and after doing the kinit:

$ ssh-cern <lxplus user name>@lxplus.cern.ch

rsyncing files to afs

rsync will copy files to an afs file system in the same way as any other file system. However if directories in the source file system have the stick bit set rsync will complain that it 'failed to set permissions'. A directory with the sticky bit set will have a 's' in the permissions bit field eg:

drwxrwsr-x  4 ppemgr ppemgr   45 Jan 14  2009 sl44

These errors can be safely ignored but they may obscure other errors when rsyncing. To remove stick bits before rsyncing use chmod:

chmod a-s _dir_

replacing _dir_ as required. To recursively remove sticky bits use the -R option.

Automatic Token Renewal

To automatically renew an AFS token for up to 30 days after logging in requires either a small addition the shell login script for ssh logins or for desktop logins a small script to be auto run. Each login change is independent of the others and all can be safely used at the same time.

ssh login

For bash shell users add the following to the top of .bash_profile:

if [ -e /bin/ps ] && [ -e /bin/grep ]
then
    kproc=`/bin/ps x -u ${USER} | /bin/grep krenew | /bin/grep ${KRB5CCNAME}`
    if [ "${kproc}" == "" ] && [ -e /usr/bin/krenew ]
    then
        /usr/bin/krenew -K 60 -t -k ${KRB5CCNAME} &
    fi
fi

For csh/tcsh users the untested equivalent is to add the following to the top of .cshrc or .tcshrc:

if ( -e /bin/ps && -e /bin/grep ) then
    setenv kproc `/bin/ps x -u ${USER} | /bin/grep krenew | /bin/grep ${KRB5CCNAME}`
    if ( "${kproc}" == "" && -e /usr/bin/krenew ) then
        /usr/bin/krenew -K 60 -t -k ${KRB5CCNAME} &
    endif
endif

KDE desktop

Gnome Desktop

-- AndrewPickford - 2009-10-14