Will Breaden Madden - 2015-02-21

CERN Git

This page contains information on the use of CERN Git repositories.

introduction

Git is better than Subversion.

CERN provides access-controlled Git repositories. In a nutshell, a repository is created usually for an individual project and its access permissions are controlled by e-groups. So, the access and editing permissions for users of the repository are inherited from an e-group. For a quick start, a recommended procedure is as follows:

• Request a new e-group.
• Become not only an owner of the e-group, but also a member.
• Request a new repository (perhaps one for testing initially).
• Wait for a time (anything from 20 minutes to 17 hours) for the e-group permissions to propagate to the repository.
• Clone your new repository using Kerberos authentication.
• Fuck shit up.
• Push to the central repository.

links

The CERN central Git service is at https://git.cern.ch/. The CERN Information Technology Department page on the service is https://information-technology.web.cern.ch/services/git-service. The CERN Service Portal Knowledge Base articles for the Git service are here https://cern.service-now.com/service-portal/faq.do?se=git-service.

web representation of repositories

CERN repositories can be represented using Gitweb.

• https://git.cern.ch/web/

create new repository

To create a GIT repository, an e-group is needed for permissions.

create an e-group

The access and editing permissions associated with a CERN Git repository are inherited from a CERN e-group. Specifically, the members of the e-group are granted repository permissions -- not owners. For the development of a group project, an existing e-group can be used. For personal or experimental projects, a new e-group can be beneficial. Create an e-group at the following link:

• https://e-groups.cern.ch/e-groups/EgroupsNewStatic.do

Note that there is a naming convention for e-group names:

• https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/MailingListNomenclatureRules

An example e-group name could be "atlas-sw-skunkworks". As of 2015-02-13, there is the start of a Git infrastructure e-group naming convention for user Git repositories:

%CODE{"bash"}% atlas-sw-git- %ENDCODE%

For the purposes of CERN Git repository permissions, do not simply be the owner of the e-group; become a member of the e-group.

• https://e-groups.cern.ch/e-groups/EgroupsSubscription.do?egroupName=atlas-sw-skunkworks

static e-groups and self-subscription policy

A static e-group means that the creator defines the policies for self-subscription and users are not granted access to the members list.

dynamic e-groups

Dynamic e-groups are defined by criteria such as experiment, department, status and location.

privacy policy

owners and administrators

There is one owner and there can be multiple administrators of an e-group. Ownership can be transferred. Owners and administrators have the same usage rights of the e-group. When an e-group is deleted, the owner only can recreate a group with the same name 30 days afterwards.

posting restrictions

By default, owners and administrators only are granted the ability to post messages. This can be changed in the mail properties configuration.

archives

E-group archives are available via a URL of the following form:

%CODE{"bash"}% http://cern.ch/group/ %ENDCODE%

using e-groups for access control

e-groups can be used for granting access to Indico events. In order to do this, access the Indico event's management page and then access the protection section.

create a repository

To create a GIT repository, access CERN Forge (a portal for the management of SVN, Git and JIRA projects) and create a request for a GIT repository:

• https://cern.ch/cernforge

The following details are needed for the request:

• the name of the repository to be created,
• the e-group name and
• the initial quota (optional).

The following procedure can be used to create a new repository:

• Select "New Project".
• This should direct to https://cernforge.cern.ch/request/git/.
• Enter the details and wait for approval (it can be within an hour).

The time for a new project on CERN Forge to change from status requested to status active on 2015-01-26 was ~7 minutes. The resulting CERN Forge page for the project could be something such as the following:

• https://cernforge.cern.ch/user/git/junk/

The resulting web listing for the project should be included here:

• https://git.cern.ch/

The link to the web representation of the project (CERN Gitweb) should be the following:

• https://git.cern.ch/web/junk.git

If the project is public, the resulting web listing for the project should be here:

• http://git.cern.ch/pubweb/

The default access rights to a CERN Git repository are restricted to members of the e-group provided in the request.

CERN Git repository usage

clone repository

For the purposes of examples, a repository with the name "junk" is used. The Kerberos interface can be used to clone a Git repository using a command such as the following:

%CODE{"bash"}% kinit ${USER}@CERN.CH git clone https://:@git.cern.ch/kerberos/junk %ENDCODE%

To clone a Git repository from the LXPLUS cluster with LDAP authentication, the username should be specified in the URL:

%CODE{"bash"}% git clone https://@git.cern.ch/ldap/junk %ENDCODE%

For a public repository, a command like the following can be used:

%CODE{"bash"}% git clone http://git.cern.ch/pub/junk %ENDCODE%

commits and pushes

For a new, empty repository, the push procedure should be initialised likely in the following way:

%CODE{"bash"}% git push origin master %ENDCODE%

Following this, standard commits and pushes should be available.

README repository summary page

Gitweb can display a README file in the summary page of a CERN Git repository. This is used typically as a short introduction to the repository.

There are some CERN Knowledge Base articles on this:

• https://cern.service-now.com/service-portal/article.do?n=KB0002461
• https://cern.service-now.com//service-portal/article.do?n=KB0002018

There are two main steps involved in setting up README summary page functionality for a CERN Git repository. The first step is setting up a script that runs after pushing to the repository of git.cern.ch which copies a HTML README file such that it is displayed on Gitweb. The second step is setting the "gitweb.description" of the project in order to display appropriate information on the repository summary page RSS banner.

set up git-hook for README

Add a post-receive hook script using the following command:

%CODE{"bash"}% git cat-file blob HEAD:README.html > $GIT_DIR/README.html %ENDCODE%

Then, when pushing to git.cern.ch, a file called "README.html" is searched for at the HEAD of the master branch of the repository and then put in the root of the directory ${GIT_DIR}. ${GIT_DIR} is a Git variable set by Git in the server that points to the root path of the repository. This is run with every push of the repository to git.cern.ch, so every time a new file README.html is pushed to the repository at git.cern.ch, Gitweb should display it.

Markdown is a good way to compose the README. A script to convert a Markdown README file to a HTML file compatible with CERN Git Gitweb (and a standaline HTML file perhaps for use with CERN SVN repositories) is available here:

• https://github.com/wdbm/CERN-projects

This script depends on the program Pandoc. The program Markdown could also be used (though it does not feature LaTeX-style mathematics rendering). The Gitweb summary page for example project "junk" would be at the following URL:

• https://git.cern.ch/web/junk.git

In order to set up the post-receive script, clone the special CERN Git repository "git-hooks". Then, create the executable script "post-receive" in the directory of the repository "git-hooks" that corresponds to the name of the repository for which a README workflow is being prepared (e.g. "junk"). Ensure that the script (e.g. git-hooks/junk/post-receive) is executable. The content of the script should be as follows:

%CODE{"bash"}% #!/bin/bash

git cat-file blob HEAD:README.html > $GIT_DIR/README.html %ENDCODE%

The HTML README file should not be a standalone HTML file because it is rendered as part of an existing web page for the repository. So, for example, the following HTML content would be appropriate:

%CODE{"bash"}%

This text is normal.

This text is bold.

This text is italic.

This text is normal.

Specifically, the content of the HTML README file is inserted into a DIV of the rendered Gitweb project summary page.

A procedure to do this is as follows:

%CODE{"bash"}% kinit ${USER}@CERN.CH git clone https://:@git.cern.ch/kerberos/git-hooks cd git-hooks touch junk/post-receive cat >> junk/post-receive<<'EOF' #!/bin/bash

git cat-file blob HEAD:README.html > ${GIT_DIR}/README.html EOF chmod 755 junk/post-receive git add junk/post-receive git commit -m 'add post-receive script for package junk' git push %ENDCODE%

set up gitweb-description

By default, the "description" of a Git repository is an arbitrary string and the "owner" is the e-group that administers the repository. These variables are for the Gitweb summary page of a repository only. They do not change any other functionality of a repository. Note that to change the owner of a repository, a request must be submitted. In order to change these variables, clone the repository "gitolite-admin". Edit the configuration file of the repository under development in the repository gitolite-admin (e.g. gitolite-admin/conf/subs/junk.conf). For this configuration file, the following default variables are defined:

%CODE{"bash"}% config gitweb.owner = "your egroup" config gitweb.description = "use this variable to change description" %ENDCODE%

The full edited, resultant content of the gitolite-admin configuration file for the repository "junk" could be something such as the following:

%CODE{"bash"}% repo junk #This is only for beautification.DOES NOT change any functionality config gitweb.owner = atlas-sw-skunkworks config gitweb.description = change gitweb.description variable in gitolite-admin RW+ = @atlas-sw-skunkworks %ENDCODE%

Changing, committing and then pushing to the CERN central Git repository updates the variables.

A procedure to do this is as follows:

%CODE{"bash"}% kinit ${USER}@CERN.CH git clone https://:@git.cern.ch/kerberos/gitolite-admin cd gitolite-admin cat > conf/subs/TTHbbLeptonic.conf<<'EOF' repo TTHbbLeptonic config gitweb.owner = atlas-sw-skunkworks config gitweb.description = development and testing of ttH, H to bb prototype analysis package for l + jets, dilepton and boosted channels RW+ = @atlas-sw-skunkworks EOF git add conf/subs/TTHbbLeptonic.conf git commit -m 'update Gitweb description' git push %ENDCODE%

README workflow

A recommended workflow for generating the README in general development (i.e. following the git-hooks setup etc.) is as follows:

• Edit the Markdown README file "README.md".
• Compile the Markdown README file to a HTML file suitable for inclusion in a DIV of a CERN Gitweb summary page and to a standalone HTML file (perhaps suitable for CERN SVN repositories) using the script "compile_documentation.py" (noting that the program Pandoc must be available as a dependency of this script -- as must be the Python module docopt).

%CODE{"bash"}% wget https://raw.githubusercontent.com/wdbm/CERN-projects/master/compile_documentation.py python compile_documentation.py rm compile_documentation.py %ENDCODE%

• Git add the edited Markdown README file and the resultant README HTML files.

%CODE{"bash"}% git add README.md git add README.html git add README_SVN.html %ENDCODE%

• Commit and push.