# HOWTO: OpenAFS fileserver cluster (new ebuilds /kernel 2.6)

## irondog

  HOWTO: OpenAFS fileserver cluster (new ebuilds /kernel 2.6)

Introduction

AFS is a distributed filesystem that enables co-operating hosts (clients and servers) to efficiently share filesystem resources across both local area and wide area networks. Clients hold a cache for often used objects (files), to get quicker access to them. 

source: http://www.gentoo.org/doc/en/openafs.xml

History

OpenAFS has a very long history: it's more than 20 years old. The most important milestone in it's history is, (at least for me) that it's open-source today. It runs on a wide range of platforms including Windows, OSX, Irix and GNU/Linux. The OpenAFS guys have been working very hard last year to make it run on 2.6 kernels.

There are people running OpenAFS on Gentoo and most of them are stuck to kernel 2.4 due the lack of stable packages and rc-scripts for kernel 2.6.

About

I've been testing OpenAFS client and server functions in the latest unstable releases. As I've been testing it with good results, I'm willing to share my experience and scripts to others. My work on OpenAFS aims to match the existing Gentoo docs for OpenAFS as scrict as possible. This howto shows OpenAFS is usable on a up-to-date Gentoo machine with a 2.6 kernel.

Warnings

* The 1.3.X releases that support 2.6 kernels are still unstable, 1.4 is underway.

* Heavily patched kernels like Gentoo-sources might not be a good choice to run OpenAFS on.

* The learning cruve of OpenAFS is steep. source: Linux Journal

* Errors generated by OpenAFS might be very confusing. Errors won't point directly to their possible cause. 

* Only x86 is tested, but x86_64 should work

Prerequisites

* For clients a ext2 partition mounted on /usr/vice/cache is needed (not strictly, but it's recommended)

* For servers a ext2 partition mounted on /vicepa is needed (yes, stop here if you haven't a spare partition)

* Some docs report non-ext2 partitions won't work. I don't know how true this is.

* /usr/src/linux points to the source of the running kernel

* You have a global ip: 222.222.222.222, your fully qualified hostname is: myhost  :Wink: 

Last words and some links

The following document is very complete, but a little outdated:

http://www.gentoo.org/doc/en/openafs.xml

I really tried to package OpenAFS to match the official document as good as possible.

The differences between my howto and the url above exist for these reasons:

* Openafs has different defaults now (document is a little outdated)

* Life is easier my way.

* Openafs was very broken (not only on Gentoo)

Please don't see this document as a replacement of the official OpenAFS docs. It's worth printing them.

http://www.gentoo.org/doc/en/openafs.xml

http://www.openafs.org/doc/index.htm

http://www.openafs.org/pages/doc/QuickStartUnix/auqbg005.htm#HDRWQ41

Overlay tarball can be downloaded here:

http://tienstra4.flatnet.tudelft.nl/~gerte/openafs-overlay-gerte.tar.gz

https://bugs.gentoo.org/show_bug.cgi?id=82075 (attachment: New ebuilds and rc-scripts for OpenAFS)

Building OpenAFS. Initial test for Global Cells Client

These steps are crucial to check AFSD (the client/ cache manager) and kernel module are working on your computers. Even if you plan to setup a server-only machine you need to have a working client first, it can be disabled later.

There is no need to edit /usr/vice/etc/CellServDB and /usr/vice/etc/ThisCell. The ebuild installs a usable configuartion out of the box.

* Configure portage to use overlay

/etc/make.conf

```
PORTDIR_OVERLAY="/usr/local/portage"

```

* Download and extract the OpenAFS overlay tarball from bugzilla/my website

```
# mkdir -p /usr/local/portage

# cd /usr/local/portage

# wget http://tienstra4.flatnet.tudelft.nl/~gerte/openafs-overlay-gerte.tar.gz

# tar -zxvf openafs-overlay-gerte.tar.gz

```

* Compile and Install Openafs

```
# mkdir -p /etc/portage

# echo 'net-fs/openafs-kernel ~x86' >> /etc/portage/package.keywords

# echo 'net-fs/openafs ~x86' >> /etc/portage/package.keywords

# echo 'net-fs/openafs' >> /etc/portage/package.unmask

# emerge -av openafs

```

* Start openafs client (afsd) and cross your fingers your kernel doesn't crash  :Wink: 

```
# /etc/init.d/afs start

# ls /afs/

```

Ok, no problems so far? Have a look at the directories in /afs. You can see directories for each cell in /usr/vice/etc/CellServDB.

Proceed to the 'Full OpenAFS client and server' section. If not: Review the steps above, check your firewall, but don't continue.

Note: Browsing /afs/<anycell> might be a little slow/shaky.

* Stop openafs client again

```
# /etc/init.d/afs stop

```

Full OpenAFS client and server

Once again, I'ld like to point to these documents for a more complete reference:

http://www.gentoo.org/doc/en/openafs.xml

http://www.openafs.org/doc/index.htm

http://www.openafs.org/pages/doc/QuickStartUnix/auqbg005.htm#HDRWQ41

Ok, enough. Lets get it working.

* Edit /etc/hosts

/etc/hosts

```
127.0.0.1        localhost

222.222.222.222       myhost

```

* Ping yourself to check you did things correctly

```
ping -c 10 222.222.222.222

ping -c 10 myhost
```

* Edit /etc/afs/CellServDB

#myhost is NOT a comment

/etc/afs/CellServDB

```
>mycelll      #Cell name

222.222.222.222    #myhost

```

I repeat: '#' is N O T a comment !!!!!!

* Edit /etc/afs/ThisCell

/etc/afs/ThisCell

```
mycell
```

* Make sure /vicepa is mounted

```
#df

/dev/bla x y z% /vicepa

```

* Start the Basic Overseer Server without security. Create some settings, and check.

```
# /usr/afs/bin/bosserver -noauth

# /usr/afs/bin/bos setcellname myhost mycell -noauth

# /usr/afs/bin/bos listhosts myhost mycell -noauth

```

* Create server processes.

```
# /usr/afs/bin/bos create myhost kaserver simple \

    /usr/afs/bin/kaserver -cell mycell -noauth

# /usr/afs/bin/bos create myhost buserver simple \

    /usr/afs/bin/buserver -cell mycell -noauth

# /usr/afs/bin/bos create myhost ptserver simple \

    /usr/afs/bin/ptserver -cell mycell -noauth

# /usr/afs/bin/bos create myhost vlserver simple \

    /usr/afs/bin/vlserver -cell mycell -noauth
```

* Check if server processes are running.

```
/usr/afs/bin/bos status myhost -noauth
```

If things are not running, stop here!! Check logs in /usr/afs/logs for some feedback.

* Initialise security/user management

Think about you passwords. These might be hard to change.

```
# /usr/afs/bin/kas -cell mycell -noauth

ka> create afs

initial_password:

Verifying, please re-enter initial_password:

ka> create admin

initial_password:

Verifying, please re-enter initial_password:

ka> examine afs

User data for afs

  key (0) cksum is 2651715259, last cpw: Mon Jun  4 20:49:30 2001

  password will never expire.

  An unlimited number of unsuccessful authentications is permitted.

  entry never expires.  Max ticket lifetime 100.00 hours.

  last mod on Mon Jun  4 20:49:30 2001 by $lt;none>

  permit password reuse

ka> setfields admin -flags admin

ka> examine admin

 

User data for admin (ADMIN)

  key (0) cksum is 2651715259, last cpw: Mon Jun  4 20:49:59 2001

  password will never expire.

  An unlimited number of unsuccessful authentications is permitted.

  entry never expires.  Max ticket lifetime 25.00 hours.

  last mod on Mon Jun  4 20:51:10 2001 by $lt;none>

  permit password reuse

ka> quit

# /usr/afs/bin/bos adduser myhost admin -cell mycell -noauth

# /usr/afs/bin/bos addkey myhost -kvno 0 -cell mycell -noauth

    input key: <use admin pass>

    Retype input key: <use admin pass>

# /usr/afs/bin/pts createuser -name admin -cell mycell -noauth

# /usr/afs/bin/pts adduser admin system:administrators -cell mycell -noauth

# /usr/afs/bin/pts membership admin -cell mycell -noauth

      Groups admin (id: 1) is a member of:

        system:administrators

```

* Restart server processes / check if server processes are running.

```
# /usr/afs/bin/bos restart myhost -all -cell mycell -noauth

# /usr/afs/bin/bos status myhost -noauth

```

If things are not running, stop here!! Check logs in /usr/afs/logs for some feedback.

* Create fileserver processes

```
# /usr/afs/bin/bos create myhost fs fs /usr/afs/bin/fileserver \

                                    /usr/afs/bin/volserver \

                                    /usr/afs/bin/salvager \

                                    -cell mycell -noauth
```

* Check if server processes are running.

```
# /usr/afs/bin/bos status myhost -noauth

# /usr/afs/bin/bos status myhost fs -noauth

# /usr/afs/bin/bos status myhost fs -long -noauth

```

If things are not running, stop here!! Check logs in /usr/afs/logs for some feedback.

* Create upserver process

```
# /usr/afs/bin/bos create myhost upserver simple "/usr/afs/bin/upserver -crypt /usr/afs/etc -clear /usr/afs/bin" \

          -cell mycell -noauth
```

* Check if server processes are running.

```
# /usr/afs/bin/bos status myhost -noauth

```

If things are not running, stop here!! Check logs in /usr/afs/logs for some feedback.

* Create a volume to store files into your newly created server.

```
# /usr/afs/bin/vos create myhost /vicepa root.cell -cell mycell -noauth
```

* Well done. You're now ready to have your Client use your own file server.

```
# /etc/init.d/afs start

# /usr/afs/bin/klog admin #use the admin password

# ls /afs/mycell
```

By default /afs/mycell is a AFS mount point to root.cell. We created root.cell on /vicepa. You can copy files to this volume if you are authenticated and if ACL's are set correctly.

* Make your server and client start when you boot your computer. As bosserver was started by hand previously, this is the right moment to configure the rc-script to use AFS server features.

/etc/conf.d/afs

```
ENABLE_SERVER="yes"
```

```
# rc-update add afs default
```

Hints

* In this brand new Gentoo OpenAFS package /usr/vice/etc/{CellServDB,ThisCell} and /usr/afs/etc/{CellServDB,ThisCell} are symlinks to /etc/afs/{CellServDB,ThisCell}

This was the best way to get the Client working out of te box. It keeps Server- and Client configuration in-sync. Besides it offers the a GNU/Linux default to store configuration in /etc without breaking the origional Transarc paths.

* /usr/vice/etc/cacheinfo is managed by the rc-script, don't edit this one, use /etc/conf.d/afs. Afsd picks it up automatically.

* Read administration documtents for further configuration and clusering OpenAFS.

* OpenAFS is not very easy to learn. Don't mess around with fileservers behind routers. This is expert stuff!!

* For clustering (I.E. more machines sharing the same files), you need to add hosts to /etc/afs/CellServDB and repeat all steps. Passwords and keys have to be the same on all machines.

----------

## irondog

Fucking nobody interested in this guide? Come on.

----------

## Hideki

I looked at AFS quite a while ago when I had no knowledge of network file sharing but then I found nfs at that time and using it now.

Maybe I should simply rtfm but what is better at afs than say compared to nfs?

Maybe this question is probably making people not go into look at afs much?

----------

## baloo12

I followed your manual with great success!

it runs perfect..

the only thing i did after emerging & editing ThisCell was setting up a few links..

like tokens, klog, fs and so on..

thank you..

good work!

----------

## irondog

 *Hideki wrote:*   

> Maybe I should simply rtfm but what is better at afs than say compared to nfs?
> 
> Maybe this question is probably making people not go into look at afs much?

 

AFS offers a few advantages above nfs:

* AFS runs on a wide range of platforms.

* AFS has a cache manager. When using AFS over a slow connection the advantages of the cache manager are big. All files fetched from a server can be stored in the cache. Fetching an unchanged file (I.E. the second time) is fast and there is almost no traffic to the real server hosting the file.

* AFS tolerates fall down of servers and does load balancing. When a volume is hosted on more than one server, the volume is accesible even if some servers are down at the moment of the client's request. AFS keeps track of the availability of servers.

 *baloo12 wrote:*   

> 
> 
> the only thing i did after emerging & editing ThisCell was setting up a few links..
> 
> like tokens, klog, fs and so on..
> ...

 

Thanks  :Smile: . I understand you could have some questions about the correctness of the paths certain executables located at.

I think it's not appropriate to place the fs executable somewhere in $PATH. 

Of tokens and klog it might be an idea to place them in /usr/sbin. I didn't have enough reasons to decide to place certain files outside of /usr/afs/bin (and into /usr/sbin). 

From what I've seen there have always been differences between AFS documentation and the available packages. I'll update the ebuild once someone has given me good reasons to change things.

Tnx again.  :Cool: 

----------

## KermitTheFragger

Nice guide, I did some fooling arround with AFS and Kerberos V couple of months back myself. Unfortunatly I havent been very busy with AFS lately. Do you have any estimate when 1.4 is going to be released? I really, really need a stable release of AFS compatible with the 2.6 kernel (Or a nice stable NFSv4, whichever comes first, I prefer AFS tough). NFSv3 really stinks on all fronts; Security (which is practicly non-existing), no clientside caching, etc.

----------

## drakkan

Hi,

I followed your howto, seems it works:

```

root@box admin # /usr/afs/bin/bos status box -noauth

Instance kaserver, currently running normally.

Instance buserver, currently running normally.

Instance ptserver, currently running normally.

Instance vlserver, currently running normally.

Instance fs, currently running normally.

    Auxiliary status is: file server running.

Instance upserver, currently running normally.

```

but I have the following error:

```

root@box admin # /usr/afs/bin/klog admin

Password:

root@box admin # ls /afs/mycell/

ls: /afs/mycell/: Permission denied

```

what's wrong?   :Sad: 

----------

## irondog

 *KermitTheFragger wrote:*   

>  Do you have any estimate when 1.4 is going to be released?

  The release cycle of the 1.3.x series has been very short. The latest verions followed each other with less than a month time between it.

 *latest announcement wrote:*   

> For UNIX, 1.3.82 is the latest version in the 1.4 release cycle.
> 
> Notable recent improvements are included in AIX 5 client support, Linux 
> 
> 2.6 client support, and Rx free packet handling in the fileserver.

 Reading this, I expect I won't take more than a few months before 1.4 is released. But I don't know. For me the 1.3.82 version isn't less stable than the 1.2.x versions.

 *Quote:*   

> 
> 
> ```
> root@box admin # /usr/afs/bin/klog admin
> 
> ...

 

What's the output of:

/usr/afs/bin/fs lsmount /afs/mycell

/usr/afs/bin/fs listacl /afs/mycell

/usr/afs/bin/tokens

/usr/afs/bin/vos listvldb

----------

## KermitTheFragger

 *irondog wrote:*   

>  The release cycle of the 1.3.x series has been very short. The latest verions followed each other with less than a month time between it.
> 
> 

 

Yeah I noticed that too, but this short release cycle has been going on for a couple of months. Yeah I know, I'm like a kid: I wanne, I wanne and I wanne itta now  :Very Happy: 

 *irondog wrote:*   

> 
> 
> Reading this, I expect I won't take more than a few months before 1.4 is released. But I don't know. For me the 1.3.82 version isn't less stable than the 1.2.x versions. 
> 
> 

 

Hmm, sounds nice. I think I'll take the latest release for a spin later this week.

----------

## zecora

So is this kind of like mounting network drives?  

Like i want to set this up on my fileserver and then have my windows boxes be able to view the drives.

like if i install this, do i need samba?

----------

## KermitTheFragger

 *Quote:*   

> 
> 
> So is this kind of like mounting network drives? 
> 
> 

 

Blasphemy !!  :Very Happy: 

AFS is much more then just a network filesystem, its a distributed network filesystem with sophisticated security and replication options.

 *Quote:*   

> 
> 
> Like i want to set this up on my fileserver and then have my windows boxes be able to view the drives.
> 
> like if i install this, do i need samba?
> ...

 

No you dont need samba for that. Just download the latest windows client release:

http://www.openafs.org/release/latest.html

----------

## drakkan

 *Quote:*   

> 
> 
> What's the output of:
> 
> /usr/afs/bin/fs lsmount /afs/mycell
> ...

 

Great howto!!!!!

I have reinstalled and now the first server works fine!!!

Can you explain more about clustering, please?

For example howto add a second server for failover and load balancing

thanks

P.S. in my setup I'm using xfs for vicepa instead of ext2 for now it works

----------

## irondog

 *Quote:*   

> No you dont need samba for that. Just download the latest windows client release:
> 
> http://www.openafs.org/release/latest.html

 Yep, but if you like you could also run samba on top of AFS.

 *Quote:*   

> Great howto!!!!!
> 
> I have reinstalled and now the first server works fine!!!
> 
> Can you explain more about clustering, please?
> ...

  Follow the same steps (listed above) on each machine. Once the machine is running make sure the timezone and clocks of both servers are in sync. Edit /etc/hosts and /etc/afs/CellServDB to have both servers in the server database. /etc/afs/CellServDB files must be identical on all machines!!.

Restart both servers and use:

```
/urs/afs/bin/vos syncserv

/usr/afs/bin/vos syncvldb 
```

to synchonise the global Volume location database.

Use /usr/afs/bin/vos addsite to create a replication site for a volume.

 *Quote:*   

> thanks
> 
> P.S. in my setup I'm using xfs for vicepa instead of ext2 for now it works

 Yep, might work. It is has been reported to cause problems, which might never occur for you.

----------

## KermitTheFragger

XFS in combination with AFS and a crash could become real messy:

 *SGI XFS FAQ wrote:*   

> 
> 
> Q: Why do I see binary NULLS in some files after recovery when I unplugged the power?
> 
> XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.
> ...

 

9 out of 10 times xfs_bmap can fix your problems. That 1 out of 10 times can become a real pain. Also i have no idea how AFS would react if you start runnen bmap on files on its vicepa partition. Perhaps it would be a good idea to mount the xfs vicepa partition with a synchronous write option. However this would impact your performace (not in a good way).

----------

## zecora

I am gonna drop samba and use this.  Sounds sweet  :Smile: 

----------

## Mango

I have also been looking for AFS for a while, I have a Client that have to locations and this solution would be perfect for him. Right now I am using Availl running under Windows 2000 Server, with some problems. I have thought about using DFS that comes with Windows Server 2003 but I am more interested in AFS under Gentoo.

1) Running Samba on top of AFS? Wont you loose some of the benefits from AFS like cache and speed?

2) How is the speed compared to Samba?

3) Have anyone any idea about how Windows DFS works compared to AFS?

----------

## nephros

 *irondog wrote:*   

> Fucking nobody interested in this guide? Come on.

 

I have bookmarked this for the point when I own Gentoo boxes in more than one continent (or heck, more than 5 meters apart!  :Smile:  ).

Thanks a lot for posting it!  :Cool: 

----------

## baloo12

big thanks!

----------

## comdata

thanx for this post and the ebuilds.

I got the first server setup very quickly with this guide, but now I am stuck in setting up additional filespace and a replicated second fileserver. any help or advice for good documentation would be nice.

--

mfg

comdata

----------

## zecora

Crashed when i tried to emerge it.

```
Vegeta ~ # emerge -av openafs

These are the packages that I would merge, in order:

Calculating dependencies ...done!

[ebuild  N    ] net-fs/openafs-1.2.11  -debug 0 kB

Total size of downloads: 0 kB

Do you want me to merge these packages? [Yes/No] y

>>> emerge (1 of 1) net-fs/openafs-1.2.11 to /

>>> md5 files   ;-) openafs-1.2.10-r1.ebuild

>>> md5 files   ;-) openafs-1.2.10-r2.ebuild

>>> md5 files   ;-) openafs-1.2.10.ebuild

>>> md5 files   ;-) openafs-1.2.11.ebuild

>>> md5 files   ;-) files/openafs-1.2.6.patch

>>> md5 files   ;-) files/CellServDB

>>> md5 files   ;-) files/README

>>> md5 files   ;-) files/ThisCell

>>> md5 files   ;-) files/afs.rc.rc6

>>> md5 files   ;-) files/openafs-pinstall-execve.patch

>>> md5 files   ;-) files/digest-openafs-1.2.10

>>> md5 files   ;-) files/digest-openafs-1.2.10-r1

>>> md5 files   ;-) files/digest-openafs-1.2.10-r2

>>> md5 files   ;-) files/digest-openafs-1.2.11

>>> md5 files   ;-) files/openafs-pinstall-execve-1.2.10.patch

>>> md5 files   ;-) files/openafs-pinstall-execve-env.patch

>>> md5 files   ;-) files/openafs-typechange.patch

>>> md5 src_uri ;-) openafs-1.2.11-src.tar.bz2

!!! ERROR: net-fs/openafs-1.2.11 failed.

!!! Function pkg_setup, Line 25, Exitcode 0

!!! OpenAFS does not yet support 2.5 and 2.6 kernels

!!! If you need support, post the topmost build error, NOT this status message.

Vegeta ~ #
```

----------

## mr-simon

 *zecora wrote:*   

> Crashed when i tried to emerge it.

 

Looks like it's trying to merge the portage version.

Did you follow the steps above about downloading the overlay?

Great guide, BTW irondog. This really helped me out.

----------

## irondog

 *zecora wrote:*   

> Crashed when i tried to emerge it.

  Note that Openafs>=1.3 has been hard masked.

So, the overlay is only used when you unmask it in /etc/portage/package.unmask

 *mr-simon wrote:*   

> Great guide, BTW irondog. This really helped me out.

 NP.

 *comdata wrote:*   

> thanx for this post and the ebuilds.

 NP.

 *Quote:*   

> I got the first server setup very quickly with this guide, but now I am stuck in setting up additional filespace and a replicated second fileserver. any help or advice for good documentation would be nice.

  Read the docs. You have to spend much time in OpenAFS before knowing all ins and outs.

 *Mango wrote:*   

> 
> 
> 1) Running Samba on top of AFS? Wont you loose some of the benefits from AFS like cache and speed?
> 
> 2) How is the speed compared to Samba?
> ...

 

1) I'm running Samba on top of my AFS clients. It provides locking through a patch I've written. So, people opening a MS Word document in Amsterdam prevent users in Rotterdam to open it. Works really nice! Check it out:

http://tienstra4.flatnet.tudelft.nl/~gerte/samba3-afsexclusivelock-fcntl.diff

2) Speed is OK. It's fast. But I've some troubles with big files.

3) DFS has same kind of features. What DFS misses is the cache manager.

----------

## zecora

SO what do i need to do?

```

# mkdir -p /etc/portage

# echo 'net-fs/openafs-kernel ~x86' >> /etc/portage/package.keywords

# echo 'net-fs/openafs ~x86' >> /etc/portage/package.keywords

# echo 'net-fs/openafs' >> /etc/portage/package.unmask

# emerge -av openafs
```

Do you mean those commands?

If So I followed the instuctions.

----------

## mr-simon

No. These commands:

* Configure portage to use overlay

/etc/make.conf

```
PORTDIR_OVERLAY="/usr/local/portage"
```

* Download and extract the OpenAFS overlay tarball from bugzilla/my website

```
# mkdir -p /usr/local/portage

# cd /usr/local/portage

# wget http://tienstra4.flatnet.tudelft.nl/~gerte/openafs-overlay-gerte.tar.gz

# tar -zxvf openafs-overlay-gerte.tar.gz 
```

----------

## mr-simon

Question: The openafs guide recommends using the uss tool for adding and removing users. docs here - It doesn't appear to be part of the openafs package... Or at least it's not installed by the ebuild.

Where/how can I get it?

----------

## mr-simon

 *mr-simon wrote:*   

> Question: The openafs guide recommends using the uss tool for adding and removing users. docs here - It doesn't appear to be part of the openafs package... Or at least it's not installed by the ebuild.
> 
> Where/how can I get it?

 

Uh... To answer my own post:

You can get it by running 

```
# ebuild /usr/local/portage/net-fs/openafs/openafs-1.3.82.ebuild compile

... output happens ...

# cp /var/tmp/portage/openafs-1.3.82/work/openafs-1.3.82/src/uss/uss /usr/afs/bin
```

I guess this should be in the ebuild?

----------

## irondog

New version of the Overlay tarball:

http://tienstra4.flatnet.tudelft.nl/~gerte/openafs-overlay-gerte-1.3.84.tar.gz

There is no need to upgrade from earlier versions, but it should be safe!

I need testers on SMP and x86_64 systems, please report results.

Changelog:

```

* Bumped from openafs 1.3.82 to 1.3.84

openafs-kernel ebuild:

  * Changed keywords

  * Determine SYSNAME in pkg_setup

  * Make modprobe work, old behaviour is still supported

  * Also Install kernel module in /lib/modules/`uname -r`/misc

openafs ebuild:

  * Make use of toolchain-funcs eclass instead of gcc eclass

  * Make use linux-info eclass instead of linux-mod eclass

  * Changed keywords

  * RESTRICT="nostrip" avoids annoying warnings

  * Determine SYSNAME in pkg_setup

  * Included /usr/afs/bin/uss

/etc/init.d/afs:

  * Make RC-script use modprobe, old behaviour is still supported

/etc/conf.d/afs:

  * Have RC script use modprobe by default, old behaviour as fallback

```

I've found some other changed ebuilds by Michael Hordijk on Bugzilla. These are not mine. I'll try to adopt the goods things he made, but I'm not so happy with the installation of many useless files. Some even collide with other packages. I'm not happy with the non-transarc defaults he uses either. Things should first become stable and well tested before exploiting it. Thanx anyway.

----------

## depontius

I just set up using your new overlay tarball. It built and starts just fine, but I have one suggestion, and one BIG problem.

Suggestion:

Add a file, "/etc/env.d/50afs" with contents: "PATH=/usr/afs/bin", adding the same for ROOTPATH is optional. This puts the afs utilities on the path, makes life easier. I notice that this is done for the regular ebuilds.

Problem:

I'm part of a BIG multi-cell company. Only problem, a bunch of these are symlinks, so we can get to cells by shortnames. So a cell of the form: "/afs/(site).(company).com" will typically have a second entry of the form: "/afs/(site)". At least, that's the way it looks on every other afs client I've ever used. The just-installed client for kernel 2.6 shows none of the short links. It becomes a problem because there is a LOT of use of symlinks, and a LOT of those link to the short form for a site, instead of the long form. For my new client install, ALL of those links are broken, because the short form doesn't exist. I could see cleaning up my own stuff to use the long form, but I don't own most of what I'm having problems with. I'm using the same CellServDB between clients running 2.4 and 2.6, as well as other OSs, but under 2.6 is the only place I don't have the short cell names. It looks to me like some aspect of client configuration automatically creates the names, but I don't know what it is, nor have I found anything in the FAQ or Wiki. A quick scan comparing your ebuilds with the regular one shows nothing obvious. A while back, a friend built OpenAFS for 2.6 on Gentoo, and didn't mention this problem, though I have no idea how thoroughly he tested.

The problem was DYNROOT=yes in the configuration file. Evidently all of those handy aliases are exported by the server, and DYNROOT tells the client to ignore the names exported, and build its own. I had thought that the client dynamically built the short names, but apparently they're built at the server.

----------

## irondog

Hm, maybe this should be set to default. I was also _missing_ it when switching from 2.4 to 2.6.

----------

## depontius

I guess I should be more helpful, and try to enumerate things better. I guess these are really changes from the stable ebuild, and some of them need to be brought forward to this ebuild.

1: No DYNROOT by default. (defaults to "no")

2: An entry for /etc/env.d to add afs binaries to the PATH

3: Standard place for afs binaries is "/usr/afsws", this ebuild places them in "/usr/afs". (more on this, next)

4: Rebuilding "findtools" (now that it has the +afs flag) fails, because it's expecting to find "/usr/afsws/lib/pam_afs.so.1". Not only is the "/usr/afsws" path part changed, but "lib" isn't there, and "pam_afs.so.1" is only in /lib/security.

To workaround these:

1: Change to DYNROOT=no, or just comment it out.

2: I made /etc/env.d/50afs with contents of PATH=/usr/afs/bin and ROOTPATH with the same.

3: I made a symlink from /usr/afs to /usr/afsws.

4: I made a new directory, /usr/afs/lib, and in that directory symlinked /lib/security/pam_afs.so.1 in.

Then I had really terrible performance after that. But that's because our site has some brain-dead networking that "works for Windows". Autonegotiation is turned off by default. Static IP workstations are configured to 100FD. Windows PCs think that they're autonegotiating, and somehow come up with 100FD. Linux tries to autonegotiate, and comes up with 100HD. I now have an mii-tool line in local.start to force the card to 100FD, since my old kernel parameters no longer work.

What's really annoying is that a call to support, and after some whining, they'll change your port to auto-negotiate. I absolutely NEED that for a laptop, for instance. But it ought to be default, IMHO.

----------

## zecora

```
 * Please upgrade your package (openafs-1.3.82) to use toolchain-funcs.eclass
```

how do i update?

----------

## irondog

That's fixed in the 1.3.84 ebuilds.

----------

## zecora

So just get the newest ebuilds and i should be alright?

----------

## kappax

Anybody know if there is a network file system that is not so much of a cluster, but tolorent for cached reads and writes ?

Such as Server A = webserver and Server B = fileserver If B goes down I would like to still make writes to the fileserver and have them populated back to the fileserver when it comes back up. And such reads form B wold work so long as A had cached them.   

I don't want A to have to hold all of the files that B has, eg B is 6 tb of data *multy disk array* ,  A only has a 400gig dids.

----------

## zecora

```

>>> original instance of package unmerged safely.

 * After installing a new kernel of any version, it is important

 * that you have the appropriate /etc/modules.autoload.d/kernel-X.Y

 * created (X.Y is the first 2 parts of your new kernel version)

 * For example, this kernel will require:

 * /etc/modules.autoload.d/kernel-2.6

 * If you are upgrading from a previous kernel, you may be interested

 * in the following documents:

 *   - General upgrade guide: http://www.gentoo.org/doc/en/kernel-upgrade.xml

 *   - 2.4 to 2.6 migration guide: http://www.gentoo.org/doc/en/migration-to-2.6.xml

 * For more info on this patchset, and how to report problems, see:

 * http://dev.gentoo.org/~dsd/genpatches

>>> Regenerating /etc/ld.so.cache...

>>> sys-kernel/gentoo-sources-2.6.12-r4 merged.

>>> clean: No packages selected for removal.

>>> emerge (132 of 149) net-fs/openafs-kernel-1.3.82 to /

>>> md5 files   ;-) openafs-kernel-1.3.82.ebuild

>>> md5 files   ;-) files/digest-openafs-kernel-1.3.82

>>> md5 src_uri ;-) openafs-1.3.82-src.tar.bz2

 * OpenAFS might cause kernel OOps

 * You have been warned!

 * Determining the location of the kernel source code

 * Found kernel source directory:

 *     /usr/src/linux

 * Found sources for kernel version:

 *     2.6.12-gentoo-r4

 * getfilevar requires 2 variables, with the second a valid file.

 *    getfilevar <VARIABLE> <CONFIGFILE>

 * Could not find a usable .config in the kernel source directory.

 * Please ensure that /usr/src/linux points to a configured set of Linux sources.

 * If you are using KBUILD_OUTPUT, please set the environment var so that

 * it points to the necessary object directory so that it might find .config.

!!! ERROR: net-fs/openafs-kernel-1.3.82 failed.

!!! Function linux-info_pkg_setup, Line 521, Exitcode 1

!!! Unable to calculate Linux Kernel version

!!! If you need support, post the topmost build error, NOT this status message.
```

what the hell?

----------

## depontius

Have you actually built the 2.6.12-r4 kernel yet?

Did you boot it when you tried to build OpenAFS?

Was the kernel source symlink correctly in place?

You might also want to get the newer 1.3.84 ebuild instead of the 1.3.82 one.

----------

## adsmith

I'd love to set this up, but I really do not want to muck around with my partitions.

Can the "partition" be a loopback device?

----------

## irondog

Actually, yes. I tried it and it even worked for /vicepa. I don't recall the additional steps I did, but yes I've had it working.

----------

## moja

Is there a way to authenicate the user of the afs through kerberos and samba?

I have both the samba and unix users in a central openldap directory. I want to be able to get the afs token then I log in to the system, but I dont want to create all the users in the AFS userdatabase. 

IS there a way to do this?  :Rolling Eyes: 

----------

## jamiethehutt

 *Quote:*   

> * For servers a ext2 partition mounted on /vicepa is needed (yes, stop here if you haven't a spare partition) 

 dd if=/dev/urandom of=/partition bs=1M count=1000

losetup /dev/loop0 /partition

mkfs -t ext2 /dev/loop0

mount /dev/loop0 /vicepa

I've not tested it but that should give you a 1GB partition.  :Wink: 

----------

## heini

Hi,

I tried the 1.3.85 ebuilds which are in portage now, but I have trouble starting afsd. I get the following error message:

```
> /etc/init.d/afs start

 * Starting AFS BOS server ...                                                 [ ok ]

 * Starting AFS client ...

afsd: All AFS daemons started.

afsd: Can't mount AFS on /afs(22)

```

/afs does exist. I authenticate against MIT Kerberos, but I don't think this is an issue here. My kernel is 2.6.12.3.

Anybody seen this before?

Thanx...

Dirk

----------

## irondog

Yes, but I don't remember anymore how to solve it. I've written my ebuilds very carefully to avoid problems like this. And I can tell you this was one of the problems I found a solution for. Unfortunately the creator of the ebuilds in portage didn't use much of my work.  :Sad: 

Building openAFS is not so tricky, but installing really is!

I added this in my ebuild's:

```

    keepdir /usr/afs/db

    keepdir /usr/afs/local

    keepdir /usr/afs/logs

    keepdir /usr/vice/cache

```

You might want to mkdir these directories and try again, but I can't give you any garantuee's this is the solution for your problem.

----------

## depontius

 *heini wrote:*   

> Hi,
> 
> I tried the 1.3.85 ebuilds which are in portage now, but I have trouble starting afsd. I get the following error message:
> 
> ```
> ...

 

I recently reinstalled a machine, and tried the 1.3.85 build, with disastrous results.

My suggestion: Install Irondog's 1.3.84 ebuild and get it running. I had to diddle a little with /usr/afs vs /usr/afsws and the location of pam_afs.so, but that was about it. Oh, I also had to turn dynroot off, for the way our enterprise is set up. But for one reasonably familiar with afs, it came up fairly easily.

Then decide if you want to move to the official 1.3.85 ebuild. First off, it doesn't look to me as if 1.3.85 is capable of a "cold" install. But it will upgrade a 1.3.84 install fairly readily, and that's how I got it running on the one machine where I use it. I had to add a CACHESIZE=(my cachesize) parameter to /etc/conf.d/afs, and for some odd reason had to create the /afs mount point. But now I have one machine running 1.3.84 and one running 1.3.85.

----------

## heini

First, thanks for the answers.

Meanwhile I've come a little bit further: I did a fresh install of the 1.3.85 ebuild after removing every trace of openafs from my machine (did some attempts earlier with 1.3.[67]x).

I'm now at the point that afs is starting up normally, but

```
=> fs setacl /afs system:anyuser rl

fs: You don't have the required access rights on '/afs'
```

I obtained kerberos ticket and afs token and the user is in system:administrators.

Has anybody seen/solved this?

Bye...

Dirk

```

```

----------

## depontius

 *heini wrote:*   

> First, thanks for the answers.
> 
> Meanwhile I've come a little bit further: I did a fresh install of the 1.3.85 ebuild after removing every trace of openafs from my machine (did some attempts earlier with 1.3.[67]x).
> 
> I'm now at the point that afs is starting up normally, but
> ...

 

What does "fs la /afs" say?

----------

## heini

 *heini wrote:*   

> First, thanks for the answers.
> 
> Meanwhile I've come a little bit further: I did a fresh install of the 1.3.85 ebuild after removing every trace of openafs from my machine (did some attempts earlier with 1.3.[67]x).
> 
> I'm now at the point that afs is starting up normally, but
> ...

 

I was able to solve it myself. Since I use MIT Kerberos 5, I needed to take special care to setup /etc/openafs/server/KeyFile correctly, using asetkey from the afs-krb5 migration kit.  So I started again, compiled asetkey before I did anything else, then followed the steps in http://www.seismo.ethz.ch/linux/afs/node6.html and everything is working fine now.

Maybe I'll put together just another HOWTO.

Bye...

Dirk

----------

## baloo12

I used your afs-install-methode on an other server before..

It was perfect.. but know someting strange is going on..

When I start /etc/init.d/afs, there are several error messages/warnings about missing files in /etc/openafs..

```

/etc/init.d/afs start

 * Starting AFS client ...

/etc/init.d/afs: line 12: [: -lt: unary operator expected

/etc/init.d/afs: line 14: [: -lt: unary operator expected

/etc/init.d/afs: line 16: [: -lt: unary operator expected

/etc/init.d/afs: line 18: [: -lt: unary operator expected

afsd: some file missing or bad in /etc/openafs

```

I created a testwise /etc/openafs/ with a ThisCell and a CellServDB (with appropriate content)

after that, it was asking for a the cacheinfo file:

```

 * Starting AFS client ...

/etc/init.d/afs: line 12: [: -lt: unary operator expected

/etc/init.d/afs: line 14: [: -lt: unary operator expected

/etc/init.d/afs: line 16: [: -lt: unary operator expected

/etc/init.d/afs: line 18: [: -lt: unary operator expected

ParseCacheInfoFile: Can't read cache info file '/etc/openafs/cacheinfo'

```

I don't know its content.. so..

I didn't need to modify any files (except ThisCell/CellServDB) last time..?!

any ideas?

----------

## depontius

 *baloo12 wrote:*   

> I used your afs-install-methode on an other server before..
> 
> It was perfect.. but know someting strange is going on..
> 
> When I start /etc/init.d/afs, there are several error messages/warnings about missing files in /etc/openafs..
> ...

 

/etc/conf.d/afs needs a "CACHESIZE=nnn" parameter tacked onto the end, or at least mine did. I'm in a sort of discussion with the author of the 1.3.85 ebuild about installation. Actually he's responded to my bug report, and I owe him a reply.

----------

## @zr@el

First I want to thank you for your guide, irondog.

I'm studying computer science and we decided to use OpenAFS for our student's /home directories in my university; of course under Gentoo.  :Wink: 

Based on your guide, we were able to build a gentoo-openafs-server, which was fully functional. ( /afs tree, gettting access from the clients, etc. ...)

However we aren't able to use the "afs-homedirectories" because graphical login managers, like gdm / kdm, aren't able to write on users home directories. :Crying or Very sad: 

Graphical login managers want to write .dotfiles and other config-files in the users home directory, but they have no permission to write on the home directory, because of the ACLs defined on the users home directory and the lack of authentication of user 'root'.

The only way described in various documents (google search), was to allow write access for the group system:anyuser on a special public folder under the home directory and make symbolic links in the users home directory pointing to the writeable files in the public directory. But this solution isn't very comfortable and seems more like a 'hack' then a real solution.

Anybody got an idea to solve this problem?

----------

## heini

 *@zr@el wrote:*   

> However we aren't able to use the "afs-homedirectories" because graphical login managers, like gdm / kdm, aren't able to write on users home directories.
> 
> Graphical login managers want to write .dotfiles and other config-files in the users home directory, but they have no permission to write on the home directory, because of the ACLs defined on the users home directory and the lack of authentication of user 'root'.
> 
> Anybody got an idea to solve this problem?

 

First, you need a pam module which gets AFS tokens at login time. I use pam_openafs_session (Note: I use MIT Kerberos V for authentification, so you may need another one), but this is not in portage, you have to get and compile it yourself (Google).

However, the README of this module said that the only Display Manager that would work with it was wdm. So I emerged it, changed DISPLAYMANAGER in /etc/rc.conf from kdm to wdm and everything worked fine.

HTH...

Dirk

----------

## @zr@el

Thanks for the quick answer.

The authentication is done via the pam_krb5 against a Windows 2003 Active Directory Server, because our admins want a centralized user administration with Active Directory Services. So there is no Kerberos on the linux side.  :Sad: 

But authentication works and AFS uses the token obtained from the pam_krb5 module to grant the user 'xyz' access to his home directory /afs/cellname/home/xyz. However the user 'root' can't authenticate against AFS, which is ok for security reasons. But the graphical login managers are executed as 'root' and have no permissions to write on the home directory of user 'xyz'.

 *Quote:*   

> 
> 
> So I emerged it, changed DISPLAYMANAGER in /etc/rc.conf from kdm to wdm and everything worked fine. 
> 
> 

 

Did you try to run KDE or GNOME with wdm?

----------

## heini

 *@zr@el wrote:*   

> 
> 
> The authentication is done via the pam_krb5 against a Windows 2003 Active Directory Server, because our admins want a centralized user administration with Active Directory Services. So there is no Kerberos on the linux side. 

 

Kerberos 5 is Kerberos 5, no matter if served from Windows or Linux (well, sort of  :Smile:  ). But that may be the cause of the problem. I also use pam_krb5 module for authentication, but it cannot get afs tokens from krb5 tickets. It still needs krb4 tickets!!!

This is the reason I use pam_openafs_session in addition to pam_krb5.

 *Quote:*   

> But authentication works and AFS uses the token obtained from the pam_krb5 module to grant the user 'xyz' access to his home directory /afs/cellname/home/xyz.

 

Did you verify this by logging in on a text console and issue a "tokens" command right after login?

 *Quote:*   

> However the user 'root' can't authenticate against AFS, which is ok for security reasons. But the graphical login managers are executed as 'root' and have no permissions to write on the home directory of user 'xyz'.
> 
> 

 

I doubt that any display manager would write to $HOME as user root. It may run as root, that's correct. But it changes identity to the user who is logging in right after successful authentification.

You can verify this by by doing

```
find $HOME -uid 0
```

It should find nothing.

 *Quote:*   

>  *Quote:*   
> 
> So I emerged it, changed DISPLAYMANAGER in /etc/rc.conf from kdm to wdm and everything worked fine. 
> 
>  
> ...

 

Yes, with KDE. It works just fine.

Bye...

Dirk

----------

## heini

A quick update on the pam_krb5 issue:

I tried version 2.1.8 from Fedora and this one seems to work fine in my setup (MIT KerberosV 1.4.1, Krb 4 disabled, users homedirs in AFS, OpenAFS 1.3.85), which means it gets AFS tokens from the Kerberos V tickets.

I have submitted an ebuild to bugzilla: https://bugs.gentoo.org/show_bug.cgi?id=103406.

This also means I don't need pam_openafs_session anymore and I could switch back from wdm to kdm.

Bye...

     Dirk

----------

## mr-simon

There are ebuilds for openafs 1.3.85 in portage. How do these compare to the ones here? I tried to use them but everything turned out very... er... different... and not working.

----------

## heini

 *mr-simon wrote:*   

> There are ebuilds for openafs 1.3.85 in portage. How do these compare to the ones here? I tried to use them but everything turned out very... er... different... and not working.

 

They work fine for me. Could you tell us what is not working?

Bye...

     Dirk

----------

## stefaan

mr-simon: they are different, but i like thinking it's not a bad thing at all.

The main difference is the adoption of FHS-paths, as is usually done throughout all of Gentoo.  The ebuild should normally take care of most of the transition, but it could be there's bugs.  Also, the transition code doesn't delete the old files (for safety reasons), so you may want to clean up afterwards (yes, this should be in the documentation, but as the ebuilds are still a moving target, documentation isn't quite underway yet).  So from now on everything in /usr/bin, /usr/sbin, /var/lib/openafs, ...  

Other changes include splitting the init-script into a client and a server one, not checking whether your cache-fs is on ext2 (for various reasons), not checking whether your /vicepx dirs are ext2-mounted (for like reasons), ...

Let me know how they work out for you, or if you need any help.  Every comment brings us one step closer to a stable 1.4-ebuild!!

stefaanLast edited by stefaan on Tue Oct 04, 2005 1:11 pm; edited 1 time in total

----------

## fnjordy

Needs to be updated for the new path layout, and the ebuild to 1.4.0 rc5.   And please split /etc/init.d/afs into two scripts, one for the client, and one for the server, the binaries are different, and nfs is split into two scripts so whats up with afs (apart from upstream issues).

----------

## heini

 *fnjordy wrote:*   

> Needs to be updated for the new path layout, and the ebuild to 1.4.0 rc5.

 

RC6, please  :Smile: 

 *Quote:*   

> And please split /etc/init.d/afs into two scripts, one for the client, and one for the server.

 

Has been done long time ago.

Bye...

Dirk

----------

## stefaan

I'm waiting to push out a new ebuild, because I'm working on updated documentation. It is really needed, as the upgrade procedure is currently undocumented, and the paths in the old documentation are wrong now.  Writing documentation proves to be a difficult task however, and I'm currently on a very constrained time-budget   :Sad: 

I hope to get something ready soon, in the meantime, you can just bump locally (to rc5 at least, haven't tried rc6 yet).

Cheers all!

Stefaan

----------

## stefaan

I've put a preliminary version of the newer documentation at

http://dev.gentoo.org/~stefaan/prerelease/openafs-guide/guide.html

Only the chapter about upgrading from pre-1.4 is worth considering.  It explains different changes in the setup since the old 1.2.10 (though I'm not stating version numbers where the changes occurred, maybe that should be on my todo list).  In case you think you may have missed some details about the upgrade, it might be worth reading. Any feedback is appreciated  :Smile: 

Stefaan

----------

## KermitTheFragger

 *stefaan wrote:*   

> I've put a preliminary version of the newer documentation at
> 
> http://dev.gentoo.org/~stefaan/prerelease/openafs-guide/guide.html
> 
> Only the chapter about upgrading from pre-1.4 is worth considering.  It explains different changes in the setup since the old 1.2.10 (though I'm not stating version numbers where the changes occurred, maybe that should be on my todo list).  In case you think you may have missed some details about the upgrade, it might be worth reading. Any feedback is appreciated 
> ...

 

Looking good  :Smile: 

Are you going to include a chapter on how to use Heimdal or MIT Kerberos 5 instead of the krb4 daemon (kaserver) shipping with AFS?

----------

## depontius

 *stefaan wrote:*   

> I've put a preliminary version of the newer documentation at
> 
> http://dev.gentoo.org/~stefaan/prerelease/openafs-guide/guide.html
> 
> Only the chapter about upgrading from pre-1.4 is worth considering.  It explains different changes in the setup since the old 1.2.10 (though I'm not stating version numbers where the changes occurred, maybe that should be on my todo list).  In case you think you may have missed some details about the upgrade, it might be worth reading. Any feedback is appreciated 
> ...

 

I like it. I wish I'd had it the first time I tried a 1.3.85 install. (Too many years of the Transarc way.)

----------

## rwallace

Some documentation on setting it up with mit-krb5 would be absolutely awesome.  So here's my vote for that!

----------

## fnjordy

I've updated Amanda 2.4.5-r1 and Amanda AFS 0.0.4.  I guess need some better docs on them.

----------

## brenden

Cool.  Thanks for this.  Should be updated to reflect the new ebuilds in portage however.

----------

## stefaan

The new Gentoo OpenAFS documentation has been put online, not too many changes since my last proposal though (busy busy busy).  The 1.4.0-ebuild will follow shortly.

 *KermitTheFragger wrote:*   

> Are you going to include a chapter on how to use Heimdal or MIT Kerberos 5 instead of the krb4 daemon (kaserver) shipping with AFS?

 

I would like to. I'll see next week if I can spare some time to do that.

If you see errors in the documentation, please do report.  It'd be nice to have the clearest possible manual online when 1.4.0 hits stable on gentoo.

----------

