The BRF operates a large compute infrastructure to assist in solving the data analysis problems arising in the CeBiTec. Typical tasks are fuzzy searches in large files, trying to figure out similarities in different organisms. These searches usually can be carried out in parallel, so a cluster of looseley interconnected nodes is a convenient approach here. The search algorithms themselves often use matrix operations, which can be modelled in hardware in a very efficient manner. Therefore, the BRF has installed some special purpose machines, running important operations in hardware.
The BRF compute cluster currently consists of approx. 770 CPUs, which deliver a total of 3750 Cores. Compute capacity is roughly 23Tflop. A very important part of the cluster hardware is its network connectivity based on a stack of interconnected switches, which are in turn uplinked via 10GB ethernet to the redundant filers. The filers offer a total storage capacity of 433Tb, backed up by 1290Tb of tape.
The scheduling system used at CeBiTec is Oracle Grid Engine - OGE (formerly known as Sun Grid Engine - SGE), most applications use the DRMAA interface to OGE, so job submission is quite portable.
Batch / Interactive Queues
To grant access to the appropriate resources, a system of various queues has been configured in OGE. The most important queue classes are 1.) Interactive queues, which are used through a special shell wrapper and offer (as the name predicts) interactive, shell based access to some of the compute hosts. 2.) batch queues, which are the fallback for all jobs submitted without any parameters. Jobs submitted here are distributed among most of the hosts, primarily on a first come/first serve basis. There is a number of other queue classes for jobs with other profiles, like long running jobs or jobs which are serving a webinterface, but these queue classes have to be specified at job submission time, and not every user is allowed to use each queue class.
Currently, the BRF operates three different architectures of specialized hardware for Life Science applications. We are using 4 TimeLogic DeCypher machines, equipped with a total of 12 DeCypher cards. The TimeLogic systems offer their own implementations of several bioinformatics applications with a significant (up to 300x ) speedup compared to general purpose intel machines. The applications are available as well documented packages. Furthermore, 3 machines are equippped wiith NVIDIA GPUs, where CuDA software, developed by the BRF is compiled and run. Last but not least we are using one Convey HC1ex hybrid core system. This is a system where an FPGA-Card is used as coprocessor. In contrast to the DeCypher systems this means that the FPGA can access the system memory with the same bandwith as the CPU does. For this system we utilize the precompiled packages for bioinformatics applications but we have also started to develop our own software tools.