[Bio-Linux] New cluster

[Tony Travis]

On 11/08/14 15:56, Alec Colebrook-Clark wrote:
> Hi Tim et all,
>
>
>
> We are looking at expanding our biolinux infrastructure and implementing
> a biolinux cluster. The first question I have is would it be better to
> use a virtual infrastructure than a physical? A colleague has suggested
> that there is a loss of power with the virtual solution but does it
> noticeably affect the work being done on biolinux?

Hi, Alec.

That depends what your objectives are: Aggregating the resources of
multiple machines into a traditional HPC Beowulf cluster or providing
virtual equivalents of individual workstations in a private cloud.

I ran a Beowulf as a Bio-Linux 7 "x2go" terminal server using "Onesis"
for stateless node provision:

  http://onesis.org/

I also took part in an OpenStack private cloud pilot at the University
of Edinburgh and before that I installed Eucalyptus on my Beowulf.

It's a lot simpler to run Bio-Linux as a terminal server on bare metal.
However, if your users each want their own instance of Bio-Linux to set
up and use exactly as they want to then it might be better to create a
private cloud. You can, of course, aggregate virtual machines to create
a Beowulf-type cluster on a public/private cloud but I'm not convinced
this is a good idea other than for 'embarrassingly' parallel comuting.

> Secondly, is there any advice on a backup solution. We currently have a
> 4TB HDD in our biolinux and back it up using the dump command to an
> external HDD. This was sufficient until we started peaking at 3TB of
> data. This causes a problem with uncompressing our dump file as we don’t
> have enough space.

I wrote the ETOH (Enhanced Towers Of Hanoi) version of Tim's Bio-Linux
backup script: I use a rule-of-thumb to have about the same amount of
space for backup and active filesystems. The dump compression is about
2:1 and this gives sufficient redundancy for about three months of
incremental dumps as long as the deltas are not too large. In that case,
I force a level 0 dump. However, I'm experimenting with "btrfs"
snapshots now and will give up using dump as soon as I can because it is
too specific for the physical device and the ext filesystem.

HTH,

  Tony.

--
Dr. A.J.Travis, University of Aberdeen, Institute of Biological and
Environmental Sciences, Cruickshank Building, St. Machar Drive, Aberdeen
AB24 3UU, Scotland, UK. tel +44(0)1224 272700, fax +44 (0)1224 272 396
http://www.abdn.ac.uk, mailto:tony.travis at abdn.ac.uk, skype:ajtravis


The University of Aberdeen is a charity registered in Scotland, No SC013683.
Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.