DogAndPanda: Linux and RAID - a working setup

Linux and RAID - a working setup

The layman's intro: people don't ulke losing data. A hard drive can fail any time - and it can be a gradual failure, a quick failure, or anything in between.

My base set-up:

Some version of linux (in my case ubuntu)
- Make sure to have a kernel compiled that supports RAID (the default one often does these days)
- Make sure to have the raid tools package installed (Ubuntu: this is called mdadm)
A motherboard with lots of hard drive ports or cards with ports.
Lots of hard drives.

The idea behind RAID is that you stripe your data across multiple drives, and have parity bits also striped, so that if drives fail, you can recover the data. With RAID5, one drive can fail. With RAID6, two can fail.

Most people like to go the tradional way of getting many identical drives and just creating a huge array across all of these drives. This creates several problems:

You need identically sized drives.
No easy upgrade path - if you buy a bigger drive, what do you do with it?
Takes a long time to sync - if there is a drive failure, you have to go over the entire drive - perhaps a terabyte or more -- and anything can happen during that time!

The solution I have come up with is having each drive partitioned into small, easily managed pieces. These days, I use 80 gig pieces, of which you can get about 16 on a terabyte drive:

# fdisk -l /dev/sdd

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x51f638ee

Device Boot Start End Blocks Id System /dev/sdd1 111876 121601 78124095 83 Linux /dev/sdd2 102150 111875 78124095 83 Linux /dev/sdd3 92424 102149 78124095 83 Linux /dev/sdd4 1 92423 742387716 5 Extended /dev/sdd5 1 9726 78124032 83 Linux /dev/sdd6 9727 19452 78124063+ 83 Linux /dev/sdd7 19453 29178 78124063+ 83 Linux /dev/sdd8 29179 38904 78124063+ 83 Linux /dev/sdd9 38905 48630 78124063+ 83 Linux /dev/sdd10 48631 58356 78124063+ 83 Linux /dev/sdd11 58357 68082 78124063+ 83 Linux /dev/sdd12 68083 77808 78124063+ 83 Linux /dev/sdd13 77809 82671 39062016 83 Linux /dev/sdd14 82672 87534 39062016 83 Linux /dev/sdd15 87535 92423 39270861 83 Linux

Note that in this case, I have some legacy 40 gig partitions as well.

Any newcomer to raid should become familiar with the mdadm command. Go look at the man page - it's better than having me explain it. I use 32 byte chunks (no real reason), and raid 6 (better redundancy). The number of drives in a stripe depends on how big I need the partition to be - I sort my partitions based on datatypes (movies, mp3s, raw audio recordings, etc).

Here is a small script I use to see the status of my raid drives:

cat /usr/sbin/mddf 
foreach bob (`df | grep /stuff | cut -f3 -d/ | cut -f1 -d\  `)
df | grep ${bob}\  
cat /proc/mdstat | grep -A1 ${bob}\  
end

To get a better idea of how I use it, see the following:

/dev/md48            230692820 118477064 100497164  55% /stuff/crit
md48 : active raid6 sda9[2] sdd8[1] sde12[0] sdf7[4] sdb8[5](F)
      234371904 blocks level 6, 32k chunk, algorithm 2 [5/4] [UUU_U]
/dev/md47            230692820  50698312 168275916  24% /stuff/homevid_old
md47 : active raid6 sdf10[4] sdd9[1] sde10[0] sdm10[3] sdl12[2]
      234371904 blocks level 6, 32k chunk, algorithm 2 [5/5] [UUUUU]
/dev/md32            173103296 134150696  30159444  82% /stuff/flacmp
md32 : active raid6 sdk8[1] sdd10[0] sdl6[2] sdg13[4] sde11[3]
      175863168 blocks level 6, 32k chunk, algorithm 2 [5/5] [UUUUU]
/dev/md23            117149116 114120832   3028284  98% /stuff/videos
md23 : active raid6 sdg9[1] sdc11[0] sdh5[4] sdi10[3] sdl5[2]
      117185664 blocks level 6, 32k chunk, algorithm 2 [5/5] [UUUUU]
/dev/md3             156209812 156204300      5512 100% /stuff/painhertz
md3 : active raid6 sdg12[5] sdd15[2] sdj5[0] sdb6[6](F) sdl9[4] sdg5[3] sdc13[1]
      156247552 blocks level 6, 32k chunk, algorithm 2 [6/6] [UUUUUU]

Note the different sizes of the partitions, and also that md48 is in a degraded state because the sdb drive has failed. Note also that each raid virtual drive does not span all physical disks - just a small selection of them. This means that if a single disk fails, not even all virtual raid drives will become degraded.

But, it sounds complicated creating a new raid array - having to find all the existing partitions not used in a raid array, figuring out their sizes. Likely, recovering from a drive failure means having to find some free partitions to stick in place of the failure.

Currently, I have the following quickly-hacked c++ program to do the job:

cat ~/src/raidcheck.cc
// (C) 2008 Alex Lozupone
// Quick hack.
// Things to add that are obvious: 
//   variables for things like fdisk-command, functions with good names, etc.
using namespace std;
char min_drive = 'a';
char max_drive = 'm';
void Tokenize(const string& str,
              vector& tokens,
              const string& delimiters = " ")
{
  // Skip delimiters at beginning.
  string::size_type lastPos = str.find_first_not_of(delimiters, 0);
  // Find first "non-delimiter".
  string::size_type pos     = str.find_first_of(delimiters, lastPos);
  while (string::npos != pos || string::npos != lastPos)
    {
      // Found a token, add it to the vector.
      tokens.push_back(str.substr(lastPos, pos - lastPos));
      // Skip delimiters.  Note the "not_of"
      lastPos = str.find_first_not_of(delimiters, pos);
      // Find next "non-delimiter"
      pos = str.find_first_of(delimiters, lastPos);
    }
}
int getsize(string ss) {
  vector tokens;
  Tokenize(ss,tokens);
  return atoi(tokens[3].c_str());
}
string getpartition(string ss) {
  vector tokens;
  Tokenize(ss,tokens);
  return tokens[0].c_str()+5;
}
bool raidfetch(string p){
  FILE* r = fopen("/proc/mdstat","r");
  while(!feof(r)) {
    char s[100];
    fgets(s,sizeof(s),r);
    if(strstr(s,p.c_str())) return 1;
  }
  return 0;
}
void checkdrive(string name){
  FILE* f= popen(("fdisk -l /dev/"+name).c_str(),"r");
  while(!feof(f)){
    char s[100];
    fgets(s,sizeof(s),f);
    string ss = s;
    if(ss.find(name) > 6) continue;
    if(ss.find("Extended") < 1000) continue;
    int z = getsize(ss);
    string p = getpartition(ss);
    bool avail = !raidfetch(p);
    if(avail) cout <int main() {
  char c;
  for(c=min_drive;c<=max_drive;c++){
    checkdrive(string("sd") + c);
  }
  return 0;
}

DogAndPanda: Linux and RAID - a working setup

Linux and RAID - a working setup

Comments