Getting hands dirty with Joyent SDC: first lesson learned

Finally getting into Joyent's private cloud technology. I'll talk more about what all of this is useful for some other time, but this post is more of a note to self / note of warning. I repurposed some beefy ESX nodes for testing out Smart Data Center. But, those didn't have disks worth anything. Instead, I took some disks that were evacuated out of ZFS pools for larger drives. They would still be fine here...

The problem arises in setting up compute nodes, and later in any re-installing if necessary of the headnode. Things would quietly fail without any errors on compute node configuration, and re-installs of the head node dug a deeper hole. Turns out that Joyent is being ever too cautious in creating data pools for the head and compute nodes, and won't attempt to create the necessary local disk pools if the disks were previously associated with active ZFS pools. Silent errors are never good.

The work around is to bring up the head node in their recovery mode, which is noted as not importing any pools. Next, associate the drives, import the pools (if fully there) or create a new pool for each individual disk, and then "zpool destroy" them. Rinse, repeat. I finally got my head node installed in a sane way, and now on to some remaining problems with compute nodes and testing out KVM and vcpu support. More on that later.