The problem arises in setting up compute nodes, and later in any re-installing if necessary of the headnode. Things would quietly fail without any errors on compute node configuration, and re-installs of the head node dug a deeper hole. Turns out that Joyent is being ever too cautious in creating data pools for the head and compute nodes, and won't attempt to create the necessary local disk pools if the disks were previously associated with active ZFS pools. Silent errors are never good.
The work around is to bring up the head node in their recovery mode, which is noted as not importing any pools. Next, associate the drives, import the pools (if fully there) or create a new pool for each individual disk, and then "zpool destroy" them. Rinse, repeat. I finally got my head node installed in a sane way, and now on to some remaining problems with compute nodes and testing out KVM and vcpu support. More on that later.