CDAP-13246 provisioner failure handling
Adding logic to handle failures during provisioning.
When a RetryableProvisionException is thrown, the method will
be retried up to a time limit. The time limit is hardcoded today,
but will be configurable per profile later.
Also added handling for scenarios where a cluster is request to
be created, but when polling for status, the cluster returns a
non-running status. In these scenarios, the cluster is usually
deleted, then the create is retried.
Also added logic to pick up tasks that were being executed while
CDAP was shut down. Each task will store state about what it is
about to do. If CDAP is shut down in the middle of a task, when
it comes back up, it will scan the state store and re-create
tasks that were in progress.
Also some cleanup to state transition logic by moving it out of
AppMetadataStore and into the respective ProgramRunStatus and
ProgramRunClusterStatus enums.
Fixing a bad test that relied on an invalid state transition, and
adding a test to make sure we can't go from pending to completed.
