Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #924 from cdapio/feature/CDAP-15767-int96-read-support
CDAP-15767 add support for reading int96 fields

CDAP-15767 add support for reading int96 fields
Added support for reading int96 fields by copying relevant classes
from the parquet library and slightly modifying them to handle
int96 fields. INT96 fields can be read as bytes or as a long.
When read as a long, the value is converted into a unix timestamp
in microseconds, which means the nanosecond component of the
original data is lost.

Fixed a bug with the existing parquet format so that it correctly
uses the user provided schema as the read schema, which does not
have to be the same as the write schema.

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

CDAP-15770 fixed a bug with encoding nullable byte buffers

Merge pull request #11553 from cdapio/feature/CDAP-15770-fix-write-nullable-bytebuffer
CDAP-15770 fixed a bug with encoding nullable byte buffers

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

CDAP-15770 fixed a bug with encoding nullable byte buffers

Merge pull request #11553 from cdapio/feature/CDAP-15770-fix-write-nullable-bytebuffer
CDAP-15770 fixed a bug with encoding nullable byte buffers

CDAP-15770 fixed a bug with encoding nullable byte buffers

Merge pull request #11553 from cdapio/feature/CDAP-15770-fix-write-nullable-bytebuffer
CDAP-15770 fixed a bug with encoding nullable byte buffers

Merge pull request #1017 from mlozbin-cybervisiontech/feature/CDAP-15643-bigtable-integration-tests
CDAP-15643 & CDAP-15734 Support all data types in BigTable Sink & Source integration tests

Merge pull request #1018 from vladhlinskiy/CDAP-15756-fix-spanner-tests
CDAP-15756 GoogleCloudSpannerTest broken due to pubsub dependency

Merge pull request #924 from cdapio/feature/CDAP-15767-int96-read-support
CDAP-15767 add support for reading int96 fields

CDAP-15767 add support for reading int96 fields
Added support for reading int96 fields by copying relevant classes
from the parquet library and slightly modifying them to handle
int96 fields. INT96 fields can be read as bytes or as a long.
When read as a long, the value is converted into a unix timestamp
in microseconds, which means the nanosecond component of the
original data is lost.

Fixed a bug with the existing parquet format so that it correctly
uses the user provided schema as the read schema, which does not
have to be the same as the write schema.

Merge pull request #11548 from cdapio/feature/CDAP-15747-fix-flaky-provision-cancel
CDAP-15747 fix a race when cancelling a KeyedExecutor task

CDAP-15747 fix a race when cancelling a KeyedExecutor task
Fixed a race condition in KeyedExecutor where the task could
get cancelled but return that it was not cancelled.
This fixes a race condition in the provisioner where it can
cancel a task, think that the task completed before it was
cancelled, then skip the logic to modify the run state.

Merge pull request #1014 from vladhlinskiy/feature/CDAP-15641-spanner-integration-tests
CDAP-15641 Spanner integration tests

Merge pull request #1014 from vladhlinskiy/feature/CDAP-15641-spanner-integration-tests
CDAP-15641 Spanner integration tests

Merge pull request #1014 from vladhlinskiy/feature/CDAP-15641-spanner-integration-tests
CDAP-15641 Spanner integration tests

Merge pull request #1013 from vladhlinskiy/fetaure/CDAP-15640-GCS-integration-tests-coverage
CDAP-15640 GCS integration tests coverage

CDAP-15373 add subnet as a dataproc provisioner property
Added a subnet property to the dataproc provisioner that lets
users specify a subnet when creating a profile. The subnet
must be in the network chosen, and it must be for the region
that the chosen zone is in.

If no subnet is given and the chosen network uses custom subnet
creation, the provisioner will now try to pick a subnet from
the network.

This fixes failures that would occur if a network was used that does
not use automatic subnet creation.

Merge pull request #11372 from cdapio/feature/CDAP-13795-remote-run-cleanup
CDAP-13795 clean up run resources on remote hadoop provisioner

Merge pull request #11360 from cdapio/bugfix_release/CDAP-15393-fix-empty-checkpoint
CDAP-15393 handle empty checkpoint dir for streaming pipelines

Merge pull request #11404 from cdapio/bugfix_release/CDAP-15373-dataproc-subnet
CDAP-15373 add subnet as a dataproc provisioner property

CDAP-13795 clean up run resources on remote hadoop provisioner
Added some cleanup logic to the remote hadoop provisioner to
delete the run directory on the remote host when 'deprovisioning'
the cluster.

Ideally, this logic would reside within CDAP instead of in the
provisioner, as CDAP is the one that knows where it copies the
files to. However, it requires a bit more work to make sure
the cleanup logic always runs, even across master death, due to
the asynchronous nature of how run status is recorded and how
deprovisioning is kicked off. There would also ideally need to be
some way for other provisioners to tell CDAP that it does not
need to run cleanup.

Merge pull request #11376 from cdapio/feature/CDAP-15384-fix-dataproc-network-auto-detect
CDAP-15384 fix dataproc network auto detect

CDAP-15393 handle empty checkpoint dir for streaming pipelines
Treat an empty checkpoint directory the same as a null checkpoint
directory instead of erroring.

CDAP-15384 fix dataproc network auto detect
Fixed the auto-detect logic for the dataproc provisioner when a
project is given that is not the same as the project that CDAP
is running in. In that scenario, changed the provisioner to use
the Compute API to find a network in the provided project.

Merge pull request #11437 from vladhlinskiy/bugfix/CDAP-15570-fix-get-decimal-bytebuffer
CDAP-15570 StructuredRecord#getDecimal fixed to work with ByteBuffer values

Merge pull request #323 from rishabhBudhouliya/feature/CDAP-15248-ADLSConnectionService
Squashed commit of the following:

CDAP-13795 clean up run resources on remote hadoop provisioner
Added some cleanup logic to the remote hadoop provisioner to
delete the run directory on the remote host when 'deprovisioning'
the cluster.

Ideally, this logic would reside within CDAP instead of in the
provisioner, as CDAP is the one that knows where it copies the
files to. However, it requires a bit more work to make sure
the cleanup logic always runs, even across master death, due to
the asynchronous nature of how run status is recorded and how
deprovisioning is kicked off. There would also ideally need to be
some way for other provisioners to tell CDAP that it does not
need to run cleanup.

CDAP-15393 handle empty checkpoint dir for streaming pipelines
Treat an empty checkpoint directory the same as a null checkpoint
directory instead of erroring.

update dynamic spark plugin to 2.2.2

Merge pull request #313 from cdapio/update-dynamic-spark-plugin
update dynamic spark plugin to 2.2.2

CDAP-13795 clean up run resources on remote hadoop provisioner
Added some cleanup logic to the remote hadoop provisioner to
delete the run directory on the remote host when 'deprovisioning'
the cluster.

Ideally, this logic would reside within CDAP instead of in the
provisioner, as CDAP is the one that knows where it copies the
files to. However, it requires a bit more work to make sure
the cleanup logic always runs, even across master death, due to
the asynchronous nature of how run status is recorded and how
deprovisioning is kicked off. There would also ideally need to be
some way for other provisioners to tell CDAP that it does not
need to run cleanup.

CDAP-15393 handle empty checkpoint dir for streaming pipelines
Treat an empty checkpoint directory the same as a null checkpoint
directory instead of erroring.

CDAP-15037 fix user scoped plugin loading in service
Added a way to get a dynamic plugin configurer for a different
namespace than the one a service is running in. Used this
capability to fix a bug in the pipeline studio service where
only system scoped plugins were able to be loaded.

Merge pull request #11174 from cdapio/feature/CDAP-15037-fix-user-plugin-loading
CDAP-15037 fix user scoped plugin loading in service

Merge pull request #11111 from cdapio/fix-compilation
fix compilation

fix compilation

Merge pull request #1001 from cdapio/fix-wrangler-tests
fix wrangler pipeline test

fix wrangler pipeline test
The wrangler pipeline test does not require the wrangler
service to be running, so removed the part that deploys and starts
the service. This was failing due to the fact that wrangler must
now be deployed to the system namespace.

Also fixed a compilation issue due to a class being moved.

CDAP-14915 fix flaky program lifecycle test
Fixed the test to wait for the cluster status to be deprovisioned,
since the update happens asynchronously.

Merge pull request #11069 from cdapio/feature/CDAP-14915-fix-flaky-test
CDAP-14915 fix flaky program lifecycle test

fix wrangler pipeline test
The wrangler pipeline test does not require the wrangler
service to be running, so removed the part that deploys and starts
the service. This was failing due to the fact that wrangler must
now be deployed to the system namespace.

Also fixed a compilation issue due to a class being moved.

Merge pull request #1001 from cdapio/fix-wrangler-tests
fix wrangler pipeline test

fix wrangler pipeline test
The wrangler pipeline test does not require the wrangler
service to be running, so removed the part that deploys and starts
the service. This was failing due to the fact that wrangler must
now be deployed to the system namespace.

Also fixed a compilation issue due to a class being moved.

Merge pull request #1001 from cdapio/fix-wrangler-tests
fix wrangler pipeline test

Merge pull request #11056 from cdapio/feature/CDAP-14916-fix-tx-runners-exception-propagation
CDAP-14916 fix TransactionRunners runtime exception propagation

CDAP-14916 fix TransactionRunners runtime exception propagation
Fixed a bug with the TransactionRunners.run() methods that take a
single runnable and no exception classes. These methods used to
throw a RuntimeException around the TransactionException around
the actual RuntimeException thrown by the TxRunnable.

fix wrangler pipeline test
The wrangler pipeline test does not require the wrangler
service to be running, so removed the part that deploys and starts
the service. This was failing due to the fact that wrangler must
now be deployed to the system namespace.

Also fixed a compilation issue due to a class being moved.

Merge pull request #1001 from cdapio/fix-wrangler-tests
fix wrangler pipeline test

Merge pull request #1001 from cdapio/fix-wrangler-tests
fix wrangler pipeline test

fix wrangler pipeline test
The wrangler pipeline test does not require the wrangler
service to be running, so removed the part that deploys and starts
the service. This was failing due to the fact that wrangler must
now be deployed to the system namespace.

Also fixed a compilation issue due to a class being moved.