CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.

Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed

Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed

CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.

Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed

CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.

Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed

CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.

Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection

CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.

CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.

Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection

Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection

CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.

Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent

CDAP-19133 fix namespaceId creation in tethering agent

Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent

CDAP-19133 fix namespaceId creation in tethering agent

Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent

CDAP-19133 fix namespaceId creation in tethering agent

Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent

CDAP-19133 fix namespaceId creation in tethering agent

Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent

CDAP-19133 fix namespaceId creation in tethering agent

CDAP-19133 fix spark launch namespace
Fix spark submission so that it sets the kubernetes namespace to
the correct value. To do this, added a way to get configuration
properties in the MasterEnvironment interface that gets the spark
submit configs.

In order to populate the configs, the namespace configs are now
stored in the system arguments for a program run. These are
populated at the start of the run and also in the TetheringAgent
when it forwards the program start message. The namespace
lookup was moved from the DistributedProgramRunner to the new
locations so that it is consistently available in the options
regardless of the program runner implementation or tethering,
o more easily access them from the SparkProgramRunner.

Also add a downward mount so that pods will contain their
namespace in a file instead of only looking in the CDAP conf for
the namespace that CDAP lives in. This fixes a bug where pods
would think their namespace is the cdap namespace instead of the
namespace they are running in.

CDAP-19167 update spark build to include gcp auth extension

Merge pull request #14250 from cdapio/CDAP-19167-include-auth-ext-in-spark-image
CDAP-19167 update spark build to include gcp auth extension

Merge pull request #14232 from cdapio/CDAP-19133-fix-program-launch-k8s-namespace
CDAP-19133 fix spark launch namespace

CDAP-19123 set batch interval for flaky test
Set batch interval to 5s for flaky stage consolidation test
instead of the default 1m. This allows the pipeline to stop much
faster and avoid the program stop timeout that was causing the
flakiness.

Merge pull request #14261 from cdapio/CDAP-19123-fix-flaky-datastreams-test
CDAP-19123 set batch interval for flaky test

CDAP-19123 set batch interval for flaky test
Set batch interval to 5s for flaky stage consolidation test
instead of the default 1m. This allows the pipeline to stop much
faster and avoid the program stop timeout that was causing the
flakiness.

Merge pull request #14261 from cdapio/CDAP-19123-fix-flaky-datastreams-test
CDAP-19123 set batch interval for flaky test

CDAP-18984 ensure app class is in manifest for integration tests
Somehow it is possible in some circumstances to find an application
jar with the wrong manifest in IntegrationTestBase. Added some
logic to always ensure that the manifest contains the required
main class attribute.

Also updating surefire-plugin version to fix errors that happen
while running tests in some environments.

Merge pull request #14165 from cdapio/CDAP-18984-ensure-app-class-manifest
CDAP-18984 ensure app class is in manifest for integration tests

Merge pull request #14183 from cdapio/CDAP-19042-fix-flaky-program-kill-test
CDAP-19042 fix flaky program kill test

CDAP-19042 fix flaky program kill test
The change to kill programs asynchronously made the test flaky.
Fixed the test to properly test that no programs are killed, or
to wait until the kill completes.

Merge pull request #1569 from cdapio/CDAP-18994-fix-avro-maps
CDAP-18994 fix errors with nullable map values

CDAP-18994 fix errors with nullable map values
Remove schema preprocessing that was causing failures when a schema
had a map with nullable values. Manually modifying the schema json is
brittle, doesn't fix the issue for parquet, and is also not performant.

CDAP schema parsing now does the correct thing so no preprocessing
is needed.

CDAP-19019 parallelize program force killing

Merge pull request #14160 from cdapio/CDAP-19019-parallelize-program-kill
CDAP-19019 parallelize program force killing

Merge pull request #14160 from cdapio/CDAP-19019-parallelize-program-kill
CDAP-19019 parallelize program force killing

CDAP-19019 parallelize program force killing

Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection

CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.

Merge pull request #14260 from cdapio/CDAP-19153-strip-artifact-scheme-for-tethering
CDAP-19153 strip artifact scheme and host for tethering

CDAP-19153 strip artifact scheme and host for tethering
This fixes a bug in tethering scenarios when one CDAP instance
has one scheme for location URI (like hdfs://) and the peered
instance has a different scheme (like gs://). In these scenarios,
the LocationFactory cannot convert the URI into a Location
because of the scheme mismatch.

To fix this, the ArtifactCache service used in tethered runs will
strip the scheme and host from the location URI so that the
Location object can be created. This is a hack until the more
complete solution of removing Location from the ArtifactDetail
class can be implemented.

CDAP-19153 strip artifact scheme and host for tethering
This fixes a bug in tethering scenarios when one CDAP instance
has one scheme for location URI (like hdfs://) and the peered
instance has a different scheme (like gs://). In these scenarios,
the LocationFactory cannot convert the URI into a Location
because of the scheme mismatch.

To fix this, the ArtifactCache service used in tethered runs will
strip the scheme and host from the location URI so that the
Location object can be created. This is a hack until the more
complete solution of removing Location from the ArtifactDetail
class can be implemented.

Merge pull request #14260 from cdapio/CDAP-19153-strip-artifact-scheme-for-tethering
CDAP-19153 strip artifact scheme and host for tethering

Merge pull request #14165 from cdapio/CDAP-18984-ensure-app-class-manifest
CDAP-18984 ensure app class is in manifest for integration tests

CDAP-18984 ensure app class is in manifest for integration tests
Somehow it is possible in some circumstances to find an application
jar with the wrong manifest in IntegrationTestBase. Added some
logic to always ensure that the manifest contains the required
main class attribute.

Also updating surefire-plugin version to fix errors that happen
while running tests in some environments.

Merge pull request #14183 from cdapio/CDAP-19042-fix-flaky-program-kill-test
CDAP-19042 fix flaky program kill test

CDAP-19042 fix flaky program kill test
The change to kill programs asynchronously made the test flaky.
Fixed the test to properly test that no programs are killed, or
to wait until the kill completes.

Merge pull request #14164 from cdapio/CDAP-12977-app-delete-trigger-not-found
CDAP-12977 fix app deletion when trigger not found

CDAP-12977 fix app deletion when trigger not found
Fixed a bug where app deletion can fail when it tries to delete
the app schedules and a trigger cannot be found. In this scenario,
it is already in the desired state, so just move on.

Merge pull request #14164 from cdapio/CDAP-12977-app-delete-trigger-not-found
CDAP-12977 fix app deletion when trigger not found

CDAP-12977 fix app deletion when trigger not found
Fixed a bug where app deletion can fail when it tries to delete
the app schedules and a trigger cannot be found. In this scenario,
it is already in the desired state, so just move on.

CDAP-18945 add more logging to program stopper

Merge pull request #14109 from cdapio/CDAP-18945-add-more-stop-logging
CDAP-18945 add more logging to program stopper

Merge pull request #14097 from cdapio/CDAP-18890-update-program-killer
CDAP-18890 update program killer to kill and skip tethered runs

CDAP-18890 update program killer to kill and skip tethered runs

Merge pull request #1487 from cdapio/bug/CDAP-15794
[CDAP-15794] making the error message consistent when delimited schema is wrong