albertshau <ashau@google.com>: Author Summary

Builds triggered by albertshau <ashau@google.com>

Builds triggered by an author are those builds which contains changes committed by the author.
1537
437 (28%)
1100 (72%)

Breakages and fixes

Broken means the build has failed but the previous build was successful.
Fixed means that the build was successful but the previous build has failed.
139 (9% of all builds triggered)
132 (9% of all builds triggered)
-7
Build Completed Code commits Tests
CDAP › RUT › #1854 1 week ago
CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.
Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed
3366 passed
CDAP › DUT › #3676 1 week ago
Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed
CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.
2135 passed
CDAP › BPP › #2199 1 week ago
Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed
CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.
Testless build
CDAP › DRC › #5938 1 week ago
Merge pull request #14314 from cdapio/CDAP-19050-remove-unneeded-gce-calls
CDAP-19050 remove GCE calls when they are not needed
CDAP-19050 refactor dataproc client to remove unneeded calls
Refactor the DataprocClient so that it only looks up network
information in the create cluster call, and it skips getting node
IPs and create times when SSH access is not needed.

Split the existing DataprocClient into two separate classes with
different logic for setting network information and getting node
information.
Testless build
CDAP › RUT › #1845 2 weeks ago
Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection
CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.
3364 passed
CDAP › DUT › #3667 2 weeks ago
CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.
Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection
2134 passed
CDAP › DRC › #5926 2 weeks ago
Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection
CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.
Testless build
CDAP › RUT › #1835 3 weeks ago
Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent
CDAP-19133 fix namespaceId creation in tethering agent
1 of 2362 failed
CDAP › DUT › #3656 3 weeks ago
Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent
CDAP-19133 fix namespaceId creation in tethering agent
2350 passed
CDAP › DRC › #5910 3 weeks ago
Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent
CDAP-19133 fix namespaceId creation in tethering agent
Testless build
Build Completed Code commits Tests
CDAP › RUT › #1835 3 weeks ago
Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent
CDAP-19133 fix namespaceId creation in tethering agent
1 of 2362 failed
CDAP › DUT › #3656 3 weeks ago
Merge pull request #14284 from cdapio/CDAP-19133-fix-tethering-runtime-namespace
CDAP-19133 fix namespaceId creation in tethering agent
CDAP-19133 fix namespaceId creation in tethering agent
2350 passed
CDAP › DUT › #3653 3 weeks ago
CDAP-19133 fix spark launch namespace
Fix spark submission so that it sets the kubernetes namespace to
the correct value. To do this, added a way to get configuration
properties in the MasterEnvironment interface that gets the spark
submit configs.

In order to populate the configs, the namespace configs are now
stored in the system arguments for a program run. These are
populated at the start of the run and also in the TetheringAgent
when it forwards the program start message. The namespace
lookup was moved from the DistributedProgramRunner to the new
locations so that it is consistently available in the options
regardless of the program runner implementation or tethering,
o more easily access them from the SparkProgramRunner.

Also add a downward mount so that pods will contain their
namespace in a file instead of only looking in the CDAP conf for
the namespace that CDAP lives in. This fixes a bug where pods
would think their namespace is the cdap namespace instead of the
namespace they are running in.
CDAP-19167 update spark build to include gcp auth extension
Merge pull request #14250 from cdapio/CDAP-19167-include-auth-ext-in-spark-image
CDAP-19167 update spark build to include gcp auth extension
Merge pull request #14232 from cdapio/CDAP-19133-fix-program-launch-k8s-namespace
CDAP-19133 fix spark launch namespace
5 of 3381 failed
CDAP › RUT › #1830 3 weeks ago
CDAP-19123 set batch interval for flaky test
Set batch interval to 5s for flaky stage consolidation test
instead of the default 1m. This allows the pipeline to stop much
faster and avoid the program stop timeout that was causing the
flakiness.
Merge pull request #14261 from cdapio/CDAP-19123-fix-flaky-datastreams-test
CDAP-19123 set batch interval for flaky test
2335 passed
CDAP › DUT › #3651 3 weeks ago
CDAP-19123 set batch interval for flaky test
Set batch interval to 5s for flaky stage consolidation test
instead of the default 1m. This allows the pipeline to stop much
faster and avoid the program stop timeout that was causing the
flakiness.
Merge pull request #14261 from cdapio/CDAP-19123-fix-flaky-datastreams-test
CDAP-19123 set batch interval for flaky test
1 of 2350 failed
CDAP › RUT › #1811 1 month ago
CDAP-18984 ensure app class is in manifest for integration tests
Somehow it is possible in some circumstances to find an application
jar with the wrong manifest in IntegrationTestBase. Added some
logic to always ensure that the manifest contains the required
main class attribute.

Also updating surefire-plugin version to fix errors that happen
while running tests in some environments.
Merge pull request #14165 from cdapio/CDAP-18984-ensure-app-class-manifest
CDAP-18984 ensure app class is in manifest for integration tests
1 of 2338 failed
CDAP › RUT › #1807 1 month ago
Merge pull request #14183 from cdapio/CDAP-19042-fix-flaky-program-kill-test
CDAP-19042 fix flaky program kill test
CDAP-19042 fix flaky program kill test
The change to kill programs asynchronously made the test flaky.
Fixed the test to properly test that no programs are killed, or
to wait until the kill completes.
1 of 2338 failed
HYP › BAD › #466 1 month ago
Merge pull request #1569 from cdapio/CDAP-18994-fix-avro-maps
CDAP-18994 fix errors with nullable map values
CDAP-18994 fix errors with nullable map values
Remove schema preprocessing that was causing failures when a schema
had a map with nullable values. Manually modifying the schema json is
brittle, doesn't fix the issue for parquet, and is also not performant.

CDAP schema parsing now does the correct thing so no preprocessing
is needed.
1 of 1369 failed
CDAP › RUT › #1802 1 month ago
CDAP-19019 parallelize program force killing
Merge pull request #14160 from cdapio/CDAP-19019-parallelize-program-kill
CDAP-19019 parallelize program force killing
2322 passed
CDAP › DUT › #3620 1 month ago
Merge pull request #14160 from cdapio/CDAP-19019-parallelize-program-kill
CDAP-19019 parallelize program force killing
CDAP-19019 parallelize program force killing
2317 passed
Build Completed Code commits Tests
CDAP › RUT › #1845 2 weeks ago
Merge pull request #14280 from cdapio/CDAP-19078-extend-format-interface-schema-detection
CDAP-19078 extend format interface for schema detection
CDAP-19078 extend format interface for schema detection
Adding a method to ValidatingInputFormat so that format plugins
have a way to detect schema given some input data. This removes
the need for format plugins to interact with various filesystems,
allowing them to focus purely on how to read data instead of
also needing to know how to find which data to process.
3364 passed
CDAP › DUT › #3652 3 weeks ago
Merge pull request #14260 from cdapio/CDAP-19153-strip-artifact-scheme-for-tethering
CDAP-19153 strip artifact scheme and host for tethering
CDAP-19153 strip artifact scheme and host for tethering
This fixes a bug in tethering scenarios when one CDAP instance
has one scheme for location URI (like hdfs://) and the peered
instance has a different scheme (like gs://). In these scenarios,
the LocationFactory cannot convert the URI into a Location
because of the scheme mismatch.

To fix this, the ArtifactCache service used in tethered runs will
strip the scheme and host from the location URI so that the
Location object can be created. This is a hack until the more
complete solution of removing Location from the ArtifactDetail
class can be implemented.
3346 passed
CDAP › RUT › #1831 3 weeks ago
CDAP-19153 strip artifact scheme and host for tethering
This fixes a bug in tethering scenarios when one CDAP instance
has one scheme for location URI (like hdfs://) and the peered
instance has a different scheme (like gs://). In these scenarios,
the LocationFactory cannot convert the URI into a Location
because of the scheme mismatch.

To fix this, the ArtifactCache service used in tethered runs will
strip the scheme and host from the location URI so that the
Location object can be created. This is a hack until the more
complete solution of removing Location from the ArtifactDetail
class can be implemented.
Merge pull request #14260 from cdapio/CDAP-19153-strip-artifact-scheme-for-tethering
CDAP-19153 strip artifact scheme and host for tethering
3346 passed
CDAP › DUT › #3631 1 month ago
Merge pull request #14165 from cdapio/CDAP-18984-ensure-app-class-manifest
CDAP-18984 ensure app class is in manifest for integration tests
CDAP-18984 ensure app class is in manifest for integration tests
Somehow it is possible in some circumstances to find an application
jar with the wrong manifest in IntegrationTestBase. Added some
logic to always ensure that the manifest contains the required
main class attribute.

Also updating surefire-plugin version to fix errors that happen
while running tests in some environments.
3332 passed
CDAP › DUT › #3626 1 month ago
Merge pull request #14183 from cdapio/CDAP-19042-fix-flaky-program-kill-test
CDAP-19042 fix flaky program kill test
CDAP-19042 fix flaky program kill test
The change to kill programs asynchronously made the test flaky.
Fixed the test to properly test that no programs are killed, or
to wait until the kill completes.
3331 passed
CDAP › DUT › #3619 1 month ago
Merge pull request #14164 from cdapio/CDAP-12977-app-delete-trigger-not-found
CDAP-12977 fix app deletion when trigger not found
CDAP-12977 fix app deletion when trigger not found
Fixed a bug where app deletion can fail when it tries to delete
the app schedules and a trigger cannot be found. In this scenario,
it is already in the desired state, so just move on.
3327 passed
CDAP › RUT › #1801 1 month ago
Merge pull request #14164 from cdapio/CDAP-12977-app-delete-trigger-not-found
CDAP-12977 fix app deletion when trigger not found
CDAP-12977 fix app deletion when trigger not found
Fixed a bug where app deletion can fail when it tries to delete
the app schedules and a trigger cannot be found. In this scenario,
it is already in the desired state, so just move on.
3327 passed
CDAP › DUT › #3593 2 months ago
CDAP-18945 add more logging to program stopper
Merge pull request #14109 from cdapio/CDAP-18945-add-more-stop-logging
CDAP-18945 add more logging to program stopper
3279 passed
CDAP › DRC › #5809 2 months ago
Merge pull request #14097 from cdapio/CDAP-18890-update-program-killer
CDAP-18890 update program killer to kill and skip tethered runs
CDAP-18890 update program killer to kill and skip tethered runs
Testless build
HYP › BAD › #422 8 months ago
Merge pull request #1487 from cdapio/bug/CDAP-15794
[CDAP-15794] making the error message consistent when delimited schema is wrong
1211 passed