Albert Shau <ashau@desktobert.svl.corp.google.com>: Author Summary

Builds triggered by Albert Shau <ashau@desktobert.svl.corp.google.com>

Builds triggered by an author are those builds which contains changes committed by the author.
101
32 (32%)
69 (68%)

Breakages and fixes

Broken means the build has failed but the previous build was successful.
Fixed means that the build was successful but the previous build has failed.
14 (14% of all builds triggered)
10 (10% of all builds triggered)
-4
Build Completed Code commits Tests
CDAP › BA › #320 2 years ago
CDAP-13521 fix profile property override
Fixed a bug where profile properties provided at runtime were not
correctly overriding whatever currently exists in the profile.
In addition, properties not set in the profile were being ignored
completely.

Added a method to ProfileService to do the property overrides and
added a unit test for it.
CDAP-13338 restrict profile program types
Make sure only workflows look up the profile from the runtime
args. All other program types will run using the native profile.
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
Testless build
CDAP › DBT › #780 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
Testless build
CDAP › RUT › #478 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
2436 passed
CDAP › DUT › #2227 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
2436 passed
CDAP › UDUT › #25 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
Testless build
CDAP › URUT › #24 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
Testless build
CDAP › CSI › #1275 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
55 passed
CDAP › DRC › #3887 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
Testless build
CDAP › RUT › #477 2 years ago
CDAP-13338 restrict profile program types
Make sure only workflows look up the profile from the runtime
args. All other program types will run using the native profile.
2435 passed
CDAP › DBT › #779 2 years ago
CDAP-13521 fix profile property override
Fixed a bug where profile properties provided at runtime were not
correctly overriding whatever currently exists in the profile.
In addition, properties not set in the profile were being ignored
completely.

Added a method to ProfileService to do the property overrides and
added a unit test for it.
CDAP-13338 restrict profile program types
Make sure only workflows look up the profile from the runtime
args. All other program types will run using the native profile.
Testless build
Build Completed Code commits Tests
CDAP › DUT › #2225 2 years ago
CDAP-13521 fix profile property override
Fixed a bug where profile properties provided at runtime were not
correctly overriding whatever currently exists in the profile.
In addition, properties not set in the profile were being ignored
completely.

Added a method to ProfileService to do the property overrides and
added a unit test for it.
CDAP-13358 rename 'default' profile to 'native'
This rename is to avoid confusion, since the word default has
meaning about how a profile is used, but not what the profile is.
For example, the default profile for a namespace is actually
whatever profile is set for that namespace, which might be a user
created profile. Changing it to 'native' so that it describes
what the profile does.
2 of 2449 failed
CDAP › RUT › #474 2 years ago
CDAP-13614 avoid a guaranteed NPE when task state is not found
We don't expect the task state to be missing, but will fix that
in a follow up PR.
1 of 2446 failed
CDAP › DUT › #2223 2 years ago
CDAP-13614 avoid a guaranteed NPE when task state is not found
We don't expect the task state to be missing, but will fix that
in a follow up PR.
1 of 1677 failed
CDAP › RUT › #472 2 years ago
CDAP-13391 log provisioner messages in program context
Modify the provisioner tasks to use a program logging context
instead of a system logging context in order to tie the logs to
the program run and not the system logs. Even though the provisioner
runs in the CDAP master, it's logs belong with the program run,
otherwise a provisioning error will cause a program run to fail
without any feedback given to the user.

Also adding a cluster end time to each run record. This is needed
because the log handler restricts the log query end time using
the end time in the run record if it exists. However, the end time
in the run record is the program run end, whereas the cluster end
time is usually minutes later. Adjusted the log handler to restrict
using the cluster end time, so that the deprovisioning logs will
be included in the program logs.

Removing the expiresAt field from the cluster proto since it is
currently not used. It will be re-introduced when we implement
the feature that allows keeping a cluster around after stop or
failure of a program run.
2 of 1677 failed
CDAP › DUT › #2217 2 years ago
CDAP-13586 set secure store provider for clusters
Set the secure store provider to 'none' for clusters. This is
to prevent errors when it is configured to be something else on
the CDAP master. In the future, this will be configurable based
on the profile.
2089 passed
CDAP › DUT › #2214 2 years ago
CDAP-13573 configure sandbox with secure storage
Include configuration in the sandbox such that the secure store
APIs will be available by default. Though secure storage is not
really a concern in local development environments, users may
still want to try out secure macros in the sandbox.
1 of 887 failed
CDAP › DUT › #2203 2 years ago
CDAP-13296 fix race in program status call
The call to get a program status was suspectible to races because
run records for the program were not read in a single transaction.
If run state happened to change in between run record reads, the
state could incorrectly be returned as stopped.

This race condition would cascade and cause other tests to fail
that would wait for a program run to complete. If one of those
tests ran into this scenario, the test would move on assuming
the program had stopped when it in fact was still running.

Fixed to read run records in a single transaction, and added
missing unit tests.

Also fixing a bug where a suspended program would have a stopped
status instead of a running status.
1 of 1635 failed
CDAP › RUT › #453 2 years ago
CDAP-13493 remove hadoop operational stats
Removing YARN, HDFS, and HBase operational stats. These turned
out to not be very useful for people, while being actively
confusing in certain cluster setups. For example, with a secure,
HA namenode, the HDFS operational stats would log warnings that
confused most users.

In addition, these stats will be confusing once we add the cloud
integrations planned for 5.0.0
CDAP-13493 remove unneeded hadoop excludes from pom
1 of 2036 failed
CDAP › RUT › #451 2 years ago
CDAP-13501 fix flaky run history tests
Fix a race condition where a call to list run records is made first,
then calls to fetch individual run records are made and the results
compared. This is subject to race conditions where the cluster state
has changed. For example, when listing, a run record might be in the
DEPROVISIONING state. When it is individually fetched, it could be
in the DEPROVISIONED state. Instead of comparing entire run records,
just comparing run record program state, since that is really all that
is important for the test.

Also performing cleanup to remove duplicate code and to remove a bunch
of warnings.
1 of 2049 failed
CDAP › RUT › #447 2 years ago
CDAP-13489 another dynamic partioning test fix
The previous fix can fail due to a race in program status
(CDAP-13296) where run records are not fetched in a single
transaction. If a run transitions state during a status call,
the status can incorrectly be returned as STOPPED.
Instead, just wait for the number of failed runs to increase,
which should always indicate that the run has completed.
1 of 1920 failed
Build Completed Code commits Tests
CDAP › DUT › #2227 2 years ago
CDAP-13623 fix flaky service test
Fix the test to wait for the service to be running before checking
that it is available.
2436 passed
CDAP › RUT › #477 2 years ago
CDAP-13338 restrict profile program types
Make sure only workflows look up the profile from the runtime
args. All other program types will run using the native profile.
2435 passed
CDAP › DUT › #2222 2 years ago
CDAP-13391 log provisioner messages in program context
Modify the provisioner tasks to use a program logging context
instead of a system logging context in order to tie the logs to
the program run and not the system logs. Even though the provisioner
runs in the CDAP master, it's logs belong with the program run,
otherwise a provisioning error will cause a program run to fail
without any feedback given to the user.

Also adding a cluster end time to each run record. This is needed
because the log handler restricts the log query end time using
the end time in the run record if it exists. However, the end time
in the run record is the program run end, whereas the cluster end
time is usually minutes later. Adjusted the log handler to restrict
using the cluster end time, so that the deprovisioning logs will
be included in the program logs.

Removing the expiresAt field from the cluster proto since it is
currently not used. It will be re-introduced when we implement
the feature that allows keeping a cluster around after stop or
failure of a program run.
2431 passed
CDAP › BA › #315 2 years ago
CDAP-13573 configure sandbox with secure storage
Include configuration in the sandbox such that the secure store
APIs will be available by default. Though secure storage is not
really a concern in local development environments, users may
still want to try out secure macros in the sandbox.
Testless build
CDAP › RUT › #457 2 years ago
CDAP-13296 fix race in program status call
The call to get a program status was suspectible to races because
run records for the program were not read in a single transaction.
If run state happened to change in between run record reads, the
state could incorrectly be returned as stopped.

This race condition would cascade and cause other tests to fail
that would wait for a program run to complete. If one of those
tests ran into this scenario, the test would move on assuming
the program had stopped when it in fact was still running.

Fixed to read run records in a single transaction, and added
missing unit tests.

Also fixing a bug where a suspended program would have a stopped
status instead of a running status.
2384 passed
CDAP › DUT › #2198 2 years ago
CDAP-13501 fix flaky run history tests
Fix a race condition where a call to list run records is made first,
then calls to fetch individual run records are made and the results
compared. This is subject to race conditions where the cluster state
has changed. For example, when listing, a run record might be in the
DEPROVISIONING state. When it is individually fetched, it could be
in the DEPROVISIONED state. Instead of comparing entire run records,
just comparing run record program state, since that is really all that
is important for the test.

Also performing cleanup to remove duplicate code and to remove a bunch
of warnings.
2374 passed
CDAP › DUT › #2192 2 years ago
CDAP-13489 another dynamic partioning test fix
The previous fix can fail due to a race in program status
(CDAP-13296) where run records are not fetched in a single
transaction. If a run transitions state during a status call,
the status can incorrectly be returned as STOPPED.
Instead, just wait for the number of failed runs to increase,
which should always indicate that the run has completed.
2388 passed
CDAP › RUT › #446 2 years ago
CDAP-13486 fix waitForStopped
The method was incorrectly waiting for the state to not be
RUNNING instead of waiting for it to be STOPPED. This is incorrect,
as waitForStopped will return if it sees the state as STARTING.
2388 passed
CDAP › CSI › #1212 2 years ago
CDAP-13489 re-fix dynamic partitioning test
The previous fix was not correct, as it was too aggressive in
removing logic. The call to start a program does not guarantee
any state change by the time the call returns. We therefore cannot
just call waitForStopped() right after calling start().

We really need to change the way a program run is handled in
unit tests.
55 passed
CDAP › CSI › #1207 2 years ago
CDAP-13405 increase default twill reserved memory
Increasing the default twill reserved memory from 300 to 768.
I have seen YARN kill containers due to physical memory limits
when this is set to 512, but not when it is set to 768.
55 passed