Build #2,524

Cask Data Application Platform - Develop Build and Test

Build: #2524 failed Child of CDAP-DRC-4361

Code commits

caskdata/cdap

  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> 2e14120c3557dd7e82964dcf6e9bec1d709afc6e

    Merge pull request #11144 from cdapio/feature/CDAP-14984-master-env-task
    (CDAP-14984) Introduce master environment task

  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> a376eb79a9d1a13bbef219b955734761d285403d

    Merge pull request #11143 from cdapio/feature/CDAP-15003-fix-preview-logs
    (CDAP-15003) Fix preview log endpoint

  • Terence Yim <terencey@google.com>

    Terence Yim <terencey@google.com> bad9e9f5a88692cbdd66fba5e04ebd719b8f2933

    (CDAP-14984) Introduce master environment task
    - Also implemented a pod killer to periodically kill the preview pod

    • cdap-kubernetes/src/main/java/co/cask/cdap/master/environment/k8s/KubeMasterEnvironment.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-kubernetes/src/main/java/co/cask/cdap/master/environment/k8s/PodKillerTask.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-kubernetes/src/test/java/co/cask/cdap/master/environment/k8s/PodKillerTaskTest.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master-spi/src/main/java/co/cask/cdap/master/spi/environment/MasterEnvironment.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master-spi/src/main/java/co/cask/cdap/master/spi/environment/MasterEnvironmentTask.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/AbstractServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/AppFabricServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/MasterTaskExecutorService.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/MessagingServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/MetadataServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/MetricsServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/PreviewServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/RouterServiceMain.java (version bad9e9f5a88692cbdd66fba5e04ebd719b8f2933)
  • Terence Yim <terencey@google.com>

    Terence Yim <terencey@google.com> e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd

    (CDAP-15003) Fix preview log endpoint
    - Read logs from inside the preview space

    • cdap-app-fabric/src/main/java/co/cask/cdap/app/guice/AppFabricServiceRuntimeModule.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-app-fabric/src/main/java/co/cask/cdap/app/preview/PreviewHttpModule.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-app-fabric/src/main/java/co/cask/cdap/app/preview/PreviewManager.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-app-fabric/src/main/java/co/cask/cdap/gateway/handlers/preview/PreviewHttpHandler.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/preview/DefaultPreviewManager.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-app-fabric/src/test/java/co/cask/cdap/internal/app/preview/DefaultPreviewManagerTest.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/PreviewServiceMain.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-standalone/src/main/java/co/cask/cdap/StandaloneMain.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-unit-test/src/main/java/co/cask/cdap/test/TestBase.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/gateway/handlers/AbstractLogHandler.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/gateway/handlers/LogHandler.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/meta/FileMetaDataReader.java (version e2373af6c4b3f6d8f323238a21c72d0b88b5c6bd)
  • Vinisha Shah <CuriousVini@users.noreply.github.com>

    Vinisha Shah <CuriousVini@users.noreply.github.com> 376029e521a0dc63bd2dcc53e81f86bb82f34597

    Merge pull request #11094 from cdapio/feature/CDAP-14881-add-log-handler
    [CDAP-14881] Adding log handler, RemoteLogAppender and LogBufferService

  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> 2e4866c4f6d5de41ad57a12296b219d415bd40ca

    Merge pull request #11138 from cdapio/feature/CDAP-14645-fix-dependencies
    (CDAP-14645) Filter out fasterxml classes in Spark classloader

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 246a3be5e58d2210512529521cbad8f74ca3d167

    Merge pull request #11141 from cdapio/feature/CDAP-14998-fix-memory-leak
    CDAP-14998 fix DatasetRuntimeContext leak

  • Andreas Neumann <andreas@cask.co>

    Andreas Neumann <andreas@cask.co> 95362edcbdac7c0971a32e0a04658b432e0cd464

    Merge pull request #11133 from cdapio/feature/CDAP-14959-metadata-client-changes
    [CDAP-14993] Implement consistent cursor semantics

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 0b2976f02d44ba3f5cbf7cff0e06a88e14d427a7 m

    CDAP-14998 fix DatasetRuntimeContext leak
    Fixed a memory leak caused by the DefaultDatasetRuntimeContext.
    The context uses a ThreadLocal variable, which means every time
    an instance is created, another ThreadLocal instance gets added
    to the Thread's local map. These variables were never getting
    removed, which means long running threads would continually
    accumulate these variables until running out of memory.

    Made the DatasetRuntimeContext implement Closeable and made sure
    datasets always close the runtime context when the dataset is
    closed. The DefaultDataRuntimeContext then removes the thread
    local variable in its close method, preventing the leak.

    • cdap-api/src/main/java/co/cask/cdap/internal/dataset/DatasetRuntimeContext.java (version 0b2976f02d44ba3f5cbf7cff0e06a88e14d427a7)
    • cdap-common/src/main/java/co/cask/cdap/common/dataset/DatasetClassRewriter.java (version 0b2976f02d44ba3f5cbf7cff0e06a88e14d427a7)
    • cdap-data-fabric/src/main/java/co/cask/cdap/data2/dataset2/DefaultDatasetRuntimeContext.java (version 0b2976f02d44ba3f5cbf7cff0e06a88e14d427a7)
  • Terence Yim <terencey@google.com>

    Terence Yim <terencey@google.com> dcf51c6692d4a91767548b44533e2c36bef23d4a m

    (CDAP-14645) Fix preview execution in distributed mode and in K8s
    - Filter out fasterxml classes for SparkRunnerClassLoader
    - Make sure preview is using local FS
    - Remove copying of Hadoop from dependency
    - Use hadoop classpath command to setup Hadoop classpath
    - Also remove unneeded dependencies

    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/preview/DefaultPreviewManager.java (version dcf51c6692d4a91767548b44533e2c36bef23d4a)
    • cdap-common/bin/functions.sh (version dcf51c6692d4a91767548b44533e2c36bef23d4a)
    • cdap-common/pom.xml (version dcf51c6692d4a91767548b44533e2c36bef23d4a)
    • cdap-master/pom.xml (version dcf51c6692d4a91767548b44533e2c36bef23d4a)
    • cdap-spark-core-base/src/main/java/co/cask/cdap/app/runtime/spark/SparkProgramRuntimeProvider.java (version dcf51c6692d4a91767548b44533e2c36bef23d4a)
    • pom.xml (version dcf51c6692d4a91767548b44533e2c36bef23d4a)
  • Vinisha Shah

    Vinisha Shah aece2892e2da8bcf9173a1581e7ddf930aa568f8 m

    [CDAP-14881] Adding log handler, RemoteLogAppender and LogBufferService

    • cdap-common/src/main/java/co/cask/cdap/common/conf/Constants.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-common/src/main/resources/cdap-default.xml (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-data-fabric/src/main/java/co/cask/cdap/store/StoreDefinition.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/appender/AbstractLogPublisher.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/appender/remote/RemoteLogAppender.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/appender/tms/TMSLogAppender.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/framework/distributed/DistributedLogFramework.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/guice/RemoteLogAppenderModule.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/logbuffer/ConcurrentLogBufferWriter.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/logbuffer/LogBufferService.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/main/java/co/cask/cdap/logging/logbuffer/handler/LogBufferHandler.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/test/java/co/cask/cdap/logging/logbuffer/ConcurrentLogBufferWriterTest.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/test/java/co/cask/cdap/logging/logbuffer/MockCheckpointManager.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
    • cdap-watchdog/src/test/java/co/cask/cdap/logging/logbuffer/handler/LogBufferHandlerTest.java (version aece2892e2da8bcf9173a1581e7ddf930aa568f8)
  • Andreas Neumann

    Andreas Neumann 0ff280076e0a0553e92c819930c19c2dbb67cc33 m

    [CDAP-14959] Implement consistent cursor semantics

    • cdap-common/src/main/java/co/cask/cdap/common/metadata/Cursor.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-common/src/test/java/co/cask/cdap/common/metadata/CursorTest.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/dataset/Cursor.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/dataset/DatasetMetadataStorage.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-data-fabric/src/test/java/co/cask/cdap/spi/metadata/dataset/DatasetMetadataStorageTest.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-elastic/src/main/java/co/cask/cdap/metadata/elastic/Cursor.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-elastic/src/main/java/co/cask/cdap/metadata/elastic/ElasticsearchMetadataStorage.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)
    • cdap-elastic/src/test/java/co/cask/cdap/metadata/elastic/ElasticsearchMetadataStorageTest.java (version 0ff280076e0a0553e92c819930c19c2dbb67cc33)