Initial commit for Spark to analyze visibility data 89/63889/1
authorearrage <eddie.arrage@huawei.com>
Mon, 22 Oct 2018 20:53:54 +0000 (13:53 -0700)
committerEddie Arrage <eddie.arrage@huawei.com>
Mon, 22 Oct 2018 21:07:24 +0000 (21:07 +0000)
commit4f95f77d4f1e1a4455360690c02e06eeb60fc9da
treea6a1295d43b1f14ca29c7ee372bbb114cdd4d03b
parent31d8ae98dc68152da3965d7ce6e3be5bc90a35da
Initial commit for Spark to analyze visibility data

- Add Apache Spark 2.3 with native Kubernetes support.
- Runs self contained within K8s cluster in clover-system
namespace. One container (clover-spark) includes Clover Spark
JAR artifact. This container interacts with the
K8s API to spawn a spark-driver pod. This pod in turn spawns executor
pods to execute Spark jobs.
- Currently JAR is included in source for convenience and must be
built with sbt (install sbt and execute sbt package)
- Includes JAR from DataStax to provide Cassandra connector to analyze
Cassandra schemas as RDDs (Resilient Distributed Dataset).
- Includes Redis interface JAR to write analyzed data back to visibility
(UI, CLI or API).
- Second container (clover-spark-submit) submits Spark jobs
continuously to allow Spark to be operated entirely within the cluster.
- Two Spark jobs (CloverSlow, CloverFast) allows some analytics to be
provided in real-time and other analytics to be provided over longer
horizons.
- Each Spark job spawns two executor pods.
- Includes yaml manifest to deploy clover-spark-submit with the
necessary RBAC permissions to interact with the K8s API.
- Data analyzed includes tracing and metrics schemas obtained by
clover-collector and written to Cassandra.
- Docker builds of clover-spark and clover-spark-submit are provided
and will be pushed as OPNFV DockerHub images in a separate patch.

Change-Id: I2e92c41fd75d4ebba948c0f8cb60face57005e50
Signed-off-by: earrage <eddie.arrage@huawei.com>
13 files changed:
clover/spark/build.sbt [new file with mode: 0644]
clover/spark/docker/clover-spark/Dockerfile [new file with mode: 0644]
clover/spark/docker/clover-spark/build.sh [new file with mode: 0755]
clover/spark/docker/clover-spark/jars/clover-spark_2.11-1.0.jar [new file with mode: 0644]
clover/spark/docker/clover-spark/jars/datastax_spark-cassandra-connector-2.3.0-s_2.11.jar [new file with mode: 0644]
clover/spark/docker/clover-spark/jars/redisclient_2.11-3.7.jar [new file with mode: 0644]
clover/spark/docker/spark-submit/Dockerfile [new file with mode: 0644]
clover/spark/docker/spark-submit/build.sh [new file with mode: 0755]
clover/spark/docker/spark-submit/runner.sh [new file with mode: 0755]
clover/spark/docker/spark-submit/runner_fast.sh [new file with mode: 0755]
clover/spark/src/main/scala/CloverFast.scala [new file with mode: 0644]
clover/spark/src/main/scala/CloverSlow.scala [new file with mode: 0644]
clover/spark/yaml/clover-spark.yaml [new file with mode: 0644]