During the Sentinel upgrade, some the of Spark application files are also updated. You must resubmit the Spark applications with these updated files by performing the following steps:
Log in to the SSDM server as the novell user and copy the files to the Spark history server where HDFS NameNode is installed:
cd /etc/opt/novell/sentinel/scalablestore
scp SparkApp-*.jar avroevent-*.avsc avrorawdata-*.avsc spark.properties log4j.properties manage_spark_jobs.sh root@<hdfs_node>:<destination_directory>
where <destination_directory> is any directory where you want to place the copied files. Also, ensure that the hdfs user has full permissions to this directory.
Log in to the <hdfs_node> server as the root user and change the ownership of the copied files to hdfs user:
cd <destination_directory>
chown hdfs SparkApp-*.jar avroevent-*.avsc avrorawdata-*.avsc spark.properties log4j.properties manage_spark_jobs.sh
Assign executable permission to the manage_spark_jobs.sh script.
Run the following command to stop data processing:
./manage_spark_jobs.sh stop
Clear the data processing checkpoint:
sudo -u hdfs hadoop fs -rm -R -skipTrash /<checkpoint_directory>
The location of the <checkpoint_directory> will be in the /etc/opt/novell/sentinel/scalablestore/spark.properties file. For example, /spark/checkpoint
Run the following script to re-submit the Spark jobs:
./manage_spark_jobs.sh start
The above command takes a while to complete the submit process.
(Optional) Run the following command to verify the status of the submitted Spark jobs:
./manage_spark_jobs.sh status