I have spark job which is reading a file, connecting to DB server doing some calculations and then create a different
Since yesterday its continue to fail with this error (even tho it creates an output file):
error WARN ShutdownHookManager: ShutdownHook ‘$anon$2’ timeout
How to fix it?
I did search and found that somebody recommend to use ‘explicitly invoke
sparkContext.stop() before exiting the application’. but the problem if I am adding this like of code to my spark code I cant compile it, on compilation it throws me an error:
not found: value sparkContext [error] sparkContext.stop() [error]
^ [error] one error found [error] (compile:compileIncremental)