Get Logs for A Spark Job

  • To get the initial logs that contain information about the pod spin up, do:

    curl -X GET -u admin:password123 http://localhost:8764/api/apollo/spark/driver/log/{jobId}
  • Get the pod ID by running:

    k get pods -l spark-role=driver -l jobConfigId=<job-id>
  • Logs from failed jobs can be obtained by using:

    kubectl logs [DRIVER-POD-NAME]
  • Logs from running containers can be tailed using the -f parameter:

    kubectl logs -f [POD-NAME]

Spark deletes failed and successful executor pods. Fusion provides a cleanup Kubernetes cron job that removes successfully completed driver pods every 15 minutes.

For related topics, see Spark Operations.