0
点赞
收藏
分享

微信扫一扫

配置spark的jupyter notebook kernel -- spark magic

陆公子521 2022-11-02 阅读 138


1、下载livy ​​https://livy.incubator.apache.org/​​​ 解压,进入livy文件夹,然后运行​​bin/livy-server​

2、安装sparkmagic

pip install sparkmagic
jupyter nbextension enable --py --sys-prefix widgetsnbextension

下边是可选部分:
​​​pip show sparkmagic​​,会出现类似如下结果:

Name: sparkmagic
Version: 0.12.5
Summary: SparkMagic: Spark execution via Livy
Home-page: https://github.com/jupyter-incubator/sparkmagic
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.org
License: BSD 3-clause
Location: /Users/xiligey/anaconda3/lib/python3.6/site-packages
Requires: tornado, notebook, autovizwidget, hdijupyterutils, nose, ipykernel, ipywidgets, pandas, requests-kerberos, numpy, mock, ipython, requests
Required-by:

进入到上边的location,运行

jupyter-kernelspec install sparkmagic/kernels/sparkkernel
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install sparkmagic/kernels/pyspark3kernel
jupyter-kernelspec install sparkmagic/kernels/sparkrkernel

3、启动jupyter notebook
​​​jupyter notebook​​ 然后新建一个Spark




配置spark的jupyter notebook kernel -- spark magic_不兼容


新建一个Spark Notebook


4、遇到的问题
在新建一个spark notebook之后报错:

The code failed because of a fatal error:
Failed to register auto viz for notebook.
Exception details:
"cannot import name 'DataError'".

Some things to try:
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.
c) Restart the kernel.

原因是目前而言,sparkmagic最新版和python pandas最新版本不兼容

可以回退一下pandas版本来解决报错:
​​​pip install pandas==0.22.0​

最终的界面:




配置spark的jupyter notebook kernel -- spark magic_spark_02


image.png


然后就可以开心的用spark交互界面码代码了:)

举报

相关推荐

0 条评论