0
点赞
收藏
分享

微信扫一扫

pyspark 读取本txt 构建RDD

王小沫 2023-01-13 阅读 89


​​pyspark 读取本txt 构建RDD​​



#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Fri Mar 8 18:51:51 2019

@author: lg
"""

from pyspark import SparkContext ,SparkConf

conf=SparkConf().setAppName("miniProject").setMaster("local[1]")
#conf=SparkConf().setAppName("lg").setMaster("spark://192.168.10.182:7077")
sc = SparkContext(conf=conf)


lines = sc.textFile("data.txt")

lineLengths = lines.map(lambda s: len(s))

print (lines.collect())
totalLength = lineLengths.reduce(lambda a, b: a + b)

sc.stop()





2019-03-08 18:59 ​​luoganttcc​​


举报

相关推荐

0 条评论