spark-1.6.3最新版改动
Sub-task(子任务)
- [SPARK-16488] - Codegen variable namespace collision for pmod and partitionBy
- [SPARK-16491] - Crc32 should use different variable names (not "checksum")
- [SPARK-16514] - RegexExtract and RegexReplace crash on non-nullable input
Bug（修复Bug）
- [SPARK-6005] - Flaky test: o.a.s.streaming.kafka.DirectKafkaStreamSuite.offset recovery
- [SPARK-11301] - filter on partitioned column is case sensitive even the context is case insensitive
- [SPARK-14209] - Application failure during preemption.
- [SPARK-15541] - SparkContext.stop throws error
- [SPARK-15606] - Driver hang in o.a.s.DistributedSuite on 2 core machine
- [SPARK-16044] - input_file_name() returns empty strings in data sources based on NewHadoopRDD.
- [SPARK-16077] - Python UDF may fail because of six
- [SPARK-16078] - from_utc_timestamp/to_utc_timestamp may give different result in different timezone
- [SPARK-16182] - Utils.scala -- terminateProcess() should call Process.destroyForcibly() if and only if Process.destroy() fails
- [SPARK-16257] - spark-ec2 script not updated for 1.6.2 release
- [SPARK-16313] - Spark should not silently drop exceptions in file listing
- [SPARK-16353] - Intended javadoc options are not honored for Java unidoc
- [SPARK-16375] - [Spark web UI]:The wrong value(numCompletedTasks) has been assigned to the variable numSkippedTasks
- [SPARK-16385] - NoSuchMethodException thrown by Utils.waitForProcess
- [SPARK-16409] - regexp_extract with optional groups causes NPE
- [SPARK-16422] - maven 3.3.3 missing from mirror, breaks older builds
- [SPARK-16440] - Undeleted broadcast variables in Word2Vec causing OoM for long runs
- [SPARK-16489] - Test harness to prevent expression code generation from reusing variable names
- [SPARK-16656] - CreateTableAsSelectSuite is flaky
- [SPARK-16664] - Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.
- [SPARK-16751] - Upgrade derby to 10.12.1.1 from 10.11.1.1
- [SPARK-16873] - force spill NPE
- [SPARK-16925] - Spark tasks which cause JVM to exit with a zero exit code may cause app to hang in Standalone mode
- [SPARK-16939] - Fix build error by using `Tuple1` explicitly in StringFunctionSuite
- [SPARK-17003] - release-build.sh is missing hive-thriftserver for scala 2.11
- [SPARK-17038] - StreamingSource reports metrics for lastCompletedBatch instead of lastReceivedBatch
- [SPARK-17245] - NPE thrown by ClientWrapper.conf
- [SPARK-17356] - A large Metadata filed in Alias can cause OOM when calling TreeNode.toJSON
- [SPARK-17404] - [BRANCH-1.6] Broken test: showDF in test_sparkSQL.R
- [SPARK-17418] - Spark release must NOT distribute Kinesis related assembly artifact
- [SPARK-17465] - Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak
- [SPARK-17531] - Don't initialize Hive Listeners for the Execution Client
- [SPARK-17547] - Temporary shuffle data files may be leaked following exception in write
- [SPARK-17617] - Remainder(%) expression.eval returns incorrect result
- [SPARK-17618] - Dataframe except returns incorrect results when combined with coalesce
- [SPARK-17678] - Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"
- [SPARK-17696] - Race in CoarseGrainedExecutorBackend shutdown can lead to wrong exit status
- [SPARK-17721] - Erroneous computation in multiplication of transposed SparseMatrix with SparseVector
- [SPARK-17884] - In the cast expression, casting from empty string to interval type throws NullPointerException
Improvement(改进)
- [SPARK-2424] - ApplicationState.MAX_NUM_RETRY should be configurable
- [SPARK-15761] - pyspark shell should load if PYSPARK_DRIVER_PYTHON is ipython an Python3
- [SPARK-16341] - [SQL] In regexp_replace function column and/or column expression should also allowed as replacement.
- [SPARK-16796] - Visible passwords on Spark environment page
- [SPARK-17316] - Don't block StandaloneSchedulerBackend.executorRemoved
- [SPARK-17378] - Upgrade snappy-java to 1.1.2.6
- [SPARK-17485] - Failed remote cached block reads can lead to whole job failure
- [SPARK-17649] - Log how many Spark events got dropped in LiveListenerBus
New Feature(新特征)
- [SPARK-16956] - Make ApplicationState.MAX_NUM_RETRY configurable
