Contents

In case of Spark2 you can enable the DEBUG logging as by invoking the 'sc.setLogLevel('DEBUG')' as following: $ export SPARKMAJORVERSION=2 $ spark-shell -master yarn -deploy-mode client SPARKMAJORVERSION is set to.
If everything is all right, then: DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library. INFO util.NativeCodeLoader - Loaded the native-hadoop library If something goes wrong, then: INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable.

Python Spark Shell

17/11/13 12:10:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable 17/11/13 12:10:22 WARN Utils: Your hostname, tutorialkart resolves to a loopback address: 127.0.0.1; using 192.168.0.104 instead (on interface wlp7s0).

Spark Shell is an interactive shell through which we can access Spark’s API. Spark provides the shell in two programming languages : Scala and Python. In this tutorial, we shall learn the usage of Python Spark Shell with a basic word count example.

Python Spark Shell

Prerequisites

Prerequisite is that Apache Spark is already installed on your local machine. If not, please refer Install Spark on Ubuntu or Install Spark on MacOS based on your Operating System.

Start Spark Interactive Python Shell

Python Spark Shell can be started through command line. To start pyspark, open a terminal window and run the following command :

For the word-count example, we shall start with option--master local[4] meaning the spark context of this spark shell acts as a master on local node with 4 threads.

If you accidentally started spark shell without options, you may kill the shell instance.

~$ pyspark--master local[4]

[GCC5.4.020160609] on linux2

Type'help', 'copyright', 'credits'or'license'formore information.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

17/11/13 12:10:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/11/13 12:10:22 WARN Utils: Your hostname, tutorialkart resolves to a loopback address: 127.0.0.1; using 192.168.0.104 instead (on interface wlp7s0)

17/11/13 12:10:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

17/11/13 12:10:40 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

____ __

_ / _ / _ `/ __/ '_/

/_/

Using Python version2.7.12 (default, Nov19201606:48:10)

>>>

Spark context Web UI would be available at http://192.168.0.104:4040 [The default port is 4040]. Open a browser and hit the url http://192.168.0.104:4040.

Spark context : You can access the spark context in the shell as variable named sc.

Macos Unable To Load Native-hadoop Library For Your Platform Download

Spark session : You can access the spark session in the shell as variable named spark.

Word-Count Example with PySpark

We shall use the following Python commands in PySpark Shell in the respective order.

input_file=sc.textFile('/path/to/text/file')

map=input_file.flatMap(lambda line:line.split(' ')).map(lambda word: (word, 1))

counts.saveAsTextFile('/path/to/output/')

Input

In this step, using Spark context variable, sc, we read a text file

Map

We can split each line of input using space ” ” as separator

and we map each word to a tuple (word, 1), 1 being the number of occurrences of word.

We use the tuple (word,1) as (key, value) in reduce stage.

Reduce

Reduce all the words based on Key. Here a, b are values and for the same key, values are reduced to a+b.

Save counts to local file

At the end, counts could be saved to a local file.

When all the commands are run in Terminal, following would be the output :

Word Count Example using Python Spark Shell - PySpark

>>>input_file=sc.textFile('/home/arjun/data.txt')

>>>map=input_file.flatMap(lambda line:line.split(' ')).map(lambda word: (word, 1))

>>>counts.saveAsTextFile('/home/arjun/output/')

Output can be verified by checking the save location.

/home/arjun/output$ls

Sample of the contents of output file, part-00000, is shown below :

/home/arjun/output$cat part-00000

(sent,1)

(tasks,4)

We have successfully counted unique words in a file with the help of Python Spark Shell – PySpark.

You can use Spark Context Web UI to check the details of the Job (Word Count) we have just run.

Navigate through other tabs to get an idea of Spark Web UI and the details about the Word Count Job.

Conclusion

In this Apache Spark Tutorial, we have learnt the usage of Spark Shell using Python programming language with the help of Word Count Example.

Native Hadoop Library

Overview

This guide describes the native hadoop library and includes a small discussion about native shared libraries.

Macos Unable To Load Native-hadoop Library For Your Platform School

Note: Depending on your environment, the term 'native libraries' could refer to all *.so's you need to compile; and, the term 'native compression' could refer to all *.so's you need to compile that are specifically related to compression. Currently, however, this document only addresses the native hadoop library (libhadoop.so).

Native Hadoop Library

Hadoop has native implementations of certain components for performance reasons and for non-availability of Java implementations. These components are available in a single, dynamically-linked native library called the native hadoop library. On the *nix platforms the library is named libhadoop.so.

Usage

It is fairly easy to use the native hadoop library:

Review the components.
Review the supported platforms.
Either download a hadoop release, which will include a pre-built version of the native hadoop library, or build your own version of the native hadoop library. Whether you download or build, the name for the library is the same: libhadoop.so
Install the compression codec developme