One of the first things to do in most programming languages is create a “Hello World!” program.\u00a0 The equivalent in Spark is to create a program that will read the contents of a file and count the number of occurrences of each word.<\/p>\n
Below I will show a basic example, so let’s start counting.<\/p>\n
<\/p>\n
I loaded the file into the hadoop file system.\u00a0 After loading the file we will use Python to process the file.<\/p>\n
lines = sc.textFile(\"hdfs:\/user\/cloudera\/words.txt\")\r\nlines.count()\r\n<\/pre>\nRead the contents of the words.txt file and store it in a variable called lines.\u00a0 Next will ask for how many lines are in the file; result is<\/p>\n
124456<\/pre>\nNext we will split the words on logical word boundaries store in the list variable words.\u00a0 In this case I am using space as the word boundary.<\/p>\n
words = lines.flatMap(lambda line : line.split(\" \"))\r\nwords.take(5)<\/pre>\nAnd then ask for the first five words in the list.<\/p>\n
['This', 'is', 'the', '100th', 'Etext']<\/pre>\nNext we will covert the list into a tuple. And with each tuple we will assign a value of 1.\u00a0 When we do the count of words, it will sum the value in the second part of the tuple and show the results.<\/p>\n
tuples = words.map(lambda word : (word, 1))\r\ntuples.take(5)<\/pre>\nAnd the result for the first five elements in the tuple are<\/p>\n
[('This', 1), ('is', 1), ('the', 1), ('100th', 1), ('Etext', 1)]<\/pre>\nIt seems to make perfect sense.\u00a0 We now have a list of tuples with each word and a value of 1. Now we can sum up the values by using key-pairs.\u00a0 Every time two matching keys are found, Spark will merge the keys and add the values in the second element.<\/p>\n
counts = tuples.reduceByKey(lambda a, b: (a+b))\r\ncounts.take(5)<\/pre>\nThe results were stored in a variable called counts.\u00a0 And the first five values are shown below.<\/p>\n
[('', 517065), ('Quince', 1), ('Corin,', 2), ('Just', 10), ('enrooted', 1)]<\/pre>\nWe can see there were 517065 non-printable characters which we can ignore; and there was 10 occurrences of the word “Just”, etc..<\/p>\n
Now we can save our results to the hdfs partition.<\/p>\n
counts.coalesce(1).saveAsTextFile(\"hdfs:\/user\/cloudera\/wordcount\/outputDir\")<\/pre>\nAnd now we have our first Spark, “Hello World!” program.<\/p>\n
I hope you found the above informative, let me know if you have any questions in the comments below.<\/p>\n