Coding Musings

Trying to share some code ideas

logo

Category: Spark

Spark – Streaming Data

Spark – Streaming Data, Capturing and Querying

Today we will look at how to capture streaming data and perform some simple queries as the data is streamed. We will use the regular expressions library and the PySpak library.  The streaming data comes from a weather station that transmits different weather at different intervals.  We will need to find the correct data out of the stream and output the results.

Let’s get started.

Read More

Spark – SQLContext

Spark – SQLContext

Today we will look at the SQLContext object from the PySpark library and how you can use it to connect to a local database.  In the example below we will:
Connect to a local PostgreSQL database and read the contents into a dataframe.
Run some simple SQL queries
And join two data frames together

Let’s get started.

Read More

Hadoop Spark – Hello World, Word Count

Hadoop Spark – Word Count

One of the first things to do in most programming languages is create a “Hello World!” program.  The equivalent in Spark is to create a program that will read the contents of a file and count the number of occurrences of each word.

Below I will show a basic example, so let’s start counting.

Read More

Powered by WordPress & Theme by Anders Norén