Development

I want to apologize for the delay on this, I wrote 95% of this article months ago then life got busy. So I am back at it now with some free time finishing it and hitting on a few new areas of knowledge to share. This is a very simple subject so this won’t be long at all.

The Code

In the same fashion as all of my coding tweets I will first give you all of the code for the entry (just make sure you insert your own values into it before using it). So here is the code:

 Prerequisites

As with most Node projects there are some modules that need to be installed. So run the following in the directory of your project:

 Code Analysis

I will be skipping the part of the code where I open up the Twitter stream and the db connection. If you wish to know how these work please see my earlier entry here.

This small block of code defines the schema for our documents that we are inserting into Mongo. It will have the tweet ID (which comes from Twitter and is unique for each tweet and also very large so we must take it as a string), the date the tweet was sent at, the text of the tweet, and the username of the person who tweeted. It’s pretty simple and self exclamatory.

This will just create an instance of the model using mongoose. Simple as that. On the name passed in as the first argument make sure you remember this as that is now your object type that you will instantiate. Thank you JavaScript for making this kind of crazy on us that don’t use you often.

Note this first line where it instantiates the object. The name is the same name you passed into your model function previously, so don’t mistype this as it will throw some rather interesting errors on you.

Next is the easy part where we are just taking out the pieces of the tweet and putting them into our model. Nothing fancy here, but just notice that our tweet_id is a string not any form of integer.

All this does is save the model to Mongo. Very simple and straight forward.

Lastly just let this run for a few hours and then check your collection to see what has been inserted. I will continue on with a more complicated example later that does some more with the Twitter API.

The Code

Here is a simple example of how to create a simple Node.js client that reads tweets off of a user stream and inserts them as JSON into a MongoDB collection.

To make it simple for people that don’t need an explanation of the code here is the code in it’s entirety (just be aware there are parts you need to change for your own data):

Now if you need to start from the get go start reading here again as I will explain exactly what this thing does. If you are new to Node.js then please read my article here before continuing as it might help with the above syntax and understanding Node.js better.

To run this simply put it into a text file and run it by running: node <filename>.js

Pre-requisites

Before running this there are a few commands you should run first. These are to install the needed modules for this code, they are pretty self-explanatory. They are:

You will also need to setup a Twitter account with an application registered for this. If you are unsure of how to do this read this document. It’s a great intro that I found when doing mine since it is not really that intuitive so maybe Twitter will make this easier to find in the future.

Notes on MongoDB Hosting

If you don’t want to host your own MongoDB instance for whatever reason I will give a huge recommendation on a hosting provider that I use. It is called MongoLab and these guys are awesome. They give you a free instance for up to 512MB of stored data and in using them for a few months they are very reliable with no downtimes so far. Plus their editor works well for seeing JSON data as will be inserted by this program.

The Code Broken Down and Dissected

Now let’s do a break down on the code itself and explain what each section does.

This is pretty easy to look at and see what it does. It creates an instance of MongoClient from the just installed module and tries to connect with it. Makes sure you change my URL string to whatever it is for yours. If it fails then it will throw an error for you to see on the command line. Next it tells the Mongo module which collection to use (if you are new to Mongo this is a rough equivalent to a database from the SQL world). You will get this from your Mongo instance.

This is the part where we connect to Twitter. You will need to log into Twitter and find the application you registered earlier and on that page is a list of keys you will use above. Just match up the labels with the keys in the code. Simple as pie (well sometimes pie isn’t that easy).

The first line of this initializes a user stream from a Twitter account and registers a callback function. The second line sets up an event-handler for when data is received from the Twitter stream and registers a callback function passing the data from the tweet into this function.

The first line of this code inserts the data (which comes from Twitter as JSON) into your Mongo collection and registers a callback function. The code inside of that callback function simply counts the number of records in the collection and spits it out to the console. The nesting of multiple callbacks can get confusing, but after writing a few it becomes pretty easy to see and if you use a good editor it will show the indentation and syntax coloring for you. I use vim as an editor, but there are plenty more out there.

 Conclusion

This was a very simple example of how to insert tweets into a MongoDB collection. Just be warned if you run this for any extended amount of time it will take up a LOT of space. For example I inserted 124 tweets and that takes up about 360KB. So just scale that and you can see it will take up a ton of space if you let it go for an extended amount of time. I recommend breaking down the JSON and inserting only what you need if you want to run this for any significant amount of time. I will probably touch on a way to do this in a future posting once I get a design created and some code created for it.

Hello World

So for the past few years there has been this crazy buzz word for a thing called Node or Node.js. As many a recruiter that has called me about jobs with it has asked what exactly is it. The simple answer is server-side JavaScript. A more complex answer is an event-based framework that runs JavaScript code using Google’s V8 JavaScript Engine. In this I will be referring to Node.js as Node or node, that’s the common term for it used around.

Now if you want to try this thing out let’s get into the process for doing that. For the purpose of simplicity I am going to assume you have a *NIX machine around such as Linux, Mac OS X, BSD, etc.

First you want to install node for your system. Go here and download the installer for your system and install it.

Next open up your favorite text editor (I prefer vim myself, but nano or pico work just as well for this exercise). Copy and paste the following code into it:

Save the file as hello.js somewhere. Do a cd to get into the directory you saved the file in.

Now run the following command: node hello.js
You will see Hello World printed on your console. If not double check you did this correctly.

That’s how to run a very simple node application. Continue on for a short demo on how to setup a very simple HTTP server along with some required knowledge for it.

Misc. Required JavaScript Knowledge

One thing that most people (like myself) that did JavaScript way back in the day never use is the event-based model in it or the functional programming aspect of it.

A huge part of programming node is a nice concept that comes from some functional programming languages. This is the ability to pass functions around as parameters to other functions. Ok yes this sounds confusing so here is a simple example:

If you run that it will output “4” to the console. It simply is creating what is called an anonymous function (think a function with no name) and passing that as a parameter just like you would the integer parameter before it. Since it is a function it can be called just like normal inside the called function. This is very useful when you want some function to perform a callback into some function you have written to perform an operation.

With that being said another important concept is that Node uses what is called an event-based model for calling functions. What this means is that it will call a function based on some event happening whether that is a connection being established to a database, a HTTP request being made to the application, closing of a file handle, etc. Note that all of this happens asynchronously meaning that execution continues and these events are triggered at any time or maybe not at all. This is where the callbacks I just talked about come in handy.

Consider the following code:

This very simple snippet creates an instance of the filesystem module from the Node standard library (it is called fs in there). Next it will try and read all contents of test.txt. If the file does not exist you will notice it throws an error about file not existing or if it does exist it will print the contents of that file. You try running it before creating a test.txt file just to see the error and how it will react.

A great way to see how it works as an event is create your test.txt file and copy about 5MB of data into it. This will take a few seconds to read the data so you will see it waiting after printing out the ‘Reading the data.’ string then when it finishes it will write out the contents of the file to the console.

Simple HTTP Server

Now that a few fundamentals are out of the way I am going to give an example of something practical, a web server.

For those of you that want to just blast through this I will just give you the code here and continue explanations later:

This code will first create an instance of the http module from the Node standard library. Next it creates the server with a very simple callback that just writes out the basic header (200 is the OK code which means no errors) and a simple text string.

If you run this simply go to http://localhost:8085 and see what it does.

Conclusion

That’s it for this installment. Hopefully this makes it a little easier to get the hang of Node.js since it is vastly different than most other programming paradigms out there nowadays. If you want to tinker around with more of the standard libraries the documentation is located here.

Next time I will try and bring in some more interesting and advanced topics like integrating with MongoDB and Twitter.