Building an SMS weather and image bot

August 11, 2016

Notice: This post uses a legacy version of our API. Visit out developer portal to view current API documentation.

Why an SMS bot?

With the rise in popularity of bots on the internet, we at Bandwidth thought it might be fun to see how one would work using the power of SMS. Another reason we thought this would be something cool to do is that it could potentially replace a number of single-function apps such as weather, translators, payment processing, etc. on your phone with “one app to rule them all”. I also think this is a particularly interesting thing to be working on because a lot of these services use Machine Learning, and they’re only going to get better allowing app developers to harness these improvements with no changes.

This tutorial doesn’t focus on developing the front-end mobile app, but rather talks about a possible way of setting up a simple back-end for an SMS Weather and Image Bot using Node.js.

What Does it Do?

The back-end service that we’re going to build has two main functions:

It tells us the weather (for the next 10 days).
It describes images by returning five relevant tags.

Prerequisites

In order to deploy the app, you’re going to want to make sure you have the relevant API keys by signing up on the following platforms.

SMS is done using the Bandwidth Communication Platform (The Bandwidth Node.js Client Library is a pretty good place to start).
Natural Language Processing is done using wit.ai (If you’re unfamiliar with wit.ai, their docs are a pretty good place to start).
Image tagging is done using Clarifai.
Weather information is received using Forecast.io.
Geocoding is done with Google Maps Geocoding API.
The app is deployed on Heroku.
Heroku uses AWS for image storage.
Express for routing requests.

What does this look like?

Code Walkthrough: Setting Up the Web Server

The code below is required to tell the Web server to start running on the port Heroku deems suitable and to parse the body of the request and make it available to us.

In addition, all POST requests to the /messages endpoint will hit this part of the code, and the logic for handling them should be inside this block.

app.set('port', (process.env.PORT || 5000));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({
    extended: true
}));

app.post('/messages', function(request, response) {
    if (request.body.media) {
        //handle the case when an image Url is present in the body
    }
    else {
        //get context from the text of the SMS to figure out what the message is
    }
});

Next we’re going to take a look at how the different requests are handled. Calls are made to the respective functions, depending on the content of the payload. In this case, when an image Url is found (by looking at the media field of the request body), the image recognition API is called. If no image is found, the weather API is called.

Get Image Tags

Clarifai requires a publicly accessible URL in order to get the image so that it can return a list of tags for the image. The Bandwidth Communications Platform, as a security measure, prevents images from being publicly viewable. In order to overcome this limitation, the app stores the image on AWS, using its S3 service, and passes this URL to Clarifai. This requires creating a bucket on S3, details on which can be found here.

The code shown below is used to make a call to the Clarifai API, which then has a callback with the results it provides. This data is sent back to where it was requested from, in order to be sent back to the user via a text message.

var getImageTags = function (pictureUrl, callback) {
  Clarifai.getTagsByUrl([
    'https://s3.amazonaws.com/' + process.env.S3_BUCKET_NAME + '/media/' + pictureUrl
  ], function (err, res) {
        //process result from Clarifai API call
  });
};

Before this code is executed, the image had to be stored in the S3 bucket designated for this app. This is done with the official AWS Node.js SDK. The code below, which again uses callbacks shows this.

var putImageS3 = function (pictureUrl, callback) {
   let url = 'https://' + process.env.BANDWIDTH_CLIENT_API_TOKEN + ':' 
       + process.env.BANDWIDTH_CLIENT_API_SECRET + '@api.catapult.inetwork.com/v1/users/' 
       + process.env.BANDWIDTH_CLIENT_USER_ID + '/media/' + pictureUrl;

   request({
       url: url,
       encoding: null
   }, function(err, res, body) {
       if (err) {
           return callback(err, res);
       }

       s3.putObject({
           Bucket: process.env.S3_BUCKET_NAME,
           Key: 'media/' + pictureUrl,
           ContentType: res.headers['content-type'],
           ContentLength: res.headers['content-length'],
           Body: body
       }, function (err, res) {
          //process result from AWS S3 call.
           }
       });
   });
}

One thing to note here is that the URL that is provided has your Bandwidth API token and API Secret Key in it, which is used to authenticate and actually get the image.

Getting the Weather

The next step is implementing the Weather API, in this Forecast.io. An interesting thing here is that an SMS doesn’t contain the location of the user, and hence needs to be specified. One way to get around this would be to include the location while sending the SMS, but this would require a specialized app on a phone. In order to get around this, the address sent or zip code sent by the user is geocoded, and its latitude and longitude is used to determine the location for the weather API. The two code snippets below show this:

geocoder.geocode(weatherInput.location, function(err, res) {    weatherInput.lat = res[0].latitude;    weatherInput.long = res[0].longitude;    getWeather(weatherInput, function (err, res) {        callback(null, res);    }); } var getWeather = function (weatherInput, callback) {    forecast.get([weatherInput.lat, weatherInput.long], function(err, res) {        if(err) {            callback(err, null);        }        else if (weatherInput.time - Date.now() < 0) {            //get today's weather        }        else {            for (it in res.daily.data) {                if (weatherInput.time/1000 - res.daily.data[it].time < 86400) {                    //get the weather for some day in the future                }            }        }    }); }

The Forecast.io API returns a JSON object that needs to be parsed, and a message that is to be sent to the user is constructed here and sent through the callback.

Sending the SMS to the User

The final step involves sending the information received by making an API call back to the user. This is done using the Bandwidth Communications Platform, and the code is written using our super simple Node.js package. Our Client Library has support for both callbacks, and the newer ES6 style promises. If you’re unfamiliar with promises, more information can be found over here.

The first step to using our Client Library is defining the client object, code for which is shown below.

var client = new CatapultClient({
   userId    : process.env.BANDWIDTH_CLIENT_USER_ID,
   apiToken  : process.env.BANDWIDTH_CLIENT_API_TOKEN,
   apiSecret : process.env.BANDWIDTH_CLIENT_API_SECRET
});

The next step, which is actually sending the SMS is also surprisingly easy and code for this is shown below. (Both you and your team must at all times comply with applicable laws and industry practices prohibiting spam and other unwanted text messages. We are not your lawyers, but – If you do not know and understand these rules – invest the time to understand them.)

This snippet uses the aforementioned concept of promises. What this means is that a portion of code is only executed once the previous section has completed. Here a section can be defined as code that is inside a then or catch block. Only once the SMS is actually sent, and a response is received from the Bandwidth Communications Platform will the code in the then block (the console.log statement in this case) execute. In case anything goes wrong either with the send or the then block, the catch block is immediately called up, and the code inside it is executed.

var sendMessage = function (sendTo, content) {

   client.Message.send({
       from : process.env.BANDWIDTH_PHONE_NUMBER, 
       to   : sendTo,
       text : content
   })
   .then(function(message) {
       console.log('Message sent with ID ' + message.id);
   })
   .catch(function(err) {
       console.log("Message failed to send: " + err);
   });
}

Once this piece of code has executed successfully, the user should have received a response to their query.

What’s next?

Now that you have a simple back-end service listening for text messages sent to a particular number, you can do pretty much anything you want! Some things I think might be interesting to integrate with the service are translation APIs, payment APIs, news APIs, flight trackers, and IoT devices.