Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for SMS conversations #7

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 30 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,19 +43,42 @@ and you've built your own hotline!
The main service runs using [AWS Lambda](https://aws.amazon.com/documentation/lambda/), a
scalable, serverless platform which removes the overhead of needing to maintain an underlying
server. The [Zappa documentation](https://github.com/Miserlou/Zappa#zappa---serverless-python-web-services)
provides detailed set-up instructions, but if you have your `~/.aws/credentials` file in order,
it should be as simple as:
provides detailed set-up instructions, but the process should be as simple as:

$ . venv/bin/activate
$ zappa init
$ zappa deploy
1. [Create a private S3 bucket](https://s3.console.aws.amazon.com/s3/home). The Rick Astley Hotline will store the state of its SMS conversations here. Take a note of its name.
2. [Create an IAM user](https://console.aws.amazon.com/iam/home?region=us-east-1#/users$new?step=details) that has access to the S3 bucket. If your bucket from the previous step was called `rick-astley-data`, the IAM policy to grant access should look something like this:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::rick-astley-data",
"arn:aws:s3:::rick-astley-data/*"
]
}
]
}

Take a note of the Access Key ID and Secret Access Key of your new IAM user.

3. Sign up for [Twilio](https://www.twilio.com), create a new project, and take a note of its SID and access token.
4. Copy `zappa_settings.example.json` to `zappa_settings.json`. Copy the IAM details, S3 bucket name and Twilio details into the appropriate places.
5. Deploy your project to AWS Lambda:

Record the URL it spits out, connect it as a webhook to your own phone number with Twilio, and you have your own serverless Rick Astley Hotline!
$ . venv/bin/activate
$ zappa init
$ zappa deploy

6. Record the URL it spits out, connect it as the incoming call webhook to your own phone number with Twilio. Set the incoming SMS webhook to the same URL, but with `/sms` on the end. (You can do this for more than one number.)

Congratulations! You now have your own serverless Rick Astley Hotline!
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation! You are amazing! ❤️


## How much do you spend bringing joy to Rick Astley fans?

As of the end of 2016, number and connection costs were averaging about $150 USD/yr. It's totally
worth it for the joy it brings others.

If you want to defray my hosting costs, there's always [bitcoin](https://blockchain.info/address/18pgvfqWGs2CvurmNvq58h499RRTPCh3mz) and [Patreon](https://www.patreon.com/_pjf).

5 changes: 4 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,17 @@ Flask==0.11.1
Jinja2==2.8
MarkupSafe==0.23
PyYAML==3.12
Unidecode==0.04.19
Unidecode==0.4.19
Werkzeug==0.11.11
argparse==1.2.1
base58==0.2.4
blinker==1.4
boto3==1.4.1
botocore==1.4.80
certifi==2016.9.26
cffi==1.9.1
click==6.6
contextlib2==0.5.5
cryptography==1.5.3
docutils==0.12
enum34==1.1.6
Expand All @@ -30,6 +32,7 @@ pycparser==2.17
python-dateutil==2.6.0
python-slugify==1.2.1
pytz==2016.7
raven==6.1.0
requests==2.12.3
s3transfer==0.1.9
six==1.10.0
Expand Down
73 changes: 72 additions & 1 deletion rickroll.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,19 @@
app = Flask(__name__)

from twilio import twiml
from twilio.rest import TwilioRestClient

import boto3
import botocore
import os
from datetime import datetime, timedelta
from raven.contrib.flask import Sentry
sentry = Sentry(app)

s3 = boto3.resource('s3')
bucket = s3.Bucket(os.environ['data_bucket'])
twilio_client = TwilioRestClient(os.environ['twilio_sid'],
os.environ['twilio_token'])

# Where we're storing all our audio files.
url_base = "https://s3-us-west-2.amazonaws.com/true-commitment/"
Expand Down Expand Up @@ -68,6 +81,14 @@
_original
]

messages = [
"We're no strangers to love.",
"You know the rules, and so do I.",
"A full commitment's what I'm thinking of.",
"You wouldn't get this from any other guy.",
"Call me?",
]

# Menu generation. I'd love to put this in its own function to be clean and
# tidy, but if I put that at the end Python gets grumpy and I'm not sure how to
# forward-declare. I could put it into a separate file and include that, but
Expand Down Expand Up @@ -105,7 +126,7 @@ def play_tune(tune):
gather = response.gather(numDigits=1, timeout=10)
gather.play(tune['url'])
gather.say(menu)

# Our goodbye triggers after gather times out.
response.say(goodbye)

Expand Down Expand Up @@ -153,5 +174,55 @@ def original():

return str(play_tune(tune))

@app.route("/sms", methods=["POST"])
def sms():
"""Adds an incoming SMS to a queue, to reply to later."""
bucket.put_object(
Key='queue/{to}/{from_}'.format(to=request.form['To'],
from_=request.form['From']),
Body='',
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this lose the message they sent us? It would be visible in the Twilio logs, but in terms of maximum amusement for line operators, being able to keep the conversation in a more accessible form would be amazing. :)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or am I correct that these queued messages get deleted later (when we process them), so storing the SMS received in them is not exactly useful. :)

Would it make more sense to have a separate logs/ folder, which just logs incoming and outbound messages and timestamps,. but doesn't play a role in message queuing at all? (Absence of a logs directory isn't a blocker for this change, but is certainly a nice-to-have!)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's exactly how it works! (I meant to add a PR comment summarising how everything works, but ran out of time.)

)
return "Hello world!"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong here, but my understanding is that Twilio requires a valid TwiML response in order to consider a message processed. If that return is going back to Twilio (which I think it is), then this will make Twilio unhappy.

We should be able to get away with just <Response></Response> as a return, which is a valid return that does nothing.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Twilio didn't generate an error message when I tested it, contrary to their docs. (Perhaps because I'm sending a response with whatever Flask's default Content-Type is (text/plain maybe?) rather than the TwiML type, it's not expecting to be able to do anything with it?)

In any case, I agree that what you're suggesting is nicer, whether it's strictly required or not.


def send_sms():
"""Empties the queue of incoming SMSes, replying to each one."""
for queue_entry in bucket.objects.filter(Prefix='queue/'):
_, our_number, their_number = queue_entry.key.split("/")
state_key = "state/{}/{}".format(our_number, their_number)

# Find out which message we sent last, if any
# Error handling taken from https://stackoverflow.com/a/33843019
try:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code climate is point out that this function is a bit complex, and this seems perfect for factoring out to a separate function which returns the next message number (or raises on error).

I'm guessing that if we do throw an exception, we should log that but still try to process the other messages in our queue, so a single bad message doesn't knock out the entire SMS queue. (This isn't a blocker for this PR, but indicates future work that may be required.)

I have no idea what the best way is to log an error from Lambda.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, no issues on separating out into a separate function. I'll do both that and what you're suggesting with error catching as well.

For logging errors from Lambda, I'd use Sentry; it's a really well-thought-out tool with a generous free tier, and the whole thing is open source too! (We run the open-source version on-prem at work.)

state_obj = s3.Object(
os.environ['data_bucket'],
state_key,
)
state_obj.load()
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
last_msg = -1
else:
raise
else:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python n00b 'woah' moment here... You can have a try/except/else statement? That's cool!

Am I correct the else happens if no exception is raised?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep!

You can also attach an else block to a for or while loop, which executes if the loop runs to completion (as opposed to hitting a break).

last_msg = int(state_obj.get()['Body'].read().decode('utf8'))

# Send the next one
next_msg = last_msg + 1
if next_msg < len(messages):
twilio_client.messages.create(
to=their_number,
from_=our_number,
body=messages[next_msg],
)

# Record which message we just sent in S3
bucket.put_object(
Key=state_key,
Body=str(next_msg),
Expires=datetime.now() + timedelta(days=7),
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I correct this expires our messages after a week, so if the same person were to text again after a week has passed the conversation would start again from the beginning?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. I wasn't sure what the threshold should be, but I felt like there should be some cut-off.

)

queue_entry.delete()

if __name__ == "__main__":
app.run()
24 changes: 24 additions & 0 deletions zappa_settings.example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"dev": {
"app_function": "rickroll.app",
// Put a random S3 bucket name that doesn't already exist here. Zappa
// will create it and put your Lambda code in it when you run
// 'zappa init'.
"s3_bucket": "insert-bucket-name-here",
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of my least favourite things about JSON is that it doesn't allow comments... 😭

While Zappa may be relaxed enough to allow them, the QA/Testing part of my soul stays awake at night when I know we're technically breaking a format standard.

I know it's awful, but can I ask for something like:

"s3_bucket": "An S3 bucket that doesn't already exist. Zappa will create it and use it for storage when you run `zappa init`"

Comments are waaay nicer, and if we were using YAML or HJSON I'd be all over them. But every linter (including github's changes view) and my psyche is going to worry every time it sees a // otherwise. :/

"aws_region": "us-east-1",
"environment_variables": {
// This should be a bucket WITHOUT public read access. Phone numbers
// will get stored in here.
"data_bucket": "rick-astley-data",
// These should be set to an IAM user with access to your S3 bucket
"aws_access_key_id": "INSERT_ACCESS_KEY_HERE",
"aws_secret_access_key": "insert-secret-access-key-here",
"twilio_sid": "insert-twilio-sid-here",
"twilio_token": "insert-twilio-token-here"
},
"events": [{
"function": "rickroll.send_sms",
"expression": "rate(4 minutes)"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oooh, this is cool! So every N minutes (n=4 in this case) we'll clear the SMS queue of messages? (So on average, we'd see a reply every two minutes?)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep! This seemed like the simplest way to have a delay before replying without actually making the Lambda function spin for a few minutes (and rack up charges in the process).

I thought about randomly skipping 50% of the queue every time send_sms runs to increase the variability, but I'm not sure whether that would make it more realistic or less.

}]
}
}