ecs-mqtt-stream

Build your own stream, step by step

get the source

Let’s start by cloning the repository and moving inside:

git clone https://gitlab.com/fuelsave-os/ecs-mqtt-stream.git & cd ecs-mqtt-stream

install dependencies

Source the virtualenv (for a quick aid on how to install it, see here). We're installing some packages to be able to use aws services, get information from them and connect to the stream we’re about to create.

source /path/to/env/bin/activate & pip install -r src/requirements.txt

running the stream locally

If you just want to run locally and skip the AWS part, spin up docker and connect with the client:

docker-compose up -d ; python src/client.py --server local

Your console should have printed the following:

Dashboard is running at http://localhost:8080
Client connected

If you want to skip the aws deployment, jump to explore-a-bit.

deploying in AWS with the Elastic Container Service

create a VPC and a ECS cluster

We are going to create every resource with CloudFormation. For that we need to: 1. set a config file with your AWS access keys like described here. 2. if you have the root keys, skip this. Otherwise see the section embed an inline policy for a user or role here, and add the following permissions: * AmazonEC2FullAccess * IAMFullAccess * AmazonS3FullAccess * AWSCloudMapFullAccess * AmazonECS_FullAccess * AmazonSSMFullAccess * AmazonVPCFullAccess * AWSCloudFormationFullAccess

Once we are done with these, we can finally create the VPC-ECS stack. There is extensive documentation on VPCs. For a shallow overview, think of a VPC as a block of addresses (IPs): an online exclusive neighborhood where your computation resources live and where you can define who (and what) goes in and out.

Then comes the Elastic Container Service (ECS) doing the job of assigning your containers to those addresses. ECS is an incredible service (and free on AWS) because nowadays people tend to ship their applications in containers and they must only care about CPU and RAM. Given those constraints (with the possibility of a much more granular control), ECS distributes the load. For the curious reader, it’s incredibly smart the abstraction made with Elastic Network Interfaces (ENI) which makes a service as powerful as ECS look trivial and native to AWS (more of it in a future article, if requested!).

So we spin this stack up with:

aws cloudformation create-stack --stack-name vpc-ecs --capabilities CAPABILITY_IAM --template-body file:///path/to/repo/stack_vpc_ecs.yml

As the stack_mqtt.yml depends on the one we've just launched, we actually have to wait until it's completed. Should just take a couple minutes. Meanwhile you can take a look of how this CloudFormation stacks work.

create the mqtt service

In essence, ECS takes a container definition and calls it a task. Then, a service is launched that ensures this task is up, gets replicated, scaled, recovered and so on. As you can imagine, you can get very granular regarding the control you want, namely on service discovery (if you want your services to meet each other internally), availability zone, task count and many more.

aws cloudformation create-stack --stack-name mqtt --capabilities CAPABILITY_IAM --template-body file:///path/to/repo/stack_mqtt.yml

Now that everything is up, run the following python script to connect to the mqtt broker you just launched: python src/client.py --server ecs

Your console should print the following:

Dashboard is running at http://{dns}:8080
Client connected

explore a bit

You can navigate to the dashboard and actually play a bit with it (how you reach the dashboard depends on the deploy mode).

In the dash, go to clients (left navigation panel), select the only connected client (should be called my_first_stream) and add a subscription to the topic stream_topic. Now your terminal should be printing all messages because you have subscribed to the topic which you’re writing messages to!

kill aws

Kill everything on aws with: * aws cloudformation delete-stack --stack-name mqtt * aws cloudformation delete-stack --stack-name vpc-ecs