Target Groups Application Pipeline Setup

1. AWS console ⇒ EC2:

2. Target Groups ⇒ Create target group

3. Create 2 groups

• Ludlow2-api-qa

• Ludlow2-api-prod

The port doesn’t matter, keep it as default: 80.

Load Balancer setup

1. AWS console ⇒ EC2 ⇒ Load Balancers ⇒ Create Load Balancer

2. Select: Application Load Balancer

3. Add 2 Listeners: HTTP / HTTPS, Select All Availability Zones.

4. Choose a certificate

5. Select an existing security group: default

6. New target group or Select existing one: Ludlow2-api-prod. Port doesn’t matter, keep it as default: 80.

7. No need to Register Targets, which will be register automatically by our ECS Services.

8. Review and Create

9. Select the new created Load Balancer: Ludlow2 ⇒ Listeners: 80/443 ⇒ View/edit rules

10. Add a rule: If Host is qa.ludlow.io forward to Target Group: Ludlow2-api-qa

EC2 Instance setup

The EC2 instance where we need to add an inbound rule letting a Load Balancer redirect the request, should be created while creating ECS Cluster below.

1. AWS console ⇒ EC2 ⇒ Instances ⇒ Ludlow2 ⇒ click Security groups

2. Inbound ⇒ Edit

3. Add Rule, Type pick All TCP, Source input the Security Group ID(sg-964aa2ef) from Load Balancer: Ludlow2

ECS Cluster setup

1. AWS console ⇒ Elastic Container Service:

2. Clusters ⇒ Create Cluster

3. Select EC2 Linux + Networking

4. Use default EC2 instance.

5. Networking use existing VPC, Subnet, and Security group. Security group should be the same as the one in Load Balancer, i.e. default.

ECR Repositories setup

1. AWS console ⇒ Elastic Container Service ⇒ Repositories ⇒ Create repository

2. Create two repositories: ludlow2-api-qa, ludlow2-api-prod.

3. Write down the commands, which will be used in CI platform: TeamCity.

ECS Task Definitions setup

1. AWS console ⇒ Elastic Container Service ⇒ Task Definitions ⇒ Create new Task Definition

2. Select EC2 as launch type compatibility

3. Configure task and container definitions

4. Add container

5. Standard configuration:

• Image points to the Repository we just created: ludlow2-api-qa

• Memory Limits should set to Hard limit for qa, just in case of affecting the prod Task’s memory

• Port mappings: Host port must be set to 0, in order to register to a Target Group with a dynamic port, which allow two different Tasks(Blue/Green Deployment) running at the same time, one for old version image, one for new version image. Container port is whatever you set in application’s Dockerfile.

6. Advanced container configuration

• STORAGE AND LOGGING ⇒ Log configuration ⇒ check Auto-configure CloudWatch Logs, this makes sure all the console log will go to CloudWatch.

ECS Services setup

1. AWS console ⇒ Elastic Container Service ⇒ Clusters ⇒ Ludlow2

2. Services ⇒ Create

3. Configure services

• Make sure Maximum percent * Number of tasks >= Number of tasks + 1, letting your new task can be started while the old one is stopping.

4. Configure network

• Load balancer type: Application Load Balancer

• Select Load Balancer: Ludlow2, Click Add to load balancer

• Target group name, pick Ludlow2-api-qa

Here at SWARM, our engineering team has agreed on the following goals for deployment architecture.

• Automatic

Deployment should happen automatically. Creating and publishing packages are for the dark ages!

• Reproducible

Production environment should be the same as the development environment. Issues in production should be easy to reproduce in development.

• Elastic

Easy to scale up or down services based upon demand.

• Smooth

Zero-downtime while scaling or upgrading the service.

• Traceable

Logging & monitoring should be in place to watch for issues

Workflow

Our workflow needs to account for several folks to coordinate in pushing out a build.

1. Developers commit code / update scripts to git. We use gitflow to keep track commits for our dev / qa / production environments.

2. Our CI Platform of choice TeamCity automatically kicks in to create a build, test it and then deploy it to QA. It then notifies our PM & QA teams to verify the latest build. Once approved

3. We use a combination of TeamCity & AWS Application Pipeline to generate builds for the production environments.

 

Deploying backend apps with AWS Application Pipeline

When we try to deploy an application in AWS, let’s say Ludlow2, we should set up an application pipeline first.

A typical pipeline includes:

• One Load Balancer

• Several Target Groups (each for one branch: dev, qa, prod)

  • Each Target Group is used to route requests to one or more registered targets by a Load Balancer listener rule.

• Several ECS services (each for one branch: dev, qa, prod)

  • ECS service is a specified number of instances of a task definition you can run and maintain simultaneously in an Amazon ECS cluster.

• Several ECR repositories (each for one branch: dev, qa, prod)

  • ECR is a managed AWS Docker registry service which supports private Docker repositories. You can use the Docker CLI to author and manage images.

• One EC2 instance

  • An EC2 instance provides scalable computing capacity in the Amazon Web Services (AWS) cloud.

Prerequisite

• AWS account

• Site domain name for your application, e.g.

• • *.ludlow.io

  • qa.ludlow.io

  • prod.ludlow.io

• Site certificate

Application Pipeline Setup

See the detail operations.

Continuous Integration (CI) with TeamCity and AWS

For CI, we set up a standalone TeamCity server instance as a docker container for each client and deploy it to AWS. The build agent is also a docker container that serves as an image for multiple on-demand build agents running on AWS.

While, the official build agent from jetbrains is great, most of our projects also need support for Node.js and AWS tools. So we created our own docker image for the TeamCity build agent where we bundled in the Node.js development environment and installed AWS CLI tools

These are available on GitHub or on Docker Hub.

In this flow we focus on building backend API application server building as another docker image.

 

Unit Testing

TeamCity is great for running unit test, which can nicely show the test result report and code coverage report. TeamCity supports different test frameworks. We pick Karma as our test framework, which can be used for backend apps (Node.js) as well as front-end (Angular) apps. Also, Karma is TeamCity friendly by using its TeamCity plugin .

Karma has a plugin karma-coverage-istanbul-reporter to generate coverage report, which is a html page, zipped them as an Artifact: coverage.zip. TeamCity could recognize it and create a Code Coverage tab for you automatically.

In an Angular project, Start a test by command: ng test –code-coverage

TeamCity could generate a nice statistic report for you based on historical unit test results.

Build

1. Building Docker Image by Docker Build Runner

2. Pushing Docker image to ECR, then deploy to ECS

 

These 2 Build Steps should be the same while building a docker image. The differences are coming from the Parameters.

Deployment

We are using ECS Deploy to do the deployment, which is triggered by the last command in Build phase:

ecs deploy %ECS_CLUSTER_NAME% %ECS_SERVICE_NAME% %AWS_REGION% %ECS_DEPLOY_OPTIONS%

This tells Amazon ECS to duplicate the current Task Definition and cause the Service to redeploy all running tasks.

The new task will be started while the old one is still running. When the new task successfully registers to the Target Group, the old one will stop and unregister(draining) from the Target Group. From the User’s perspective, the application(Ludlow2) is upgraded without any downtime.

TODO

• All the env files are commit to GitHub, convenience yet insecurity. Instead of being put into version control, they should be generated in TeamCity Build phase.

Integration with IDE

If you are using IntelliJ-based IDEs, e.g. WebStorm, you can install the TeamCity Plugin, which take advantage of all the features provided by TeamCity as a continuous integration server without leaving the context of the IDE.

The coolest function is Remote Run, which is similar as git commit. Instead of committing to your github repository, it just commit to the TeamCity server and do a CI cycle based on your local change.

⇒ ⇒

Logging

We are using CloudWatch for logging.

When setting up the ECS Task Definitions in Application Pipeline, we already redirect application output to CloudWatch by awslogs. In this way, the application doesn’t need to change anything and the console.log messages will go to CloudWatch.

In case you want to control the log flow, e.g. log to different stream based on the session, you could use AWS Log API, e.g. winston-cloudwatch.

Alerts

CloudWatch has Alarms, you can create one based on build-in or your own customized Metrics. We will show below how to send an email to develop team when there is a error happened in production server.

1. Create a Metric Filter based on a Log Group

2. Create Alarm based on the Metric

3. Define how to trigger alert.

 

With WebRTC technology, people can easily stream their live video and audio content just using a web browser. If you have a cloud video streaming idea and want to build a Minimum Viable Product (MVP), Kurento is the choice. With Kurento, you’d be able to handle the streaming audio/video easily, including analyzing, mixing, augmentation, etc. Kurento is a WebRTC server infrastructure, based on gstreamer. With its seamless OpenCV integration, you can process the video frame by frame quite handily. Kurento now is a developing project and updates very often, so fasten your seatbelt. I am trying to show you how Kurento works by a simple demo program: license plate detector. Here’s how to build a Kurento MVP:

Let’s get our hands dirty

To run the demo program: license plate detector, all we need are an Ubuntu box and a web browser. (OK, not any browser, just Chrome, Firefox or Opera.)

Install Kurento server on the Ubuntu box:

echo “deb http://ubuntu.kurento.org trusty-dev kms6″ | sudo tee /etc/apt/sources.list.d/kurento-dev.list
wget -O –http://ubuntu.kurento.org/kurento.gpg.key | sudo apt-key add –
1. sudo apt-get update

2. sudo apt-get install kurento-media-server-6.0-dev

3. sudo apt-get install kms-platedetector-6.0

4. sudo apt-get install libboost-all-dev libjson-glib-dev bison flex uuid-dev libsoup2.4-dev build-essential libtool autotools-dev  automake git libtesseract-dev

5. sudo service kurento-media-server-6.0 start

Install kurento tutorial nodejs on the Ubuntu box:

curl -sL https://deb.nodesource.com/setup | sudo bash –
sudo apt-get install -y nodejs

sudo npm install npm -g

git clonehttps://github.com/Kurento/kurento-tutorial-js.git

cd kurento-tutorial-js/kurento-platedetector

http-server -p 8443 -S -C keys/server.crt -K keys/server.key

npm install

npm start 

Open link https://localhost:8443 in web browser and check the result:

Hmmm, not so accurate, but it does work.

⇒  

 

Close-up

Let’s dip into the source project to see how that works. Pull the project from github and build it.

git clone https://github.com/Kurento/kms-platedetector.git

cd kms-platedetector/src

cmake ..

make

 

There are two folders: kms-platedetector/src

gst-plugins/ implements a gstreamer plugin: platedetector

server/ implements a kurento plugin: PlateDetectorFilter

The Kurento server is controlled by web browser using Kurento Protocol, based on WebSocket and JSON-RPC. The kurento plugin is the interface of the protocol, it receives the remote call from web browser and creates a gstreamer plugin to do the real job, i.e. analyze every frame from the live video stream, find where the plate locate, and recognize the numbers and characters.

The Kurento Server will do the WebRTC stuff for you, so you don’t need to worry about the details of stream encode/decode, NAT traversal, which are really, really fuzzy.

Kurento provides a tool to create the above plugin structure, which is kurento-module-scaffold. You can use this tool to create two flavors of Kurento modules:

1. OpenCV module:

kurento-module-scaffold.sh <module_name> <output_directory> opencv_filter

1. Gstreamer module:

kurento-module-scaffold.sh <module_name> <output_directory>

 

Kurento is based on two concepts that act as building blocks for application developers:

Media Elements

• A Media element is a functional unit performing a specific action on a media stream. Media elements are a way of every capability is represented as a self-contained “black box” (the media element) to the application developer, who does not need to understand the low-level details of the element for using it. Media elements are capable of receiving media from other elements (through media sources) and of sending media to other elements (through media sinks). Depending on their function, media elements can be split into different groups:
  • Input Endpoints: Media elements capable of receiving media and injecting it into a pipeline. There are several types of input endpoints. File input endpoints take the media from a file, Network input endpoints take the media from the network, and Capture input endpoints are capable of capturing the media stream directly from a camera or other kind of hardware resource.
  • Filters: Media elements in charge of transforming or analyzing media. Hence there are filters for performing operations such as mixing, muxing, analyzing, augmenting, etc.
  • Hubs: Media Objects in charge of managing multiple media flows in a pipeline. A Hub has several hub ports where other media elements are connected. Depending on the Hub type, there are different ways to control the media. For example, there are a Hub called Composite that merge all input video streams in a unique output video stream with all inputs in a grid.
  • Output Endpoints: Media elements capable of taking a media stream out of the pipeline. Again, there are several types of output endpoints specialized in files, network, screen, etc.

Media Pipeline:

• A Media Pipeline is a chain of media elements, where the output stream generated by one element (source) is fed into one or more other elements input streams (sinks). Hence, the pipeline represents a “machine” capable of performing a sequence of operations over a stream.

You can regard the OpenCV module and Gstreamer module as Filters that you need to implement. Media Pipeline and other Media Elements are already implemented by Kurento. More details here.

How it Works

Kurento use GStreamer to do the real streaming media job. Actually, Kurento Server is a gstreamer session manager, maintaining a bunch of gstreamer pipelines. In order to understand how kms-platedetector plugin works, we have to look deep into GStreamer.

GStreamer is a framework for creating streaming media applications. The fundamental design comes from the video pipeline at Oregon Graduate Institute, as well as some ideas from DirectShow.

The GStreamer framework is designed to make it easy to write applications that handle audio or video or both. The pipeline design is made to have little overhead above what the applied filters induce. This makes GStreamer a good framework for designing even high-end audio applications which put high demands on latency. 

 

Many of the virtues of the GStreamer framework come from its modularity: GStreamer can seamlessly incorporate new plugin modules. But because modularity and power often come at a cost of greater complexity, writing new applications is not always easy.

Kurento Server is one of the gstreamer tools, which means you can use any plugins it includes, or you could create a new one based on these off-the-shelf. Most of them are open source and you can customize them to fit your specified requirement, if you can find one. More importantly, Kurento has done most of the difficult part of GStreamer framework for you, so all you need are writing the code into the plugin structure kurento-module-scaffold created for you. Of course, understanding the GStreamer’s basic concepts will help you do a better and easier job.

Now, let’s take a look at the functions(in gst-plugins/platedetector/kmsplatedetector.c) you mostly need to implement:

static void class_init ()

It’s the function called when a plugin first created, and usually do some class initialization jobs like installing some override methods, such as getter/setter, finalize/dispose, adding pad template, registering private structure, etc.

static void init ()

It’s the function called every time a plugin created, and usually do some object initialization jobs like setting default value to plugin’s parameters, creating other resources, such as plugins, files, fonts, etc.

static GstFlowReturn transform_frame_ip (GstVideoFilter *filter, GstVideoFrame *frame)

It’s the function called every time a new frame comes, and usually do some image operation like analyzing the incoming frame, drawing something on it, mixing other input frames into one, etc.

This is the most important function you need to handle, but the incoming frame is presented as GstVideoFrame, which is not so easy to deal with, unless you really fancy the bitwise operations. Here comes OpenCV, which is a convenient tool to process the image with its rich computer vision and machine learning algorithms, and it’s straightforward and no-performance-cost to transform GstVideoFrame to Image objects, like Mat, IplImage, in OpenCV.

 

The End

Using Kurento, you can build a MVP to verify your video cloud streaming idea quickly, although the performance of Kurento is still an issue. Since Kurento is under developing, if you don’t make a good backup of the current version, your project may not compile at all after updating to a new version of Kurento.

Some tips for developing Kurento applications:

• Backup the version of Kurento you are working on, because the Kurento develop team don’t do that for you.
• Debug your Kurento application using the runtime pipeline map. If you have some problem, try to dump the pipeline runtime to a dot file and check it out. e.g.

presenter.pipeline.getGstreamerDot(function(err, ret){

var fs = require(‘fs’);

fs.writeFile(“pipeline.dot”, ret, function(err) {

console.log(“The file pipeline.dot was saved!”);

});

});

• Find a similarly plugin first before you want to create a new one, because there are plenty of open source plugin there, even you can not find a suitable one, you can learn something from these source code.

Let me know if you use this method to create a Kurento MVP and how it goes for you!

Check out our other articles on tech creation here.