Last modified October 27, 2016
Microservices & Custom Metrics Example
In this example, we show how to build a simple Microservice application that offers custom Prometheus metrics.
The multiple components work hand in hand to collect URLs mentioned on Twitter and create a hotlist of popular URLs. We call this example internally Twitter Hot URLs Example - short THUX.
This component consumes the Twitter Stream API, looking for tweets containing the strings
https to fetch all tweets with links. The tweets are then parsed for contained URLs.
The URLs found are stored in the
inbox redis database.
This component is a simple Redis database that receives all found URLs from the
tracker component. It makes use of the official Redis Docker image.
This component consciously does not provide a volume, which means that whenever this component is restarted, the database content is lost.
The script inside this component reads URLs from the
inbox Redis database and creates requests to those URLs in order to resolve redirects, to reveal the actual target URL. The resulting URL is stored in the
hotlist Redis database.
To prevent accessing the same URL several times, a cache is maintained in the
resolver component can be thought of as a worker, processing jobs from a queue. Since resolving URLs is in many cases a time-consuming job, there can be multiple instances of this component working in parallel.
This second Redis database component stores all resolved URLs together with scoring information. It also contains the cache for the
resolver. Just like the
inbox component, we use the official Redis Docker image here.
In contrast to the
inbox component, the
hotlist provides a volume to persist the database throughout restarts.
This component contains a little helper that periodically removes outdated information from the
hotlist Redis database.
This is a Python/Flask web application that offers a JSON API to fetch the resulting URL hotlist.
rebrow component offers a web-based user interface (“rebrow” stands for “redis browser”) to debug the content of both Redis databases. It makes use of a third party Docker image.
Getting Credentials to Access Twitter API
Name: thux Description: Tracks URLs mentioned on Twitter and creates a ranked list Website: https://github.com/giantswarm/twitter-hot-urls-example Callback URL: <leave this field blank>
After that also create an Access Token under “Keys and Access Tokens”. Edit
secrets/twitter-api-secret.yaml and fill all four data fields with the corresponding
base64 encoded values.
printf "exampletokenxyz" | base64
Deploying the Application
Note: To get custom metrics you need to have our Monitoring recipe deployed.
Then, you can deploy the app by running:
kubectl apply --filename https://raw.githubusercontent.com/giantswarm/twitter-hot-urls-example/master/manifests-all.yaml kubectl apply --filename secrets/twitter-api-secret.yaml
After a while, you should be able to see the custom metrics appearing in Grafana.
You can collaborate on this recipe on GitHub.