Getting Started with Realtime Events and Streaming Data (in JS)
About this Series
In this series, I’ll be exploring how to develop an event-driven web applications in Javascript. We’ll use Vue on the frontend and FeathersJS on the backend to accomplish this. If you’re interested in learning more about developing real-time and streaming data applications, follow this series. This will include developing chat-based apps, streaming videos or audio apps, device-driven apps, such as those in the Internet-of-Things space, and much more.
This series assumes familiarity with Javascript. It also assumes familiarity with basic Vue and Node.js concepts, such as setting up a simple project in Vue/Node. I’ll do my best to break down more complex concepts. If anything is unclear, please leave a comment so the article can be updated for clarity.
What is Realtime and Streaming Data?
Real-time data is data that is delivered immediately after collection. With ongoing improvements in hardware and computing power, it has become increasingly common for businesses to provide real-time analysis of data to identify potential issues or opportunities. Data collected can be transformed “on-the-fly” and presented to decision-makers instantly.
Traditionally, this was seen with devices and Geographical Information Systems (GIS) which would frequently emit sensor and/or location data.
With the increasing use of AI and data warehousing techniques, it’s fairly common now to see real-time processed data. For high-volume applications, it’s increasingly important to be able to update sites in real-time as the state of the system changes.
Real-time Events and Data Scenarios
Applications of real-time data will only continue to increase with time. Here are a few common ones to think about:
Health and Fitness Devices
As technology continues to improve, the advent of devices providing instant feedback will continue to increase in order to optimize the care and service doctors can provide. Medical equipment full of sensors will often need to transmit the information instantly in order to provide the doctor and patient the information needed to make informed decisions. In the past, X-rays would take days to process and develop. Now it’s available within minutes. Other similar diagnostic procedures are increasingly becoming available to provide near real-time feedback for doctors to make decisions on a patient’s condition.
Similarly, fitness trackers can transmit data such as heart rate and record your activity as you exercise or sleep. It can alert you when you’ve hit your daily step goals or warn you that you’re working too hard. These devices all rely on real-time updates to keep the user informed as an event occurs.
E-Commerce & Scheduling
Managing inventory is important for customer satisfaction. Inventory is also finite, so when a user makes a purchase, the item is typically deducted from inventory. This generally works fine on low volume sites where a single user may only make one purchase for a single item at any given time. But what happens when multiple users try to purchase the same item at the same time?
Only one person will be able to complete the purchase. The other orders will need to be canceled once it’s been discovered that the product is no longer available. This can lead to a terrible customer experience if the time taken to handle this exceeds a customer’s patience and expectations.
Through real-time event updates, customers can be informed the product has been purchased and the item can be removed from their shopping cart before the purchase can be completed. This would help better manage customer expectations. The same can be applied to booking or scheduling applications.
Operational Awareness
Sometimes, monitoring data in real-time is important for business operations. This is generally true for any kind of heuristics or diagnostic platform for computing systems. For example, in preventing and mitigating cyberattacks, it’s often necessary to track the flow of traffic entering a network.
The sooner an attack is discovered, the more likely a business can recover from the attack or defend against an attack. In such cases, real-time updates are important to accurately display the current situation.
Working with Realtime Data
The most common way to receive real-time updates on a website is through a real-time transport such as a socket. Sockets maintain an open channel with the server, allowing data and event notifications to pass through.
Socket.io is a popular library to support such connections. FeathersJS supports this out-of-the-box and provides additional scaffolding features for building a robust backend to support real-time applications.
Getting Started with FeathersJS
Getting started with Feathers is easy. I’ll briefly go over how to create your own project so you can start using it. Afterward, I’ll be using a pre-built project template to demonstrate different use cases. You can either create your own project or follow along using the same template.
Feathers Command Line Interface (CLI)
Feathers provides a CLI to enable you to quickly generate a new application. Globally install the Feathers CLI to generate an app:
npm install @feathersjs/cli -g
Create a folder for your project.
mkdir feathers-realtime
cd feathers-realtime/
feathers generate appThe Feathers CLI will prompt you with questions to help you configure your project, including auth, test frameworks, and data source providers. Adjust these based on your preferences. Make sure to select socket.io for your project when asked about APIs. Once complete, the CLI will automatically generate the project directory structure and files.
To learn more about the generated files, visit the docs.
Project Template
To start with a bit more functionality, I’m going to work from existing project templates within the FeathersJS community and build off these examples.
For the frontend, we’ll use the feathers-vuex-chat frontend as a starting point which leverages the feathers-vuex library.
For the backend, we’ll use the feathers-chat backend as a starting point.
The combined repo for this post can be found here: https://github.com/meditatingdragon/realtime-feathers.
Real-time Transports
As mentioned above, Feathers supports Socket.io as a real-time transport. It also supports Primus, which is a wrapper for real-time frameworks, making it possible to adapt Feathers to existing real-time frameworks used by other parts of the business.
Hello World — Pushing Messages to the Frontend
To kick off this project, I’m going to mock up some data on the backend to demonstrate real-time updates on the front end. We’ll create a simple dashboard with different charts to display real-time data. We’ll dive into more use-cases in the next series of posts.
Running the Project
This template uses vue on the frontend. To run the development server, use yarn serve within the feathers-chat-vuex directory. This will start on port 8080 by default. Navigate to your browser, http://localhost:8080/ to view the web app.
This project uses FeatherJS on the backend. To run the development server, use npm run dev. This will start on http://localhost:3030 by default.
The frontend should already be configured to connect to the backend on port 3030 through the /src/feathers-client.js setup.
Mocking the Data
To keep this first post simple, I’ll mock up data to be sent by the Feathers backend at regular intervals. We’ll start by creating a custom service called Metrics Service using the Feathers CLI to generate a service.
feathers generate serviceFollow the prompts:
What kind of service is it? A custom service
? What is the name of the service? metrics
? Which path should the service be registered on? /metrics
? Does the service require authentication? No
create src/services/metrics/metrics.service.js
force src/services/index.js
create src/services/metrics/metrics.class.js
create src/services/metrics/metrics.hooks.js
create test/services/metrics.test.jsDefine the Metrics Service as a custom service that can send data every 5 seconds.
const { getRandomInt } = require("../../utils/dataGenerator");
/* eslint-disable no-unused-vars */
exports.Metrics = class Metrics {
async create(data) {
return data;
}
setup() {
let logins = [
getRandomInt(50, 70),
getRandomInt(50, 70),
getRandomInt(50, 70),
getRandomInt(50, 70),
getRandomInt(50, 70),
getRandomInt(50, 70),
getRandomInt(50, 70),
];
setInterval(() => {
console.log("Sending new data");
logins.shift();
logins.push(getRandomInt(50, 70));
this.create({
messageCount: getRandomInt(1000, 10000) + getRandomInt(0, 100),
activeChatRooms: getRandomInt(5, 100),
recentLogins: logins,
openTickets: getRandomInt(0, 100),
closedTickets: getRandomInt(0, 100),
unassignedTickets: getRandomInt(0, 100),
});
}, 5000);
}
};The data is randomly generated using a helper function, getRandomInt. The data created is what I'll use to update the charts.
For a more realistic use-case, this data could be provided by a service or other data source. Consider sending logs which may provide a constant stream of log data. Or maybe send binary data to display to the user, like an audio clip or video generated by another user.
When you run npm run dev to start the server, the service can be called by the frontend to receive the data at regular intervals.
Sockets Handling in the Frontend
Feathers provides a wrapper for the socket.io client that works seamlessly with a Feathers backend. Feathers-vuex integrates this library and also provides real-time socket event support within the vuex store. To get started, add the following libraries if not already in your project:
yarn add @feathersjs/feathers @feathersjs/socketio-client @feathersjs/authentication-client socket.io-client @vue/composition-api feathers-vuex feathers-hooks-commonThese packages have already been added to the project template. @feathersjs/feathers, @feathersjs/socketio-client, @feathersjs/authentication-client, and socket.io-client provide the connection framework to communicate with your Feathers server through the socket.io real-time transport. The remaining libraries provide support for Vue/Vuex on the frontend.
By default, the feathers-vuex library defaults to real-time connections (the alternative being REST API calls, which you can also configure).
If this is your first time using Feathers-Vuex, I would recommend reviewing the docs, which documents the setup process and key concepts, such as the Auth Plugin, the Service Plugin, and Data Modeling. While this series will cover concepts relevant to the use-cases described, it will not be possible to cover everything.
Dashboard
To demonstrate the continuous stream of data, I’ll be creating a simple dashboard filled with charts.
Creating a Dashboard View
// /src/views/Dashboard.vue<template>
<main class="home container">
<div class="row text-center">
<h1>Dashboard</h1>
</div>
<div class="row">
<div class="col-6">
<h3>Messages Sent</h3>
<BarChart :chart-data="barchartdata" :options="options" />
</div>
<div class="col-6">
<h3>Active Chat Rooms</h3>
<BarChart :chart-data="barchartdata2" :options="options" />
</div>
</div>
<div class="row">
<h3>Recent Logins</h3>
<LineChart :chart-data="linechartdata" :options="options" />
</div>
<div class="row">
<h3>Current Tasks</h3>
<DonutChart :chart-data="donutchartdata" :options="doptions" />
</div>
<div class="row">
<h3>DEBUG</h3>
{{ serverMessage }}
</div>
</main>
</template>
You may notice chart components added to this dashboard view. We’ll create these down below.
Adding the View to the Routes
const routes = [
...
{ path: '/chat', name: 'Chat', component: Chat },
{ path: '/dashboard', name: 'Dashboard', component: Dashboard },
...
];Adding a Dashboard Link to the Chat View
<div class="title-wrapper block center-element">
<img
class="logo"
src="http://feathersjs.com/img/feathers-logo-wide.png"
alt="Feathers Logo"
/>
<span class="title">Chat</span>
</div>
<router-link class="float-right link" to="/dashboard">
Dashboard
</router-link>Displaying the Data
To visualize the flow of data, we’ll use charts to display data updates. I’m going to use the vue wrapper library vue-chartjs for Charts.js, which provides a simple yet customizable charting library.
yarn add vue-chartjs chart.js
Creating Chart Components
vue-chartjs makes it easy to add charts as a chart component within a single vue component file. View the documentation to learn more about how it can be used within a vue app.
Here is an example of the bar chart component.
// /src/components/BarChart.vue<script>
import { Bar, mixins } from 'vue-chartjs';
const { reactiveProp } = mixins;export default {
extends: Bar,
mixins: [reactiveProp],
props: ['chartData', 'options'],
mounted() {
this.renderChart(this.chartData, this.options);
},
};
</script>
Make sure you include the mixins and reactiveProp. The reactiveProp mixin adds a watcher to the chartData variable, enabling updates as data changes.
Listening to Events
With the new metrics service, we can use the feathers front-end client library to listen for the ‘created’ event from the metrics service:
mounted() {
// add an event listener to dataAvailable event
this.establishConnection();
},
methods: {
establishConnection() {
feathersClient.service('metrics').on('created', data => {
console.log('Receiving data from server: ', JSON.stringify(data));
// update variables to the data received from the server
this.messageCount = data.messageCount;
this.recentLogins = data.recentLogins;
this.activeChatRooms = data.activeChatRooms;
this.openTickets = data.openTickets;
this.closedTickets = data.closedTickets;
this.unassignedTickets = data.unassignedTickets;
this.serverMessage = data;
});
},
},Now, when you run the vue app and navigate to the /dashboard page, you should see the charts updating every 5 seconds.
Check Your Work
The final code for this post is on the metrics-service branch of this repo.
Coming Up Next: Channels
To handle real-time events in future posts, we’ll be making use of channels. If you want to get a jump start, take a look at the docs.
Let me know — how will you leverage real-time events in your application?
