Open in app

Sign in

Write

Sign in

Building Scalable Knowledge & Social Networks: Based on GraphDB, Gremlin, Python Fast API Server & Flutter Mobile SDK

Lokendra Lodha
6 min readOct 17, 2020

Building an efficient , big data based knowledge network could be challenging using traditional relational databases and even no-sql databases. The use-cases requires plenty of recommendations, analytics , metrics and relationships where the GraphDB based provider might be the Best fit.

The tutorial proposes the new architecture stack composing of: Gremlin compliant GraphDB provider such as JanusGraph or Amazon Neptune, Python Fast API for web layer, and Google Flutter Based user interface apps for building the Android/iOS and PWA applications.

Gremlin based property graphs to be used for data and relationships.
Gremlin based property graphs to be used for graph modeling of data and relationships.
JanusGraph : Scalable implementation of Gremlin provider and shall be used for for GraphDB storage and data access.
Python Fast API : An emerging REST API server shall be used web layer.
Google Flutter based Mobile App for Android/iOS/and a Sample PWA App generation

This is 3 part series of articles

  1. In this 1st part we cover the foundation stack setup, installation and define the foundation REST API and data models i.e Vertex in graph db terminology and loads a sample initial graph data to JanusGraph.
  2. The 2nd part of this series shall extend the code to define knowledge social network relationships. These relationships shall be consumed via FastAPI REST api for end users use-cases to generate the recommendations, and social features such a likes, followings, recommended, popular, trending,…
  3. The final part of this series builds a sample Flutter sample android app for conclusion.
# Knowledge Network - the story we are going to build
Large scale solution for Knowledge & Mind Map typical applications based on Graph database

Some of the below use-cases shall be used for the demo graph application.


## Writers & articles
Every articles are based on certain root belief
  -- An example could be : Hinduism, Buddhism, Christianity, Islam, Gandhi, and so on

An article can be related to other facts, experiences and root beliefs.

Readers and writers participate and share there reading experiences via likes, following, agreements etcThe network establishes and identifies readers and writers affinity and promotes wisely future engaging content.

## Relations
A writer post an article
A writer shares the articles to all or to his followers

The readers reviews the articles and share there experiences and likeness
The readers network shall be notified on his/her activities
The writer shall be notified of readers activities and article stats

The readers may follow writers
The reader and writers may follow root beliefs

## social network

## Show article and writer insights

For every visited article by a reader:
Show most recent 5 readers that this reader's knows and who liked this article and option to show:
   all readers this reader knows and who liked this article
   all readers this reader does not knows and who liked this article and are most trending users
   all readers this reader does not knows and who liked this thought and are most popular users
   all readers this reader does not knows and who liked this thought and are newly added users


## Show recommendations
Show recent articles that this current reader network of readers/writers liked or written and not read by this mind
Show recent articles of trending writers that and not read by this current reader having common root beliefs
Show recent thoughts of popular writers that and not read by this current reader having common root beliefs
Show recent thoughts of new writers that and not read by this mind having common core beliefs


### Explore new & unknown writers and articles

JanusGraph installation and setup:

This article assumes we have an ubuntu 16 xenial OS and JanusGraph docker shall be utilized for quick turnaround.

## Install docker on ubuntu 16 xenial
https://docs.docker.com/engine/install/ubuntu/

## Install docker image for janus
https://hub.docker.com/r/janusgraph/janusgraph

 $ docker pull janusgraph/janusgraph

## run janusgraph docker container

https://docs.janusgraph.org/getting-started/installation/

 $ docker run -it -p 8182:8182 janusgraph/janusgraph

### verify logs
 7540 [main] INFO  org.apache.tinkerpop.gremlin.server.AbstractChannelizer  - application/json already has org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0 configured - it will not be replaced by org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, change order of serialization configuration if this is not desired.
 7639 [gremlin-server-boss-1] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
 7640 [gremlin-server-boss-1] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Channel started at port 8182.

Install python gremlin

https://docs.janusgraph.org/connecting/python/

sudo apt-get update

sudo apt-get upgrade python3
sudo apt-get install python3.7-venv## Create python virtual env - Let's name it as gapp inside /root/graphdb/gapp/ directorypython3.7 -m venv gapp

source /root/graphdb/gapp/bin/activate

pip install gremlinpython==3.4.6Connecting from Python
Gremlin traversals can be constructed with Gremlin-Python just like in Gremlin-Java or Gremlin-Groovy. Refer to Gremlin Query Language for an introduction to Gremlin and pointers to further resources.

Important

Some Gremlin step and predicate names are reserved words in Python. Those names are simply postfixed with _ in Gremlin-Python, e.g., in() becomes in_(), not() becomes not_(), and so on. The other names affected by this are: all, and, as, from, global, is, list, or, and set.

Install FastAPI server

$ pip install fastapi
$ pip install uvicorn

Optionaly install to serve HTML content:
$ pip install jinja2

Optionaly install to serve static content:
$ pip install aiofiles
Create a FAST API based project:
cd /home/workspace/travelo/gapp/
source /root/graphdb/gapp/bin/activate

create a 'templates' and 'static' directories under root of your projectcreate a main.py for your main web application:from typing import Optionalfrom fastapi import FastAPI,Header, Request, Formfrom fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templatesfrom gremlin_python import statics
from gremlin_python.structure.graph import Graph
from gremlin_python.process.graph_traversal import __
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnectionfrom typing import Optional
from pydantic import BaseModel
import datetime
import uuidclass ProductModel(BaseModel):
 title: str
 description: Optional[str] = None
 price: float
 tax: Optional[float] = Noneclass PostModel(BaseModel):
 gid: Optional[str] = None
 status: Optional[int] = 200
 tenant: Optional[str] = 'demo'
 title: str
 description: Optional[str] = None
 author_id: intclass RegisterUser(BaseModel):
 gid: Optional[str] = None
 email: Optional[str] = None
 phone: Optional[str] = None
 password: Optional[str] = None
 first_name: str
 last_name: Optional[str] = None
 tenant: Optional[str] = 'demo'class LoginUser(BaseModel):
 email: Optional[str] = None
 phone: Optional[str] = None
 password: Optional[str] = None
 tenant: Optional[str] = 'demo'graph = Graph()
connection = DriverRemoteConnection('ws://localhost:8182/gremlin', 'g')
# The connection should be closed on shut down to close open connections with connection.close()
g = graph.traversal().withRemote(connection)
# Reuse 'g' across the applicationapp = FastAPI()app.mount("/static", StaticFiles(directory="static"), name="static")templates = Jinja2Templates(directory="templates")def getTraversal():
 graph_l = Graph()
 connection_l = DriverRemoteConnection('ws://localhost:8182/gremlin', 'g')
 # The connection should be closed on shut down to close open connections with connection.close()
 g_l = graph_l.traversal().withRemote(connection_l)
 # Reuse 'g' across the applicationreturn g_l@app.get("/items/{id}", response_class=HTMLResponse)
async def read_item(request: Request, id: str):
 return templates.TemplateResponse("item.html", {"request": request, "id": id})@app.get("/")
def read_root():
 return {"Hello": "World"}@app.get("/beliefs")
def read_beliefs(token: Optional[str] = Header(None)):
 label = 'BELIEF'
 
 beliefs = g.V().hasLabel(label).has('status',200).valueMap(True).toList()
 return beliefs@app.get("/users")
def users(token: Optional[str] = Header(None)):
 label = 'USER'
 
 list = g.V().hasLabel(label).valueMap(True).toList()
 return listdef addV(label, data):
 print("label:",label, "data:", data)
 #return
 g_l = getTraversal()if 'gid' not in data or data['gid'] is None:
  data['gid'] = uuid.uuid1()
 if  'tenant' not in data or data['tenant'] is None:
  data['tenant'] = 'opac'
 if  'status' not in data or data['status'] is None:
  data['status'] = 200exists = g_l.V().hasLabel(label).has('gid',data['gid']).hasNext()if  exists :
  v = g_l.V().hasLabel(label).has('gid',data['gid'])\
  .property('status',data['status'])\
  .next()  
 else:
 
  g_l.addV(label).property('gid',data['gid']) \
   .property('created_at',datetime.datetime.now())\
   .property('updated_at',datetime.datetime.now())\
   .next() 
  
  v = g_l.V().hasLabel(label).has('gid',data['gid']).next()qu = g_l.V().hasLabel(label).has('gid',data['gid'])
 for key in data.keys():
   qu.property(key,data[key]) 
 qu.next()return {'s':200}
 
@app.get("/load")
def load(token: Optional[str] = Header(None)):
 list1 = [{'status':200,'title':'Hinduism','slug':'hinduism','gid':'hinduism'},
    {'status':200,'title':'Buddhism','slug':'buddhism','gid':'buddhism'},
    {'status':200,'title':'Islam','slug':'islam','gid':'islam'},
    {'status':200,'title':'Christianity','slug':'christianity','gid':'Christianity'},
    {'status':200,'title':'Ghandhism','slug':'ghandhism','gid':'ghandhism'},
    {'status':200,'title':'Truth','slug':'truth','gid':'truth'},
    {'status':200,'title':'Logic','slug':'logic','gid':'logic'},
    {'status':200,'title':'Science','slug':'science','gid':'science'},
    {'status':200,'title':'Faith','slug':'faith','gid':'faith'}]for item in list1:
  addV('BELIEF', item)users = [
   {'status':200, 'first_name':'John','last_name':' QA', 'email':'sqa1@g.com','password':'qa', 'phone':'+91-1234567891','tenant':'opac','is_admin':'Y'},
   {'status':200, 'first_name':'Loki','last_name':' QA', 'email':'sqa2@g.com','password':'qa', 'phone':'+91-1234567892','tenant':'opac','is_admin':'Y'},
   {'status':200, 'first_name':'Tina','last_name':' QA', 'email':'sqa3@g.com','password':'qa', 'phone':'+91-1234567893','tenant':'opac','is_admin':'N'},
   {'status':200, 'first_name':'Vayu','last_name':' QA', 'email':'sqa4@g.com','password':'qa', 'phone':'+91-1234567894','tenant':'opac','is_admin':'N'},
   {'status':200, 'first_name':'Beet','last_name':' QA', 'email':'sqa5@g.com','password':'qa', 'phone':'+91-1234567895','tenant':'opac','is_admin':'N'},
   {'status':200, 'first_name':'Sam','last_name':' QA', 'email':'sqa6@g.com','password':'qa', 'phone':'+91-1234567896','tenant':'opac','is_admin':'N'},
   {'status':200, 'first_name':'Nora','last_name':' QA', 'email':'sqa7@g.com','password':'qa', 'phone':'+91-1234567897','tenant':'opac','is_admin':'N'},
   {'status':200, 'first_name':'Daniel','last_name':' QA', 'email':'sqa8@g.com','password':'qa', 'phone':'+91-1234567898','tenant':'opac','is_admin':'N'},]for item in users:
  addV('USER', item)

Run and verify the FastAPI server:

uvicorn main:app --reload --host <IP or hostname> --port 8000

Hurray !
Your 1st application is now Live at:
http://<IP or hostname>:8000

REST endpoint docs and try it now editors are available at :
http://<IP or hostname>:8000/docs

Load the test data in Graph — a few sample users and root beliefs:

http://<IP or hostname>:8000/load

Verify and test few API:

List all available beliefs in graph DB:

http://<IP or hostname>:8000/beliefs

List all available users in graph DB:

http://<IP or hostname>:8000/users

That’s it in this part!!

Graphdb
Gremlins 2
Janusgraph
Fastapi
Flutter

--

--

Lokendra Lodha

Written by Lokendra Lodha

15 Followers

Founder/CTO at OpacLabs

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams