This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

1 - Bootstrapper

A library to make an application at enterprise level.

1.1 - Concept

RK provide bootstrapper for popular frameworks in order to save time for learning complex initializing process.

What is it?

RK bootstrapper is a library which support bootstrapping server at runtime.

Why do I want it?

  • Build application with unified project layout at enterprise level .

  • Build API with the unified format of logging, metrics, tracing, authorization at enterprise level.

  • Make application replace core dependencies quickly.

  • Save learning time of writing initializing procedure of popular frameworks and libraries.

  • User defined Entry for customization.

Where should I go next?

1.2 - Getting started

Create first service with RK bootstrapper at enterprise level.

Frameworks supported

Lang Name Status Demo
Go Gin Stable Demo
Go GRPC Stable Demo
Go Echo Stable Demo

1.2.1 - Gin-golang

Create a Gin server with RK bootstrapper.

Overview

This is an example demonstrate how to configure boot.yaml file to start a Gin server with bellow functionality.

Full demo: https://github.com/rookie-ninja/rk-demo/gin/getting-started

Create server

1.Install dependency

$ go get github.com/rookie-ninja/rk-boot

2.Create boot.yaml

---
gin:
  - name: greeter       # Name of gin entry
    port: 8080          # Port of gin entry
    enabled: true       # Enable gin entry
    sw:
      enabled: true     # Enable swagger UI
    commonService:
      enabled: true     # Enable common service
    tv:
      enabled:  true    # Enable RK TV

3.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

4.Start server

$ go run main.go
...
2021-07-02T04:53:25.921+0800    INFO    boot/gin_entry.go:630   Bootstrapping GinEntry. {"eventId": "8fd734a2-549a-4b81-bea2-fe8e94666ab7", "entryName": "greeter", "entryType": "GinEntry", "port": 8080, "interceptorsCount": 1, "swEnabled": true, "tlsEnabled": false, "commonServiceEnabled": true, "tvEnabled": true, "swPath": "/sw/"}
------------------------------------------------------------------------
endTime=2021-07-02T04:53:25.921604+08:00
startTime=2021-07-02T04:53:25.919344+08:00
elapsedNano=2259975
timezone=CST
ids={"eventId":"8fd734a2-549a-4b81-bea2-fe8e94666ab7"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GinEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.6","os":"darwin","realm":"*","region":"*"}
payloads={"commonServiceEnabled":true,"entryName":"greeter","entryType":"GinEntry","interceptorsCount":1,"port":8080,"swEnabled":true,"swPath":"/sw/","tlsEnabled":false,"tvEnabled":true}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost
operation=bootstrap
resCode=OK
eventStatus=Ended
EOE

5.Validate

Swagger: http://localhost:8080/sw

TV: http://localhost:8080/rk/v1/tv

Cheers

Add an API

We will define a Greeter API with the path of “/v1/greeter”.

1.Register API

We need gin.Engine in order to register HandlerFunc.

In GinEntry, gin.Engine will be initialized by bootstrapper with the name of Router.

package main

import (
	"context"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"net/http"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Register handler before bootstrap!
	boot.GetGinEntry("greeter").Router.GET("/v1/greeter", Greeter)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

// Gin handler function
func Greeter(ctx *gin.Context) {
	ctx.JSON(http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", ctx.Query("name")),
	})
}

// Response.
type GreeterResponse struct {
	Message string
}

2.Validate

$ curl "http://localhost:8080/v1/greeter?name=rk-dev"
{"Message":"Hello rk-dev!"}

Cheers

Support Swagger UI

We will use swag to generate swagger UI config files.

1.Install swag

Option 1:

From swag the official repository:

$ go get -u github.com/swaggo/swag/cmd/swag

Option 2:

With RK CMD

# Install RK CMD
$ go get -u github.com/rookie-ninja/rk/cmd/rk

# Install swag with rk
$ rk install swag

2.Add comments

package main

import (
	"context"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"net/http"
)

// @title RK Swagger for Gin
// @version 1.0
// @description This is a greeter service with rk-boot.

// Application entrance.
func main() {...}

// @Summary Greeter service
// @Id 1
// @version 1.0
// @produce application/json
// @Param name query string true "Input name"
// @Success 200 {object} GreeterResponse
// @Router /v1/greeter [get]
func Greeter(ctx *gin.Context) {...}

3.Generate swagger config

$ swag init

# swagger.json, swagger.yaml, docs.go files will be generated under ./docs folder.
$ tree
.
├── boot.yaml
├── docs
│   ├── docs.go
│   ├── swagger.json
│   └── swagger.yaml
├── go.mod
├── go.sum
└── main.go

1 directory, 7 files

4.Add path to boot.yaml

In order to make rk-boot finds out available swagger config files, we need to add gin.sw.jsonPath in boot.yaml file.

---
gin:
  - name: greeter
    ...
    sw:
      enabled: true
      jsonPath: "docs"  # Boot will look for swagger config files from this folder
    ...

5.Validate

Swagger: http://localhost:8080/sw

TV: http://localhost:8080/rk/v1/tv

Cheers

1.2.2 - GRPC-golang

Create a GRPC server with RK bootstrapper.

Overview

This is an example demonstrate how to configure boot.yaml file to start a gin server with bellow functionality.

Full demo: https://github.com/rookie-ninja/rk-demo/tree/master/grpc/getting-started

Create server

1.Install dependency

$ go get github.com/rookie-ninja/rk-boot

2.Create boot.yaml

By default, grpc gateway and grpc will use same port. rk-grpc will distinguish connection between grpc and http requests.

---
grpc:
  - name: greeter            # Required, Name of grpc entry
    port: 8080               # Required, Port of grpc entry
    enabled: true            # Required, Enable grpc entry
    enableReflection: true   # Optional, Enable grpc server reflection, https://github.com/grpc/grpc/blob/master/doc/server-reflection.md
    enableRkGwOption: true   # Optional, Enable grpc gateway server option as RK style
    commonService:
      enabled: true          # Optional, Enable common service
    tv:
      enabled: true          # Optional, Enable RK TV
    sw:
      enabled: true          # Optional, Enable Swagger UI

3.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

4.Start server

$ go run main.go
...
2021-09-11T04:43:23.219+0800    INFO    boot/grpc_entry.go:829  Bootstrapping grpcEntry.        {"eventId": "2df7fa50-3d1f-44cb-9db6-c4c26d9a29ce", "entryName": "greeter", "entryType": "GrpcEntry", "port": 8080, "swEnabled": true, "tvEnabled": true, "promEnabled": true, "commonServiceEnabled": true, "tlsEnabled": false, "reflectionEnabled": true, "swPath": "/sw/", "headers": {}, "tvPath": "/rk/v1/tv"}
------------------------------------------------------------------------
endTime=2021-09-11T04:43:23.217156+08:00
startTime=2021-09-11T04:43:23.217145+08:00
elapsedNano=11496
timezone=CST
ids={"eventId":"2df7fa50-3d1f-44cb-9db6-c4c26d9a29ce"}
app={"appName":"rk-grpc","appVersion":"master-bd63f29","entryName":"greeter","entryType":"GrpcPromEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.6","os":"darwin","realm":"*","region":"*"}
payloads={"entryName":"greeter","entryType":"GrpcPromEntry","path":"/metrics","port":8080}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost
operation=bootstrap
resCode=OK
eventStatus=Ended
EOE

5.Validate

Swagger: http://localhost:8080/sw

TV: http://localhost:8080/rk/v1/tv

Cheers

Add an API

We will define a Greeter API with the path of “/v1/greeter” with protocol buffer.

1.Install tools

We recommend use rk install them easily.

# Install RK CLI
$ go get -u github.com/rookie-ninja/rk/cmd/rk

# List available installation
$ rk install
COMMANDS:
    buf                      install buf on local machine
    cfssl                    install cfssl on local machine
    cfssljson                install cfssljson on local machine
    gocov                    install gocov on local machine
    golangci-lint            install golangci-lint on local machine
    mockgen                  install mockgen on local machine
    pkger                    install pkger on local machine
    protobuf                 install protobuf on local machine
    protoc-gen-doc           install protoc-gen-doc on local machine
    protoc-gen-go            install protoc-gen-go on local machine
    protoc-gen-go-grpc       install protoc-gen-go-grpc on local machne
    protoc-gen-grpc-gateway  install protoc-gen-grpc-gateway on local machine
    protoc-gen-openapiv2     install protoc-gen-openapiv2 on local machine
    swag                     install swag on local machine
    rk-std                   install rk standard environment on local machine
    help, h                  Shows a list of commands or help for one command

# Install buf, protoc-gen-go, protoc-gen-go-grpc, protoc-gen-grpc-gateway, protoc-gen-openapiv2
$ rk install protoc-gen-go
$ rk install protoc-gen-go-grpc
$ rk install protoc-gen-go-grpc-gateway
$ rk install protoc-gen-openapiv2
$ rk install buf
Tool Description Installation
buf Help you create consistent Protobuf APIs that preserve compatibility and comply with design best-practices. Install
protoc-gen-go Plugin for the Google protocol buffer compiler to generate Go code.. Install
protoc-gen-go-grpc This project aims to provide that HTTP+JSON interface to your gRPC service. Install
protoc-gen-grpc-gateway plugin for Google protocol buffer compiler to generate a reverse-proxy, which converts incoming RESTful HTTP/1 requests gRPC invocation. Install
protoc-gen-openapiv2 plugin for Google protocol buffer compiler to generate open API config file. Install

2.Create greeter.proto

Create greeter.proto under api/v1 folder

syntax = "proto3";

package api.v1;

option go_package = "api/v1/greeter";

service Greeter {
  rpc Greeter (GreeterRequest) returns (GreeterResponse) {}
}

message GreeterRequest {
  string name = 1;
}

message GreeterResponse {
  string message = 1;
}

3.Create compilation config file

Create buf.yaml

version: v1beta1
name: github.com/rk-dev/rk-boot
build:
  roots:
    - api

Create buf.gen.yaml

version: v1beta1
plugins:
  - name: go
    out: api/gen
    opt:
     - paths=source_relative
  - name: go-grpc
    out: api/gen
    opt:
      - paths=source_relative
  - name: grpc-gateway
    out: api/gen
    opt:
      - paths=source_relative
      - grpc_api_configuration=api/v1/gw_mapping.yaml
  - name: openapiv2
    out: api/gen
    opt:
      - grpc_api_configuration=api/v1/gw_mapping.yaml

Create api/v1/gw_mapping.yaml

Please refer google.api.Http in https://github.com/googleapis/googleapis/blob/master/google/api/http.proto file for details.

type: google.api.Service
config_version: 3

http:
  rules:
    - selector: api.v1.Greeter.Greeter
      get: /api/v1/greeter

Run buf

$ buf generate
# By configuration in buf.gen.yaml, generated files would be write to api/gen folder
$ tree
.
├── api
│   ├── gen
│   │   └── v1
│   │       ├── greeter.pb.go
│   │       ├── greeter.pb.gw.go
│   │       ├── greeter.swagger.json
│   │       └── greeter_grpc.pb.go
│   └── v1
│       ├── greeter.proto
│       └── gw_mapping.yaml
├── boot.yaml
├── buf.gen.yaml
├── buf.yaml
├── go.mod
├── go.sum
└── main.go

4.Register API

buf will help us generate the interface of grpc service and grpc-gateway registration function.

We need to implement the interface of grpc service and register it to bootstrapper with grpc-gateway by calling:

  • AddRegFuncGrpc
  • AddRegFuncGw
package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Get grpc entry with name
	grpcEntry := boot.GetGrpcEntry("greeter")
	grpcEntry.AddRegFuncGrpc(registerGreeter)
	grpcEntry.AddRegFuncGw(greeter.RegisterGreeterHandlerFromEndpoint)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}

5.Validate

$ curl "http://localhost:8080/api/v1/greeter?name=rk-dev"
{"Message":"Hello rk-dev!"}

Cheers

Support Swagger UI

With above example, we already generated greeter.swagger.json file.

What we need to do is add lines in boot.yaml to make process load the files into memory.

1.Add path to boot.yaml

---
grpc:
  - name: greeter
    ...
    gwMappingFilePaths:
      - "api/v1/gw_mapping.yaml"  # Boot will look for gateway mapping files and load information into memory
    sw:
      enabled: true
      jsonPath: "api/gen/v1"      # Boot will look for swagger config files from this folder

2.Validate

Cheers

1.2.3 - Echo-golang

Create a Echo server with RK bootstrapper.

Overview

This is an example demonstrate how to configure boot.yaml file to start a Echo server with bellow functionality.

Full demo: https://github.com/rookie-ninja/rk-demo/echo/getting-started

Create server

1.Install dependency

$ go get github.com/rookie-ninja/rk-boot

2.Create boot.yaml

---
echo:
  - name: greeter       # Name of gin entry
    port: 8080          # Port of gin entry
    enabled: true       # Enable gin entry
    sw:
      enabled: true     # Enable swagger UI
    commonService:
      enabled: true     # Enable common service
    tv:
      enabled:  true    # Enable RK TV

3.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

4.Start server

$ go run main.go
...
2021-11-02T05:13:03.960+0800    INFO    boot/echo_entry.go:693  Bootstrapping EchoEntry.        {"eventId": "4ca7e0de-a5ff-4d94-91fa-dc5f75c599fb", "entryName": "greeter", "entryType": "EchoEntry", "port": 8080}
------------------------------------------------------------------------
endTime=2021-11-02T05:13:03.960021+08:00
startTime=2021-11-02T05:13:03.958277+08:00
elapsedNano=1743749
timezone=CST
ids={"eventId":"4ca7e0de-a5ff-4d94-91fa-dc5f75c599fb"}
app={"appName":"gin-demo","appVersion":"master-8ad197d","entryName":"greeter","entryType":"EchoEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.6","os":"darwin","realm":"*","region":"*"}
payloads={"entryName":"greeter","entryType":"EchoEntry","port":8080}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost
operation=bootstrap
resCode=OK
eventStatus=Ended
EOE

5.Validate

Swagger: http://localhost:8080/sw

TV: http://localhost:8080/rk/v1/tv

Cheers

Add an API

We will define a Greeter API with the path of “/v1/greeter”.

1.Register API

We need echo.Echo in order to register HandlerFunc.

In EchoEntry, echo.Echo will be initialized by bootstrapper with the name of Echo.

package main

import (
	"context"
	"fmt"
	"github.com/labstack/echo/v4"
	"github.com/rookie-ninja/rk-boot"
	"net/http"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Register handler
	boot.GetEchoEntry("greeter").Echo.GET("/v1/greeter", Greeter)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

// Echo handler
func Greeter(ctx echo.Context) error {
	return ctx.JSON(http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", ctx.QueryParam("name")),
	})
}

// Response.
type GreeterResponse struct {
	Message string
}

2.Validate

$ curl "http://localhost:8080/v1/greeter?name=rk-dev"
{"Message":"Hello rk-dev!"}

Cheers

Support Swagger UI

We will use swag to generate swagger UI config files.

1.Install swag

Option 1:

From swag the official repository:

$ go get -u github.com/swaggo/swag/cmd/swag

Option 2:

With RK CMD

# Install RK CMD
$ go get -u github.com/rookie-ninja/rk/cmd/rk

# Install swag with rk
$ rk install swag

2.Add comments

package main

import (
	"context"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"net/http"
)

// @title RK Swagger for Echo
// @version 1.0
// @description This is a greeter service with rk-boot.

// Application entrance.
func main() {...}

// @Summary Greeter service
// @Id 1
// @version 1.0
// @produce application/json
// @Param name query string true "Input name"
// @Success 200 {object} GreeterResponse
// @Router /v1/greeter [get]
func Greeter(ctx echo.Context) error {...}

3.Generate swagger config

$ swag init

# swagger.json, swagger.yaml, docs.go files will be generated under ./docs folder.
$ tree
.
├── boot.yaml
├── docs
│   ├── docs.go
│   ├── swagger.json
│   └── swagger.yaml
├── go.mod
├── go.sum
└── main.go

1 directory, 7 files

4.Add path to boot.yaml

In order to make rk-boot finds out available swagger config files, we need to add echo.sw.jsonPath in boot.yaml file.

---
gin:
  - name: greeter
    ...
    sw:
      enabled: true
      jsonPath: "docs"  # Boot will look for swagger config files from this folder
    ...

5.Validate

Swagger: http://localhost:8080/sw

TV: http://localhost:8080/rk/v1/tv

Cheers

1.3 - User guide

RK provide bootstrapper for popular frameworks in order to save time for learning complex initializing process.

1.3.1 - Gin

User guide for Gin framework in Go.

1.3.1.1 - Basic

Basic user guide which could operate by user turn on/off by modifying boot.yaml.

Overview

In basic user guide, we will introduce how to enable bellow functionalities by modifying boot.yaml.

Functionality Description
Swagger UI Enable swagger UI
Common Service Enable common service
RK TV Enable RK TV
Prometheus Client Enable prometheus client
Logging Enable logging middleware
Metrics Enable prometheus middleware
Meta Enable meta middleware
Tracing Enable openTelemetry tracing middleware
Auth Enable basic & ApiKey auth middleware

1.3.1.1.1 - Swagger UI

Enable swagger UI for server.

Prerequisite

We will use swag to generate swagger UI config files.

Option 1: With RK CMD

# Install RK CMD
$ go get -u github.com/rookie-ninja/rk/cmd/rk

# Install swag with rk
$ rk install swag

Option 2: From swag the official repository:

$ go get -u github.com/swaggo/swag/cmd/swag

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Swagger options

name description type default value
gin.sw.enabled Enable swagger service over gin server boolean false
gin.sw.path The path access swagger service from web string /sw
gin.sw.jsonPath Where the swagger.json files are stored locally string ""
gin.sw.headers Headers would be sent to caller as scheme of [key:value] []string []

Quick start

1.Create boot.yaml

In order to make rk-boot finds out available swagger config files, we need to add gin.sw.jsonPath in boot.yaml file.

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    sw:
      enabled: true
      jsonPath: "docs"
#      path: "sw"        # Default value is "sw", change it as needed
#      headers: []       # Headers that will be set while accessing swagger UI main page.

2.Create main.go

In order to generate swagger config with swag, we need to add comments as bellow.

package main

import (
	"context"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"net/http"
)

// @title RK Swagger for Gin
// @version 1.0
// @description This is a greeter service with rk-boot.

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Register handler
	boot.GetGinEntry("greeter").Router.GET("/v1/greeter", Greeter)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

// @Summary Greeter service
// @Id 1
// @version 1.0
// @produce application/json
// @Param name query string true "Input name"
// @Success 200 {object} GreeterResponse
// @Router /v1/greeter [get]
func Greeter(ctx *gin.Context) {
	ctx.JSON(http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", ctx.Query("name")),
	})
}

// Response.
type GreeterResponse struct {
	Message string
}

3.Generate swagger config

$ swag init

# swagger.json, swagger.yaml, docs.go files will be generated under ./docs folder.
$ tree
.
├── boot.yaml
├── docs
│   ├── docs.go
│   ├── swagger.json
│   └── swagger.yaml
├── go.mod
├── go.sum
└── main.go

1 directory, 7 files

4.Validate

Swagger: http://localhost:8080/sw

Cheers

1.3.1.1.2 - Common Service

Enable common service for server.

Overview

Common service contains builtin API commonly used by user.

Path Description
/rk/v1/apis List APIs in current GinEntry.
/rk/v1/certs List CertEntry.
/rk/v1/configs List ConfigEntry.
/rk/v1/deps List dependencies related application, entire contents of go.mod file would be returned.
/rk/v1/entries List all Entries.
/rk/v1/gc Trigger GC
/rk/v1/healthy Get application healthy status.
/rk/v1/info Get application and process info.
/rk/v1/license Get license related application, entire contents of LICENSE file would be returned.
/rk/v1/logs List logger related entries.
/rk/v1/git Get git information.
/rk/v1/readme Get contents of README file.
/rk/v1/req List prometheus metrics of requests.
/rk/v1/sys Get OS stat.
/rk/v1/tv Get HTML page of /tv.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

CommonService options

name description type default value
gin.commonService.enabled Enable embedded common service boolean false

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true     # Enable common service

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

4.Enable swagger UI

In order to visualise common service in swagger UI, we need to enable swagger in boot.yaml as bellow.

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true     # Enable common service
    sw:
      enabled: true     # Enable swagger UI

Validate

http://localhost:8080/sw

sw-common

Cheers

1.3.1.1.3 - RK TV

Enable RK TV for server.

Overview

RK TV is a web UI contains service information including APIs, process info, metrics etc.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

TV options

name description type default value
gin.tv.enabled Enable RK TV boolean false

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    tv:
      enabled: true     # Enable TV

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Validate

http://localhost:8080/rk/v1/tv

tv

Cheers

1.3.1.1.4 - Prometheus client

Enable prometheus client for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Prometheus options

name description type default value
gin.prom.enabled Enable prometheus boolean false
gin.prom.path Path of prometheus string /metrics
gin.prom.pusher.enabled Enable prometheus pusher bool false
gin.prom.pusher.jobName Job name would be attached as label while pushing to remote pushgateway string ""
gin.prom.pusher.remoteAddress PushGateWay address, could be form of http://x.x.x.x or x.x.x.x string ""
gin.prom.pusher.intervalMs Push interval in milliseconds string 1000
gin.prom.pusher.basicAuth Basic auth used to interact with remote pushgateway, form of [user:pass] string ""
gin.prom.pusher.cert.ref Reference of rkentry.CertEntry string ""

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    prom:
      enabled: true     # Enable prometheus client
#      path: "metrics"   # Default value is "metrics", set path as needed.

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Validate

http://localhost:8080/metrics

prom

Cheers

4.Add value to prometheus client

In order to add custom metrics into current prometheus client, we need to clarify bellow concept.

prom

Name Description
MetricsSet RK defined data structure could be used register Counter, Gauge, Histogram and Summary
Prometheus Registerer User need to register MetricsSet into registerer
Prometheus Counter A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
Prometheus Gauge A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
Prometheus Histogram A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
Prometheus Summary Similar to a histogram, a summary samples observations (usually things like request durations and response sizes).
Prometheus Namespace Prometheus metrics was consist of namespace_subSystem_metricsName
Prometheus SubSystem Prometheus metrics was consist of namespace_subSystem_metricsName
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-prom"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Create a metrics set into prometheus.Registerer
	set := rkprom.NewMetricsSet("rk", "demo", boot.GetGinEntry("greeter").PromEntry.Registerer)

	// Register counter, gauge, histogram, summary
	set.RegisterCounter("my_counter", "label")
	set.RegisterGauge("my_gauge", "label")
	set.RegisterHistogram("my_histogram", []float64{}, "label")
	set.RegisterSummary("my_summary", rkprom.SummaryObjectives, "label")

	// Increase counter, gauge, histogram, summary with label value
	set.GetCounterWithValues("my_counter", "value").Inc()
	set.GetGaugeWithValues("my_gauge", "value").Add(1.0)
	set.GetHistogramWithValues("my_histogram", "value").Observe(0.1)
	set.GetSummaryWithValues("my_summary", "value").Observe(0.1)

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

5.Validate metrics

Validate

http://localhost:8080/metrics

prom

Cheers

6.Push metrics to Pushgateway

Turn on pusher in boot.yaml.

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    prom:
      enabled: true                         # Enable prometheus client
      pusher:
        enabled : true                      # Enable backend job push metrics to remote pushgateway
        jobName: "demo"                     # Name of current push job
        remoteAddress: "localhost:9091"     # Remote address of pushgateway
        intervalMs: 2000                    # Push interval in milliseconds
#        basicAuth: "user:pass"             # Basic auth of pushgateway
#        cert:
#          ref: "ref"                       # Cert reference defined in CertEntry. Please see advanced user guide for details.

Start pushgateway locally for testing

$ docker run prom/pushgateway -p 9091:9091

Validate metrics at local pushgateway

http://localhost:9091

pushgateway

Cheers

1.3.1.1.5 - Middleware logging

Enable RPC logging interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Logging options

name description type default value
gin.interceptors.loggingZap.enabled Enable log interceptor boolean false
gin.interceptors.loggingZap.zapLoggerEncoding json or console string console
gin.interceptors.loggingZap.zapLoggerOutputPaths Output paths []string stdout
gin.interceptors.loggingZap.eventLoggerEncoding json or console string console
gin.interceptors.loggingZap.eventLoggerOutputPaths Output paths []string false

Concept

In order to monitor every RPC request, we introduce EventLogger and ZapLogger

ZapLogger

RK bootstrapper adopt zap as the logger. lumberjack as the logger rotation.

Example

2021-07-05T23:35:17.104+0800    INFO    boot/gin_entry.go:631   Bootstrapping GinEntry. {"eventId": "08d11a34-472a-4785-a98c-144a3417d7f0", "entryName": "greeter", "entryType": "GinEntry", "port": 8080, "interceptorsCount": 1, "swEnabled": false, "tlsEnabled": false, "commonServiceEnabled": false, "tvEnabled": false, "promPath": "/metrics", "promPort": 8080}

EventLogger

RK bootstrapper treat RPC request as an Event, and record every RPC request into Event type in rk-query.

Field Description
endTime As name described
startTime As name described
elapsedNano Elapsed time for RPC in nanoseconds
timezone As name described
ids Contains three different ids(eventId, requestId and traceId). If meta interceptor was enabled or event.SetRequestId() was called by user, then requestId would be attached. eventId would be the same as requestId if meta interceptor was enabled. If trace interceptor was enabled, then traceId would be attached.
app Contains appName, appVersion, entryName, entryType.
env Contains arch, az, domain, hostname, localIP, os, realm, region. realm, region, az, domain were retrieved from environment variable named as REALM, REGION, AZ and DOMAIN. “*” means empty environment variable.
payloads Contains RPC related metadata
error Contains errors if occur
counters Set by calling event.SetCounter() by user.
pairs Set by calling event.AddPair() by user.
timing Set by calling event.StartTimer() and event.EndTimer() by user.
remoteAddr As name described
operation RPC method name
resCode Response code of RPC
eventStatus Ended or InProgress

Example

------------------------------------------------------------------------
endTime=2021-06-25T01:30:45.144023+08:00
startTime=2021-06-25T01:30:45.143767+08:00
elapsedNano=255948
timezone=CST
ids={"eventId":"3332e575-43d8-4bfe-84dd-45b5fc5fb104","requestId":"3332e575-43d8-4bfe-84dd-45b5fc5fb104","traceId":"65b9aa7a9705268bba492fdf4a0e5652"}
app={"appName":"rk-gin","appVersion":"master-xxx","entryName":"greeter","entryType":"GinEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"apiMethod":"GET","apiPath":"/rk/v1/healthy","apiProtocol":"HTTP/1.1","apiQuery":"","userAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost:60718
operation=/rk/v1/healthy
resCode=200
eventStatus=Ended
EOE

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      loggingZap:
        enabled: true        # Enable logging interceptor/middleware, by default, stdout would be destination of log

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

Log

------------------------------------------------------------------------
endTime=2021-07-05T23:42:35.588164+08:00
startTime=2021-07-05T23:42:35.588095+08:00
elapsedNano=69414
timezone=CST
ids={"eventId":"9b874eea-b16b-4c46-b0f5-d2b7cff6844e"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GinEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"apiMethod":"GET","apiPath":"/rk/v1/healthy","apiProtocol":"HTTP/1.1","apiQuery":"","userAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost:56274
operation=/rk/v1/healthy
resCode=200
eventStatus=Ended
EOE

Cheers

4.JSON format

---
gin:
  - name: greeter
    ...
    interceptors:
      loggingZap:
        ...
        zapLoggerEncoding: "json"          # Override to json format, option: json or console
        eventLoggerEncoding: "json"        # Override to json format, option: json or console

Cheers

5.Output paths

By default, logs will be rotated by 1GB and compressed.

---
gin:
  - name: greeter
    ...
    interceptors:
      loggingZap:
        ...
        zapLoggerOutputPaths: ["logs/app.log"]        # Override output paths, option: json or console
        eventLoggerOutputPaths: ["logs/event.log"]    # Override output paths, option: json or console

Cheers

5.Get RPC scope logger

A new zap logger instance with requestId(if exist enabled by meta interceptor) will be created for every RPC request.

func Greeter(ctx *gin.Context) {
	rkginctx.GetLogger(ctx).Info("Received request")

	ctx.JSON(http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", ctx.Query("name")),
	})
}
2021-07-06T02:04:55.306+0800    INFO    basic/main.go:39        Received request        {"requestId": "b8522178-9fac-47d3-b866-ce43e119f7a3"}

Cheers

6.Modify Event

A new event logger instance will be created for every RPC request.

User can add pairs, counters, errors in event which will be logged as soon as RPC finish.

func Greeter(ctx *gin.Context) {
	event := rkginctx.GetEvent(ctx)
	event.AddPair("key", "value")

	ctx.JSON(http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", ctx.Query("name")),
	})
}
------------------------------------------------------------------------
endTime=2021-07-06T02:08:40.929323+08:00
startTime=2021-07-06T02:08:40.929191+08:00
elapsedNano=132017
timezone=CST
ids={"eventId":"8f5cfe20-12c7-4208-995a-b113c7fc46a1","requestId":"8f5cfe20-12c7-4208-995a-b113c7fc46a1"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GinEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"apiMethod":"GET","apiPath":"/v1/greeter","apiProtocol":"HTTP/1.1","apiQuery":"name=rk-dev","userAgent":"curl/7.64.1"}
error={}
counters={}
pairs={"key":"value"}
timing={}
remoteAddr=localhost:57311
operation=/v1/greeter
resCode=200
eventStatus=Ended
EOE

Cheers

1.3.1.1.6 - Middleware metrics

Enable RPC metrics interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Metrics options

name description type default value
gin.interceptors.metricsProm.enabled Enable metrics interceptor boolean false

Concept

By default, metrics interceptor/middleware will record bellow metrics.

Metrics name Metrics type Description
elapsedNano Summary The time elapsed for RPC.
resCode Counter The counter of RPC with resCode.
errors Counter The counter of RPC with errors if occurs.

All of three metrics have the same labels as bellow:

Label name Description
entryName Gin entry name
entryType Gin entry type
realm OS environment variable: REALM, eg: rk
region OS environment variable: REGION, eg: beijing
az OS environment variable: AZ, eg: beijing-1
domain OS environment variable: DOMAIN, eg: prod
instance Hostname
appVersion Retrieved from AppInfoEntry
appName Retrieved from AppInfoEntry
restMethod Restful API method, eg: GET
restPath Restful API path, eg: /rk/v1/healthy
resCode Response code, eg: 200

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    prom:
      enabled : true         # Enable prometheus client in order to export metrics
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      metricsProm:
        enabled: true        # Enable metrics interceptor/middleware

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

Prometheus client:

http://localhost:8080/metrics

prom-inter

Cheers

1.3.1.1.7 - Middleware meta

Enable RPC meta interceptor/middleware for server.

Overview

Meta interceptor/middleware will attach bellow headers to server response.

Header key Description
X-Request-Id Request id generated by the interceptor.
X-[Prefix]-App Application name.
X-[Prefix]-App-Version Version of application.
X-[Prefix]-App-Unix-Time Unix time of current application.
X-[Prefix]-Request-Received-Time Time of current request received by application.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Meta options

name description type default value
gin.interceptors.meta.enabled Enable meta interceptor boolean false
gin.interceptors.meta.prefix Header key was formed as X--XXX string RK

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      meta:
        enabled: true        # Enable meta interceptor/middleware

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -vs -X GET localhost:8080/rk/v1/healthy
  ...
  < X-Request-Id: c2ec237e-e141-45af-a4ff-89ce8a9d0636
  < X-Rk-App-Name: rk-demo
  < X-Rk-App-Unix-Time: 2021-07-06T02:46:31.977902+08:00
  < X-Rk-App-Version: master-f414049
  < X-Rk-Received-Time: 2021-07-06T02:46:31.977902+08:00
  ...
  {"healthy":true}

Cheers

4.Override requestId

func Greeter(ctx *gin.Context) {
    // Override request id
	rkginctx.SetHeaderToClient(ctx, rkginctx.RequestIdKey, "request-id-override")
    // We expect new request id attached to logger
	rkginctx.GetLogger(ctx).Info("Received request")

	ctx.JSON(http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", ctx.Query("name")),
	})
}
$ curl -vs -X GET "localhost:8080/v1/greeter?name=rk-dev"
...
< X-Request-Id: request-id-override
< X-Rk-App-Name: rk-demo
< X-Rk-App-Unix-Time: 2021-07-06T02:49:34.27756+08:00
< X-Rk-App-Version: master-f414049
< X-Rk-Received-Time: 2021-07-06T02:49:34.27756+08:00
...
{"Message":"Hello rk-dev!"}

If we enabled logging interceptor/middleware, then we expect request as bellow

2021-07-06T02:49:34.277+0800    INFO    basic/main.go:41        Received request        {"requestId": "request-id-override"}
------------------------------------------------------------------------
endTime=2021-07-06T02:49:34.27773+08:00
startTime=2021-07-06T02:49:34.277544+08:00
elapsedNano=185412
timezone=CST
ids={"eventId":"request-id-override","requestId":"request-id-override"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GinEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"apiMethod":"GET","apiPath":"/v1/greeter","apiProtocol":"HTTP/1.1","apiQuery":"name=rk-dev","userAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost:57600
operation=/v1/greeter
resCode=200
eventStatus=Ended
EOE

Cheers

5.Override header prefix

---
gin:
  - name: greeter
    ...
    interceptors:
      meta:
        enabled: true        # Enable meta interceptor/middleware
        prefix: "Override"   # Override prefix which formed as X-[Prefix]-xxx
$ curl -vs -X GET localhost:8080/rk/v1/healthy
...
< X-Override-App-Name: rk-demo
< X-Override-App-Unix-Time: 2021-07-06T02:52:53.035941+08:00
< X-Override-App-Version: master-f414049
< X-Override-Received-Time: 2021-07-06T02:52:53.035941+08:00
< X-Request-Id: c3097361-1833-4bba-9867-e8d1f2fb2207
...
{"healthy":true}

Cheers

1.3.1.1.8 - Middleware tracing

Enable RPC tracing interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Tracing options

name description type default value
gin.interceptors.tracingTelemetry.enabled Enable tracing interceptor boolean false
gin.interceptors.tracingTelemetry.exporter.file.enabled Enable file exporter boolean false
gin.interceptors.tracingTelemetry.exporter.file.outputPath Export tracing info to files string stdout
gin.interceptors.tracingTelemetry.exporter.jaeger.agent.enabled Export tracing info to jaeger agent boolean false
gin.interceptors.tracingTelemetry.exporter.jaeger.agent.host As name described string localhost
gin.interceptors.tracingTelemetry.exporter.jaeger.agent.port As name described int 6831
gin.interceptors.tracingTelemetry.exporter.jaeger.collector.enabled Export tracing info to jaeger collector boolean false
gin.interceptors.tracingTelemetry.exporter.jaeger.collector.endpoint As name described string http://localhost:16368/api/trace
gin.interceptors.tracingTelemetry.exporter.jaeger.collector.username As name described string ""
gin.interceptors.tracingTelemetry.exporter.jaeger.collector.password As name described string ""

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      tracingTelemetry:
        enabled: true        # Enable tracing interceptor/middleware
        exporter:
          file:
            enabled: true
            outputPath: "stdout"

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}
[
        {
                "SpanContext": {
                        "TraceID": "4b59250ca328450045efe841a443e77d",
                        "SpanID": "4d3545e0c0da05c3",
                        "TraceFlags": "01",
                        "TraceState": null,
                        "Remote": false
                },
                "Parent": {
                        "TraceID": "00000000000000000000000000000000",
                        "SpanID": "0000000000000000",
                        "TraceFlags": "00",
                        "TraceState": null,
                        "Remote": true
                },
                "SpanKind": 2,
                "Name": "/rk/v1/healthy",
                "StartTime": "2021-07-06T03:06:50.871097+08:00",
                "EndTime": "2021-07-06T03:06:50.871167677+08:00",
                ....
                "InstrumentationLibrary": {
                        "Name": "greeter",
                        "Version": "semver:0.20.0"
                }
        }
]

Cheers

4.Export tracing log to file

There will be delay for the couple of seconds before flushing to log file.

---
gin:
  - name: greeter
    ...
    interceptors:
      tracingTelemetry:
        enabled: true                       # Enable tracing interceptor/middleware
        exporter:
          file:
            enabled: true
            outputPath: "logs/tracing.log"  # Log to files

Cheers

5.Export to jaeger

Start jaeger-all-in-one for testing

$ docker run -d --name jaeger \
    -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one:1.23
---
gin:
  - name: greeter
    ...
    interceptors:
      tracingTelemetry:
        enabled: true                                          # Enable tracing interceptor/middleware
        exporter:
          jaeger:
            agent:
              enabled: true                                    # Export to jaeger agent
#              host: ""                                        # Optional, default: localhost
#              port: 0                                         # Optional, default: 6831
#            collector:
#              enabled: true                                   # Optional, default: false
#              endpoint: ""                                    # Optional, default: http://localhost:14268/api/traces
#              username: ""                                    # Optional, default: ""
#              password: ""                                    # Optional, default: ""

Jaeger:

http://localhost:16686/

jaeger

Cheers

1.3.1.1.9 - Middleware auth

Enable RPC auth interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Auth options

name description type default value
gin.interceptors.auth.enabled Enable auth interceptor boolean false
gin.interceptors.auth.basic Basic auth credentials as scheme of user:pass []string []
gin.interceptors.auth.apiKey API key auth []string []
gin.interceptors.auth.ignorePrefix The paths of prefix that will be ignored by interceptor []string []

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      auth:
        enabled: true        # Enable auth interceptor/middleware
        basic: ["user:pass"] # Enable basic auth

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

$ curl  -X GET localhost:8080/rk/v1/healthy
# This is RK style error code if unauthorized
{
    "error":{
        "code":401,
        "status":"Unauthorized",
        "message":"Missing authorization, provide one of bellow auth header:[Basic Auth]",
        "details":[]
    }
}

Cheers

4.With X-API-Key auth type

---
gin:
  - name: greeter
    ...
    interceptors:
      auth:
        enabled: true        # Enable auth interceptor/middleware
        apiKey: ["token"]    # Enable X-API-Key auth

Cheers

5.Ignore paths

---
gin:
  - name: greeter
    ...
    interceptors:
      auth:
        enabled: true                     # Enable auth interceptor/middleware
        basic: ["user:pass"]              # Enable basic auth
        ignorePrefix: ["/rk/v1/healthy"]  # Ignoring path with prefix

Cheers

1.3.1.1.10 - Middleware rate limit

Enable rate limit interceptor/middleware for the server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Rate limit options

name description type default value
gin.interceptors.rateLimit.enabled Enable rate limit interceptor boolean false
gin.interceptors.rateLimit.algorithm Provide algorithm, tokenBucket and leakyBucket are available options string tokenBucket
gin.interceptors.rateLimit.reqPerSec Request per second globally int 0
gin.interceptors.rateLimit.paths.path Full path string ""
gin.interceptors.rateLimit.paths.reqPerSec Request per second by full path int 0

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      rateLimit:
        enabled: true
        algorithm: "leakyBucket"
        reqPerSec: 100
        paths:
          - path: "/rk/v1/healthy"
            reqPerSec: 0

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/healthy
{
    "error":{
        "code":429,
        "status":"Too Many Requests",
        "message":"",
        "details":[
            "slow down your request"
        ]
    }
}

Cheers

1.3.1.1.11 - Middleware timeout

Enable timeout interceptor/middleware for the server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Timeout options

name description type default value
gin.interceptors.timeout.enabled Enable timeout interceptor boolean false
gin.interceptors.timeout.timeoutMs Global timeout in milliseconds. int 5000
gin.interceptors.timeout.paths.path Full path string ""
gin.interceptors.timeout.paths.timeoutMs Timeout in milliseconds by full path int 5000

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true                                 # Enable common service for testing
    interceptors:
      timeout:
        enabled: true                               # Optional, default: false
        timeoutMs: 5000                             # Optional, default: 5000
        paths: 
          - path: "/rk/v1/gc"                       # Optional, default: ""
            timeoutMs: 1                            # Optional, default: 5000

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/gc
{
    "error":{
        "code":408,
        "status":"Request Timeout",
        "message":"Request timed out!",
        "details":[]
    }
}

Cheers

1.3.1.1.12 - Middleware gzip

Enable gzip interceptor/middleware for the server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a gin server with rk-boot

name description type default value
gin.name The name of gin server string N/A
gin.port The port of gin server integer nil, server won’t start
gin.enabled Enable gin entry bool false
gin.description Description of gin entry. string ""

Gzip options

name description type default value
gin.interceptors.gzip.enabled Enable gzip interceptor boolean false
gin.interceptors.gzip.level Provide level of compression, options are noCompression, bestSpeed, bestCompression, defaultCompression, huffmanOnly. string defaultCompression

Quick start

1.Create boot.yaml

---
gin:
  - name: greeter                     # Required
    port: 8080                        # Required
    enabled: true                     # Required
    commonService:
      enabled: true                   # Optional, default: false
    interceptors:
      gzip:
        enabled: true                 # Optional, default: false
        level: bestSpeed              # Optional, options: [noCompression, bestSpeed, bestCompression, defaultCompression, huffmanOnly]

2.Create main.go

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"io"
	"net/http"
	"strings"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Register handler
	boot.GetGinEntry("greeter").Router.POST("/v1/post", post)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

// PostResponse Response of Greeter.
type PostResponse struct {
	ReceivedMessage string
}

// post Handler.
func post(ctx *gin.Context) {
	buf := new(strings.Builder)
	io.Copy(buf, ctx.Request.Body)

	ctx.JSON(http.StatusOK, &PostResponse{
		ReceivedMessage: fmt.Sprintf("Received %s!", buf.String()),
	})
}

3.Validate

Send request

$ echo 'this is message' | gzip | curl --compressed --data-binary @- -H "Content-Encoding: gzip" -H "Accept-Encoding: gzip" localhost:8080/v1/post
{"ReceivedMessage":"this is message\n"}

Cheers

1.3.1.2 - Advanced

RK provide bootstrapper for popular frameworks in order to save time for learning complex initializing process.

Overview

In advanced user guide, we will introduce how to enable bellow functionalities by modifying boot.yaml.

Functionality Description
Locale How to distinguish environment based os OS.Environment
Logging Use user defined logging
TLS Enable TLS support
Config Use viper config
AppInfo Use custom application information
Multiple entries Start multiple Gin entries
Shutdown hook Register shutdown hook functions
Error type Use standard error type
Override bootstrapper Override bootstrapper
Trace RPC with logs Trace RPC logs based on traceId

1.3.1.2.1 - Locale

Distinguish entries based on different environment.

Overview

Lets assuming we need to use different config files based on the region like Singapore or Frankfurt.

Since bootstrapper support multiple ConfigEntries in boot.yaml, we need a mechanism to distinguish ConfigEntries.

Concept

How entries selected by bootstrapper?

RK use REALM, REGION, AZ, DOMAIN environment variable to distinguish environments.

Quick start

1.Create config files

config/singapore.yaml

---
region: sinpapore

config/frankfurt.yaml

---
region: frankfurt

config/default.yaml

---
region: default

2.Create boot.yaml

config:
  # Load this config if nothing matched
  - name: my-config
    locale: "*::*::*::*"
    path: config/default.yaml
  # Load this config if REGION=singapore
  - name: my-config
    locale: "*::singapore::*::*"
    path: config/singapore.yaml
  # Load this config if REGION=frankfurt
  - name: my-config
    locale: "*::frankfurt::*::*"
    path: config/frankfurt.yaml
gin:
  - name: greeter
    port: 8080
    enabled: true

3.With REGION=singapore

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"os"
)

// Application entrance.
func main() {
    // Set REGION=singapore
	os.Setenv("REGION", "singapore")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

singapore

4.With REGION=frankfurt

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"os"
)

// Application entrance.
func main() {
    // Set REGION=singapore
	os.Setenv("REGION", "frankfurt")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

frankfurt

Cheers

5.With REGION=not-matched

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"os"
)

// Application entrance.
func main() {
    // Set REGION=singapore
	os.Setenv("REGION", "not-matched")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

default

Cheers

6.With REGION=""

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

default

Cheers

1.3.1.2.2 - Logging

Customise logging.

Architecture

ZapLoggerEntry

ZapLoggerEntry is used for initializing zap logger.

// ZapLoggerEntry contains bellow fields.
// 1: EntryName: Name of entry.
// 2: EntryType: Type of entry which is ZapLoggerEntryType.
// 3: EntryDescription: Description of ZapLoggerEntry.
// 4: Logger: zap.Logger which was initialized at the beginning.
// 5: LoggerConfig: zap.Logger config which was initialized at the beginning which is not accessible after initialization..
// 6: LumberjackConfig: lumberjack.Logger which was initialized at the beginning.
type ZapLoggerEntry struct {
	EntryName        string             `yaml:"entryName" json:"entryName"`
	EntryType        string             `yaml:"entryType" json:"entryType"`
	EntryDescription string             `yaml:"entryDescription" json:"entryDescription"`
	Logger           *zap.Logger        `yaml:"-" json:"-"`
	LoggerConfig     *zap.Config        `yaml:"zapConfig" json:"zapConfig"`
	LumberjackConfig *lumberjack.Logger `yaml:"lumberjackConfig" json:"lumberjackConfig"`
}

YAML

ZapLoggerEntry follows zap and lumberjack YAML hierarchy.

Please refer to zap and lumberjack site for details.

---
zapLogger:
  - name: zap-logger                      # Required
    description: "Description of entry"   # Optional
    zap:
      level: info                         # Optional, default: info, options: [debug, DEBUG, info, INFO, warn, WARN, dpanic, DPANIC, panic, PANIC, fatal, FATAL]
      development: true                   # Optional, default: true
      disableCaller: false                # Optional, default: false
      disableStacktrace: true             # Optional, default: true
      sampling:                           # Optional, default: empty map
        initial: 0
        thereafter: 0
      encoding: console                   # Optional, default: "console", options: [console, json]
      encoderConfig:
        messageKey: "msg"                 # Optional, default: "msg"
        levelKey: "level"                 # Optional, default: "level"
        timeKey: "ts"                     # Optional, default: "ts"
        nameKey: "logger"                 # Optional, default: "logger"
        callerKey: "caller"               # Optional, default: "caller"
        functionKey: ""                   # Optional, default: ""
        stacktraceKey: "stacktrace"       # Optional, default: "stacktrace"
        lineEnding: "\n"                  # Optional, default: "\n"
        levelEncoder: "capitalColor"      # Optional, default: "capitalColor", options: [capital, capitalColor, color, lowercase]
        timeEncoder: "iso8601"            # Optional, default: "iso8601", options: [rfc3339nano, RFC3339Nano, rfc3339, RFC3339, iso8601, ISO8601, millis, nanos]
        durationEncoder: "string"         # Optional, default: "string", options: [string, nanos, ms]
        callerEncoder: ""                 # Optional, default: ""
        nameEncoder: ""                   # Optional, default: ""
        consoleSeparator: ""              # Optional, default: ""
      outputPaths: [ "stdout" ]           # Optional, default: ["stdout"], stdout would be replaced if specified
      errorOutputPaths: [ "stderr" ]      # Optional, default: ["stderr"], stderr would be replaced if specified
      initialFields:                      # Optional, default: empty map
        key: "value"
    lumberjack:                           # Optional
      filename: "rkapp-event.log"         # Optional, default: It uses <processname>-lumberjack.log in os.TempDir() if empty.
      maxsize: 1024                       # Optional, default: 1024 (MB)
      maxage: 7                           # Optional, default: 7 (days)
      maxbackups: 3                       # Optional, default: 3 (days)
      localtime: true                     # Optional, default: true
      compress: true                      # Optional, default: true

Access

// Access entry
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger")

// Access zap logger
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLogger()

// Access zap logger config
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLoggerConfig()

// Access lumberjack config
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLumberjackConfig()

EventLoggerEntry

RK bootstrapper treat RPC request as an Event, and record every RPC request into Event type in rk-query.

// EventLoggerEntry contains bellow fields.
// 1: EntryName: Name of entry.
// 2: EntryType: Type of entry which is EventLoggerEntryType.
// 3: EntryDescription: Description of EventLoggerEntry.
// 4: EventFactory: rkquery.EventFactory was initialized at the beginning.
// 5: EventHelper: rkquery.EventHelper was initialized at the beginning.
// 6: LoggerConfig: zap.Config which was initialized at the beginning which is not accessible after initialization.
// 7: LumberjackConfig: lumberjack.Logger which was initialized at the beginning.
type EventLoggerEntry struct {
	EntryName        string                `yaml:"entryName" json:"entryName"`
	EntryType        string                `yaml:"entryType" json:"entryType"`
	EntryDescription string                `yaml:"entryDescription" json:"entryDescription"`
	EventFactory     *rkquery.EventFactory `yaml:"-" json:"-"`
	EventHelper      *rkquery.EventHelper  `yaml:"-" json:"-"`
	LoggerConfig     *zap.Config           `yaml:"zapConfig" json:"zapConfig"`
	LumberjackConfig *lumberjack.Logger    `yaml:"lumberjackConfig" json:"lumberjackConfig"`
}

Fields

Field Description
endTime As name described
startTime As name described
elapsedNano Elapsed time for RPC in nanoseconds
timezone As name described
ids Contains three different ids(eventId, requestId and traceId). If meta interceptor was enabled or event.SetRequestId() was called by user, then requestId would be attached. eventId would be the same as requestId if meta interceptor was enabled. If trace interceptor was enabled, then traceId would be attached.
app Contains appName, appVersion, entryName, entryType.
env Contains arch, az, domain, hostname, localIP, os, realm, region. realm, region, az, domain were retrieved from environment variable named as REALM, REGION, AZ and DOMAIN. “*” means empty environment variable.
payloads Contains RPC related metadata
error Contains errors if occur
counters Set by calling event.SetCounter() by user.
pairs Set by calling event.AddPair() by user.
timing Set by calling event.StartTimer() and event.EndTimer() by user.
remoteAddr As name described
operation RPC method name
resCode Response code of RPC
eventStatus Ended or InProgress

Example

------------------------------------------------------------------------
endTime=2021-06-25T01:30:45.144023+08:00
startTime=2021-06-25T01:30:45.143767+08:00
elapsedNano=255948
timezone=CST
ids={"eventId":"3332e575-43d8-4bfe-84dd-45b5fc5fb104","requestId":"3332e575-43d8-4bfe-84dd-45b5fc5fb104","traceId":"65b9aa7a9705268bba492fdf4a0e5652"}
app={"appName":"rk-gin","appVersion":"master-xxx","entryName":"greeter","entryType":"GinEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"apiMethod":"GET","apiPath":"/rk/v1/healthy","apiProtocol":"HTTP/1.1","apiQuery":"","userAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost:60718
operation=/rk/v1/healthy
resCode=200
eventStatus=Ended
EOE

YAML

EventLoggerEntry needs application name while creating event log. The application name retrieved from go.mod file. If there is go.mod file missing, then default one would be used.

---
eventLogger:
  - name: event-logger                 # Required
    description: "This is description" # Optional
    encoding: console                  # Optional, default: console, options: console and json
    outputPaths: ["stdout"]            # Optional
    lumberjack:                        # Optional
      filename: "rkapp-event.log"      # Optional, default: It uses <processname>-lumberjack.log in os.TempDir() if empty.
      maxsize: 1024                    # Optional, default: 1024 (MB)
      maxage: 7                        # Optional, default: 7 (days)
      maxbackups: 3                    # Optional, default: 3 (days)
      localtime: true                  # Optional, default: true
      compress: true                   # Optional, default: true

Access

// Access entry
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger")

// Access event factory
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventFactory()

// Access event helper
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventHelper()

// Access lumberjack config
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetLumberjackConfig()

Quick start

1.Custom ZapLoggerEntry

---
zapLogger:
  - name: zap-logger                      # Required
    description: "Description of entry"   # Optional
    zap:
      encoding: json
gin:
  - name: greeter
    port: 8080
    enabled: true
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get custom ZapLoggerEntry
	rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLogger().Info("Custom zap logger")

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
{"level":"INFO","ts":"2021-07-06T05:06:16.252+0800","msg":"Custom zap logger"}

Cheers

2.Reference ZapLoggerEntry

Bootstrapper will use this logger by default including interceptor.

---
zapLogger:
  - name: zap-logger
gin:
  - name: greeter
    port: 8080
    enabled: true
    logger:
     zapLogger:
        ref: zap-logger

Cheers

3.Custom EventLoggerEntry

---
eventLogger:
  - name: event-logger
gin:
  - name: greeter
    port: 8080
    enabled: true
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get custom EventLoggerEntry
	helper := rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventHelper()
	event := helper.Start("demo")
	event.AddPair("key", "value")
	helper.Finish(event)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
------------------------------------------------------------------------
endTime=2021-07-06T05:10:08.660629+08:00
startTime=2021-07-06T05:10:08.660625+08:00
elapsedNano=4034
timezone=CST
ids={"eventId":"545693bb-6c61-4486-957a-cdbea19649c7"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"","entryType":""}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={}
error={}
counters={}
pairs={"key":"value"}
timing={}
remoteAddr=localhost
operation=demo
resCode=OK
eventStatus=Ended
EOE

Cheers

4.Reference EventLoggerEntry

Bootstrapper will use this logger by default including interceptor.

---
eventLogger:
  - name: event-logger
gin:
  - name: greeter
    port: 8080
    enabled: true
    logger:
      eventLogger:
        ref: event-logger

Cheers

5.Set env at EventLoggerEntry

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
    // Set env
	os.Setenv("REALM", "my-realm")
	os.Setenv("REGION", "my-region")
	os.Setenv("AZ", "my-az")
	os.Setenv("DOMAIN", "my-domain")

	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get custom EventLoggerEntry
	helper := rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventHelper()
	event := helper.Start("demo")
	helper.Finish(event)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
------------------------------------------------------------------------
...
env={"arch":"amd64","az":"my-az","domain":"my-domain","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"my-realm","region":"my-region"}
...

Cheers

6.Access ZapLoggerEntry & EventLoggerEntry

User can access by calling rkentry.GlobalAppCtx.GetEventLoggerEntry() & rkentry.GlobalAppCtx.GetZapLoggerEntry().

1.3.1.2.3 - TLS/SSL

Enable TLS/SSL for the server.

Overview

In order to enable TLS/SSL on the server, we need server certificate and server key at least.

Bootstrapper will read CertEntry section in boot.yaml file and load certificates into memory.

We support four types of CertEntry retrievers. We will extend retriever future.

  • LocalFs
  • RemoteFs
  • Consul
  • Etcd

Architecture

Locale

CertEntry support concept of locale.

Generate Self-Signed Certificate

There is a convenient way to generate certificates with cfssl

1.Download cfssl & cfssljson

Install cfssl with rk cli

$ go get -u github.com/rookie-ninja/rk/cmd/rk
$ rk install cfssl
$ rk install cfssljson

2.Generate CA

$ cfssl print-defaults config > ca-config.json
$ cfssl print-defaults csr > ca-csr.json

Adjust ca-config.json and ca-csr.json as needed.

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

3.Generate server certificates

server.csr, server.pem and server-key.pem would be created.

$ cfssl gencert -config ca-config.json -ca ca.pem -ca-key ca-key.pem -profile www csr.json | cfssljson -bare server

Quick start

1.localFs

Name Description Default
cert.localFs.name Name of localFs retriever ""
cert.localFs.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.localFs.serverCertPath Path of server cert in local file system. ""
cert.localFs.serverKeyPath Path of server key in local file system. ""
cert.localFs.clientCertPath Path of client cert in local file system. ""
cert.localFs.clientCertPath Path of client key in local file system. ""
---
cert:
  - name: "local-cert"                     # Required
    description: "Description of entry"    # Optional
    provider: "localFs"                    # Required, etcd, consul, localFs, remoteFs are supported options
    locale: "*::*::*::*"                   # Required, default: ""
    serverCertPath: "cert/server.pem"      # Optional, default: "", path of certificate on local FS
    serverKeyPath: "cert/server-key.pem"   # Optional, default: "", path of certificate on local FS
gin:
  - name: greeter
    port: 8080
    enabled: true
    cert:
      ref: "local-cert"
    commonService: 
      enabled: true
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

2.remoteFs

Name Description Default
cert.remoteFs.name Name of remoteFileStore retriever ""
cert.remoteFs.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.remoteFs.endpoint Endpoint of remoteFileStore server, http://x.x.x.x or x.x.x.x both acceptable. N/A
cert.remoteFs.basicAuth Basic auth for remoteFileStore server, like user:pass. ""
cert.remoteFs.serverCertPath Path of server cert in remoteFs server. ""
cert.remoteFs.serverKeyPath Path of server key in remoteFs server. ""
cert.remoteFs.clientCertPath Path of client cert in remoteFs server. ""
cert.remoteFs.clientCertPath Path of client key in remoteFs server. ""
---
cert:
  - name: "remote-cert"                    # Required
    description: "Description of entry"    # Optional
    provider: "remoteFs"                   # Required, etcd, consul, localFs, remoteFs are supported options
    endpoint: "localhost:8081"             # Required, both http://x.x.x.x or x.x.x.x are acceptable
    locale: "*::*::*::*"                   # Required, default: ""
    serverCertPath: "cert/server.pem"      # Optional, default: "", path of certificate on local FS
    serverKeyPath: "cert/server-key.pem"   # Optional, default: "", path of certificate on local FS
gin:
  - name: greeter
    port: 8080
    enabled: true
    cert:
      ref: "remote-cert"
    commonService: 
      enabled: true
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

3.consul

Name Description Default
cert.consul.name Name of consul retriever ""
cert.consul.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.consul.endpoint Endpoint of Consul server, http://x.x.x.x or x.x.x.x both acceptable. N/A
cert.consul.datacenter consul datacenter. ""
cert.consul.token Token for access consul. ""
cert.consul.basicAuth Basic auth for consul server, like user:pass. ""
cert.consul.serverCertPath Path of server cert in Consul server. ""
cert.consul.serverKeyPath Path of server key in Consul server. ""
cert.consul.clientCertPath Path of client cert in Consul server. ""
cert.consul.clientCertPath Path of client key in Consul server. ""
---
cert:
  - name: "consul-cert"                    # Required
    provider: "consul"                     # Required, etcd, consul, localFS, remoteFs are supported options
    description: "Description of entry"    # Optional
    locale: "*::*::*::*"                   # Required, default: ""
    endpoint: "localhost:8500"             # Required, http://x.x.x.x or x.x.x.x both acceptable.
    datacenter: "dc1"                      # Optional, default: "", consul datacenter
    serverCertPath: "server.pem"           # Optional, default: "", key of value in consul
    serverKeyPath: "server-key.pem"        # Optional, default: "", key of value in consul
gin:
  - name: greeter
    port: 8080
    enabled: true
    cert:
      ref: "consul-cert"
    commonService: 
      enabled: true
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

4.etcd

Name Description Default
cert.etcd.name Name of etcd retriever ""
cert.etcd.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.etcd.endpoint Endpoint of etcd server, http://x.x.x.x or x.x.x.x both acceptable. N/A
cert.etcd.basicAuth Basic auth for etcd server, like user:pass. ""
cert.etcd.serverCertPath Path of server cert in etcd server. ""
cert.etcd.serverKeyPath Path of server key in etcd server. ""
cert.etcd.clientCertPath Path of client cert in etcd server. ""
cert.etcd.clientCertPath Path of client key in etcd server. ""
---
cert:
  - name: "etcd-cert"                      # Required
    description: "Description of entry"    # Optional
    provider: "etcd"                       # Required, etcd, consul, localFs, remoteFs are supported options
    locale: "*::*::*::*"                   # Required, default: ""
    endpoint: "localhost:2379"             # Required, http://x.x.x.x or x.x.x.x both acceptable.
    serverCertPath: "server.pem"           # Optional, default: "", key of value in etcd
    serverKeyPath: "server-key.pem"        # Optional, default: "", key of value in etcd
gin:
  - name: greeter
    port: 8080
    enabled: true
    cert:
      ref: "etcd-cert"
    commonService: 
      enabled: true
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

5.Access CertEntry

User can access CertEntry by calling rkentry.GlobalAppCtx.GetCertEntry().

1.3.1.2.4 - Config

How to read configs in local file system.

Overview

Bootstrapper will try to read config files described in [config] section in boot.yaml file and load the file content with viper.

As a result, bellow file type can be supported.

  • JSON
  • TOML
  • YAML
  • HCL
  • envfile
  • Java propertie

Architecture

Locale

ConfigEntry support concept of locale.

Quick start

1.Create config

Create config file and specify in boot.yaml file

config/default.yaml

region: default

boot.yaml

config:
  - name: my-config
    locale: "*::*::*::*"
    path: config/default.yaml
gin:
  - name: greeter
    port: 8080
    enabled: true

2.Refer config path

main.go

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
	"github.com/rookie-ninja/rk-gin/interceptor/context"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Get ConfigEntry and get viper instance
	fmt.Println(rkentry.GlobalAppCtx.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Access ConfigEntry

User can access ConfigEntry by calling rkentry.GlobalAppCtx.GetConfigEntry().

1.3.1.2.5 - AppInfo

How to specify application info in boot.yaml?

Overview

Application info consist of bellow fields.

app:
  description: "this is description"  # Optional, default: ""
  keywords: ["rk", "golang"]          # Optional, default: []
  homeUrl: "http://example.com"       # Optional, default: ""
  iconUrl: "http://example.com"       # Optional, default: ""
  docsUrl: ["http://example.com"]     # Optional, default: []
  maintainers: ["rk-dev"]             # Optional, default: []

Quick start

app:
  description: "this is description"  # Optional, default: ""
  keywords: ["rk", "golang"]          # Optional, default: []
  homeUrl: "http://example.com"       # Optional, default: ""
  iconUrl: "http://example.com"       # Optional, default: ""
  docsUrl: ["http://example.com"]     # Optional, default: []
  maintainers: ["rk-dev"]             # Optional, default: []
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true                   # Enable for validation
    tv: 
      enabled: true                   # Enable for validation

1.Access from RK TV

http://localhost:8080/rk/v1/tv/info

Cheers

2.Access from common service

$ curl localhost:8080/rk/v1/info
{
    "appName":"rk-demo",
    "version":"master-f414049",
    "description":"this is description",
    "keywords":[
        "rk",
        "golang"
    ],
    "homeUrl":"http://example.com",
    "iconUrl":"http://example.com",
    "docsUrl":[
        "http://example.com"
    ],
    "maintainers":[
        "rk-dev"
    ],
    "uid":"501",
    "gid":"20",
    "username":"XXX",
    "startTime":"2021-07-08T00:38:51+08:00",
    "upTimeSec":13,
    "upTimeStr":"13 seconds",
    "region":"",
    "az":"",
    "realm":"",
    "domain":""
}

Cheers

3.Access from AppInfoEntry

rkentry.GlobalAppCtx.GetAppInfoEntry()

type AppInfoEntry struct {
	EntryName        string   `json:"entryName" yaml:"entryName"`
	EntryType        string   `json:"entryType" yaml:"entryType"`
	EntryDescription string   `json:"description" yaml:"description"`
	AppName          string   `json:"appName" yaml:"appName"`
	Version          string   `json:"version" yaml:"version"`
	Lang             string   `json:"lang" yaml:"lang"`
	Keywords         []string `json:"keywords" yaml:"keywords"`
	HomeUrl          string   `json:"homeUrl" yaml:"homeUrl"`
	IconUrl          string   `json:"iconUrl" yaml:"iconUrl"`
	DocsUrl          []string `json:"docsUrl" yaml:"docsUrl"`
	Maintainers      []string `json:"maintainers" yaml:"maintainers"`
	License          string   `json:"-" yaml:"-"`
	Readme           string   `json:"-" yaml:"-"`
	GoMod            string   `json:"-" yaml:"-"`
	UtHtml           string   `json:"-" yaml:"-"`
}

Cheers

1.3.1.2.6 - Multiple entries

How to start multiple Gin server with different port in one process?

Overview

With bootstrapper, user can start multiple GinEntry at the same time. Event for multiple different entries like GRPC.

Quick start

gin:
  - name: alice
    port: 8080
    enabled: true
    commonService:
      enabled: true
  - name: bob
    port: 8081
    enabled: true
    commonService:
      enabled: true

1.Access entries

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-gin/interceptor/context"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get alice
	boot.GetGinEntry("alice")
    // Get bob
	boot.GetGinEntry("bob")

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Cheers

1.3.1.2.7 - Error type

What is the best way to return an RPC error?

Overview

It is always not easy to define a user-friendly API. However, errors could happen anywhere.

In order to return standard error type to user, it is important to define an error type.

By default, panic middleware/interceptor will be attached which will catch panic and return internal server error as bellow:

Error type used by bootstrapper was defined in rkerror.ErrorResp

func Greeter(ctx *gin.Context) {
	panic("Panic manually!")
}
{
    "error":{
        "code":500,
        "status":"Internal Server Error",
        "message":"Panic manually!",
        "details":[]
    }
}

Quick start

Return errors

Here is the way how to return errors in user RPC implementation.

func Greeter(ctx *gin.Context) {
	err := rkerror.New(
		rkerror.WithHttpCode(http.StatusAlreadyReported),
		rkerror.WithMessage("Trigger manually!"),
		rkerror.WithDetails("This is detail.", false, -1, 0.1))

	ctx.JSON(http.StatusAlreadyReported, err)
}
$ curl "localhost:8080/v1/greeter?name=rk-dev"
{
    "error":{
        "code":208,
        "status":"Already Reported",
        "message":"Trigger manually!",
        "details":[
            "This is detail.",
            false,
            -1,
            0.1
        ]
    }
}

Cheers

1.3.1.2.8 - Shutdown hook

How to add shutdown hook function while receiving shutdown signal?

Overview

We need to understand how bootstrapper stop process.

  1. Add shutdown hook functions by user
  2. Call functions as soon as receive signal from outside

Getting started

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-gin/interceptor/context"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Add shutdown hook function
	boot.AddShutdownHookFunc("shutdown-hook", func() {
		fmt.Println("shutting down")
	})

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
...
shutting down
...

Cheers

1.3.1.2.9 - Override bootstrapper

Is there any way to override boot.yaml or values in boot.yaml at start time?

Overview

Bootstrapper support two kinds of ways to override bootstrapper configs.

  • Override config file (by --rkboot)
  • Override values in config file (by --rkset)

Quick start

1.Override bootstrapper config file

In order to override bootstrapper file path, --rkboot needs to be passed.

boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true

boot-override.yaml

---
gin:
  - name: greeter
    port: 8081
    enabled: true
    commonService:
      enabled: true

main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Start server with --rkboot args

$ go build main.go
$ ./main --rkboot boot-override.yaml

Send request to 8080

$ curl localhost:8080/rk/v1/healthy
curl: (7) Failed to connect to localhost port 8080: Connection refused

Send request to 8081

$ curl localhost:8081/rk/v1/healthy
{"healthy":true}

Cheers

2.Override values in boot.yaml

In order to override bootstrapper file path, --rkset needs to be passed.

Use comma to separate multiple overrides.

Type Example
Map app.description=“This is description”
List gin[0].name=“alice”,gin[0].port=8081

boot.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true

main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Start server with --rkset args

$ go build main.go
$ ./main --rkset "gin[0].port=8081"

Send request to 8080

$ curl localhost:8080/rk/v1/healthy
curl: (7) Failed to connect to localhost port 8080: Connection refused

Send request to 8081

$ curl localhost:8081/rk/v1/healthy
{"healthy":true}

Cheers

1.3.1.2.10 - Trace RPC with logs

How to trace RPC with logs?

Overview

Sometimes, we don’t want to install any of tracing service like jaeger because of cost. In that case, we need a way to trace RPC with logs only.

Bootstrapper introduce a way to trace RPC with openTelemetry library but without tracing service.

Concept

RK bootstrapper will use traceId to track each RPC in distributed system which will be logged into RPC log while tracing and log interceptor enabled in bootstrapper.

EventId

EventId would be generated if logging interceptor/middleware was enabled bellow:

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    interceptors:
      loggingZap:
        enabled: true
------------------------------------------------------------------------
...
ids={"eventId":"cd617f0c-2d93-45e1-bef0-95c89972530d"}
...

RequestId

Generated by meta interceptor/middleware automatically if user enabled bellow:

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    interceptors:
      meta:
        enabled: true
      loggingZap:
        enabled: true
------------------------------------------------------------------------
...
ids={"eventId":"8226ba9b-424e-4e19-ba63-d37ca69028b3","requestId":"8226ba9b-424e-4e19-ba63-d37ca69028b3"}
...

If meta interceptor/middleware was enabled or requestId was enabled by user, then eventId will be override by requestId.

Simply, eventId will always the same as requestId

  rkginctx.SetHeaderToClient(ctx, rkginctx.RequestIdKey, "overridden-request-id")
------------------------------------------------------------------------
...
ids={"eventId":"overridden-request-id","requestId":"overridden-request-id"}
...

TraceId

Generated if user enable tracing interceptor/middleware by user as bellow:

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    interceptors:
      meta:
        enabled: true
      loggingZap:
        enabled: true
      tracingTelemetry:
        enabled: true
------------------------------------------------------------------------
...
ids={"eventId":"dd19cf9a-c7be-486c-b29d-7af777a78ebe","requestId":"dd19cf9a-c7be-486c-b29d-7af777a78ebe","traceId":"316a7b475ff500a76bfcd6147036951c"}
...

Quick start

1.Create ServerA at 8080

bootA.yaml

---
gin:
  - name: greeter
    port: 8080
    enabled: true
    interceptors:
      meta:
        enabled: true
      loggingZap:
        enabled: true
      tracingTelemetry:
        enabled: true

serverA.go

package main

import (
	"context"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-gin/interceptor/context"
	"net/http"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot(rkboot.WithBootConfigPath("bootA.yaml"))

	// Register handler
	boot.GetGinEntry("greeter").Router.GET("/v1/hello", func(ctx *gin.Context) {
		client := http.DefaultClient

        // Construct request point to Server B
		req, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://localhost:8081/v1/hello", nil)

		// Inject parent trace info into request header
		rkginctx.InjectSpanToHttpRequest(ctx, req)

        // Send request to Server B
		client.Do(req)

		ctx.JSON(http.StatusOK, "Hello!")
	})

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

2.Create ServerB at 8081

bootB.yaml

---
gin:
  - name: greeter
    port: 8081
    enabled: true
    interceptors:
      meta:
        enabled: true
      loggingZap:
        enabled: true
      tracingTelemetry:
        enabled: true

serverB.go

package main

import (
	"context"
	"github.com/gin-gonic/gin"
	"github.com/rookie-ninja/rk-boot"
	"net/http"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot(rkboot.WithBootConfigPath("bootB.yaml"))

	// Register handler
	boot.GetGinEntry("greeter").Router.GET("/v1/hello", func(ctx *gin.Context) {
		ctx.JSON(http.StatusOK, "Hello!")
	})

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Start ServerA and ServerB

$ go run serverA.go
$ go run serverB.go

4.Send request to ServerA

$ curl localhost:8080/v1/hello

5.Validate logs

Two server will have same traceId in event log but different requestId and eventId.

What we need to do is grep same traceId.

ServerA

------------------------------------------------------------------------
...
ids={"eventId":"9b2e9278-0e19-4936-9afe-966ff7fce1c9","requestId":"9b2e9278-0e19-4936-9afe-966ff7fce1c9","traceId":"fc3801be6de88116a7c47fafca6a7328"}
...

ServerB

------------------------------------------------------------------------
...
ids={"eventId":"7b9585de-eaa3-4ad5-ae7a-e3be7fdb312e","requestId":"7b9585de-eaa3-4ad5-ae7a-e3be7fdb312e","traceId":"fc3801be6de88116a7c47fafca6a7328"}
...

1.3.1.3 - Developer

How to implement a custom entry?

Overview

Every entries in builtin bootstrapper implement Entry interface in rk-entry package.

// Entry interface which must be implemented for bootstrapper to bootstrap
type Entry interface {
	// Bootstrap entry
	Bootstrap(context.Context)

	// Interrupt entry
	// Wait for shutdown signal and wait for draining incomplete procedure
	Interrupt(context.Context)

	// Get name of entry
	GetName() string

	// Get type of entry
	GetType() string

	// Get description of entry
	GetDescription() string

	// print entry as string
	String() string
}

Quick start

An entry could be any kinds of services or pieces of codes which needs to be bootstrap/initialized while application starts.

A third party entry could be implemented and inject to rk-boot via boot.yaml file.

How to create a new custom entry?

  1. Construct your own entry YAML struct as needed
  2. Create a struct which implements Entry interface
  3. Implements EntryRegFunc
  4. Register your reg function in init() in order to register your entry while application starts

How entry interact with rk-boot.Bootstrapper?

  1. Entry will be created and registered into rkentry.GlobalAppCtx
  2. Bootstrap will be called from Bootstrapper.Bootstrap() function
  3. Application will wait for shutdown signal
  4. Interrupt will be called from Bootstrapper.Interrupt() function

1.Implement Entry

type MyEntry struct {
	EntryName        string                    `json:"entryName" yaml:"entryName"`
	EntryType        string                    `json:"entryType" yaml:"entryType"`
	EntryDescription string                    `json:"entryDescription" yaml:"entryDescription"`
}

func (entry *MyEntry) Bootstrap(context.Context) {}

func (entry *MyEntry) Interrupt(context.Context) {}

func (entry *MyEntry) GetName() string {
	return entry.EntryName
}

func (entry *MyEntry) GetType() string {
	return entry.EntryType
}

func (entry *MyEntry) GetDescription() string {
	return entry.EntryDescription
}

func (entry *MyEntry) String() string {
	bytes, _ := json.Marshal(entry)
	return string(bytes)
}

2.Create bootstrapper config

This structure will match myEntry section in boot.yaml file

// A struct which is for unmarshalled YAML
type BootConfig struct {
	MyEntry struct {
		Enabled     bool   `yaml:"enabled" json:"enabled"`
		Name        string `yaml:"name" json:"name"`
		Description string `yaml:"description" json:"description"`
	} `yaml:"myEntry" json:"myEntry"`
}
myEntry:
  enabled: true
  name: "xxx"
  descriptin: "xxx"

3.Implements EntryRegFunc

Although we only need RegisterMyEntriesFromConfig() function, we strongly recommend implementing as bellow in order to run our entry from code either.

// An implementation of:
// type EntryRegFunc func(string) map[string]rkentry.Entry
func RegisterMyEntriesFromConfig(configFilePath string) map[string]rkentry.Entry {
	res := make(map[string]rkentry.Entry)

	// 1: decode config map into boot config struct
	config := &BootConfig{}
	rkcommon.UnmarshalBootConfig(configFilePath, config)

	// 3: construct entry
	if config.MyEntry.Enabled {
		entry := RegisterMyEntry(
			WithName(config.MyEntry.Name),
			WithDescription(config.MyEntry.Description),
		res[entry.GetName()] = entry
	}

	return res
}

func RegisterMyEntry(opts ...MyEntryOption) *MyEntry {
	entry := &MyEntry{
		EntryName:        "myEntry",
		EntryType:        "myEntry",
		EntryDescription: "Please contact maintainers to add description of this entry.",
	}

	for i := range opts {
		opts[i](entry)
	}

	if len(entry.EntryName) < 1 {
		entry.EntryName = "my-default"
	}

	if len(entry.EntryDescription) < 1 {
		entry.EntryDescription = "Please contact maintainers to add description of this entry."
	}

	rkentry.GlobalAppCtx.AddEntry(entry)

	return entry
}

type MyEntryOption func(*MyEntry)

func WithName(name string) MyEntryOption {
	return func(entry *MyEntry) {
		entry.EntryName = name
	}
}

func WithDescription(description string) MyEntryOption {
	return func(entry *MyEntry) {
		entry.EntryDescription = description
	}
}

4.Create init() function

Register your reg function in init() in order to register your entry while application starts

func init() {
	rkentry.RegisterEntryRegFunc(RegisterMyEntriesFromConfig)
}

5.Full example

boot.yaml

---
myEntry:
  enabled: true
  name: my-entry
  description: "This is my entry."

main.go

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-common/common"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	fmt.Println(rkentry.GlobalAppCtx.GetEntry("my-entry"))

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}


// Register entry, must be in init() function since we need to register entry at beginning
func init() {
	rkentry.RegisterEntryRegFunc(RegisterMyEntriesFromConfig)
}

// A struct which is for unmarshalled YAML
type BootConfig struct {
	MyEntry struct {
		Enabled     bool   `yaml:"enabled" json:"enabled"`
		Name        string `yaml:"name" json:"name"`
		Description string `yaml:"description" json:"description"`
	} `yaml:"myEntry" json:"myEntry"`
}

// An implementation of:
// type EntryRegFunc func(string) map[string]rkentry.Entry
func RegisterMyEntriesFromConfig(configFilePath string) map[string]rkentry.Entry {
	res := make(map[string]rkentry.Entry)

	// 1: decode config map into boot config struct
	config := &BootConfig{}
	rkcommon.UnmarshalBootConfig(configFilePath, config)

	// 3: construct entry
	if config.MyEntry.Enabled {
		entry := RegisterMyEntry(
			WithName(config.MyEntry.Name),
			WithDescription(config.MyEntry.Description))
		res[entry.GetName()] = entry
	}

	return res
}

func RegisterMyEntry(opts ...MyEntryOption) *MyEntry {
	entry := &MyEntry{
		EntryName:        "default",
		EntryType:        "myEntry",
		EntryDescription: "Please contact maintainers to add description of this entry.",
	}

	for i := range opts {
		opts[i](entry)
	}

	if len(entry.EntryName) < 1 {
		entry.EntryName = "my-default"
	}

	if len(entry.EntryDescription) < 1 {
		entry.EntryDescription = "Please contact maintainers to add description of this entry."
	}

	rkentry.GlobalAppCtx.AddEntry(entry)

	return entry
}

type MyEntryOption func(*MyEntry)

func WithName(name string) MyEntryOption {
	return func(entry *MyEntry) {
		entry.EntryName = name
	}
}

func WithDescription(description string) MyEntryOption {
	return func(entry *MyEntry) {
		entry.EntryDescription = description
	}
}

type MyEntry struct {
	EntryName        string                    `json:"entryName" yaml:"entryName"`
	EntryType        string                    `json:"entryType" yaml:"entryType"`
	EntryDescription string                    `json:"entryDescription" yaml:"entryDescription"`
}

func (entry *MyEntry) Bootstrap(context.Context) {}

func (entry *MyEntry) Interrupt(context.Context) {}

func (entry *MyEntry) GetName() string {
	return entry.EntryName
}

func (entry *MyEntry) GetDescription() string {
	return entry.EntryDescription
}

func (entry *MyEntry) GetType() string {
	return entry.EntryType
}

func (entry *MyEntry) String() string {
	bytes, _ := json.Marshal(entry)
	return string(bytes)
}

5.Validate

$ go run main.go
{"entryName":"my-entry","entryType":"myEntry","entryDescription":"This is my entry."}

1.3.2 - GRPC

User guide for GRPC framework in Go.

1.3.2.1 - Basic

Basic user guide which could operate by user turn on/off by modifying boot.yaml.

Overview

In basic user guide, we will introduce how to enable bellow functionalities by modifying boot.yaml.

Functionality Description
GRPC Gateway Enable grpc-gateway
Swagger UI Enable swagger UI
Common Service Enable common service
RK TV Enable RK TV
Prometheus Client Enable prometheus client
Logging Enable logging middleware
Metrics Enable prometheus middleware
Meta Enable meta middleware
Tracing Enable openTelemetry tracing middleware
Auth Enable basic & ApiKey auth middleware

1.3.2.1.1 - GRPC Gateway

Enable grpc-gateway.

Overview

In traditional way, we need to add a bunch of codes in order to enable grpc-gateway,

With bootstrapper, grpc-gateway will be automatically started and bind to one single port with grpc for convenience.

Steps to enable grpc-gateway:

  1. Write gw_mapping.yaml
  2. Write buf.yaml & buf.gen.yaml
  3. Generate gateway file based on gw_mapping.yaml
  4. Register gateway function into GRPC server

Prerequisite

Install required tools.

We recommend use rk install them easily.

# Install RK CLI
$ go get -u github.com/rookie-ninja/rk/cmd/rk

# List available installation
$ rk install
COMMANDS:
    buf                      install buf on local machine
    cfssl                    install cfssl on local machine
    cfssljson                install cfssljson on local machine
    gocov                    install gocov on local machine
    golangci-lint            install golangci-lint on local machine
    mockgen                  install mockgen on local machine
    pkger                    install pkger on local machine
    protobuf                 install protobuf on local machine
    protoc-gen-doc           install protoc-gen-doc on local machine
    protoc-gen-go            install protoc-gen-go on local machine
    protoc-gen-go-grpc       install protoc-gen-go-grpc on local machne
    protoc-gen-grpc-gateway  install protoc-gen-grpc-gateway on local machine
    protoc-gen-openapiv2     install protoc-gen-openapiv2 on local machine
    swag                     install swag on local machine
    rk-std                   install rk standard environment on local machine
    help, h                  Shows a list of commands or help for one command

# Install buf, protoc-gen-go, protoc-gen-go-grpc, protoc-gen-grpc-gateway, protoc-gen-openapiv2
$ rk install protoc-gen-go
$ rk install protoc-gen-go-grpc
$ rk install protoc-gen-go-grpc-gateway
$ rk install protoc-gen-openapiv2
$ rk install buf
Tool Description Installation
protobuf protocol buffer Install
buf Help you create consistent Protobuf APIs that preserve compatibility and comply with design best-practices. Install
protoc-gen-go Plugin for the Google protocol buffer compiler to generate Go code.. Install
protoc-gen-go-grpc This project aims to provide that HTTP+JSON interface to your gRPC service. Install
protoc-gen-grpc-gateway plugin for Google protocol buffer compiler to generate a reverse-proxy, which converts incoming RESTful HTTP/1 requests gRPC invocation. Install
protoc-gen-openapiv2 plugin for Google protocol buffer compiler to generate open API config file. Install

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Quick start

1.Create api/v1/greeter.proto

syntax = "proto3";

package api.v1;

option go_package = "api/v1/greeter";

service Greeter {
  rpc Greeter (GreeterRequest) returns (GreeterResponse) {}
}

message GreeterRequest {
  string name = 1;
}

message GreeterResponse {
  string message = 1;
}

2.Create api/v1/gw_mapping.yaml

type: google.api.Service
config_version: 3

# Please refer google.api.Http in https://github.com/googleapis/googleapis/blob/master/google/api/http.proto file for details.
http:
  rules:
    - selector: api.v1.Greeter.Greeter
      get: /api/v1/greeter

3.Create buf.yaml

version: v1beta1
name: github.com/rk-dev/rk-demo
build:
  roots:
    - api

4.Create buf.gen.yaml

version: v1beta1
plugins:
  # protoc-gen-go needs to be installed, generate go files based on proto files
  - name: go
    out: api/gen
    opt:
     - paths=source_relative
  # protoc-gen-go-grpc needs to be installed, generate grpc go files based on proto files
  - name: go-grpc
    out: api/gen
    opt:
      - paths=source_relative
      - require_unimplemented_servers=false
  # protoc-gen-grpc-gateway needs to be installed, generate grpc-gateway go files based on proto files
  - name: grpc-gateway
    out: api/gen
    opt:
      - paths=source_relative
      - grpc_api_configuration=api/v1/gw_mapping.yaml
  # protoc-gen-openapiv2 needs to be installed, generate swagger config files based on proto files
  - name: openapiv2
    out: api/gen
    opt:
      - grpc_api_configuration=api/v1/gw_mapping.yaml

5.Compile proto file

$ buf generate

There will be bellow files generated.

$ tree api/gen 
api/gen
└── v1
    ├── greeter.pb.go
    ├── greeter.pb.gw.go
    ├── greeter.swagger.json
    └── greeter_grpc.pb.go
 
1 directory, 4 files

6.Create boot.yaml

grpc.gw.gwMappingFilePaths is not required, however, we strongly recommend specify this path since bootstrapper will use this in bellow:

  1. /rk/v1/apis
  2. RK TV
---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    gwMappingFilePaths:
      - "api/v1/gw_mapping.yaml"    # Boot will look for gateway mapping files and load information into memory

7. Create main.go

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

    // ***************************************
    // ******* Register GRPC & Gateway *******
    // ***************************************

	// Get grpc entry with name
	grpcEntry := boot.GetGrpcEntry("greeter")
    // Register grpc registration function
	grpcEntry.AddRegFuncGrpc(registerGreeter)
    // Register grpc-gateway registration function
	grpcEntry.AddRegFuncGw(greeter.RegisterGreeterHandlerFromEndpoint)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

// Implementation of [type GrpcRegFunc func(server *grpc.Server)]
func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

// Implementation of grpc service defined in proto file
type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}

8.Full structure

$ tree
.
├── api
│   ├── gen
│   │   └── v1
│   │       ├── greeter.pb.go
│   │       ├── greeter.pb.gw.go
│   │       ├── greeter.swagger.json
│   │       └── greeter_grpc.pb.go
│   └── v1
│       ├── greeter.proto
│       └── gw_mapping.yaml
├── boot.yaml
├── buf.gen.yaml
├── buf.yaml
├── go.mod
├── go.sum
└── main.go

4 directories, 12 files

9.Validate

$ go run main.go
$ curl "localhost:8080/api/v1/greeter?name=rk-dev"
{"message":"Hello rk-dev!"}

Cheers

Gateway server option

RK introduced gateway server options with bellow functionality.

rkgrpc.RkGwServerMuxOptions

Functionality Description
HttpErrorHandler Mainly copied from original gateway error handler. RK wrap errors to RK style error response.
MarshallerOption protojson.MarshalOptions{UseProtoNames: false, EmitUnpopulated: true},UnmarshalOptions: protojson.UnmarshalOptions{}}
Metadata Inject x-forwarded-method, x-forwarded-path, x-forwarded-scheme, x-forwarded-user-agent and x-forwarded-remote-addr into grpc metadata
OutgoingHeaderMatcher Bypass all grpc metadata to http header without prefix
IncomingHeaderMatcher Bypass all http header to grpc metadata without prefix

1.Enable rkServerOption

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    enableRkGwOption: true          # Enable rk style grpc gateway server option
    gwMappingFilePaths:
      - "api/v1/gw_mapping.yaml"    # Bootstrapper will look for gateway mapping files and load information into memory
    interceptors:
      loggingZap:
        enabled: true               # Enable logging interceptor for validation

2.Validate log

$ curl "localhost:8080/api/v1/greeter?name=rk-dev"

gwMethod, gwPath, gwScheme, gwUserAgent would be logged

------------------------------------------------------------------------
endTime=2021-07-09T21:03:43.518106+08:00
...
payloads={"grpcMethod":"Greeter","grpcService":"api.v1.Greeter","grpcType":"unaryServer","gwMethod":"GET","gwPath":"/v1/greeter","gwScheme":"http","gwUserAgent":"curl/7.64.1"}
...

3.Validate incoming metadata

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
    // Print incoming headers
	fmt.Println(rkgrpcctx.GetIncomingHeaders(ctx))

	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}
map[:authority:[0.0.0.0:8080] accept:[*/*] content-type:[application/grpc] user-agent:[grpc-go/1.38.0] x-forwarded-for:[::1] x-forwarded-host:[localhost:8080] x-forwarded-method:[GET] x-forwarded-path:[/v1/greeter] x-forwarded-remote-addr:[[::1]:57082] x-forwarded-scheme:[http] x-forwarded-user-agent:[curl/7.64.1]]

4.Override gateway server option for marshaller

In some cases, we may hope to override marshaller options. For example, returning underscore style restful result instead of camelcase.

Please refer to bellow repository for detailed explanations.

grpc:
  - name: greeter                                     # Required
    port: 8080                                        # Required
    enabled: true                                     # Required
    enableRkGwOption: true                            # Optional, default: false
    gwOption:                                         # Optional, default: nil
      marshal:                                        # Optional, default: nil
        multiline: false                              # Optional, default: false
        emitUnpopulated: false                        # Optional, default: false
        indent: ""                                    # Optional, default: false
        allowPartial: false                           # Optional, default: false
        useProtoNames: false                          # Optional, default: false
        useEnumNumbers: false                         # Optional, default: false
      unmarshal:                                      # Optional, default: nil
        allowPartial: false                           # Optional, default: false
        discardUnknown: false                         # Optional, default: false

Cheers

Error mapping

It is important to understand grpc-gateway error to grpc error mapping.

Here is the default error mapping defind in grpc-gateway.

GRPC Error GRPC Error Str Gateway(Http) Error Code Gateway(Http) Error Str
0 OK 200 OK
1 CANCELLED 408 Request Timeout
2 UNKNOWN 500 Internal Server Error
3 INVALID_ARGUMENT 400 Bad Request
4 DEADLINE_EXCEEDED 504 Gateway Timeout
5 NOT_FOUND 404 Not Found
6 ALREADY_EXISTS 409 Conflict
7 PERMISSION_DENIED 403 Forbidden
8 RESOURCE_EXHAUSTED 429 Too Many Requests
9 FAILED_PRECONDITION 400 Bad Request
10 ABORTED 409 Conflict
11 OUT_OF_RANGE 400 Bad Request
12 UNIMPLEMENTED 501 Not Implemented
13 INTERNAL 500 Internal Server Error
14 UNAVAILABLE 503 Service Unavailable
15 DATA_LOSS 500 Internal Server Error
16 UNAUTHENTICATED 401 Unauthorized

1.Validate error(Standard go error)

Based on default error mapping, we expect 500 response code.

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return nil, errors.New("error triggered manually")
}
$ curl "localhost:8080/v1/greeter?name=rk-dev"
{
    "error":{
        "code":500,
        "status":"Internal Server Error",
        "message":"error triggered manually",
        "details":[]
    }
}

2.Validate error(grpc error)

grpc needs to specify errors with status.New().

We recommend use rkerror to generate errors as bellow.

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return nil, rkerror.PermissionDenied("permission denied manually").Err()
}
curl "localhost:8080/v1/greeter?name=rk-dev"
{
    "error":{
        "code":403,
        "status":"Forbidden",
        "message":"permission denied manually",
        "details":[
            {
                "code":7,
                "status":"PermissionDenied",
                "message":"[from-grpc] permission denied manually"
            }
        ]
    }
}

Cheers

1.3.2.1.2 - Swagger UI

Enable swagger UI for server.

Prerequisite

In order to enable swagger, we need to enable grpc-gateway first.

Follow steps at prerequisite

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Swagger options

name description type default value
grpc.sw.enabled Enable swagger service over gRpc server boolean false
grpc.sw.path The path access swagger service from web string /sw
grpc.sw.jsonPath Where the swagger.json files are stored locally string ""
grpc.sw.headers Headers would be sent to caller as scheme of [key:value] []string []

Quick start

1.Create api/v1/greeter.proto

syntax = "proto3";

package api.v1;

option go_package = "api/v1/greeter";

service Greeter {
  rpc Greeter (GreeterRequest) returns (GreeterResponse) {}
}

message GreeterRequest {
  string name = 1;
}

message GreeterResponse {
  string message = 1;
}

2.Create api/v1/gw_mapping.yaml

type: google.api.Service
config_version: 3

# Please refer google.api.Http in https://github.com/googleapis/googleapis/blob/master/google/api/http.proto file for details.
http:
  rules:
    - selector: api.v1.Greeter.Greeter
      get: /api/v1/greeter

3.Create buf.yaml

version: v1beta1
name: github.com/rk-dev/rk-demo
build:
  roots:
    - api

4.Create buf.gen.yaml

version: v1beta1
plugins:
  # protoc-gen-go needs to be installed, generate go files based on proto files
  - name: go
    out: api/gen
    opt:
     - paths=source_relative
  # protoc-gen-go-grpc needs to be installed, generate grpc go files based on proto files
  - name: go-grpc
    out: api/gen
    opt:
      - paths=source_relative
      - require_unimplemented_servers=false
  # protoc-gen-grpc-gateway needs to be installed, generate grpc-gateway go files based on proto files
  - name: grpc-gateway
    out: api/gen
    opt:
      - paths=source_relative
      - grpc_api_configuration=api/v1/gw_mapping.yaml
  # protoc-gen-openapiv2 needs to be installed, generate swagger config files based on proto files
  - name: openapiv2
    out: api/gen
    opt:
      - grpc_api_configuration=api/v1/gw_mapping.yaml

5.Compile proto file

$ buf generate

There will be bellow files generated.

$ tree api/gen 
api/gen
└── v1
    ├── greeter.pb.go
    ├── greeter.pb.gw.go
    ├── greeter.swagger.json
    └── greeter_grpc.pb.go
 
1 directory, 4 files

6.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    enableRkGwOption: true          # Enable RK style server options
    gwMappingFilePaths:
      - "api/v1/gw_mapping.yaml"    # Bootstrapper will look for gateway mapping files and load information into memory
    sw:
      enabled: true                 # Enable swagger
      jsonPath: "api/gen/v1"            # Provide swagger config file path

7. Create main.go

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

    // ***************************************
    // ******* Register GRPC & Gateway *******
    // ***************************************

	// Get grpc entry with name
	grpcEntry := boot.GetGrpcEntry("greeter")
    // Register grpc registration function
	grpcEntry.AddRegFuncGrpc(registerGreeter)
    // Register grpc-gateway registration function
	grpcEntry.AddRegFuncGw(greeter.RegisterGreeterHandlerFromEndpoint)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

// Implementation of [type GrpcRegFunc func(server *grpc.Server)]
func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

// Implementation of grpc service defined in proto file
type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}

8.Full structure

$ tree
.
├── api
│   ├── gen
│   │   └── v1
│   │       ├── greeter.pb.go
│   │       ├── greeter.pb.gw.go
│   │       ├── greeter.swagger.json
│   │       └── greeter_grpc.pb.go
│   └── v1
│       ├── greeter.proto
│       └── gw_mapping.yaml
├── boot.yaml
├── buf.gen.yaml
├── buf.yaml
├── go.mod
├── go.sum
└── main.go

4 directories, 12 files

9.Validate

Swagger: http://localhost:8080/sw

Cheers

1.3.2.1.3 - Common Service

Enable common service for server.

Overview

Common service contains builtin API commonly used by user.

grpc-gateway Path GRPC Method Description
/rk/v1/apis rk.api.v1.RkcCommonService.Apis List APIs in current GrpcEntry.
/rk/v1/certs rk.api.v1.RkcCommonService.Certs List CertEntry.
/rk/v1/configs rk.api.v1.RkcCommonService.Configs List ConfigEntry.
/rk/v1/deps rk.api.v1.RkcCommonService.Deps List dependencies related application, entire contents of go.mod file would be returned.
/rk/v1/entries rk.api.v1.RkcCommonService.Entries List all Entries.
/rk/v1/gc rk.api.v1.RkcCommonService.Gc Trigger GC
/rk/v1/healthy rk.api.v1.RkcCommonService.Healthy Get application healthy status.
/rk/v1/info rk.api.v1.RkcCommonService.Info Get application and process info.
/rk/v1/license rk.api.v1.RkcCommonService.License Get license related application, entire contents of LICENSE file would be returned.
/rk/v1/logs rk.api.v1.RkcCommonService.Logs List logger related entries.
/rk/v1/git rk.api.v1.RkcCommonService.Git Get git information.
/rk/v1/readme rk.api.v1.RkcCommonService.Readme Get contents of README file.
/rk/v1/req rk.api.v1.RkcCommonService.Req List prometheus metrics of requests.
/rk/v1/sys rk.api.v1.RkcCommonService.Sys Get OS stat.
/rk/v1/tv rk.api.v1.RkcCommonService.Tv Get HTML page of /tv.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

CommonService options

name description type default value
grpc.commonService.enabled Enable embedded common service boolean false

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter
    port: 8080
    enabled: true
    enableReflection: true    # Enable reflection in order to use grpcurl for validation
    commonService:
      enabled: true           # Enable common service

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Install grpcurl from the official site. https://github.com/fullstorydev/grpcurl

# List grpc services at port 8080 without TLS
# Expect RkCommonService since we enabled common services.
$ grpcurl -plaintext localhost:8080 list                           
grpc.reflection.v1alpha.ServerReflection
rk.api.v1.RkCommonService

# List grpc methods in rk.api.v1.RkCommonService
$ grpcurl -plaintext localhost:8080 list rk.api.v1.RkCommonService            
rk.api.v1.RkCommonService.Apis
rk.api.v1.RkCommonService.Certs
rk.api.v1.RkCommonService.Configs
rk.api.v1.RkCommonService.Deps
rk.api.v1.RkCommonService.Entries
rk.api.v1.RkCommonService.Gc
rk.api.v1.RkCommonService.GwErrorMapping
rk.api.v1.RkCommonService.Healthy
rk.api.v1.RkCommonService.Info
rk.api.v1.RkCommonService.License
rk.api.v1.RkCommonService.Logs
rk.api.v1.RkCommonService.Readme
rk.api.v1.RkCommonService.Req
rk.api.v1.RkCommonService.Sys
rk.api.v1.RkCommonService.Git

# Send request to rk.api.v1.RkCommonService.Healthy
$ grpcurl -plaintext localhost:8080 rk.api.v1.RkCommonService.Healthy
{
    "healthy": true
}

Cheers

4.Enable swagger UI

In order to visualise common service in swagger UI, we need to enable swagger in boot.yaml as bellow.

---
grpc:
  - name: greeter
    port: 8080
    enabled: true
    enableReflection: true    # Enable reflection in order to use grpcurl for validation
    commonService:
      enabled: true           # Enable common service
    sw:
      enabled: true

Validate

http://localhost:8080/sw

sw-common

Cheers

1.3.2.1.4 - RK TV

Enable RK TV for server.

Overview

RK TV is a web UI contains service information including APIs, process info, metrics etc.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

TV options

name description type default value
grpc.tv.enabled Enable RK TV boolean false

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    tv:
      enabled: true                 # Enable TV

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Validate

http://localhost:8080/rk/v1/tv

tv

Cheers

1.3.2.1.5 - Prometheus client

Enable prometheus client for the server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Prometheus options

name description type default value
grpc.prom.enabled Enable prometheus boolean false
grpc.prom.path Path of prometheus string /metrics
grpc.prom.pusher.enabled Enable prometheus pusher bool false
grpc.prom.pusher.jobName Job name would be attached as label while pushing to remote pushgateway string ""
grpc.prom.pusher.remoteAddress PushGateWay address, could be form of http://x.x.x.x or x.x.x.x string ""
grpc.prom.pusher.intervalMs Push interval in milliseconds string 1000
grpc.prom.pusher.basicAuth Basic auth used to interact with remote pushgateway, form of [user:pass] string ""
grpc.prom.pusher.cert.ref Reference of rkentry.CertEntry string ""

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    prom:
      enabled: true                 # Enable prometheus client
#      path: "metrics"              # Default value is "metrics", set path as needed.

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Validate

http://localhost:8080/metrics

prom

Cheers

4.Add value to prometheus client

In order to add custom metrics into current prometheus client, we need to clarify bellow concept.

prom

Name Description
MetricsSet RK defined data structure could be used register Counter, Gauge, Histogram and Summary
Prometheus Registerer User need to register MetricsSet into registerer
Prometheus Counter A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
Prometheus Gauge A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
Prometheus Histogram A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
Prometheus Summary Similar to a histogram, a summary samples observations (usually things like request durations and response sizes).
Prometheus Namespace Prometheus metrics was consist of namespace_subSystem_metricsName
Prometheus SubSystem Prometheus metrics was consist of namespace_subSystem_metricsName
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-prom"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Create a metrics set into prometheus.Registerer
	set := rkprom.NewMetricsSet("rk", "demo", boot.GetGrpcEntry("greeter").GwEntry.PromEntry.Registerer)

	// Register counter, gauge, histogram, summary
	set.RegisterCounter("my_counter", "label")
	set.RegisterGauge("my_gauge", "label")
	set.RegisterHistogram("my_histogram", []float64{}, "label")
	set.RegisterSummary("my_summary", rkprom.SummaryObjectives, "label")

	// Increase counter, gauge, histogram, summary with label value
	set.GetCounterWithValues("my_counter", "value").Inc()
	set.GetGaugeWithValues("my_gauge", "value").Add(1.0)
	set.GetHistogramWithValues("my_histogram", "value").Observe(0.1)
	set.GetSummaryWithValues("my_summary", "value").Observe(0.1)

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

5.Validate metrics

Validate

http://localhost:8080/metrics

prom

Cheers

6.Push metrics to Pushgateway

Turn on pusher in boot.yaml.

---
grpc:
  - name: greeter                             # Name of grpc entry
    port: 8080                                # Port of grpc entry
    enabled: true                             # Enable grpc entry
    prom:
      enabled: true                           # Enable prometheus client
      pusher:
        enabled : true                        # Enable backend job push metrics to remote pushgateway
        jobName: "demo"                       # Name of current push job
        remoteAddress: "localhost:9091"       # Remote address of pushgateway
        intervalMs: 2000                      # Push interval in milliseconds
#        basicAuth: "user:pass"               # Basic auth of pushgateway
#        cert:
#          ref: "ref"                         # Cert reference defined in CertEntry. Please see advanced user guide for details.

Start pushgateway locally for testing

$ docker run prom/pushgateway -p 9091:9091

Validate metrics at local pushgateway

http://localhost:9091

pushgateway

Cheers

1.3.2.1.6 - Interceptor logging

Enable RPC logging interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Logging options

name description type default value
grpc.interceptors.loggingZap.enabled Enable log interceptor boolean false
grpc.interceptors.loggingZap.zapLoggerEncoding json or console string console
grpc.interceptors.loggingZap.zapLoggerOutputPaths Output paths []string stdout
grpc.interceptors.loggingZap.eventLoggerEncoding json or console string console
grpc.interceptors.loggingZap.eventLoggerOutputPaths Output paths []string stdout

Concept

In order to monitor every RPC request, we introduce EventLogger and ZapLogger

ZapLogger

RK bootstrapper adopt zap as the logger. lumberjack as the logger rotation.

Example

2021-07-09T23:52:13.667+0800    INFO    boot/grpc_entry.go:694  Bootstrapping grpcEntry.        {"eventId": "9bc192fb-567c-45d4-8775-7a097b0dab04", "entryName": "greeter", "entryType": "GrpcEntry", "grpcPort": 8080, "commonServiceEnabled": true, "tlsEnabled": false, "gwEnabled": true, "reflectionEnabled": false, "swEnabled": false, "tvEnabled": false, "promEnabled": false, "gwClientTlsEnabled": false, "gwServerTlsEnabled": false}

EventLogger

RK bootstrapper treat RPC request as an Event, and record every RPC request into Event type in rk-query.

Field Description
endTime As name described
startTime As name described
elapsedNano Elapsed time for RPC in nanoseconds
timezone As name described
ids Contains three different ids(eventId, requestId and traceId). If meta interceptor was enabled or event.SetRequestId() was called by user, then requestId would be attached. eventId would be the same as requestId if meta interceptor was enabled. If trace interceptor was enabled, then traceId would be attached.
app Contains appName, appVersion, entryName, entryType.
env Contains arch, az, domain, hostname, localIP, os, realm, region. realm, region, az, domain were retrieved from environment variable named as REALM, REGION, AZ and DOMAIN. “*” means empty environment variable.
payloads Contains RPC related metadata
error Contains errors if occur
counters Set by calling event.SetCounter() by user.
pairs Set by calling event.AddPair() by user.
timing Set by calling event.StartTimer() and event.EndTimer() by user.
remoteAddr As name described
operation RPC method name
resCode Response code of RPC
eventStatus Ended or InProgress

Example

------------------------------------------------------------------------
endTime=2021-07-09T23:44:09.81483+08:00
startTime=2021-07-09T23:44:09.814784+08:00
elapsedNano=46065
timezone=CST
ids={"eventId":"67d64dab-f3ea-4b77-93d0-6782caf4cfee"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Healthy","grpcService":"rk.api.v1.RkCommonService","grpcType":"unaryServer","gwMethod":"","gwPath":"","gwScheme":"","gwUserAgent":""}
error={}
counters={}
pairs={"healthy":"true"}
timing={}
remoteAddr=localhost:58205
operation=/rk.api.v1.RkCommonService/Healthy
resCode=OK
eventStatus=Ended
EOE

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    commonService:
      enabled: true                 # Enable common service for testing
    interceptors:
      loggingZap:
        enabled: true

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

Log

------------------------------------------------------------------------
endTime=2021-07-09T23:44:09.81483+08:00
startTime=2021-07-09T23:44:09.814784+08:00
elapsedNano=46065
timezone=CST
ids={"eventId":"67d64dab-f3ea-4b77-93d0-6782caf4cfee"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Healthy","grpcService":"rk.api.v1.RkCommonService","grpcType":"unaryServer","gwMethod":"","gwPath":"","gwScheme":"","gwUserAgent":""}
error={}
counters={}
pairs={"healthy":"true"}
timing={}
remoteAddr=localhost:58205
operation=/rk.api.v1.RkCommonService/Healthy
resCode=OK
eventStatus=Ended
EOE

Cheers

4.JSON format

---
grpc:
  - name: greeter
    ...
    interceptors:
      loggingZap:
        ...
        zapLoggerEncoding: "json"          # Override to json format, option: json or console
        eventLoggerEncoding: "json"        # Override to json format, option: json or console

Cheers

5.Output paths

By default, logs will be rotated by 1GB and compressed.

---
grpc:
  - name: greeter
    ...
    interceptors:
      loggingZap:
        ...
        zapLoggerOutputPaths: ["logs/app.log"]        # Override output paths, option: json or console
        eventLoggerOutputPaths: ["logs/event.log"]    # Override output paths, option: json or console

Cheers

5.Get RPC scope logger

A new zap logger instance with requestId(if exist enabled by meta interceptor) will be created for every RPC request.

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	rkgrpcctx.GetLogger(ctx).Info("Received request")

	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}
2021-07-09T23:50:39.318+0800    INFO    basic/main.go:36        Received request        {"requestId": "c33698f2-3071-48d4-9d92-b1aa311e6c06"}

Cheers

6.Modify Event

A new event logger instance will be created for every RPC request.

User can add pairs, counters, errors in event which will be logged as soon as RPC finish.

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	event := rkgrpcctx.GetEvent(ctx)
	event.AddPair("key", "value")

	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}
------------------------------------------------------------------------
endTime=2021-07-09T23:52:39.103351+08:00
startTime=2021-07-09T23:52:39.10332+08:00
elapsedNano=31154
timezone=CST
ids={"eventId":"92001951-80c1-4dda-8f14-f920834f5c61","requestId":"92001951-80c1-4dda-8f14-f920834f5c61"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Greeter","grpcService":"api.v1.Greeter","grpcType":"unaryServer","gwMethod":"","gwPath":"","gwScheme":"","gwUserAgent":""}
error={}
counters={}
pairs={"key":"value"}
timing={}
remoteAddr=localhost:58269
operation=/api.v1.Greeter/Greeter
resCode=OK
eventStatus=Ended
EOE

Cheers

1.3.2.1.7 - Interceptor metrics

Enable RPC metrics interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Logging options

name description type default value
grpc.interceptors.metricsProm.enabled Enable metrics interceptor boolean false

Concept

By default, metrics interceptor/middleware will record bellow metrics.

Metrics name Metrics type Description
elapsedNano Summary The time elapsed for RPC.
resCode Counter The counter of RPC with resCode.
errors Counter The counter of RPC with errors if occurs.

All of three metrics have the same labels as bellow:

Label name Description
entryName gRPC entry name
entryType gRPC entry type
realm OS environment variable: REALM, eg: rk
region OS environment variable: REGION, eg: beijing
az OS environment variable: AZ, eg: beijing-1
domain OS environment variable: DOMAIN, eg: prod
instance Hostname
appVersion Retrieved from AppInfoEntry
appName Retrieved from AppInfoEntry
restMethod Restful API method, eg: GET
restPath Restful API path, eg: /rk/v1/healthy
type Entry type, eg: grpc
resCode Response code, eg: OK

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    commonService:
      enabled: true                 # Enable common service for testing
    prom:
      enabled: true                 # Enable prometheus client
    interceptors:
      metricsProm:
        enabled: true

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

Prometheus client:

http://localhost:8080/metrics

prom-inter

Cheers

1.3.2.1.8 - Interceptor meta

Enable RPC meta interceptor/middleware for server.

Overview

Meta interceptor/middleware will attach bellow headers to server response.

Header key Description
X-Request-Id Request id generated by the interceptor.
X-[Prefix]-App Application name.
X-[Prefix]-App-Version Version of application.
X-[Prefix]-App-Unix-Time Unix time of current application.
X-[Prefix]-Request-Received-Time Time of current request received by application.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Meta options

name description type default value
grpc.interceptors.meta.enabled Enable meta interceptor boolean false
grpc.interceptors.meta.prefix Header key was formed as X--XXX string RK

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    commonService:
      enabled: true                 # Enable common service for testing
    interceptors:
      meta:
        enabled: true

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

At above boot.yaml, we didn’t enable enableRkGwOption, as a result, the headers returned from server will have prefix of Grpc-Metadata

$ curl -vs -X GET localhost:8080/rk/v1/healthy
  ...
  < Grpc-Metadata-Content-Type: application/grpc
  < Grpc-Metadata-X-Request-Id: cb8c0c77-39cd-4e15-89d2-66ba30fa7a46
  < Grpc-Metadata-X-Rk-App-Name: rk-demo
  < Grpc-Metadata-X-Rk-App-Unix-Time: 2021-07-10T00:19:58.170222+08:00
  < Grpc-Metadata-X-Rk-App-Version: master-f414049
  < Grpc-Metadata-X-Rk-Received-Time: 2021-07-10T00:19:58.170222+08:00
  ...
  {"healthy":true}

If we enable enableRkGwOption, then Grpc-Metadata will be removed!

---
grpc:
  ...
  enableRkGwOption: true 
$ curl -vs -X GET localhost:8080/rk/v1/healthy
  ...
  < X-Request-Id: 2ae18df6-d132-42ce-841c-571f19a88787
  < X-Rk-App-Name: rk-demo
  < X-Rk-App-Unix-Time: 2021-07-10T00:24:10.281868+08:00
  < X-Rk-App-Version: master-f414049
  < X-Rk-Received-Time: 2021-07-10T00:24:10.281868+08:00
  ...
  {"healthy":true}

Cheers

4.Override requestId

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	// Override request id
	rkgrpcctx.AddHeaderToClient(ctx, rkginctx.RequestIdKey, "request-id-override")
	// We expect new request id attached to logger
	rkgrpcctx.GetLogger(ctx).Info("Received request")

	return &greeter.GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.Name),
	}, nil
}

The headers bellow look weird, since there are two X-Request-Id, this is because grpc will merge metadata if we set it multiple times.

However, at client side, RK will read the last one with same key.

$ curl -vs -X GET "localhost:8080/v1/greeter?name=rk-dev"
...
< X-Request-Id: 4e858644-f2e2-48b2-adec-a1d329497abf
< X-Request-Id: request-id-override
< X-Rk-App-Name: rk-demo
< X-Rk-App-Unix-Time: 2021-07-10T00:29:19.030555+08:00
< X-Rk-App-Version: master-f414049
< X-Rk-Received-Time: 2021-07-10T00:29:19.030555+08:00
...
{"Message":"Hello rk-dev!"}

If we enabled logging interceptor/middleware, then we expect request as bellow

2021-07-10T00:29:19.030+0800    INFO    basic/main.go:40        Received request        {"requestId": "request-id-override"}
------------------------------------------------------------------------
endTime=2021-07-10T00:29:19.030673+08:00
startTime=2021-07-10T00:29:19.030535+08:00
elapsedNano=138133
timezone=CST
ids={"eventId":"request-id-override","requestId":"request-id-override"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Greeter","grpcService":"api.v1.Greeter","grpcType":"unaryServer","gwMethod":"GET","gwPath":"/v1/greeter","gwScheme":"http","gwUserAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost:58601
operation=/api.v1.Greeter/Greeter
resCode=OK
eventStatus=Ended
EOE

Cheers

5.Override header prefix

---
grpc:
  - name: greeter
    ...
    interceptors:
      meta:
        enabled: true        # Enable meta interceptor/middleware
        prefix: "Override"   # Override prefix which formed as X-[Prefix]-xxx
$ curl -vs -X GET localhost:8080/rk/v1/healthy
...
< X-Override-App-Name: rk-demo
< X-Override-App-Unix-Time: 2021-07-10T00:34:59.974205+08:00
< X-Override-App-Version: master-f414049
< X-Override-Received-Time: 2021-07-10T00:34:59.974205+08:00
< X-Request-Id: 4deded12-2d39-42a5-b740-b4f74dd73c90
...
{"healthy":true}

Cheers

1.3.2.1.9 - Interceptor rate limit

Enable rate limit interceptor/middleware for the server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Rate limit options

name description type default value
grpc.interceptors.rateLimit.enabled Enable rate limit interceptor boolean false
grpc.interceptors.rateLimit.algorithm Provide algorithm, tokenBucket and leakyBucket are available options string tokenBucket
grpc.interceptors.rateLimit.reqPerSec Request per second globally int 0
grpc.interceptors.rateLimit.paths.path gRPC full name string ""
grpc.interceptors.rateLimit.paths.reqPerSec Request per second by gRPC full method name int 0

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      rateLimit:
        enabled: false
        algorithm: "leakyBucket"
        reqPerSec: 0
        paths:
          - path: "/rk.api.v1.RkCommonService/Healthy"
            reqPerSec: 0

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ grpcurl -plaintext localhost:8080 rk.api.v1.RkCommonService.Healthy
Error invoking method "rk.api.v1.RkCommonService.Healthy": rpc error: code = ResourceExhausted desc = failed to query for service descriptor "rk.api.v1.RkCommonService": Slow down your request
$ curl -X GET localhost:8080/rk/v1/healthy
{
    "error":{
        "code":429,
        "status":"Too Many Requests",
        "message":"Slow down your request.",
        "details":[
            {
                "code":8,
                "status":"ResourceExhausted",
                "message":"[from-grpc] Slow down your request."
            }
        ]
    }
}

Cheers

1.3.2.1.10 - Interceptor tracing

Enable RPC tracing interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Tracing options

name description type default value
grpc.interceptors.tracingTelemetry.enabled Enable tracing interceptor boolean false
grpc.interceptors.tracingTelemetry.exporter.file.enabled Enable file exporter boolean false
grpc.interceptors.tracingTelemetry.exporter.file.outputPath Export tracing info to files string stdout
grpc.interceptors.tracingTelemetry.exporter.jaeger.agent.enabled Export tracing info to jaeger agent boolean false
grpc.interceptors.tracingTelemetry.exporter.jaeger.agent.host As name described string localhost
grpc.interceptors.tracingTelemetry.exporter.jaeger.agent.port As name described int 6831
grpc.interceptors.tracingTelemetry.exporter.jaeger.collector.enabled Export tracing info to jaeger collector boolean false
grpc.interceptors.tracingTelemetry.exporter.jaeger.collector.endpoint As name described string http://localhost:16368/api/trace
grpc.interceptors.tracingTelemetry.exporter.jaeger.collector.username As name described string ""
grpc.interceptors.tracingTelemetry.exporter.jaeger.collector.password As name described string ""

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    commonService:
      enabled: true                 # Enable common service for testing
    interceptors:
      tracingTelemetry:
        enabled: true
        exporter:
          file:
            enabled: true
            outputPath: "stdout"

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ grpcurl -plaintext localhost:8080 rk.api.v1.RkCommonService.Healthy
{
    "healthy": true
}

By default, tracing information will be exported to stdout

[
        {
                "SpanContext": {
                        "TraceID": "479bd29a93042619372da50163ae75e3",
                        "SpanID": "3b8e3ceba6741c23",
                        "TraceFlags": "01",
                        "TraceState": null,
                        "Remote": false
                },
                "Parent": {
                        "TraceID": "00000000000000000000000000000000",
                        "SpanID": "0000000000000000",
                        "TraceFlags": "00",
                        "TraceState": null,
                        "Remote": true
                },
                "SpanKind": 2,
                "Name": "/rk.api.v1.RkCommonService/Healthy",
                "StartTime": "2021-07-10T00:42:56.302578+08:00",
                "EndTime": "2021-07-10T00:42:56.30263999+08:00",
                ....
                "InstrumentationLibrary": {
                        "Name": "greeter",
                        "Version": "semver:0.20.0"
                }
        }
]

Cheers

4.Export tracing log to file

There will be delay for the couple of seconds before flushing to log file.

---
grpc:
  - name: greeter
    ...
    interceptors:
      tracingTelemetry:
        enabled: true                       # Enable tracing interceptor/middleware
        exporter:
          file:
            enabled: true
            outputPath: "logs/tracing.log"  # Log to files

Cheers

5.Export to jaeger

Start jaeger-all-in-one for testing

$ docker run -d --name jaeger \
    -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one:1.23
---
grpc:
  - name: greeter
    ...
    interceptors:
      tracingTelemetry:
        enabled: true                                          # Enable tracing interceptor/middleware
        exporter:
          jaeger:
            agent:
              enabled: true                                    # Export to jaeger agent
#              host: ""                                        # Optional, default: localhost
#              port: 0                                         # Optional, default: 6831
#            collector:
#              enabled: true                                   # Optional, default: false
#              endpoint: ""                                    # Optional, default: http://localhost:14268/api/traces
#              username: ""                                    # Optional, default: ""
#              password: ""                                    # Optional, default: ""

Jaeger:

http://localhost:16686/

jaeger

Cheers

1.3.2.1.11 - Interceptor auth

Enable RPC auth interceptor/middleware for server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Auth options

name description type default value
grpc.interceptors.auth.enabled Enable auth interceptor boolean false
grpc.interceptors.auth.basic Basic auth credentials as scheme of user:pass []string []
grpc.interceptors.auth.apiKey API key auth []string []
grpc.interceptors.auth.ignorePrefix The paths of prefix that will be ignored by interceptor []string []

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter
    port: 8080
    enabled: true
    enableRkGwOption: true
    commonService:
      enabled: true          # Enable common service for testing
    interceptors:
      auth:
        enabled: true        # Enable auth interceptor/middleware
        basic: ["user:pass"] # Enable basic auth

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

$ curl  -X GET localhost:8080/rk/v1/healthy
# This is RK style error code if unauthorized
{
    "error":{
        "code":401,
        "status":"Unauthorized",
        "message":"Missing authorization, provide one of bellow auth header:[Basic Auth]",
        "details":[
            {
                "code":16,
                "status":"Unauthenticated",
                "message":"[from-grpc] Missing authorization, provide one of bellow auth header:[Basic Auth]"
            }
        ]
    }
}

Cheers

4.With X-API-Key auth type

---
grpc:
  - name: greeter
    ...
    interceptors:
      auth:
        enabled: true        # Enable auth interceptor/middleware
        apiKey: ["token"]    # Enable X-API-Key auth

Cheers

5.Ignore paths

---
grpc:
  - name: greeter
    ...
    interceptors:
      auth:
        enabled: true                                         # Enable auth interceptor/middleware
        basic: ["user:pass"]                                  # Enable basic auth
        ignorePrefix: ["/rk.api.v1.RkCommonService/Healthy"]  # Ignoring path with prefix

Cheers

1.3.2.1.12 - Interceptor timeout

Enable timeout interceptor/middleware for the server.

Installation

go get github.com/rookie-ninja/rk-boot

General options

These are general options to start a grpc server with rk-boot

name description type default value Required
grpc.name The name of grpc server string “”, server won’t start Required
grpc.port The port of grpc server integer 0, server won’t start Required
grpc.enabled Enable grpc entry bool false Required
grpc.description Description of grpc entry. string "" Optional
grpc.enableReflection Enable grpc server reflection boolean false Optional
grpc.enableRkGwOption Enable RK style gateway server options. boolean false Optional
grpc.noRecvMsgSizeLimit Disable grpc server side receive message size limit boolean false Optional
grpc.gwMappingFilePaths The grpc gateway mapping file path []string [] Optional

Timeout options

name description type default value
grpc.interceptors.timeout.enabled Enable timeout interceptor boolean false
grpc.interceptors.timeout.timeoutMs Global timeout in milliseconds. int 5000
grpc.interceptors.timeout.paths.path Full path string ""
grpc.interceptors.timeout.paths.timeoutMs Timeout in milliseconds by full path int 5000

Quick start

1.Create boot.yaml

---
grpc:
  - name: greeter
    port: 8080
    enabled: true
    commonService:
      enabled: true                                 # Enable common service for testing
    interceptors:
      timeout:
        enabled: true                               # Optional, default: false
        timeoutMs: 5000                             # Optional, default: 5000
        paths: 
          - path: "/rk.api.v1.RkCommonService/Gc"   # Optional, default: ""
            timeoutMs: 1                            # Optional, default: 5000

2.Create main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3.Validate

Send request

$ grpcurl -plaintext localhost:8080 rk.api.v1.RkCommonService.Gc
ERROR:
  Code: Canceled
  Message: Request timed out!
  Details:
  1)	{"@type":"type.googleapis.com/rk.api.v1.ErrorDetail","code":1,"message":"[from-grpc] Request timed out!","status":"Canceled"}
$ curl -X GET localhost:8080/rk/v1/gc
{
    "error":{
        "code":408,
        "status":"Request Timeout",
        "message":"Request timed out!",
        "details":[
            {
                "code":1,
                "status":"Canceled",
                "message":"[from-grpc] Request timed out!"
            }
        ]
    }
}

Cheers

1.3.2.2 - Advanced

RK provide bootstrapper for popular frameworks in order to save time for learning complex initializing process.

Overview

In advanced user guide, we will introduce how to enable bellow functionalities by modifying boot.yaml.

Functionality Description
Locale How to distinguish environment based os OS.Environment
Logging Use user defined logging
TLS Enable TLS support
Config Use viper config
AppInfo Use custom application information
Multiple entries Start multiple GRPC entries
Shutdown hook Register shutdown hook functions
Error type Use standard error type
Override bootstrapper Override bootstrapper
Trace RPC with logs Trace RPC logs based on traceId
Custom routes in grpc-gateway Add custom routes to grpc-gateway
File uploads Support API uploading files to server.

1.3.2.2.1 - Locale

Distinguish entries based on different environment.

Overview

Lets assuming we need to use different config files based on the region like Singapore or Frankfurt.

Since bootstrapper support multiple ConfigEntries in boot.yaml, we need a mechanism to distinguish ConfigEntries.

Concept

How entries selected by bootstrapper?

RK use REALM, REGION, AZ, DOMAIN environment variable to distinguish environments.

Quick start

1.Create config files

config/singapore.yaml

---
region: sinpapore

config/frankfurt.yaml

---
region: frankfurt

config/default.yaml

---
region: default

2.Create boot.yaml

config:
  # Load this config if nothing matched
  - name: my-config
    locale: "*::*::*::*"
    path: config/default.yaml
  # Load this config if REGION=singapore
  - name: my-config
    locale: "*::singapore::*::*"
    path: config/singapore.yaml
  # Load this config if REGION=frankfurt
  - name: my-config
    locale: "*::frankfurt::*::*"
    path: config/frankfurt.yaml
grpc:
  - name: greeter
    port: 8080
    enabled: true

3.With REGION=singapore

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"os"
)

// Application entrance.
func main() {
    // Set REGION=singapore
	os.Setenv("REGION", "singapore")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

singapore

Cheers

4.With REGION=frankfurt

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"os"
)

// Application entrance.
func main() {
    // Set REGION=singapore
	os.Setenv("REGION", "frankfurt")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

frankfurt

Cheers

5.With REGION=not-matched

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"os"
)

// Application entrance.
func main() {
    // Set REGION=singapore
	os.Setenv("REGION", "not-matched")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

default

Cheers

6.With REGION=""

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	fmt.Println(boot.GetConfigEntry("my-config").GetViper().GetString("region"))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

Output

default

Cheers

1.3.2.2.2 - Logging

Customise logging.

Architecture

ZapLoggerEntry

ZapLoggerEntry is used for initializing zap logger.

// ZapLoggerEntry contains bellow fields.
// 1: EntryName: Name of entry.
// 2: EntryType: Type of entry which is ZapLoggerEntryType.
// 3: EntryDescription: Description of ZapLoggerEntry.
// 4: Logger: zap.Logger which was initialized at the beginning.
// 5: LoggerConfig: zap.Logger config which was initialized at the beginning which is not accessible after initialization..
// 6: LumberjackConfig: lumberjack.Logger which was initialized at the beginning.
type ZapLoggerEntry struct {
	EntryName        string             `yaml:"entryName" json:"entryName"`
	EntryType        string             `yaml:"entryType" json:"entryType"`
	EntryDescription string             `yaml:"entryDescription" json:"entryDescription"`
	Logger           *zap.Logger        `yaml:"-" json:"-"`
	LoggerConfig     *zap.Config        `yaml:"zapConfig" json:"zapConfig"`
	LumberjackConfig *lumberjack.Logger `yaml:"lumberjackConfig" json:"lumberjackConfig"`
}

YAML

ZapLoggerEntry follows zap and lumberjack YAML hierarchy.

Please refer to zap and lumberjack site for details.

---
zapLogger:
  - name: zap-logger                      # Required
    description: "Description of entry"   # Optional
    zap:
      level: info                         # Optional, default: info, options: [debug, DEBUG, info, INFO, warn, WARN, dpanic, DPANIC, panic, PANIC, fatal, FATAL]
      development: true                   # Optional, default: true
      disableCaller: false                # Optional, default: false
      disableStacktrace: true             # Optional, default: true
      sampling:                           # Optional, default: empty map
        initial: 0
        thereafter: 0
      encoding: console                   # Optional, default: "console", options: [console, json]
      encoderConfig:
        messageKey: "msg"                 # Optional, default: "msg"
        levelKey: "level"                 # Optional, default: "level"
        timeKey: "ts"                     # Optional, default: "ts"
        nameKey: "logger"                 # Optional, default: "logger"
        callerKey: "caller"               # Optional, default: "caller"
        functionKey: ""                   # Optional, default: ""
        stacktraceKey: "stacktrace"       # Optional, default: "stacktrace"
        lineEnding: "\n"                  # Optional, default: "\n"
        levelEncoder: "capitalColor"      # Optional, default: "capitalColor", options: [capital, capitalColor, color, lowercase]
        timeEncoder: "iso8601"            # Optional, default: "iso8601", options: [rfc3339nano, RFC3339Nano, rfc3339, RFC3339, iso8601, ISO8601, millis, nanos]
        durationEncoder: "string"         # Optional, default: "string", options: [string, nanos, ms]
        callerEncoder: ""                 # Optional, default: ""
        nameEncoder: ""                   # Optional, default: ""
        consoleSeparator: ""              # Optional, default: ""
      outputPaths: [ "stdout" ]           # Optional, default: ["stdout"], stdout would be replaced if specified
      errorOutputPaths: [ "stderr" ]      # Optional, default: ["stderr"], stderr would be replaced if specified
      initialFields:                      # Optional, default: empty map
        key: "value"
    lumberjack:                           # Optional
      filename: "rkapp-event.log"         # Optional, default: It uses <processname>-lumberjack.log in os.TempDir() if empty.
      maxsize: 1024                       # Optional, default: 1024 (MB)
      maxage: 7                           # Optional, default: 7 (days)
      maxbackups: 3                       # Optional, default: 3 (days)
      localtime: true                     # Optional, default: true
      compress: true                      # Optional, default: true

Access

// Access entry
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger")

// Access zap logger
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLogger()

// Access zap logger config
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLoggerConfig()

// Access lumberjack config
rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLumberjackConfig()

EventLoggerEntry

RK bootstrapper treat RPC request as an Event, and record every RPC request into Event type in rk-query.

// EventLoggerEntry contains bellow fields.
// 1: EntryName: Name of entry.
// 2: EntryType: Type of entry which is EventLoggerEntryType.
// 3: EntryDescription: Description of EventLoggerEntry.
// 4: EventFactory: rkquery.EventFactory was initialized at the beginning.
// 5: EventHelper: rkquery.EventHelper was initialized at the beginning.
// 6: LoggerConfig: zap.Config which was initialized at the beginning which is not accessible after initialization.
// 7: LumberjackConfig: lumberjack.Logger which was initialized at the beginning.
type EventLoggerEntry struct {
	EntryName        string                `yaml:"entryName" json:"entryName"`
	EntryType        string                `yaml:"entryType" json:"entryType"`
	EntryDescription string                `yaml:"entryDescription" json:"entryDescription"`
	EventFactory     *rkquery.EventFactory `yaml:"-" json:"-"`
	EventHelper      *rkquery.EventHelper  `yaml:"-" json:"-"`
	LoggerConfig     *zap.Config           `yaml:"zapConfig" json:"zapConfig"`
	LumberjackConfig *lumberjack.Logger    `yaml:"lumberjackConfig" json:"lumberjackConfig"`
}

Fields

Field Description
endTime As name described
startTime As name described
elapsedNano Elapsed time for RPC in nanoseconds
timezone As name described
ids Contains three different ids(eventId, requestId and traceId). If meta interceptor was enabled or event.SetRequestId() was called by user, then requestId would be attached. eventId would be the same as requestId if meta interceptor was enabled. If trace interceptor was enabled, then traceId would be attached.
app Contains appName, appVersion, entryName, entryType.
env Contains arch, az, domain, hostname, localIP, os, realm, region. realm, region, az, domain were retrieved from environment variable named as REALM, REGION, AZ and DOMAIN. “*” means empty environment variable.
payloads Contains RPC related metadata
error Contains errors if occur
counters Set by calling event.SetCounter() by user.
pairs Set by calling event.AddPair() by user.
timing Set by calling event.StartTimer() and event.EndTimer() by user.
remoteAddr As name described
operation RPC method name
resCode Response code of RPC
eventStatus Ended or InProgress

Example

------------------------------------------------------------------------
endTime=2021-07-10T03:00:12.153392+08:00
startTime=2021-07-10T03:00:12.153261+08:00
elapsedNano=130727
timezone=CST
ids={"eventId":"c9a1f6b0-b9ec-4e46-9ed4-238c3c6759ab","requestId":"c9a1f6b0-b9ec-4e46-9ed4-238c3c6759ab","traceId":"5441ff5c3855f03b573e95d81139123b"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Greeter","grpcService":"api.v1.Greeter","grpcType":"unaryServer","gwMethod":"GET","gwPath":"/v1/greeter","gwScheme":"http","gwUserAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost:59631
operation=/api.v1.Greeter/Greeter
resCode=OK
eventStatus=Ended
EOE

YAML

EventLoggerEntry needs application name while creating event log. The application name retrieved from go.mod file. If there is go.mod file missing, then default one would be used.

---
eventLogger:
  - name: event-logger                 # Required
    description: "This is description" # Optional
    encoding: console                  # Optional, default: console, options: console and json
    outputPaths: ["stdout"]            # Optional
    lumberjack:                        # Optional
      filename: "rkapp-event.log"      # Optional, default: It uses <processname>-lumberjack.log in os.TempDir() if empty.
      maxsize: 1024                    # Optional, default: 1024 (MB)
      maxage: 7                        # Optional, default: 7 (days)
      maxbackups: 3                    # Optional, default: 3 (days)
      localtime: true                  # Optional, default: true
      compress: true                   # Optional, default: true

Access

// Access entry
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger")

// Access event factory
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventFactory()

// Access event helper
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventHelper()

// Access lumberjack config
rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetLumberjackConfig()

Quick start

1.Custom ZapLoggerEntry

---
zapLogger:
  - name: zap-logger                      # Required
    description: "Description of entry"   # Optional
    zap:
      encoding: json
grpc:
  - name: greeter
    port: 8080
    enabled: true
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get custom ZapLoggerEntry
	rkentry.GlobalAppCtx.GetZapLoggerEntry("zap-logger").GetLogger().Info("Custom zap logger")

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
{"level":"INFO","ts":"2021-07-06T05:06:16.252+0800","msg":"Custom zap logger"}

Cheers

2.Reference ZapLoggerEntry

Bootstrapper will use this logger by default including interceptor.

---
zapLogger:
  - name: zap-logger
grpc:
  - name: greeter
    port: 8080
    enabled: true
    logger:
     zapLogger:
        ref: zap-logger

Cheers

3.Custom EventLoggerEntry

---
eventLogger:
  - name: event-logger
grpc:
  - name: greeter
    port: 8080
    enabled: true
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get custom EventLoggerEntry
	helper := rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventHelper()
	event := helper.Start("demo")
	event.AddPair("key", "value")
	helper.Finish(event)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
------------------------------------------------------------------------
endTime=2021-07-10T03:02:58.540804+08:00
startTime=2021-07-10T03:02:58.540803+08:00
elapsedNano=568
timezone=CST
ids={"eventId":"55f9dd94-9bb8-42a0-b3ed-9164d5f00a64"}
app={"appName":"rk-demo","appVersion":"master-f414049","entryName":"","entryType":""}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={}
error={}
counters={}
pairs={"key":"value"}
timing={}
remoteAddr=localhost
operation=demo
resCode=OK
eventStatus=Ended
EOE

Cheers

4.Reference EventLoggerEntry

Bootstrapper will use this logger by default including interceptor.

---
eventLogger:
  - name: event-logger
grpc:
  - name: greeter
    port: 8080
    enabled: true
    logger:
      eventLogger:
        ref: event-logger

Cheers

5.Set env at EventLoggerEntry

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-entry/entry"
)

// Application entrance.
func main() {
    // Set env
	os.Setenv("REALM", "my-realm")
	os.Setenv("REGION", "my-region")
	os.Setenv("AZ", "my-az")
	os.Setenv("DOMAIN", "my-domain")

	// Create a new boot instance.
	boot := rkboot.NewBoot()
    
    // Get custom EventLoggerEntry
	helper := rkentry.GlobalAppCtx.GetEventLoggerEntry("event-logger").GetEventHelper()
	event := helper.Start("demo")
	helper.Finish(event)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}
------------------------------------------------------------------------
...
env={"arch":"amd64","az":"my-az","domain":"my-domain","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"my-realm","region":"my-region"}
...

Cheers

6.Access ZapLoggerEntry & EventLoggerEntry

User can access by calling rkentry.GlobalAppCtx.GetEventLoggerEntry() & rkentry.GlobalAppCtx.GetZapLoggerEntry().

1.3.2.2.3 - TLS/SSL

Enable TLS/SSL for the server.

Overview

In order to enable TLS/SSL on the server, we need server certificate and server key at least.

Bootstrapper will read CertEntry section in boot.yaml file and load certificates into memory.

We support four types of CertEntry retrievers. We will extend retriever future.

  • LocalFs
  • RemoteFs
  • Consul
  • Etcd

Architecture

Locale

CertEntry support concept of locale.

Generate Self-Signed Certificate

There is a convenient way to generate certificates with cfssl

1.Download cfssl & cfssljson

Install cfssl with rk cli

$ go get -u github.com/rookie-ninja/rk/cmd/rk
$ rk install cfssl
$ rk install cfssljson

2.Generate CA

$ cfssl print-defaults config > ca-config.json
$ cfssl print-defaults csr > ca-csr.json

Adjust ca-config.json and ca-csr.json as needed.

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

3.Generate server certificates

server.csr, server.pem and server-key.pem would be created.

$ cfssl gencert -config ca-config.json -ca ca.pem -ca-key ca-key.pem -profile www csr.json | cfssljson -bare server

Quick start

1.localFs

Name Description Default
cert.localFs.name Name of localFs retriever ""
cert.localFs.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.localFs.serverCertPath Path of server cert in local file system. ""
cert.localFs.serverKeyPath Path of server key in local file system. ""
cert.localFs.clientCertPath Path of client cert in local file system. ""
cert.localFs.clientCertPath Path of client key in local file system. ""

Enable TLS on grpc and gateway

---
cert:
  - name: "local-cert"                     # Required
    description: "Description of entry"    # Optional
    provider: "localFs"                    # Required, etcd, consul, localFs, remoteFs are supported options
    locale: "*::*::*::*"                   # Required, default: ""
    serverCertPath: "cert/server.pem"      # Optional, default: "", path of certificate on local FS
    serverKeyPath: "cert/server-key.pem"   # Optional, default: "", path of certificate on local FS
grpc:
  - name: greeter
    port: 8080
    enabled: true
    enableReflection: true
    cert:
      ref: "local-cert"                    # Enable grpc TLS
    commonService: 
      enabled: true
# try call http service
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}
# try call grpc service
grpcurl -insecure localhost:8080 rk.api.v1.RkCommonService.Healthy
{
    "healthy": true
}

Cheers

2.remoteFs

Name Description Default
cert.remoteFs.name Name of remoteFileStore retriever ""
cert.remoteFs.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.remoteFs.endpoint Endpoint of remoteFileStore server, http://x.x.x.x or x.x.x.x both acceptable. N/A
cert.remoteFs.basicAuth Basic auth for remoteFileStore server, like user:pass. ""
cert.remoteFs.serverCertPath Path of server cert in remoteFs server. ""
cert.remoteFs.serverKeyPath Path of server key in remoteFs server. ""
cert.remoteFs.clientCertPath Path of client cert in remoteFs server. ""
cert.remoteFs.clientCertPath Path of client key in remoteFs server. ""
---
cert:
  - name: "remote-cert"                    # Required
    description: "Description of entry"    # Optional
    provider: "remoteFs"                   # Required, etcd, consul, localFs, remoteFs are supported options
    endpoint: "localhost:8081"             # Required, both http://x.x.x.x or x.x.x.x are acceptable
    locale: "*::*::*::*"                   # Required, default: ""
    serverCertPath: "cert/server.pem"      # Optional, default: "", path of certificate on local FS
    serverKeyPath: "cert/server-key.pem"   # Optional, default: "", path of certificate on local FS
grpc:
  - name: greeter
    port: 8080
    enabled: true
    cert:
      ref: "remote-cert"                   # Enable grpc TLS
    commonService: 
      enabled: true
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

3.consul

Name Description Default
cert.consul.name Name of consul retriever ""
cert.consul.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.consul.endpoint Endpoint of Consul server, http://x.x.x.x or x.x.x.x both acceptable. N/A
cert.consul.datacenter consul datacenter. ""
cert.consul.token Token for access consul. ""
cert.consul.basicAuth Basic auth for consul server, like user:pass. ""
cert.consul.serverCertPath Path of server cert in Consul server. ""
cert.consul.serverKeyPath Path of server key in Consul server. ""
cert.consul.clientCertPath Path of client cert in Consul server. ""
cert.consul.clientCertPath Path of client key in Consul server. ""
---
cert:
  - name: "consul-cert"                    # Required
    provider: "consul"                     # Required, etcd, consul, localFS, remoteFs are supported options
    description: "Description of entry"    # Optional
    locale: "*::*::*::*"                   # Required, default: ""
    endpoint: "localhost:8500"             # Required, http://x.x.x.x or x.x.x.x both acceptable.
    datacenter: "dc1"                      # Optional, default: "", consul datacenter
    serverCertPath: "server.pem"           # Optional, default: "", key of value in consul
    serverKeyPath: "server-key.pem"        # Optional, default: "", key of value in consul
grpc:
  - name: greeter
    port: 8080
    enabled: true
    cert:
      ref: "consul-cert"                   # Enable grpc TLS
    commonService: 
      enabled: true
$ curl -X GET --insecure https://localhost:8080/rk/v1/healthy
{"healthy":true}

Cheers

4.etcd

Name Description Default
cert.etcd.name Name of etcd retriever ""
cert.etcd.locale Represent environment of current process follows schema of <realm>::<region>::<az>::<domain> ""
cert.etcd.endpoint Endpoint of etcd server, http://x.x.x.x or x.x.x.x both acceptable. N/A
cert.etcd.basicAuth Basic auth for etcd server, like user:pass. ""
cert.etcd.serverCertPath Path of server cert in etcd server. ""
cert.etcd.serverKeyPath Path of server key in etcd server. ""
cert.etcd.clientCertPath Path of client cert in etcd server. ""
cert.etcd.clientCertPath Path of client key in etcd server. ""
---
cert:
  - name: "etcd-cert"                      # Required
    description: "Description of entry"    # Optional
    provider: "etcd"                       # Required, etcd, consul, localFs, remoteFs are supported options
    locale: "*::*::*::*"                   # Required, default: ""
    endpoint: "localhost:2379"             # Required, http://x.x.x.x or x.x.x.x both acceptable.
    serverCertPath: "server.pem"           # Optional, default: "", key of value in etcd
    serverKeyPath: "server-key.pem"        # Optional, default: "", key of value in etcd
grpc:
  - name: greeter
    port: 8080
    enabled