Commit b68a9c7e authored by Muhammad Hamza's avatar Muhammad Hamza
Browse files

Add README for proto3 generation and usage

parent 9b42281a
Loading
Loading
Loading
Loading
Loading

proto3-gen.md

0 → 100644
+171 −0
Original line number Original line Diff line number Diff line
# Protobuf Schema Generation

[OpenAPI Generator](https://openapi-generator.tech) is used to generate protobuf schema (`.proto3`) files from OpenAPI specifications of MEC015 Traffic Management APIs.

>**NOTE:** At the time of writing, the tool does not support OAS 3.1 version and we have to first convert the [Traffic Management APIs](./BwManagementApi.yaml and ./TrafficSteeringApi.yaml) to OAS 3.0 for generating protobuf schema.

1. Convert OAS for [Traffic Management APIs](./BwManagementApi.yaml and ./TrafficSteeringApi.yaml) from 3.1 to 3.0​

   - Change the value of `openapi` field from 3.1.0 to 3.0.0​

   - Use this [VS code extension](https://marketplace.visualstudio.com/items?itemName=42Crunch.vscode-openapi) to see the errors in the downgraded YAML (v3.0)​

   - Manually fix the errors​
     - mostly related to `examples` <--> `example` interchange​
     - or some 3.1 fields that are not supported in 3.0​ (comment them out)

2. Generate proto files
   - Install the `openapi-generator-cli.jar` using the installation procedure mentioned [here](https://openapi-generator.tech/docs/installation#jar).
   - Generate the proto files using the following commands: 
        ```sh
        $ java -jar openapi-generator-cli.jar generate -i BwManagementApi.yaml -g protobuf-schema -o proto3/ --package-name mec015

        $ java -jar openapi-generator-cli.jar generate -i TrafficSteeringApi.yaml -g protobuf-schema -o proto3/ --package-name mec015
        ```

3. Carefully inspect the generated `.proto` files for any inconsistencies. Some of the things to look out for:
    - Proto3 generated files for enumerations, structures containing allOf, oneOf, anyOf etc. may need to be touched manually
    - Check that all the nested models are being _imported_ correctly in their parent models
    - Remove redundant proto files


4. Validate protobuf schema by generating code from proto3 descriptions in different languages. See [this section](#code-generation-from-proto3) for more details.

# Code Generation from proto3

Below are some code generation examples for Python, Go and Ruby. For other languages, please refer to https://grpc.io/docs/quickstart/.

### Python

1. Install the grpcio-tools package
    ```sh
    $ pip install grpcio-tools
    ```

2. Create a directory for generated Python stubs
    ```sh
    $ mkdir python-stubs 
    ```

3. Run the following commands from the root of the directory containing this README that you are reading.

   - Models:
  
        ```sh
        $ python -m grpc_tools.protoc -I./BwManagementProto3/proto3 --python_out=./python-stubs ./BwManagementProto3/proto3/models/*

        $ python -m grpc_tools.protoc -I./TrafficSteeringProto3/proto3 --python_out=./python-stubs ./TrafficSteeringProto3/proto3/models/*
        ```
    
        The above commands will generate .py files for all the data models in the ./models directory

   - Services:

        ```sh
        $ python -m grpc_tools.protoc -I./BwManagementProto3/proto3 --python_out=./python-stubs --grpc_python_out=./python-stubs ./BwManagementProto3/proto3/services/bwm_service.proto

        $ python -m grpc_tools.protoc -I./TrafficSteeringProto3/proto3 --python_out=./python-stubs --grpc_python_out=./python-stubs ./TrafficSteeringProto3/proto3/services/mts_service.proto
        ```

        The above commands will generate two files for the BWM and MTS service:
        - _bwm_service_pb2.py_: containing the python data models used in the BWM service file
        - _bwm_service_pb2_grpc.py_: containing all the classes and functions needed for the supported HTTP methods in the BWM API
        And two files for the MTS service:
        - _mts_service_pb2.py_: containing the python data models used in the MTS service file
        - _mts_service_pb2_grpc.py_: containing all the classes and functions needed for the supported HTTP methods in the MTS API

### Go

1. Install protocol buffer compiler
    ```sh
    $ apt install -y protobuf-compiler
    ```
2. Install Go plugins for `protoc`
    ```sh
    $ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.1
    ```
    ```sh
    $ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
    ```
3. Update `PATH` so `protoc` can find the plugins
    ```sh
    $ export PATH="$PATH:$(go env GOPATH)/bin"
    ```
4. Define a go package by appending `option go_package = "./mec015.services.bwmservice";` or `option go_package = "./mec015.services.mtsservice";` in .proto files like this:

    ```Go
    ...

    syntax = "proto3";

    package mec015.services.bwmservice;

    option go_package = "./mec015.services.bwmservice";

    import public "models/<xyz>.proto";

    ...

    ...

    syntax = "proto3";

    package mec015.services.mtsservice;

    option go_package = "./mec015.services.mtsservice";

    import public "models/<xyz>.proto";

    ...
    ```
5. Generate Go code for models and services
    ```sh 
    $ mkdir go-stubs

    $ protoc --go_out=./go-stubs ./BwManagementProto3/proto3/models/* -I./BwManagementProto3/proto3
    
    $ protoc --go_out=./go-stubs ./BwManagementProto3/proto3/services/* --go-grpc_out=go-stubs -I./BwManagementProto3/proto3

    ...

    $ protoc --go_out=./go-stubs ./TrafficSteeringProto3/proto3/models/* -I./TrafficSteeringProto3/proto3
    
    $ protoc --go_out=./go-stubs ./TrafficSteeringProto3/proto3/services/* --go-grpc_out=go-stubs -I./TrafficSteeringProto3/proto3
    ```


    > The generated `<data_model>.pb.go` files will contain all the protocol buffer code to populate, serialize, and retrieve request and response message types defined in the `models` folder.

    > The `bwm_service_grpc.pb.go` will contain the stubs for the methods defined in the `bwm_service.proto` file. And The `mts_service_grpc.pb.go` will contain the stubs for the methods defined in the `mts_service.proto` file.

### Ruby

1. Install gRPC Ruby Plugin and required tools
   ```sh
   $ gem install grpc
   $ sudo apt install ruby-grpc-tools
   ```

2. Generate code
    ```sh
    $ mkdir ruby-stubs
    ```
    
    Run the following commands to create Ruby modules for all the data models defined in the proto files.

    ```sh
    $ grpc_tools_ruby_protoc -I./BwManagementProto3/proto3 --ruby_out=ruby-stubs ./BwManagementProto3/proto3/models/*

    ...

    $ grpc_tools_ruby_protoc -I./TrafficSteeringProto3/proto3 --ruby_out=ruby-stubs ./TrafficSteeringProto3/proto3/models/*
    ```
    Run the following commands to generate `bwm_service_pb.rb`, `bwm_service_services_pb.rb` and `mts_service_pb.rb`, `mts_service_services_pb.rb` files, containing stub and service classes for the endpoints and methods defined in MEC015 Traffic Management APIs.

    ```sh
    $ grpc_tools_ruby_protoc -I./BwManagementProto3/proto3 --ruby_out=ruby-stubs --grpc_out=ruby-stubs ./BwManagementProto3/proto3/services/*
    
    ...

    $ grpc_tools_ruby_protoc -I./TrafficSteeringProto3/proto3 --ruby_out=ruby-stubs --grpc_out=ruby-stubs ./TrafficSteeringProto3/proto3/services/*
    ```
 No newline at end of file