1. Introduction

Adapters are proxy-like services that are used by SGNL to communicate to their respective Systems of Record (SoR). The figure above depicts where adapters are placed in the ingestion flow from the SoRs to the SGNL platform. Adapters communicate with SoR APIs in a protocol the SoR supports and retrieves data about resources and their attributes. The received response from the SoR is transformed to a SGNL format that allows it to ingest data from the SoR into the SGNL graph.

There are two types of adapters: Standard and Custom. This help page details the steps required to write your own custom adapters. In most cases, SGNL will already have an adapter for most SoRs currently available. Custom adapters are built by users to support their proprietary/custom Systems of Record. Users are responsible for developing, managing and maintaining these adapters. Each SGNL client’s adapters are unique to the client and are not visible or usable to other SGNL clients.

2. About Adapters in SGNL

As mentioned above, adapters are proxy-like services which serve two main functions:

1. Implement functionality to make requests to a specific System of Record. The SoR may use any protocol. SGNL supports ReST today - SGNL’s Adapter Framework is open-source and may be easily extended to support SOAP, GraphQL, gRPC or any proprietary protocol.
2. Implement functionality to transform the response received from the SoR to a form that can be consumed by SGNL.

Adapters are stateless, by default. However, it’s up to the user implementing the adapter to make it stateful if required.

3. Writing Custom Adapters in SGNL

SGNL has made it very easy to write custom adapters. SGNL provides an adapter template written in Golang that abstracts out all of the complexity of transforming responses from the SoR APIs.

Prerequisites:

1. Basic knowledge of Golang, Docker
2. Knowledge of Git and Git tools set up on your development machine
3. Go and Git installed on your development machine. Golang installation instructions can be found here.
4. IDE on which you can develop Golang code. We recommend Visual Studio Code with Golang plugin installed

We will walk you through the entire process of writing a custom adapter to implement the following use-case:

“We want to restrict access to Job Applications based on the department an employee belongs to - for instance, only employees belonging to Human Resources can access Job Applicant information.”

At the end of this exercise, you should have created a BambooHR SoR in SGNL and successfully ingested the Employee directory and Applicants. You can then replicate these steps to implement an adapter for an SoR used in your organization.

The main repository you will be working with is the adapter-template repository in Github.

The following steps detail the process of writing a custom adapter:

Determine the System of Record and Entities to ingest

Before we start implementing the adapter, it is important to determine which SoRs need to be ingested to implement policy. Based on the use-case, we will need to ingest Employee data from a Employee Directory and Applications data from an Applicant Tracking System. Assuming the company uses BambooHR as the System of Record for both Employee data and Applications, we can use BambooHR’s Employee Directory API to ingest employee information and the Applications API to ingest applications information. In the context of SGNL, Employees and Applications are Entities and their properties are their Attributes.

Exercise the System of Record API

Exercise the API to understand the request and response structure and authentication requirements. Sometimes services will provide a Postman collection which makes it easier to explore the API. BambooHR does not provide a Postman collection, but we have created one that can be downloaded here for the “Get Employee Directory” and “Get Applications” APIs to retrieve data on all employees in a BambooHR instance.

Generate the API key for your BambooHR instance.

Use Postman to send a GET request to the Employee Directory API and study the response. Note the Attributes for the Employee Entity and narrow down to the subset of attributes that would be useful to ingest into the SGNL graph and can be used in the SGNL policy.

{
  "fields": [
    {
      "id": "displayName",
      "type": "text",
      "name": "Display name"
    },
    {
      "id": "firstName",
      "type": "text",
      "name": "First name"
    },
    {
      "id": "lastName",
      "type": "text",
      "name": "Last name"
    },
    {
      "id": "preferredName",
      "type": "text",
      "name": "Preferred name"
    },
    {
      "id": "jobTitle",
      "type": "list",
      "name": "Job title"
    },
    {
      "id": "workPhone",
      "type": "text",
      "name": "Work Phone"
    },
    {
      "id": "mobilePhone",
      "type": "text",
      "name": "Mobile Phone"
    },
    {
      "id": "workEmail",
      "type": "email",
      "name": "Work Email"
    },
    {
      "id": "department",
      "type": "list",
      "name": "Department"
    },
    {
      "id": "location",
      "type": "list",
      "name": "Location"
    },
    {
      "id": "division",
      "type": "list",
      "name": "Division"
    },
    {
      "id": "linkedIn",
      "type": "text",
      "name": "LinkedIn URL"
    },
    {
      "id": "instagram",
      "type": "text",
      "name": "Instagram URL"
    },
    {
      "id": "pronouns",
      "type": "list",
      "name": "Pronouns"
    },
    {
      "id": "workPhoneExtension",
      "type": "text",
      "name": "Work Ext."
    },
    {
      "id": "supervisor",
      "type": "employee",
      "name": "Manager"
    },
    {
      "id": "photoUploaded",
      "type": "bool",
      "name": "Employee photo exists"
    },
    {
      "id": "photoUrl",
      "type": "url",
      "name": "Employee photo url"
    },
    {
      "type": "bool",
      "name": "",
      "id": "canUploadPhoto"
    }
  ],
  "employees": [
    {
      "id": "4",
      "displayName": "Charlotte Abbott",
      "firstName": "Charlotte",
      "lastName": "Abbott",
      "preferredName": null,
      "jobTitle": "Sr. HR Administrator",
      "workPhone": "801-724-6600",
      "mobilePhone": "801-724-6600",
      "workEmail": "cabbott@efficientoffice.com",
      "department": "Human Resources",
      "location": "Lindon, Utah",
      "division": "North America",
      "linkedIn": "www.linkedin.com",
      "instagram": "@instagram",
      "pronouns": null,
      "workPhoneExtension": "1272",
      "supervisor": "Jennifer Caldwell",
      "photoUploaded": true,
      "photoUrl": "https://images7.bamboohr.com/585431/photos/4-6-4.jpg?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9pbWFnZXM3LmJhbWJvb2hyLmNvbS81ODU0MzEvKiIsIkNvbmRpdGlvbiI6eyJEYXRlR3JlYXRlclRoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwNDIyMzE4M30sIkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNzA2ODE1MTkzfX19XX0_&Signature=kcySXIqawaRqZuP6AtSNaotVZVuMfNhFxsC~v~vtvnbn7wFs24iGeeJpNRlUQlprDA097NO8WxzfRmCecD3EqEXqcETMW05OUQ0gRuMRPiokxSSVE8JZA3-nEDr7RYEMT3vTdpLbEtGBqmgP9TwoYs7Xmq6pTKsEMpS7SFg48GStEBnFeEuD7KnPfQv7TDoOGkiTBZo67m-NeOxsGkg5ucviYhPiBF51KStE4m~cAy~6KCvntLOGlEsj8brTxdQDYPY~ScXm6Wuo8QC~Ya6S0ngGlvyyMbt4v-3hqInnUqIUXZ2E162Q9JNXoD9azjfTMRnfvRBjOCNPc6DiA7D8Yg__&Key-Pair-Id=APKAIZ7QQNDH4DJY7K4Q",
      "canUploadPhoto": 1
    },
 …
 …
]

Based on the Get Employee API response above, the below table that outlines what the corresponding SoR Entity Attribute Fields for Employees would be in SGNL:

For more information on ExternalId, Type, UniqueId, Indexed values, please refer to SGNL’s SoR Entity Attributes Fields Help Documentation.

{
  "paginationComplete": true,
  "applications": [
    {
      "id": 48,
      "appliedDate": "2024-02-29T17:08:59+00:00",
      "status": {
        "id": 1,
        "label": "New"
      },
      "rating": null,
      "applicant": {
        "id": 110,
        "firstName": "Janet",
        "lastName": "Lewis",
        "avatar": "https://resources.bamboohr.com/employees/photos/initials.php?initials=JL",
        "email": "jlewis@efficientoffice.com",
        "source": "other"
      },
      "job": {
        "id": 19,
        "title": {
          "id": null,
          "label": "General Application"
        }
      }
    },
    {
      "id": 56,
      "appliedDate": "2024-02-21T22:46:01+00:00",
      "status": {
        "id": 1,
        "label": "New"
      },
      "rating": null,
      "applicant": {
        "id": 114,
        "firstName": "James",
        "lastName": "Garcia",
        "avatar": "https://resources.bamboohr.com/employees/photos/initials.php?initials=JG",
        "email": "jamesgarcia@efficientoffice.com",
        "source": "ZipRecruiter"
      },
      "job": {
        "id": 21,
        "title": {
          "id": null,
          "label": "Marketing Manager"
        }
      }
    }
    ...
    ...
  ],
  "nextPageUrl": null
}

Based on the above, the attributes table for Applications would be:

Once you have this information, it’s time to start coding the adapter.

4. Implementing the Custom BambooHR Adapter

Adapters Call Flow

The following diagram depicts a typical SGNL->Adapter->SoR call flow using BambooHR and Employees Entity as an example.

Adapter Template Code Call Flow

The call flow through the adapter is depicted in the diagram above and works as follows:

1. SGNL sends a gRPC message requesting ingestion for a particular entity is sent to the adapter. Only one entity is requested in each gRPC message.
2. The gRPC message is validated in config.go and validation.go. If the message passes validation, it’s then sent to adapter.go
3. In adapter.go, the gRPC message gets parsed into a Request data structure, which contains all the information needed to construct a request that gets sent to the System of Record
4. The Request data structure is then sent to datasource.go, which uses this information to construct and send the HTTPS request to the System of Record.
5. The response from the SoR is then parsed in datasource.go which converts the HTTPS response to a Response data structure.
6. The Response data structure is sent to adapter.go
7. adapter.go returns it to our adapter-framework, which converts the Response to a gRPC message in a format SGNL understands.

Coding the Adapter

1. Clone the adapter-template repository from Github and update references to adapter-template and sgnl-ai

git clone https://github.com/SGNL-ai/adapter-template.git bamboohr-sgnl-adapter

2. Figure out code to update

   1. Change your directory to the cloned repository

      cd bamboohr-sgnl-adapter
      

   2. In the directory, find all occurrences of the string ‘sgnl-ai/adapter-template’ with

      egrep --include \*.go -irs "sgnl-ai/adapter-template" .
      

   3. Replace all occurrences of “sgnl-ai/adapter-template” with “<your-org>/bamboohr-sgnl-adapter”
      Note: <your-org> is your company’s GitHub’s organization where the repository will reside.

      Example: The file cmd/adapter/main.go imports github.com/sgnl-ai/adapter-template/pkg/adapter. If <your-org> is acme, replace github.com/sgnl-ai/adapter-template/pkg/adapter with github.com/acme/adapter-template/pkg/adapter

   4. Create a list of all code locations that will need to be updated to support your SoR: (Search for SCAFFOLDING in the bamboohr-sgnl-adapter directory)

      egrep --include \*.go -irs "SCAFFOLDING" .
      

This command will return all files that contain SCAFFOLDING pointers that you need to update as noted in this document. The SCAFFOLDING pointers are also listed below:

We will update all underlying code near SCAFFOLDING comments to implement the BambooHR adapter. For each of the following steps, you can copy and paste the code directly for the corresponding SCAFFOLDING in the adapter-template source file. You can also check out the code directly in the bamboohr-sgnl-adapter branch in Github.

3. Update cmd/adapter/main.go

   SCAFFOLDING #1

   // SCAFFOLDING #1 - cmd/adapter/main.go: Pass options to configure TLS, connection parameters.
   s := grpc.NewServer()
   

   grpc.NewServer() creates a new gRPC server that listens for connections from SGNL. You don’t need to update any code here for BambooHR.

   SCAFFOLDING #2

   // SCAFFOLDING #2 - cmd/adapter/main.go: Update Adapter type.
   // The Adapter type below must be unique across all registered Adapters and match the Adapter type configured on the Adapter object via the SGNL Config API.If you need to run multiple adapters on the same gRPC server, they can be registered here.
   err = server.RegisterAdapter(adapterServer, "BambooHR-1.0.0", adapter.NewAdapter(adapter.NewClient(*Timeout)))
   

   The Adapter type has been updated from Test-1.0.0 placeholder to BambooHR-1.0.0

4. Update pkg/adapter/config.go

   SCAFFOLDING #3

   Adapter config allows us to define configuration data in JSON format that’s sent to the adapter. This is done at SoR configuration time. SGNL sends the JSON in every request to the adapter. The adapter can then use the config in any way it chooses. Examples of configuration are SoR API version, headers that must be passed, connection timeout parameters, etc.

   // SCAFFOLDING #3 - pkg/adapter/config.go - pass Adapter config fields.
   // Every field MUST have a `json` tag.
   APIVersion   string `json:"apiVersion,omitempty"`
   AcceptHeader string `json:"acceptHeader,omitempty"`
   

   SCAFFOLDING #4

   BambooHR returns responses in XML format, by default. Therefore, we will add the AcceptHeader as one of the config fields that must be passed to the BambooHR adapter.

   // ValidateConfig validates that a Config received in a GetPage call is valid.
   func (c *Config) Validate(_ context.Context) error {
   // SCAFFOLDING #4 - pkg/adapter/config.go: Validate fields passed in Adapter config.
   // Update the checks below to validate the fields in Config.
   switch {
       case c == nil:
           return fmt.Errorf("request contains no config")
       case c.APIVersion == "":
           return fmt.Errorf("apiVersion is not set")
       case c.AcceptHeader != "application/json":
           return fmt.Errorf("acceptHeader is invalid and not supported by this adapter: %s", c.AcceptHeader)
       default:
           return nil
       }
   }
   

5. Update pkg/adapter/client.go

   SCAFFOLDING #5

   We need to use the Accept header specified in the config passed to the BambooHR adapter. Add the Config field in SCAFFOLDING #5

   // SCAFFOLDING #5 - pkg/adapter/client.go: Add/Remove/Update any fields to model the request for the SoR API.
   
   // Request is a request to the datasource.
   type Request struct {
   // BaseURL is the Base URL of the datasource to query.
   BaseURL string
   
   // Username is the username to use to authenticate with the datasource.
   Username string
   
   // Password is the password to use to authenticate with the datasource.
   Password string
   
   // PageSize is the maximum number of objects to return from the entity.
   PageSize int64
   
   // EntityExternalID is the external ID of the entity.
   // The external ID should match the API's resource name.
   EntityExternalID string
   
   // Cursor identifies the first object of the page to return, as returned by
   // the last request for the entity.
   // Optional. If not set, return the first page for this entity.
   Cursor string
   
   // Config that contains the header information
   Config *Config
   }
   

   SCAFFOLDING #6

   For BambooHR, no fields need to be added/modified for the response because the adapter-template covers the required fields in both the request and response data received from the BambooHR API.

   // SCAFFOLDING #6 - pkg/adapter/client.go: Add/Remove/Update any fields to model the response from the SoR API.
   

6. Update pkg/adapter/validation.go

   SCAFFOLDING #7

   MaxPageSize does not need to be updated. The two entities we are interested in do not take a pageSize parameter. The adapter code, by default, does not make use of MaxPageSize. However, if your SoR does support pagination, you can pass this as one of the parameters. Refer to your SoR API documentation for information on pagination.

      const (
      // MaxPageSize is the maximum page size allowed in a GetPage request.
      // SCAFFOLDING #7 - pkg/adapter/validation.go: Update this limit to match the limit of the SoR.
      MaxPageSize = 100
   )
   

   SCAFFOLDING #8

   BambooHR supports Basic authentication. Since the validation code here already checks for Basic authentication, it does not need to be updated. SGNL also supports OAuth2 Client Credentials and Bearer authentication methods. If your SoR supports OAuth2 Client Credentials

      // SCAFFOLDING #8 - pkg/adapter/validation.go: Modify this validation to match the authn mechanism(s) supported by the SoR.
      if request.Auth == nil || request.Auth.Basic == nil {
          return &framework.Error{
              Message: "Provided datasource auth is missing required basic credentials.",
              Code:    api_adapter_v1.ErrorCode_ERROR_CODE_INVALID_DATASOURCE_CONFIG,
          }
      }
   

   SCAFFOLDING #9

   BambooHR’s Employee or Applications entity does not need child entities. This validation check does not need to be modified.

      // Validate that no child entities are requested.
      //
      // SCAFFOLDING #9 - pkg/adapter/validation.go: Modify this validation if the entity contains child entities.
      if len(request.Entity.ChildEntities) > 0 {
          return &framework.Error{
              Message: "Requested entity does not support child entities.",
              Code:    api_adapter_v1.ErrorCode_ERROR_CODE_INVALID_ENTITY_CONFIG,
          }
      }
   

   SCAFFOLDING #10

   BambooHR does not support ordered responses. This code does not need to be modified.

      // SCAFFOLDING #10 - pkg/adapter/validation.go: Check for Ordered responses.
      // If the datasource doesn't support sorting results by unique ID
      // attribute for the requested entity, check instead that Ordered is set to
      // false.
      if request.Ordered {
          return &framework.Error{
              Message: "Ordered must be set to false.",
              Code:    api_adapter_v1.ErrorCode_ERROR_CODE_INVALID_ENTITY_CONFIG,
          }
      }
   

7. Update pkg/adapter/datasource.go

   SCAFFOLDING #11

   Add ‘directory’ and ‘applications’ as the valid entity types for Employee and Applications entities.

   const (
      // SCAFFOLDING #11 - pkg/adapter/datasource.go: Update the set of valid entity types this adapter supports.
      // Using Directory API https://documentation.bamboohr.com/reference/get-employees-directory-1
      Directory    string = "directory"
      Applications string = "applications"
   )
   

   SCAFFOLDING #12

   SCAFFOLDING #12 does not need to be updated. Entity can be updated to hold any entity specific information. This could be information about the attribute that is a unique identifier (uniqueIDAttrExternalID), or endpoint information if the SoR’s URL Path structures are inconsistent across different entities, or any other information required to ingest the entity.

   // Entity contains entity specific information, such as the entity's unique ID attribute and the endpoint to query that entity.
   type Entity struct {
      // SCAFFOLDING #12 - pkg/adapter/datasource.go: Update Entity fields used to store entity specific information
      // Add or remove fields as needed. This should be used to store entity specific information
      // such as the entity's unique ID attribute name and the endpoint to query that entity.
   
      // uniqueIDAttrExternalID is the external ID of the entity's uniqueId attribute.
      uniqueIDAttrExternalID string
   }
   

   SCAFFOLDING #13

   SCAFFOLDING #13 does not need to be updated. It is used to unmarshal a response. For instance, certain APIs might have extra information in fields returned in the response that might be useful to process. Such fields can be added here.

   SCAFFOLDING #14

   Rename Objects to Employees and objects to employees in SCAFFOLDING #14.

   type DatasourceResponse struct {
      // SCAFFOLDING #13  - pkg/adapter/datasource.go: Add or remove fields in the response as necessary. This is used to unmarshal the response from the SoR.
   
      // SCAFFOLDING #14 - pkg/adapter/datasource.go: Update `objects` with field name in the SoR response that contains the list of objects.
      Employees    []map[string]any `json:"employees,omitempty"`
      Applications []map[string]any `json:"applications,omitempty"`
   }
   

   This is because the Employee API response has the following structure:

   {
        "employees": [
       {
         "id": "1",
         "displayName": "John Doe",
         …
       },
       {
         "id": "4",
         "displayName": "Charlotte Abbott",
         …
       }
     ]
   }
   

   All objects returned in the employees list will be unmarshalled into the Employees list.

   Similarly, the Applications API response has all records returned in an applications list. Those will be unmarshalled into the Applications list.

   SCAFFOLDING #15

   Update uniqueIdAttrExternalId to the name of the attribute that is marked as unique in the Attributes table for each entity. For both Employees and Applications, the name of the unique attribute is “id”

   var (
      // SCAFFOLDING #15 - pkg/adapter/datasource.go: Update the set of valid entity types supported by this adapter. Used for validation.
   
      // ValidEntityExternalIDs is a map of valid external IDs of entities that can be queried.
      // The map value is the Entity struct which contains the unique ID attribute.
      ValidEntityExternalIDs = map[string]Entity{
          Directory: {
              uniqueIDAttrExternalID: "id",
          },
          Applications: {
              uniqueIDAttrExternalID: "id",
          },
      }
   )
   

   SCAFFOLDING #16

   Construct the URL for both entities. SGNL sends one request for each entity to the adapter. The if condition checks whether the incoming SGNL request is either for employees or applications, and constructs the URL accordingly using the following information:

   1. BaseURL: this is the URL specified during SoR creation. For BambooHR, this is https:/api.bamboohr.com/api/gateway.php/<companyDomain>
   2. API Version passed in the adapter configuration at the time of SoR creation
   3. External ID of the entities (Directory and Applications constants defined in SCAFFOLDING #11)

      // SCAFFOLDING #16 - pkg/adapter/datasource.go: Create the SoR API URL
      // Populate the request with the appropriate path, headers, and query parameters to query the
      // datasource.
      // The BambooHR Directory URL is as follows:
      // https://api.bamboohr.com/api/gateway.php/{companyDomain}/v1/employees/directory
      // The BambooHR Applications URL is as follows:
      // https://api.bamboohr.com/api/gateway.php/{companyDomain}/v1/applicant_tracking/applications
   
      // User must pass api.bamboohr.com/api/gateway.php/<companyDomain> as the BaseURL
   
      url := fmt.Sprintf("%s/%s", request.BaseURL, request.Config.APIVersion)
      if request.EntityExternalID == "directory" {
          url = fmt.Sprintf("%s/employees/%s", url, Directory)
      } else if request.EntityExternalID == "applications" {
          url = fmt.Sprintf("%s/applicant_tracking/%s", url, Applications)
      }
   

   SCAFFOLDING #17

   Set up the Accept header. We use the Accept header passed along in the adapter configuration.

      // SCAFFOLDING #17 - pkg/adapter/datasource.go: Add any headers required to communicate with the SoR APIs.
      // Add headers to the request, if any.
      req.Header.Add("Accept", request.Config.AcceptHeader)
   

   SCAFFOLDING #17-1

   Next, we need to add code to parse the responses we receive from the Employees and Applications APIs. The following code calls ParseEmployeesResponse() or ParseApplicationsResponse() depending on the entity being ingested.

      // SCAFFOLDING #17-1 - pkg/adapter/datasource.go: To add support for multiple entities that require different parsing functions
      // Add code to call different ParseResponse functions for each entity response.
      var parseErr *framework.Error
      var objects []map[string]any
      var nextCursor string
   
      // Start: Based on the entity being processed, call the corresponding ParseResponse functions
      if request.EntityExternalID == "directory" {
          objects, nextCursor, parseErr = ParseEmployeesResponse(body)
      } else if request.EntityExternalID == "applications" {
          objects, nextCursor, parseErr = ParseApplicationsResponse(body)
      }
      if parseErr != nil {
          return nil, parseErr
      }
      // End
   
      response.Objects = objects
   

   SCAFFOLDING #18 and SCAFFOLDING #19

   Next, we’ll need to duplicate the ParseResponse() function in the template and ensure we’re returning Employee and Application objects. Replace the entire ParseResponse() function, with the following code. These return data.Employee and data.Applications that are json tagged to “employees” and “applications”, enabling us to unmarshal the responses returned by the corresponding APIs.

   SCAFFOLDING #18 and #19 do not need to be updated for parsing Employees. However, in the ParseApplicationsResponse() function, note the special handling of the “id” field in SCAFFOLDING #18. SGNL requires the unique attribute to be of type String, but since BambooHR Applications return “id”s as Float64, we need to convert the type to String.

   func ParseEmployeesResponse(body []byte) (objects []map[string]any, nextCursor string, err *framework.Error) {
      var data *DatasourceResponse
   
      unmarshalErr := json.Unmarshal(body, &data)
      if unmarshalErr != nil {
          return nil, "", &framework.Error{
              Message: fmt.Sprintf("Failed to unmarshal the datasource response: %v.", unmarshalErr),
              Code:    api_adapter_v1.ErrorCode_ERROR_CODE_INTERNAL,
          }
      }
   
      // SCAFFOLDING #18 - pkg/adapter/datasource.go: Add response validations.
      // Add necessary validations to check if the response from the datasource is what is expected.
   
      // SCAFFOLDING #19 - pkg/adapter/datasource.go: Populate next page information (called cursor in SGNL adapters).
      // Populate nextCursor with the cursor returned from the datasource, if present.
      nextCursor = ""
   
      return data.Employees, nextCursor, nil
   }
   
   func ParseApplicationsResponse(body []byte) (objects []map[string]any, nextCursor string, err *framework.Error) {
      var data *DatasourceResponse
   
      unmarshalErr := json.Unmarshal(body, &data)
      if unmarshalErr != nil {
          return nil, "", &framework.Error{
              Message: fmt.Sprintf("Failed to unmarshal the datasource response: %v.", unmarshalErr),
              Code:    api_adapter_v1.ErrorCode_ERROR_CODE_INTERNAL,
          }
      }
   
      // SCAFFOLDING #18 - pkg/adapter/datasource.go: Add response validations.
      // Add necessary validations to check if the response from the datasource is what is expected.
      for _, applicationsMap := range data.Applications {
          if id, ok := applicationsMap["id"].(float64); ok {
              // Convert float64 to int
              intID := int(id)
              // Convert int to string
              applicationsMap["id"] = strconv.Itoa(intID)
          }
      }
   
      // SCAFFOLDING #19 - pkg/adapter/datasource.go: Populate next page information (called cursor in SGNL adapters).
      // Populate nextCursor with the cursor returned from the datasource, if present.
      nextCursor = ""
   
      return data.Applications, nextCursor, nil
   }
   

8. Update pkg/adapter/adapter.go

   SCAFFOLDING #20 and SCAFFOLDING #21

   SCAFFOLDING #20 and SCAFFOLDING #21 do not need to be modified

   // Adapter implements the framework.Adapter interface to query pages of objects
   // from datasources.
   type Adapter struct {
      // SCAFFOLDING #20 - pkg/adapter/adapter.go: Add or remove fields to configure the adapter.
   
      // Client provides access to the datasource.
      Client Client
   }
   
   // NewAdapter instantiates a new Adapter.
   //
   // SCAFFOLDING #21 - pkg/adapter/adapter.go: Add or remove parameters to match field updates above.
   func NewAdapter(client Client) framework.Adapter[Config] {
      return &Adapter{
          Client: client,
      }
   }
   

   SCAFFOLDING #22

   Add Config to the request object being constructed. We are merely passing on the adapter configuration (that contains the Accept Header and API Version information) to GetPage() in pkg/adapter/datasource.go to construct the correct URL and Accept header.

   Also notice that Username field is set to request.Auth.Basic.Password, and Password is set to request.Auth.Basic.Username. This is because BambooHR expects the API Key to be sent as the username, and any random string as the password. See BambooHR Authentication.

      // SCAFFOLDING #22 - pkg/adapter/adapter.go: Modify implementation to query your SoR.
      // If necessary, update this entire method to query your SoR. All of the code in this function
      // can be updated to match your SoR requirements.
   
      if !strings.HasPrefix(request.Address, "https://") {
          request.Address = "https://" + request.Address
      }
      req := &Request{
          BaseURL:          request.Address,
          Username:         request.Auth.Basic.Password,
          Password:         request.Auth.Basic.Username,
          PageSize:         request.PageSize,
          EntityExternalID: request.Entity.ExternalId,
          Cursor:           request.Cursor,
          Config:           request.Config,
      }
   

   SCAFFOLDING #23, SCAFFOLDING #24 and SCAFFOLDING #25

   SCAFFOLDING #23, #24 and #25 do not need to be modified for BambooHR

           // SCAFFOLDING #23 - pkg/adapter/adapter.go: Disable JSONPathAttributeNames.
          // Disable JSONPathAttributeNames if your datasource does not support
          // JSONPath attribute names. This should be enabled for most datasources.
          web.WithJSONPathAttributeNames(),
   
          // SCAFFOLDING #24 - pkg/adapter/adapter.go: List datetime formats supported by your SoR.
          // Provide a list of datetime formats supported by your datasource if
          // they are known. This will optimize the parsing of datetime values.
          // If this is not known, you can omit this option which will try
          // a list of common datetime formats.
          web.WithDateTimeFormats(
              []web.DateTimeFormatWithTimeZone{
                  {Format: time.RFC3339, HasTimeZone: true},
                  {Format: time.RFC3339Nano, HasTimeZone: true},
                  {Format: "2006-01-02T15:04:05.000Z0700", HasTimeZone: true},
                  {Format: "2006-01-02", HasTimeZone: false},
              }...,
          ),
   
          // SCAFFOLDING #25 - pkg/adapter/adapter.go: Uncomment to set the default timezone in case the SoR datetime attribute does not have timezone specified.
          // This can be provided to be used as a default value when parsing
          // datetime values lacking timezone info. This defaults to UTC.
          // web.WithLocalTimeZoneOffset(-7),
      )
      if parserErr != nil {
          return framework.NewGetPageResponseError(
              &framework.Error{
                  Message: fmt.Sprintf("Failed to convert datasource response objects: %v.", parserErr),
                  Code:    api_adapter_v1.ErrorCode_ERROR_CODE_INTERNAL,
              },
          )
      }
   

Generating an Authentication Token for SGNL-to-Adapter Authentication

To verify that the adapter receives ‘GetPage()’ requests exclusively from SGNL, we employ shared tokens for authentication. These tokens are stored in a JSON file and are provided to the adapter upon startup. Additionally, the same authentication token is incorporated into the Custom Adapter at the time of its creation in the SGNL Console.

To generate the authentication token, run the following command in your terminal:

openssl rand 64 | openssl enc -base64 -A

The openssl utility is available in most Unix based operating systems. If it’s not available, you can install it from the OpenSSL.org page.

Create a authtokens.json file and add the following line:

["GENERATED AUTHENTICATION TOKEN"]

Get the path to the authtokens.json file by running:

ls "`pwd`/authTokens.json"

Note: Do not commit the authTokens.json to git. You want this file stored in a secure location.

5. Testing the Adapter on your Development machine

Once code is committed, you can run and test the adapter locally by executing in the bamboohr-sgnl-adapter directory as follows:

1. Set environment variable AUTH_TOKENS_PATH

   export AUTH_TOKENS_PATH=<PATH TO authtokens.json file>
   

2. In the bamboohr-sgnl-adapter directory, execute the following command to start the adapter running locally on port 8080

   go run cmd/adapter/main.go
   

3. Use Postman to send a gRPC call to the adapter

   1. Download the SGNL Adapter gRPC service definition from here and save it as adapter.proto
   2. Set up a new gRPC request in Postman and Import the proto file into the Postman gRPC request as shown below:

   

   

   

   3. Set address to 127.0.0.1:8080 and select GetPage as the method

   

   4. The test message for Employees and Applications to be sent to the locally running adapter can be found in the Appendix. Copy the message and paste in Postman

   

   5. Update the API Key and Company Domain for your BambooHR instance
   6. Add the token generated in the previous section to the request metadata with key=token and the generated token as the value

   

   7. Hitting Invoke will send a gRPC message to the adapter running locally. The adapter then relays the request to BambooHR, and responds to the gRPC message as shown below:

   

   8. You can repeat the process to test your adapter for Applications entities as well. Generate a new request and follow the steps above. Use the test Applications message available in the Appendix.

6. Deploying the Adapter in AWS EC2

The adapter-template repo comes with a Dockerfile that will enable you to build and deploy the adapter in a Docker container. You could choose to either run it in a virtual machine in AWS/GCP/Azure, or as part of your organization’s k8s cluster. Ingestion will work as long the adapter is reachable on a public IP address and the port 8080 is exposed. You are not tied to 8080 - it can be updated in cmd/adapter/main.go

We focus on demonstrating building the adapter and running it in an AWS EC2 instance. The assumption is that the EC2 instance is publicly reachable over a public IP address and the port 8080 that the adapter is listening on is open and reachable from the Internet. For more information on setting up your EC2 instance, please refer to the AWS EC2 documentation.

In your EC2 instance, please follow the instructions below:

Copy the bamboohr-sgnl-adapter directory to EC2

tar -czf adapter-template.tgz adapter-template/
scp -i <your-aws-key-pem-file> adapter-template.tgz ec2-user@EC2-INSTANCE-PUBLIC-IP:~/

# Copy the authTokens.json file to the EC2 instance
scp -i <your-aws-key-pem-file> authTokens.json ec2-user@EC2-INSTANCE-PUBLIC-IP:~/

Building the adapter:

In the EC2 instance,

# Untar the file
tar -xzf adapter-template.tgz
cd adapter-template

# set the AUTH_TOKENS_PATH environment variable
export AUTH_TOKENS_PATH=/path/to/authTokens.json

# build the adapter docker image
docker build -t adapter:latest .

# run the adapter
docker run -p 8080:8080 --rm -it -e AUTH_TOKENS_PATH=<path/to/authTokens.json adapter:latest

You should be able to point your Postman gRPC test requests to <EC2-INSTANCE-PUBLIC-IP>:8080 to test the adapter.

Next, we will walk through setting up SGNL to ingest Employee and Applications data from BambooHR through the adapter.

7. Creating Custom BambooHR Adapter in SGNL Console

In the SGNL Console, go to Admin->Adapters. Click on Add Adapter

Enter Adapter configuration. For BambooHR, enter “BambooHR-1.0.0” for the type because that’s what we defined as the type in the adapter code in **cmd/adapter/main.go. **Click Save

8. Creating BambooHR SoR in SGNL Console

Copy BambooHR SoR Template

In the SGNL Console, go to “System of Record” and click on Add. Click on “Create Custom SoR”

Paste the BambooHR template YAML into the text box and click Continue

SGNL Console should render all information from the template. Fill in the BambooHR domain, Username and Password and click on Continue

The Employee and Application entity will be created but ingestion will be disabled, by default. Enable Ingestion on each Entity and finally on the SoR as shown below.

You can view the entities and relationships in the Visualizer page.

On the Systems of Record page, you should see all Employees and Applications ingested into SGNL.

Once ingestion is complete and BambooHR data is in the SGNL graph, you can use Data Lens to explore the SGNL graph.

9. Appendix

Test Employee gRPC message to be sent to BambooHR adapter

{
    "datasource": {
        "address": "https://api.bamboohr.com/api/gateway.php/<YourBambooHRDomain>",
        "auth": {
            "basic": {
                "password": "BambooHR API Key",
                "username": "randomstring"
            }
        },
        "config":"ewogICJhcGlWZXJzaW9uIjogInYxIiwKICAiYWNjZXB0SGVhZGVyIjogImFwcGxpY2F0aW9uL2pzb24iCn0K",
        "id": "38c96d7c-c042-4823-9337-cf658ad20000",
        "type": "BambooHR-1.0.0"
    },
    "entity": {
        "attributes": [
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "id",
                "id": "38c96d7c-c042-4823-9337-cf658ad20001",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "displayName",
                "id": "38c96d7c-c042-4823-9337-cf658ad20002",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "firstName",
                "id": "38c96d7c-c042-4823-9337-cf658ad20003",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "lastName",
                "id": "38c96d7c-c042-4823-9337-cf658ad20004",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "preferredName",
                "id": "38c96d7c-c042-4823-9337-cf658ad20005",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "jobTitle",
                "id": "38c96d7c-c042-4823-9337-cf658ad20006",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "workEmail",
                "id": "38c96d7c-c042-4823-9337-cf658ad20007",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "department",
                "id": "38c96d7c-c042-4823-9337-cf658ad20008",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "location",
                "id": "38c96d7c-c042-4823-9337-cf658ad20009",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "division",
                "id": "38c96d7c-c042-4823-9337-cf658ad20010",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "supervisor",
                "id": "38c96d7c-c042-4823-9337-cf658ad20011",
                "list": false,
                "ordered": false
            }
        ],
        "child_entities": [],
        "external_id": "directory",
        "id": "38c96d7c-c042-4823-9337-cf658ad21000",
        "ordered": false
    },
    "page_size": "100"
}

Test Applications gRPC message to be send to BambooHR adapter

{
    "datasource": {
        "address": "https://api.bamboohr.com/api/gateway.php/<YourBambooHRDomain>",
        "auth": {
            "basic": {
                "password": "BambooHR API Key",
                "username": "randomstring"
            }
        },
        "config":"ewogICJhcGlWZXJzaW9uIjogInYxIiwKICAiYWNjZXB0SGVhZGVyIjogImFwcGxpY2F0aW9uL2pzb24iCn0K",
        "id": "38c96d7c-c042-4823-9337-cf658ad20000",
        "type": "BambooHR-1.0.0"
    },
    "entity": {
        "attributes": [
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "id",
                "id": "38c96d7c-c042-4823-9337-cf658ad20001",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_DATE_TIME",
                "external_id": "appliedDate",
                "id": "38c96d7c-c042-4823-9337-cf658ad20002",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "applicant__firstName",
                "id": "38c96d7c-c042-4823-9337-cf658ad20003",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "applicant_lastName",
                "id": "38c96d7c-c042-4823-9337-cf658ad20004",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "status__label",
                "id": "38c96d7c-c042-4823-9337-cf658ad20005",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "applicant__email",
                "id": "38c96d7c-c042-4823-9337-cf658ad20006",
                "list": false,
                "ordered": false
            },
            {
                "type": "ATTRIBUTE_TYPE_STRING",
                "external_id": "job__title__label",
                "id": "38c96d7c-c042-4823-9337-cf658ad20007",
                "list": false,
                "ordered": false
            }
        ],
        "child_entities": [],
        "external_id": "applications",
        "id": "38c96d7c-c042-4823-9337-cf658ad21000",
        "ordered": false
    },
    "page_size": "100"
}