Golang from Scratch

Golang has become a language of the web. It’s simple usage for servers and web applications make it an ideal candidate for new projects. Here at PMG, we started our first Golang application mid-2017 and have enjoyed working with it. The application is running well and development is straightforward.

However, when the project first started, and with prior attempts being unsuccessful, there was uncertainty in how to structure and build the application. Which packages should we use and how do we arrange our own? What persistence mechanisms should we use? How do we represent our domain types and business logic in the code?

In this post, I hope to clarify some ways to structure a Golang application and how everything can fit together. The following is a single example application that has been developed for the purpose of this post with the lessons learned from all of our development team. As with all software, there are many different ways to accomplish one thing.

A couple things this is:

A fully working, example, small Golang application.

Some general tips and tricks that have been picked up by our development team around development as a whole.

Some things this is not:

The be-all-end-all way of doing things.

A production-grade application.

An example of managed infrastructure or development operations.

An example of a secure API. Securing the API and authenticating requests is considered outside the scope of this application.

All code can be found at https://github.com/AgencyPMG/go-from-scratch/tree/20170717.1. Installation and running instructions can be found in the README.

A few things up front:

We use Glide in this example to manage our external Golang dependencies.

We use DBSchema to manage our database schema.

We use Docker Compose to manage our backing services. Currently, there is just a Postgresql database. Using Docker Compose is a great way to avoid having to install development stacks on all dev computers. Installing Docker and spinning up the containers will do the job just fine. The containers must be running before the application runs.

Configuration must be set in the environment before the application runs.

The app/internal directory is what holds all Golang code.

The app/internal/data package is the root directory for all of our domain types.

The app/internal/data/user package holds the

User

type.

The app/internal/data/client package holds the

Client

type.

The app/internal/cli directory holds all of our executables. I like having them all in one directory to make build scripts and locating them easier. In this case there is just one. In some other applications, this could grow to much more since there could be an executable for difference services within the application.

The app/internal/gfsweb package holds the types and logic used in the

gfsweb

executable. We will see the connection between this package and the

cli/gfsweb

package later.

The app/schema directory holds the schema definitions for our relational database.

The bin directory holds utility scripts. On more advanced projects, this could include things like deployment, infrastructure, and Docker scripts.

The etc directory holds our simple configuration. It could be extended to hold more complicated or application specific configuration.

The var directory is used as a data directory for runtime artifacts. In this case we don’t have any of those. We use it for storing our Docker Compose volumes.

If we were to take the main entry point of the application, we would start looking at app/internal/cli/gfswebb/main.go but that doesn’t tell much about what the application does. To understand more of what the application does, we need to look at the data package – this is where we find our domain types.

Inside the data directory (package) we house all of our domain types. The best reason is that this structure keeps all of these types under a single roof instead of mixed in with the rest of our library and logic packages. Additionally, the name data signifies that these are the data types that the application cares about. Another reasonable name would be domain.

Inside the

data

One level inside the

data

user

User

User

Id

data.Id

Email

time.Time

Enabled

Also in the user package we find our first

QueryRepo

Repo

user_repo.go

//QueryRepo provides methods for retrieving Users.
type QueryRepo interface {
	//Get should return the User whose Id equals id.
	//
	//An error should be returned if the User does not exist or there was an error
	//attempting to load the User.
	Get(ctx context.Context, id data.Id) (*User, error)

	//GetEmail should return the User whose Email equals email.
	//
	//An error should be returned if the User does not exist or there was an error
	//attempting to load the User.
	GetEmail(ctx context.Context, email string) (*User, error)

	//List should return all Users in the application.
	//They should be sorted by their lexicographic Email order.
	List(ctx context.Context) ([]*User, error)
}

//Repo provides method for creating and updating Users
//as well as promotes the QueryRepo interface.
type Repo interface {
	//QueryRepo is promoted here to indicate a Repo contains all query methods.
	QueryRepo

	//Add should add u to the underlying storage repository.
	Add(ctx context.Context, u User) error

	//Set should update all stored fields of u in the underlying storage repository.
	//The update should use u.Id for determining which entity to update.
	Set(ctx context.Context, u User) error
}

I want to make some points here about the interfaces.

There are two interfaces all for interacting with

Users

. One is called

QueryRepo

and is used to retrieve

Users

, the other is

Repo

and is used for modifying Users. We go ahead and include the

QueryRepo

inside

Repo

because most of the time we need to modify an entity, we will also need to retrieve it first. We will see this when we look at our command handlers. These two interfaces also point to and make it easier to implement command query responsibility segregation.

The methods in

QueryRepo

return pointer types. We are assuming that the repo is creating a new instance of the type, and once it is returned, the calling code is the owner of the variable. Thus the calling code is free to examine and modify what it gets back.

On the other hand, the methods in

Repo

accept struct parameters, not pointer values. This is to avoid the repositories from modifying the calling variables while they do their work. In essence these methods should read and store what they are called with.

Now we can look at the

data/client

user

Client

We just saw the

QueryRepo

Repo

In our example, we use Postgres as a storage mechanism and the

database/sql

usersql

clientsql

sql

Once we have the repositories set up and working, or at least defined, we can go ahead and use them in our command handlers and business logic. We are going to do this in the data/*/*cmd packages. These packages are named as such because they handle commands to alter the state of the application in some way. These are commands in the sense of CQRS.

As an example, let’s look at part of the clientcmd package.

client_handler.go

//Handler is a handler type that understands how to work with Client commands.
type Handler struct {
	//Clients is the Client repository to use within Command handling.
	Clients client.Repo
}

//CreateClient attempts to create a new Client and add it to h.Clients.
//Cmd must be of type *CreateClientCommand.
//The result, if not nil and without error, will be a *client.Client.
func (h *Handler) CreateClient(ctx context.Context, cmd cbus.Command) (interface{}, error) {
	createClient := cmd.(*CreateClientCommand)

	client, err := createClient.newClient()
	if err != nil {
		return nil, err
	}

	err = h.Clients.Add(ctx, *client)

	return client, err
}

//CreateClientCommand is Command that should be used to create a new Client and
//add it to a repository.
type CreateClientCommand struct {
	//Name is the name of the Client to create.
	Name string
}

Above we see a

Handler

cbus

cmd.(*CreateClientCommand)

All the command handler does, in this case, is create a new

client.Client

Add

h

This is a stripped down version of the data packages we use in our applications. For instance, in our production applications, there are a lot more entity types and relationships each with a cmd and SQL sub-package.

One idea that I want to present is that of specific error types for use cases that could be hit in a normal application. Two that we can point to in this example are 1) data.ErrEntityDoesNotExist and 2)user.ErrClientWithIdDoesNotExist These can be simple sentinel error values or more complicated error types with more functionality if need be. The point though is to be able to have these error values/types passed around the application. Since so much code relies on the data packages, for instance, the API packages, that code can examine errors to figure out what went wrong.

For instance, the API package could send 404 status codes if a command results in a data.ErrEntityDoesNotExist error because it knows the request is attempting to modify an entity that doesn’t exist. Or in the second case, respond with specific error messages or codes if a caller attempts to update a user with a client that isn’t in the application.

This package is how external clients get things done within our application. By making requests to through the API they are able to access and modify entities within the application. So how does it work?

The package is located at app/internal/gfsweb/handler/api. Inside it we have the

API

cbus.Bus

The single exported method on

API

Handler

http.Handler

http.Server

Handler

Now let’s look at two Handler functions for two different endpoints in the API.

clients_api_handler.go

//getClient retrieves and sends a single Client.
func (a *API) getClient(w http.ResponseWriter, r *http.Request) {
	client, ok := a.getClientEntity(w, r)
	if !ok {
		return
	}

	a.sendData(w, client, http.StatusOK)
}

//createClient attempts to create a new Client from a dto.CreateClient form
//and executing a clientcmd.CreateClientCommand.
func (a *API) createClient(w http.ResponseWriter, r *http.Request) {
	f := &dto.CreateClient{}
	if ok := a.parseForm(w, r, f); !ok {
		return
	}

	command := &clientcmd.CreateClientCommand{
		Name: f.Name,
	}

	client, err := a.Bus.ExecuteContext(r.Context(), command)
	a.clientCommandResponse(w, client, err, http.StatusCreated)
}

The getClient method is called for GET requests when a client id is supplied in the request URL. All it’s doing (with some help) is retrieving the Client specified by the route id and sending it in the response.
The createClient method is called for POST requests on the client collection request URL. All it’s doing (with some other help) is parsing a CreateClient form, creating a command from the form, executing the command, and finally sending the response.

Those two prior explanations are supposed to seem short and simple. We want the handlers to be “dumb” and without logic. They are straightforward in that they receive input and write an output. Obviously, there are helper methods that get used quite frequently, but this is done to try to keep the handlers as simple as possible. There is no knowledge of storage in the API, just that there is a QueryRepo. There is no business logic in the API – knowing about a command bus is all it needs.

Just like this example’s

data

api

Also, there is a whole other sub-package,

dto

user.User

With these separate packages, we lastly need to look at how all of them come together so that each part gets what it needs and the application becomes fully functional. For that we have the

gfsweb

gfsweb

App

AppBuilder

App

AppBuilder

app_builder.go

//Build uses config and its factory functions to build a new App.
//If err is nil, the returned App is ready to Run.
func (ab *AppBuilder) Build(config *config.Config) (app *App, err error) {
	var sqlRepo *sqlrepo.Repo
	var dbCloser io.Closer

	defer func() {
		if err != nil && dbCloser != nil {
			dbCloser.Close()
		}
	}()

	sqlRepo, dbCloser, err = ab.SQLRepoFactory(config)
	if err != nil {
		return nil, err
	}

	repos := NewRepos(sqlRepo)

	bus := ab.CBusFactory(repos)

	api := ab.APIFactory(bus, repos)

	server := ab.ServerFactory(config)

	return &App{
		server:   server,
		handler:  api.Handler(),
		dbCloser: dbCloser,
	}, nil
}

We should note that this is where concrete implementations of all of our interfaces are built. In this case, the repositories are built from the data/*/*SQL packages. If, for instance, you decided to use Redis for some of your domain storage, then this is where you would create a Redis backed repository and send that interface throughout the application.

Finally, the App type needs 1) a server 2) a Handler for that server and 3) something to close the database connection. If there were more open connections, then we’d place those on the App as well so that the App can close them when it is told to shut down. This

App

What I like about this package is that it changes rarely, or at least in a way that is very clearly defined. There may be additional repositories, additional storage backends that need connecting to, more commands required, but once this package is solidified and working, the bulk of functionality comes from the

data

api

This section assumes that you have followed the installation and build instructions in the repository’s README. It also assumes there is no data in the database before running the following.

Start by running ./gfsweb in a terminal.

In another terminal, run

curl localhost:8080/users

and you should get an empty JSON array indicating there are no users yet.

Next, create a User via

curl localhost:8080/users -X POST --data '{"email": "email1@example.com", "enabled": true }'

The result of this command is the api/dto JSON representation of the User that is created.

Next, we can create a Client via

curl localhost:8080/clients -X POST --data '{"name": "Client 1"}'

Then run

curl localhost:8080/users/<id_of_user_above> -X PATCH --data '{"client_ids": ["<id_of_client_above>"]}'

Notice that because of the way our API, forms, and commands are built we don’t have to specify the entirety of the entity to update it, just what we want to change.

Try running curl localhost:8080/users/1234

You should see an error indicating an invalid Id value. Although error processing and messages leave something to be desired in this example, it is at least present and easy to build on top of.

There are other commands to run and things to play with, but we leave that as an exercise for the reader.

Hopefully, this has been educational in some fashion. Yes, it is small and an example, but it is functional and completed. Feel free to reach out with questions or comments about things left out or recommendations for improvement. Each of the sections in this post or sections left out could become a post on its own.