Written by: Kristjan Eljand | Technology Scout
The jobs of the future
In near future, digital agents will start to participate in our working environment, making decisions and taking actions to carry out different tasks. The role of the human expert won’t disappear, but it’ll change significantly. This blog post gives one possible version about our role next to digital agents. In the context of this article, Digital agent is defined as a program that is trained with reinforcement learning to take optimal actions in some specific process.
Our technological capability has reached the level, where programs are able to win human experts in chess, Go and in different video games. The state-of-the-art solutions are developed with Reinforcement Learning algorithms. It’s a field of machine learning with following elements:
1. Digital environment that enables to carry out actions and gives information about the state (where we are) and reward.
2. Digital agent that makes actions in the environment and receives feedback about the results.
In chess, the environment is a board with 8x8 squares where we are able to make actions based on the rules of the chess and the result of the game is either win, loss or tie. The goal of the Digital agent would be to find the optimal policy for winning. To learn, the agent plays thousands or millions of games against itself or other programs and calculates the optimal behavior pattern.
Digital Environment Architect
For digital agent to be able to learn the relationship between activities and result, we would need to define the environment that includes information about the variables that affect the results. In chess, the environment is simple — determined by the number of squares, pieces and rules to move the pieces. Defining the business environment is not that simple. For example, the activities and the results of the marketing campaign are dependent on other campaigns, budget of the marketing unit, business strategy, the activity of the competitors, economic growth, etc.
The goal of the Digital environment architect is to define the information that is relevant for choosing the right actions and connecting them to results. Daily routine includes looking for new influencing factors and analyzing the correctness of the data. It is a suitable task for today’s marketing specialists.
Environment digitizer
The goal of the Environment digitizer is to make the information suitable for digital agent. Daily routine includes building data pipelines and testing the validity of the data.
To understand this job, let’s think how we could give our digital marketing agent the information about the activity of the competitors. Human marketing specialist would look at the existing campaigns and make notes about what style is used and values emphasized. Sadly, computers don’t understand such an abstract information (yet) and thus this info needs to be quantified and structured.
Digitizing the environment is a one-time task but managing it is part of day-to-day work. Architect needs to make sure that all relevant information sources are included, and digitizer helps to make this information readable for the agent.
Action designer
The goal of the Action designer is to create a list of actions that the digital agent is allowed to carry out. Daily routine includes designing the actions, analyzing the results and making the necessary changes. The initial action design takes place at the same time as creating the environment because the environment needs to be able to give reward for every action and to reflect the current state (where our agent is currently located).
For digital marketing agent, the actions could include 1. Starting a new campaign, 2. Changing the existing campaign, 3. Increasing/decreasing the budget, 4. Closing the campaign, 5. Selecting the marketing channels, etc. In short, the Action designer needs to define all the actions that he/she would do in digital marketing and the rules of carrying out those actions.
Reward designer
Reward designer aims to define the motivation system of the digital agent. In other words, what kind of reward or punishment needs to be assigned to the agent for its activities.
Let’s use the digital marketing example again to understand this role. Assume, that digital agent starts the marketing campaign for the new product that results to 20 new visitors on the product page during the next day. We could potentially give +20 points to the agent as a reward. But is it a good idea!? Probably not, because our agent might be trying to optimize how to bring maximum number of people to the product page but our real goal is to increase the revenue. Thus, it would be better idea to give reward only when the activity of the agent results in real purchase. Reinforcement learning agents are somewhat notorious in their ability to achieve high score but doing it completely differently than we planned (it’s called the specification gaming).
Take a look at the example below. The goal was to train an agent to pick up the red Lego block and place it on top of the blue one. The agent was rewarded when the bottom of the red block was higher than the bottom of the blue block. With this reward design, the agent learnt to achieve high reward by turning the red block over.
The second example is about the agent that was aimed to walk in the simulated environment. The agent was rewarded for moving forward. Sadly, it found the mistake in the simulator and achieved the high score by sliding weirdly.
The role of the Reward designer will be extremely important because the digital agents will give you exactly what you asked them to do which might not always be what you wanted them to do.
Critic/Trainer
Critic is responsible for the initial training of the digital agent. Daily routine includes analyzing the actions of the agent and giving the feedback in the form of rewards.
Digital agent is able to learn the optimal behavior pattern from the scratch. Initially, it’ll try random activities and then it starts to select the ones that has brought the best result. For video games, this training approach is valid because the results don’t have effect on the real world. In business, we can’t allow our digital agent to make thousands or millions of random actions to learn what makes sense and what not. The role of the Critic is to solve this problem by carrying out the initial training of the agent.
Let’s assume that the environment and the activities have been designed and our marketing agent is ready for training. Critic will do the initial training before letting the agent loose to the real world. It can potentially happen as follows: Digital agent will pick a random action and if this action is a good one, the Critic will give +1 points to the agent. If the random action is a bad one, the reward is -1. These points should be coherent with the reward system created by the Reward designer. Based on the feedback from the Critic, our agent can learn the approximate value of actions.
Once the initial training is finalized, the agent can be connected to the real world and the training process can continue with the feedback from the actual process.
Summary
In future, digital agents will start to participate in our working environment, making decisions and taking actions to carry out different tasks. The role of the human expert won’t disappear, but it’ll change significantly:
- Digital environment architect — defines the information that is relevant for choosing the right actions and connecting them to results.
- Environment digitizer — makes the information suitable for digital agent.
- Action designer — creates a list of actions that the digital agent is allowed to carry out.
- Reward designer — aims to define the motivation system of the digital agent. In other words, what kind of reward or punishment needs to be assigned to the agent for its activities.
- Critic/Trainer — carries out the initial training of the digital agent before letting it loose to the real world.
Today, we are used to gather information, analyze it and make decisions. In the future, our role is to give guidance and follow the actions of the digital agents.
