on 23-Jun-2022 (Thu)

Sugarscape model:

In order to find the rules for agent behaviour, we proceed as follows. In the Initialization phase, agents are positioned randomly on the grid. Their input, i.e. the information they have access to is defined: The agents can observe the amount of sugar on their patch and the amount of sugar on each of the 4 neighbouring patches. In addition, they can also see the number of agents on their current patch and on each neighbouring patch. The score, which is used to determine if a decision was good or not, is the amount of sugar they gathered in their turn. The decision the agent faces is which of the 5 possible actions it should perform: remaining stationary, moving north, moving south, moving east or moving west

Sugarscape model:

During the Experience phase agents make random decisions and add new entries to the database consisting of a vector with all their sensory input, the randomly chosen action and the result, i.e. if the score increased, decreased or stayed the same due to this decision. To gather a sufficient amount of data, the Experience phase lasted for 5000 time steps, which is computationally relatively cheap and can be calculated in a few seconds using a single core. This number was chosen to ensure that there is enough data for training. See Section 3 for more details on the amount of data required to successfully train the Neural Network. Note, that the sensory input consists of just numerical values arranged in a vector, as depicted in (Figure 2): There is no information about the meaning of the individual elements of the vector, or connections between the elements. For example, Element 1 is the sugar amount on the patch directly north of the agent and Element 5 is the information how many agents are currently on this patch. In the database however, this structure is completely unknown and needs to be learned by the Neural Network in the following phase. This can be seen as a significant difference to reinforcement learning, where one would process the input data in order to obtain better results, e.g. by calculating the effective sugar amount of each patch using ((1)) and the competition factor. In contrast to that, we provide only input that comes directly from an agents senses.

Sugarscape model:

The Training phase uses the gathered data to train a Hidden Layer Neural Network to solve the underlying classification problem. Here, we use a Multi-Layer Perceptron [52], but other methods could be used as well. Utilizing a Random Forest approach, for example, leads to nearly identical results, with the main difference being computation time, which is not the focus of this study.

Sugarscape model:

For the Application phase, all agents are again positioned randomly. In every step they observe their environment and use the trained Neural Network to classify each of the actions they could perform. To decide on the best action, each action is rated on the following basis: WZÓR powyżej

with Cp being the confidence of the decision being positive and

Cn the confidence of the action being negative.

Of all the possible actions, the action with the highest rating R is chosen. Ties are resolved randomly.

Sugarscape model: harder version

We see that at a certain amount of training data, the positive decisions during training saturate close to 1 and show no improvement with more data. The relative number of good decisions during the Applications phase, however, remains low throughout and does not show any improvement with more training data. This means, that additional training data collected in the same manner will not solve this problem. Since during the Experience phase the agents mainly encounter system states that are not relevant for the system, they have no chance of learning about the system, as they will encounter it during the Application phase. A modification to the framework is needed

3.3. Expanding the framework In the previous section, we found an example in which the presented framework fails, because the states encountered during Experience phase and Application phase differ too much. We will now adapt the framework, in order to increase its scope to such systems. Instead of a sequential approach of training followed by application, we switch to an iterative approach: (1) Initial Experience (Random decisions) (2) Training of Neural Network (NN) (3) Advanced Experience (decision based on NN + random decisions) (4) Training of Neural network (NN) (5) Go to 3 The first two steps are identical to the original framework, but in step 3 we combine experience and application: The agents make their decision based on the current NN, but additionally choose a random, different action. The random action is only evaluated and stored as experience, but not actually performed. This ensures, that the encountered states are those relevant for the current NN while also giving the possibility to explore different actions. It would also be possible to evaluate all possible actions, but the chosen approach has the significant advantage, that calculation time does not scale with the number of available actions. In this Advanced Experience phase, the agents encounter new system states, which are more relevant for application, since they are based on decisions that are more realistic than random choices. Nevertheless, we cannot assume that those states are the ones encountered in the application, since the NN will change after training and so will the encountered states. Thus, we have to use an iterative approach. Only if the NN does not change significantly, the process converged and the encountered states in the last experience phase will be similar to those encountered during application

The number of iterations that is needed for convergence is not known in the beginning of the process. Strictly speaking, one has to check how much the NN changed between two steps, i.e. if it leads to the same agent behaviour. In practise, the use of much simpler proxies is possible to check convergence. Here, we use the average agent score during each iterative step, shown in (Figure 9). Convergence of this observable is a strong hint for the convergence of agent behaviour, although strictly speaking it is possible to think systems in which different behaviour leads to the same score. For such systems, other proxies need to be investigated as well, or the actual NN needs to be checked for differences

we expanded the framework[WHAT TYPE OF FRAMEWORK????] to an iterative process, thereby increasing its scope to systems that cannot be explored well using random agent decisions

4. Conclusion We showed that by using the presented framework it is possible to implement an agent- based model without the need to manually find rules or equations for agent behaviour, which is the most challenging step for most agent-based models. Within the framework, agents first make random decisions and gather experience. Then a Neural Network is trained to be able to judge a combination of (sensory) input and a decision, classifying this decision as positive, negative or neutral. Here, the Neural Network is not used as a form of optimization, but rather as a realistic depiction of a decision process, including the possibility of errors in judgement. We demonstrated the advantages of this approach by applying it to reproduce the results of the prominent Sugarscape model. To show the flexibility of the framework, we then made slight changes to the modelled system by removing the competition between the agents. While a traditional approach to agent-based modelling would require a reformulation of the rules for agent behaviour, here the Neural Network is automati- cally retrained to accommodate the changes in the system and we naturally end up with realistic agent behaviour. We also explored the limits of the framework and found that the original approach fails, once system states that are relevant, if agents act to reach a goal, do not appear during an Experience phase that only features random decisions. Therefore, we expanded the approach to include an iterative learning process and a way to check for convergence. Thus, we expanded the framework to a much larger class of systems

Thus, we expanded the framework to an iterative process, thereby increasing its scope to systems that cannot be explored well using random agent decisions. 4. Conclusion We showed that by using the presented framework it is possible to implement an agent- based model without the need to manually find rules or equations for agent behaviour, which is the most challenging step for most agent-based models. Within the framework, agents first make random decisions and gather experience. Then a Neural Network is trained to be able to judge a combination of (sensory) input and a decision, classifying this decision as positive, negative or neutral. Here, the Neural Network is not used as a form of optimization, but rather as a realistic depiction of a decision process, including the possibility of errors in judgement. We demonstrated the advantages of this approach by applying it to reproduce the results of the prominent Sugarscape model. To show the flexibility of the framework, we then made slight changes to the modelled system by removing the competition between the agents. While a traditional approach to agent-based modelling would require a reformulation of the rules for agent behaviour, here the Neural Network is automati- cally retrained to accommodate the changes in the system and we naturally end up with realistic agent behaviour. We also explored the limits of the framework and found that the original approach fails, once system states that are relevant, if agents act to reach a goal, do not appear during an Experience phase that only features random decisions. Therefore, we expanded the approach to include an iterative learning process and a way to check for convergence. Thus, we expanded the framework to a much larger class of systems. This work only serves as a stepping stone to application on systems that cannot be investigated using reinforcement learning. Many expansions are possible to extend the scope of the framework. Currently, agents only think one step ahead, which works well for the Sugarscape model, but may not be enough for other systems. However, the framework can be expanded for thinking ahead straightforwardly. Currently, the deci- sion the agents face is what action to take next. This can be easily expanded to the decision what the next 2 or 3 actions should be. The concept would not change, the training database would only gain one additional entry for each turn the agents think ahead. In the same manner, memory could be included, so that agents also have access to the information which decisions they made earlier

The next step on the path to a framework that can be used universally for agent-based models of arbitrary systems is to apply it to different, more complicated systems. For now, only systems that can be solved via reinforcement learning have been investigated, to give a simple way of model validation. The main application area of this framework, however, will be systems without clear solutions. Cooperation games, social dilemmas, public good games and similar systems are interesting future applications. Here we could compare against data collected in social experiments and analyse if the framework leads to better results than traditional application of machine learning or agent-based modelling techniques

For example, suppose that the causal DAG includes an unmeasured common cause of A and Y, U and also a measured variable L that is an effect of U.

The causal graph for interventional distributions is simply the same graph that was used for the observational joint distribution, but with all of the edges to the intervened node(s) removed. This is because the probability for the intervened factor has been set to 1, so we can just ignore that factor

In this paper, we present our study of consumer purchase behaviour, wherein, we establish a data-driven framework to predict whether a consumer is going to purchase an item within a certain time frame using e-commerce retail data. To model this relationship, we create a sequential time-series data for all relevant consumer-item combinations. We then build generalized non-linear models by generating features at the intersection of consumer, item, and time.

Abstract

Consumer behaviour in e-commerce can be described by sequences of interactions with a webshop. We show that recurrent neural networks (RNNs) are a natural fit for modelling and predicting consumer behaviour. In multiple aspects, RNNs offer advantages over existing methods that are relevant for real-world production systems. Applying RNNs directly to sequences of consumer actions yields the same or higher prediction accuracy than vector-based methods like logistic regression. Unlike the latter, the application of RNNs comes without the need for extensive feature engineering. In addition, we show that RNNs help us link individual actions directly to predictions in an intuitive way. This allows us to understand the implications consumer actions have on predicted probabilities over the course of the consumer’s history. We demonstrate the advantages of RNNs on the empirical data of a large European online fashion platform

In this paper, we show that recurrent neural networks (RNNs) are promising to overcome both shortcomings of vector-based methods, tedious feature engineering and lack of explainability.

RNNs operate on sequences of varying lengths and therefore provide an appropriate match to consumer histories.

We apply RNNs directly to series of captured consumer actions. RNNs maintain a latent state that is updated with each action. RNNs are trained to detect and preserve the predictive signals in the consumer histories.

RNNs allow to link individual actions to predictions in a straightforward manner. To our knowledge, in this work we provide the first visualizations that exploit this in the context of e-commerce. This allows to quantify how predictions are affected by specific actions or action sequences conducted by the consumer. These insights are drawn on a quantitative empirical basis— in contrast to vague intuitions that often drive product development in e-commerce

To summarize, our contributions are the following: (i) we show how consumer behavior can be predicted without sophisticated feature engineering by using RNNs; (ii) we provide an empirical comparison of prediction performance on real-world e-commerce data; and (iii) we demonstrate how RNNs are helpful in explaining the predictions for individual consumers

Vector-based machine learning methods like logistic regression take vectors f = (f 1 , . . . , f n ) of fixed length n as inputs. Applying these methods on consumer histories of arbitrary length requires feature engineering: a fixed set of identifiers f i has to be designed to capture the essence of an individual consumer history. Only signals that are encoded in the feature vector can be picked up by the prediction model. Defining expressive features often requires both, domain knowledge as well as data-science intuition.

While preprocessing is an important tool to improve model performance, it artificially increases the dimensionality of the input vector. Also, the resulting binary features can be strongly correlated. Both outcomes make it difficult to tell which action patterns in the underlying consumer histories have a strong impact on the prediction outcome

Event-stream RNNs

We propose to model the behavior of consumers with RNNs. Consumer histories are inherently sequential and of varying lengths T , making RNNs a natural model choice. In e-commerce, available data sources and prediction scenarios often change, making the generality of RNNs appealing as no problem-specific feature engineering has to take place.

RNNs can be applied to predict future consumer behavior in regression and classification settings, for example, to predict interest in fashion brands or consumer lifetime value. We focus on predicting the probability P (o^u | x^u ₁ , . . . , x u T ) of a consumer u to place an order o^u , which we model as a binary classification problem. For instance, we could be interested in orders in general or of specific products. The resulting probability estimates can be used in a recommender system to deliver appropriate product recommendations and webshop contents.

Recognizing relevant patterns in long input streams (> 100 actions) can turn out to be difficult for the human mind. To achieve better explainability, in many e-commerce applications consumer behavior can be viewed on the level of sessions. A session is a well-defined visit of a consumer to a web- shop: a subsequence of events within the consumer’s history that lay no further apart than a predefined time difference. Here, we split sequences of events into different sessions if there is a time gap of more than 30 minutes in-between subsequent events.

RNN details

We use a simple RNN architecture with a single LSTM layer and ten-dimensional cell states. The hidden state at the last time-step is combined with binary non-history features to make the final prediction in a logistic layer. Thus, the final prediction of the RNN is linear in the learned and non-history features. The non-history features describe time, weekday, and behavioral gender and are also provided to the baseline methods. Instead of absolute timestamps, the time differences ∆(x t−1 , x t ) to the previous inputs x t−1 are fed to the RNN at each time- step t. Furthermore, the difference between the last event x T and the prediction time (the session start) is provided to the final prediction layer

For event-stream RNNs, history inputs x_t ∈ R²⁰ consist of a one-hot encoding of the action type and the time difference. For session-stream RNNs, history inputs s_t ∈ R²³ represent sessions with binary indicators which action types occurred, the time difference to the previous session and the characteristics described in Sec. 3.2. Time differences and, in case of session-stream RNNs, the total session event counts are logarithmized.

All models are trained to minimize negative log-likelihood (NLL).

As probability estimates are required directly in many practical applications, we use NLL also for evaluation. In some applications, the resulting ranking of consumers is more important than the probabilities themselves. For this reason, we also report the area under the ROC curve (AUC)