Discussion Graph Tool

In the discussion graph tool framework, a co-occurrence analysis consists of the following key steps:

For more details on the core concepts behind discussion graphs, we recommend reading our ICWSM 2014 paper.

A note on projecting weighted data

Often, feature values are weighted. For example, the affect classifier produces a weighted feature value indicating how likely a message is to be expressing joviality, sadness, etc. (In other cases, the use of the WEIGHT BY command implicitly creates a weighted value).

When it encounters a weighted feature value in its target domains, the PROJECT TO command treats the weights as probabilities of a feature value having occurred. For example, let’s continue our analysis of activity and location mentions such as in the following message:

"I'm having fun hiking tiger mountain" tweeted by Alice on a Saturday at 10am

Let’s say our mood analysis indicates that the message has joviality with a weight of “0.8”, serenity has a weight of “0.4” in this message, in addition to the other discrete features:

The two weighted features are interpreted as independent probabilities. That is, there is an 80% likelihood of this message being jovial and a 20% likelihood of not being jovial. Independently, there is a 40% likelihood of the message being serene, and 60% chance of not being serene.

If we project this single message to the relationship between location and mood (PROJECT TO Mood, Location;) this message will expand to the following 4 projected edges::

Of course, when analyzing a larger corpus of social messages, each message will be expanded individually and the results aggregated.

The discussion graph tool’s scripting language currently supports the following commands.

Note that square brackets [ ] indicate optional elements of the command. Italicizedterms indicate user-specified arguments, variable names, etc. of the command.

LOAD

Syntax: LOAD Datasource([arguments]);

Example: LOAD MultiLine(path:”productreviews.txt”);

The LOAD command loads social media data from some datasource. The required arguments are datasource-specific. Generally, datasources require a path to the input file as well as schema information to interpret the file. See the Common things you’ll want to do section below for examples of loading TSV, Multiline record, JSON and Twitter files.

EXTRACT

Syntax: EXTRACT [PRIMARY] field|FeatureExtractor([arguments]),… [FROM varname];

Example: EXTRACT PRIMARY hashtag, Gender(), AffectDetector();

The EXTRACT command runs a series of feature extractors against the raw social media messages loaded from a data source via the LOAD command.

Extracting a field will pass through a field from the raw data unmodified.

Extracting a feature using a FeatureExtractor() will run the specified feature extractor against the social media message. Feature extractors may generate 0, 1 or more feature values for each message they process, and the domain of the feature need not match the name of the feature extractor. For example, the AffectDetector() generates features in several domains (Subjective, Mood and PosNegAffect), and other feature extractors, such as Phrases() can generate features in custom domains.

The PRIMARY flag acts as a kind of filter on the raw social media data. EXTRACT must find at least one PRIMARY field or feature in a message, otherwise the message will be ignored. If no fields or features are marked as PRIMARY, then EXTRACT will not filter messages.

FROM varname tells the EXTRACT command where to get its input data. If not specified, EXTRACT will read from the output of the previous command.

WEIGHT BY

Syntax: WEIGHT BY featureDomain[, …] [FROM varname];

Example: WEIGHT BY userid;

The WEIGHT BY command reweights the data from social media messages. By default, every social media message counts as a single observation.  If we see a co-occurrence relationship occurring in 2 social media messages, then the co-occurrence relationship will have a weight of 2.  We can change this using the WEIGHT BY command so that every unique user (or location or other feature value) counts as a single observation.  So, for example, if a co-occurrence relationship is expressed by 2 unique users, then it will have a weight of 2.  Conversely, if a single user expresses 2 distinct co-occurrence relationships, each relationship will have a weight of only 0.5.

Note that we can WEIGHT BY one feature but RELATE BY another feature.

RELATE BY

Syntax: RELATE BY featureDomain [FROM varname];

Example: RELATE BY userid;

The RELATE BY command declares the domain that defines a co-occurrence relationship. All features that co-occur with the same feature value in this domain are considered to have co-occurred.

FROM varname tells the RELATE BY command where to get its input data. If not specified, RELATE BY will read from the output of the previous command.

Note that we can WEIGHT BY one feature but RELATE BY another feature.

PROJECT

Syntax: PROJECT TO [featureDomain, …] [FROM varname];

Variants: PLANAR PROJECT TO [featureDomain, …] [FROM varname];

Variant: PLANAR BIPARTITE PROJECT TO [featureDomain, …] [FROM varname];

Example: PROJECT TO hashtag;

The PROJECT TO command will project an initial hyper-graph to focus on only relationships among the specified feature domains. That is, only edges which connect 1 or more nodes in the specified domains will be kept, and any nodes in other feature domains will be removed from the structure of the graph. By default, the PROJECT TO command generates a hyper-graph. This means that nodes that do not co-occur with other nodes will still be described by a degenerate 1-edge. Also, if many nodes simultaneously co-occur together, their relationship will be described by a k-edge (where k == the number of co-occurring nodes)

Often, especially for ease of visualization, it is useful to restrict the discussion graph to be a planar graph (where every edge in the graph connects exactly 2 nodes). The PLANAR PROJECT TO command achieves this. All hyper-edges will be decomposed and re-aggregated into their corresponding 2-edges.

Furthermore, it can be useful to restrict the graph to be bipartite, where only edges that cross domains are kept. For example, we may only care about the relationship between users and the hashtags they use, and not care about the relationship among hashtags themselves. The PLANAR BIPARTITE PROJECT TO command achieves this. Semantically, this is the equivalent of doing a planar projection and then dropping all edges that connect nodes are in the same domain.

MERGE

Syntax: MERGE varname1,varname2[,…];

Example: MERGE MentionAndUserGraph,HashTagAndUserGraph;

The MERGE command overlays two discussion graphs atop each other. Nodes with the same feature domain and values will be merged.

OUTPUT

Syntax: OUTPUT TO “filename.graph” [FROM varname];

Example: OUTPUT TO “mentions.graph”;

The OUTPUT TO command saves a discussion graph to the specified file.

File’s are saved in DGT’s native format. This format consists of 3 tab-separated columns. The first column is the edge identifier: the comma-separated list of nodes connected by this edge. The second column is the count of the number of times this co-occurrence relationship was observed to occur. The third column is a JSON-formatted representation of the context of the relationship or, in other words, the distribution of feature values conditioned on the co-occurrence relationship.

Naming variables

We can assign the result of commands to variables, and use these variables in later commands:

Syntax:

var x = COMMAND1;

COMMAND2 FROM x;

Example:

var reviewData = LOAD Multiline(path:”finefoods.tar.gz”);

var reviewFeatures = EXTRACT AffectDetector(),reviewscore FROM reviewData;

Here’s a current list of feature extractors included in the discussion graph tool release.