The accompanying code is here.

CloudFormation

AWS SAM provides direct support for DynamoDB, but not for more traditional databases like PostgreSQL, so that means dropping into CloudFormation. This is, sadly, rather wordy, because we'll need to configure all the neccessary AWS machinery (including setting up a VPC, security group, and database credentials) ourselves, but it's mostly standard boilerplate which is pretty well documented:

• Giving Lambda functions access to resources in an Amazon VPC.
• Creating a Secrets Manager secret for a master password.

You can see the full CloudFormation template here. We'll explain the various sections in more detail below.

VPC

To avoid having to make our database publicly visible, we're going to put both our Lambda function and database in a shared VPC. Here's how we create that VPC:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16

We also need to define a couple of subnets (RDS requires at least two subnets, in two different avaialability zones):

  Subnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.1.0/24
      AvailabilityZone: !Select [0, Fn::GetAZs: !Ref "AWS::Region"]
  Subnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.2.0/24
      AvailabilityZone: !Select [1, Fn::GetAZs: !Ref "AWS::Region"]

This is using a little CloudFormation magic to select the first two availability zones in whichever region we're deploying to. You could just as easily hardcode the availability zones if you prefer.

And we need a security group which allows things within the VPC to access Postgres:

  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub "Security group for ${AWS::StackName}"
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          CidrIp: !GetAtt VPC.CidrBlock

We put our Lambda function in the VPC we've created by adding the following to its Properties:

      VpcConfig:
        SecurityGroupIds: [!Ref SecurityGroup]
        SubnetIds: [!Ref Subnet1, !Ref Subnet2]
      Policies: [AWSLambdaVPCAccessExecutionRole]

Database

We now have everything we need to create a database instance:

  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: !Sub "DBSubnet group for ${AWS::StackName}"
      SubnetIds: [!Ref Subnet1, !Ref Subnet2]
  Database:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: db.t4g.micro
      Engine: postgres
      EngineVersion: 14.15
      DBName: example_lambda_app
      AllocatedStorage: 20
      StorageEncrypted: true
      ManageMasterUserPassword: true
      MasterUsername: postgres
      KmsKeyId: !Ref DatabaseKey
      VPCSecurityGroups: [!Ref SecurityGroup]
      DBSubnetGroupName: !Ref DBSubnetGroup

Most of this is pretty obvious: we're creating an RDS database running on a db.t4g.micro instance, with 20GB of storage, encrypted at rest, and with a master user called postgres. We're adding it to the security group we created earlier, and letting it know about the subnets we created via a DBSubnetGroup.

Key Management

We've asked RDS to manage the database password for us (ManageMasterUserPassword) and store the credentials in a Secrets Manager (KMS) secret. Here's how we create that secret:

  DatabaseKey:
    Type: AWS::KMS::Key
    Properties:
      Description: DatabaseKey
      EnableKeyRotation: false
      KeyPolicy:
        Version: 2012-10-17
        Id: !Sub "key-${AWS::StackName}"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: ["kms:*"]
            Resource: "*"

I've chosen to disable key rotation because I'll be passing the key to the Lambda function as an environment variable. An alternative would be to modify the Lambda function to use the Secrets Manager API to retrieve the password, but I wanted to keep the code as simple as possible. The rest is simple boilerplate taken from the article mentioned above.

To use this secret in our Lambda function, we add the following to the function's Properties:

      Environment:
        Variables:
          DB_HOST: !GetAtt Database.Endpoint.Address
          DB_PASSWORD: !Sub "{{resolve:secretsmanager:${Database.MasterUserSecret.SecretArn}:SecretString:password}}"

Deployment

Deploying is exactly the same as before: build the uberjar and then sam deploy. The first time you do this it'll take a while because it's creating the database instance, but subsequent deployments will be much quicker.

This article describes how to get a simple Clojure Ring application running. A followup article shows how to connect it to a database. The code is available here.

The Code

Let's start with a simple Clojure Ring app with a sprinkling of interactivity through HTMX:

(ns example.lambda-app
  (:require [compojure.core :refer [defroutes GET]]
            [compojure.route :as route]
            [hiccup2.core :refer [html]]
            [ring.logger :refer [wrap-with-logger]]
            [ring.middleware.defaults :refer [site-defaults wrap-defaults]]
            [ring.middleware.params :refer [wrap-params]]))

(defn index-page
  []
  (str (html [:head [:title "HTMX Example"]
              [:script {:src "https://unpkg.com/htmx.org@2.0.4"}]]
             [:body [:h1 "HTMX Example"]
              [:div#greeting {:hx-get "/greet" :hx-trigger "load"}]])))

(defn greet [] (str (html [:div "Hello, World!"])))

(defroutes app-routes
  (GET "/" [] (index-page))
  (GET "/greet" [] (greet))
  (route/not-found "Not Found"))

(def app
  (-> app-routes
      wrap-params
      (wrap-defaults site-defaults)
      wrap-with-logger))

This is all entirely standard: nothing different from any other Ring app. We're going to implement two different ways to serve this app, one using ring-jetty-adapter for local development, and one using ring-lambda-adapter for deployment to AWS Lambda.

For local development, we'll create dev/user.clj:

(ns user
  (:require [ring.adapter.jetty :as jetty]
            [example.lambda-app :refer [app]]))

(defn -main [& _]
  (jetty/run-jetty #'app {:port 8080 :host "0.0.0.0" :join? false}))

And for deployment to AWS Lambda, we'll create lambda.clj:

(ns example.lambda
  (:gen-class :implements
              [com.amazonaws.services.lambda.runtime.RequestStreamHandler])
  (:require [paulbutcher.ring-lambda-adapter :refer [handle-request]]
            [example.lambda-app :refer [app]]))

(defn -handleRequest [_ is os _] (handle-request app is os))

This implements the RequestStreamHandler interface defined by the AWS Java Lambda runtime. The handle-request function is provided by ring-lambda-adapter and does the work of converting the input and output streams to and from Ring requests and responses.

Finally, here's our deps.edn:

{:paths ["src" "resources"]
 :deps {com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.3"}
        com.amazonaws/aws-xray-recorder-sdk-slf4j {:mvn/version "2.18.2"}
        com.paulbutcher/ring-lambda-adapter {:mvn/version "1.0.7"}
        compojure/compojure {:mvn/version "1.7.1"}
        hiccup/hiccup {:mvn/version "2.0.0-RC4"}
        org.clojure/clojure {:mvn/version "1.12.0"}
        ring-logger/ring-logger {:mvn/version "1.1.1"}
        ring/ring-core {:mvn/version "1.13.0"}
        ring/ring-defaults {:mvn/version "0.5.0"}}
 :aliases {:run {:main-opts ["-m" "user"]}
           :build {:deps {io.github.clojure/tools.build {:mvn/version "0.10.6"}}
                   :ns-default build}
           :dev {:extra-paths ["dev"]
                 :extra-deps {org.slf4j/slf4j-simple {:mvn/version "2.0.16"}
                              ring/ring-jetty-adapter {:mvn/version "1.13.0"}}}}}

This is all very standard apart from:

• aws-lambda-java-core which provides the AWS Java Lambda runtime.
• aws-xray-recorder-sdk-slf4j which turns SLF4J logs into AWS X-Ray traces.
• ring-lambda-adapter which provides a Ring adapter for AWS Lambda.

You should now be able to run locally with clojure -M:dev:run.

Deployment

To deploy to AWS, you'll need to have the AWS SAM CLI installed. This sits on top of AWS CloudFormation and simplifies the process of deploying serverless applications. Our application is described via a template.yaml file:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: target/standalone.jar
      Handler: example.lambda::handleRequest
      Runtime: java21
      FunctionUrlConfig:
        AuthType: NONE
      AutoPublishAlias: live
      SnapStart:
        ApplyOn: PublishedVersions
      Timeout: 20
      MemorySize: 512
      Tracing: Active
    Metadata:
      SkipBuild: true

Outputs:
  Endpoint:
    Value: !GetAtt FunctionUrl.FunctionUrl

• CodeUri points to an uberjar file we're going to build (the SkipBuild metadata tells SAM not to try to build the jar file itself).
• Handler is our Lambda function entry point.
• AuthType is NONE because this is a public web app (not an API that's sitting behind some kind of authentication).
• We're only using a single alias (live), but we could have multiple aliases for different environments (e.g. staging etc.).
• SnapStart takes a snapshot of our function which reduces cold-start times to less than a second.
• Tracing enables AWS X-Ray tracing.

To deploy, first build the jar file with clojure -T:build uber (see build.clj in GitHub), then deploy with sam deploy --guided. This will ask you some questions such as which region you want to deploy to and then eventually output the URL of your function's endpoint. Connect to that URL, and you should see exactly what you saw when you ran locally.

Monitoring

There are a number of different ways to keep an eye on how your Lambda function is performing.

• You can watch logs locally in real time with sam logs --tail.
  • You'll need to specify the region and stack name you already specified when deploying. You can avoid having to do so every time by adding a [default.global.parameters] section to your samconfig.toml file.
• From the AWS console, you can see logs in CloudWatch. Most interestingly, you can use X-Ray traces to see detailed timing information for each request.

Development

Local development works just like any other Ring app. To deploy a new version, just build a new jar file and run sam deploy.

Conclusion

The combination of SAM, which makes Lambda function deployment so simple, and SnapStart, which removes the cold start problem, means that AWS Lambda is my new default for getting started quickly.

In the next article, we'll look at how to connect our Lambda function to a database.

Credits

This was all heavily inspired by A Recipe for Plain Clojure Lambdas.

The code to accompany this series is available here.

In the previous part of this series, we saw how to use lattice semantics to calculate all our degrees of separation within the Game of Thrones dataset in one pass. But we didn't quite get to our final solution because we weren't yet calculating counts of the number of characters at each degree.

In this part, we'll tie off that loose end, and discover why lattice semantics are "lattice" semantics in the process.

Of course, we could just count up our values in Flix (not Datalog) code, and that would be a perfectly reasonable approach. But as we'll see Datalog gives us a really elegant solution by leveraging partial ordering.

Partial Ordering

So far we've been working with simple integers, and integers are (surprise!) ordered. So 1 is less than 2 and 3 is less than 126 and so on.

In fact, not only are integers ordered, they totally ordered. You can pick any pair of numbers a and b, and at least one of a <= b or b <= a will be true (they might both be true if a equals b).

But, not every set of values is totally ordered. A real world example of a partial ordering is ancestry; given two people a and b, it might easily be the case that neither "a is an ancestor of b" nor "b is an ancestor of a" is true.

For example, "Eddard Stark is an ancestor of Arya Stark" is true. But neither "Cersei Lannister is an ancestor of Tyrion Lannister" nor "Tyrion Lannister is an ancestor of Cersei Lannister" are true.

However, we can ask "who is the most recent common ancestor of Cersei and Tyrion" (in this case Tywin Lannister). We generally refer to this as the least upper bound.

Least Upper Bound

In part 3, we said that lattice semantics chose the "maximum" value from all possible values. This was a simplification: they actually chose the least upper bound.

For integers, which we were working with in part 3, the least upper bound is the maximum value. But for other types, those that are partially ordered (as we saw with ancestry above), the least upper bound could be something else.

An interesting example is sets. It may not be the case that either "set a is a subset of set b" or "set b is a subset of set a" is true. But they will always have a least upper bound which is the union of a and b.

For example, the least upper bound of Set#{"Tyrion Lannister", "Cersei Lannister"} and Set#{"Tyrion Lannister", "Arya Stark"} is Set#{"Tyrion Lannister", "Cersei Lannister", "Arya Stark"}.

This is where the "lattice" in lattice semantics comes from: a partially ordered set which defines a least upper bound is called a lattice) in mathematics.

Happily, this is exactly what we need to solve our degrees of separation problem.

Six Degrees of Separation, Take 3

As a reminder, here were the rules we used in part 3 of this series:

        Degree(x; Down(0)) :- Root(x).
        Degree(x; n + Down(1)) :- Degree(y; n), Related(y, x).

Here are the new rules we're going to add:

        AggregatedDegree(n; Set#{x}) :- fix Degree(x; n).
        DegreeCount(n, Set.size(s)) :- fix AggregatedDegree(n; s).

We start by inferring new AggregatedDegree facts from the Degree facts we calculated previously. We're using the fact that the least upper bound of a collection of sets is the union of those sets, so the value on the right side of the semicolon will be the union of all of the character names in Degree.

You might be wondering what the fix is for in our new rules? If you look at the two sides of the rule (either side of the :-) you can see that on the left we're using n as a normal value (it's on the left of the semicolon), whereas on the right it's a lattice value (it's on the right of the semicolon). Flix requires that we use fix if we ever mix a value this way; it ensures that we completely calculate all the Degree facts before starting to create AggregateDegree facts.

So now we have a number of AggregatedDegree facts, one for each degree of separation, where the right hand side is a set of all the characters of that degree. The final step is to convert those facts into DegreeCount facts by finding the size of each set.

Here's the whole thing. As you can see, it's even more elegant than the solution we came up with in part 2 (which was already much simpler than we could have achieved without using Datalog).

def main(): Unit \ IO =

    let relationshipTypes = "parents" :: "parentOf" :: "killed" :: "killedBy" ::
        "serves" :: "servedBy" :: "guardianOf" :: "guardedBy" :: "siblings" ::
        "marriedEngaged" :: "allies" :: Nil;
    let relationships = Json.getRelationships(relationshipTypes);

    let related = inject relationships into Related;

    let rules = #{
        Degree(x; Down(0)) :- Root(x).
        Degree(x; n + Down(1)) :- Degree(y; n), Related(y, x).
        AggregatedDegree(n; Set#{x}) :- fix Degree(x; n).
        DegreeCount(n, Set.size(s)) :- fix AggregatedDegree(n; s).
    };

    let root = inject "Tyrion Lannister" :: Nil into Root;

    query rules, related, root select (d, c) from DegreeCount(d, c) |>
        List.foreach(match (d, c) -> println("Separated by degree ${d}: ${c}"))

And, for completeness, here's what it outputs:

Separated by degree 6: 2
Separated by degree 5: 14
Separated by degree 4: 60
Separated by degree 3: 104
Separated by degree 2: 56
Separated by degree 1: 6
Separated by degree 0: 1

Conclusion

That's it for our journey through Datalog and Flix. Please do experiment with other problems which can leverage Datalog: I'd love to see what you come up with!

[Part 1 | Part 2 | Part 3 | Part 4]

The code to accompany this series is available here.

Harder, Better, Faster, Stronger

It's very common that we want to find the "best" solution to a problem. Perhaps we want the shortest route between two points? Or the fastest? Or the most efficient? Or...?

Flix's implementation of Datalog provides a feature known as "lattice semantics" (we'll see why "lattice" later) which allows us to solve exactly these kinds of problems. To illustrate, consider the following function:

def foo(): List[(String, Int32)] =
    let rules = #{
        Foo("x", 1).
        Foo("x", 2).
        Foo("x", 5).
        Foo("y", 2).
        Foo("y", 3).
    };
    query rules select (x, y) from Foo(x, y)

This, unsurprisingly, just returns exactly the facts that we specified:

(x, 1) :: (x, 2) :: (x, 5) :: (y, 2) :: (y, 3) :: Nil

But now, let's modify it so that it uses lattice semantics:

def foo(): List[(String, Int32)] =
    let rules = #{
        Foo("x"; 1).
        Foo("x"; 2).
        Foo("x"; 5).
        Foo("y"; 2).
        Foo("y"; 3).
    };
    query rules select (x, y) from Foo(x; y)

All we've done is replace the commas (,) in our original Foo rules with semicolons (;). What it now returns is:

(x, 5) :: (y, 3) :: Nil

Switching to semicolons instructs Datalog to aggregate all the facts that have the same values on the left of the semicolon into a single fact, where the value on the right is the "maximum" of all the facts that are being combined. In our example above, the maximum value is 5 when the LHS is "x", and 3 when it's "y".

There is some subtlety in what we mean when we say "maximum", as we'll see later, but for simple integers then what we get is exactly the same as if we had used Int32.max.

It seems like a small change, but it's a small change that dramatically increases the power at our fingertips.

Minimums

Finding maximum values is great, but what if we want to find minimums instead? Flix allows us to reverse the sorting order of any type by using Down, so Down[Int32] sorts in the opposite order to Int32:

flix> 1 < 3
true
flix> 4 < 2
false
flix> Down(1) < Down(3)
false
flix> Down(3) < Down(2)
true

Using Down in Datalog lattice semantics has exactly the effect you might imagine, allowing us to find minimum values instead of maximum values:

def bar(): List[(String, Down[Int32])] =
    let rules = #{
        Bar("x"; Down(1)).
        Bar("x"; Down(2)).
        Bar("x"; Down(5)).
        Bar("y"; Down(2)).
        Bar("y"; Down(3)).
    };
    query rules select (x, y) from Bar(x; y)

Which returns:

(x, 1) :: (y, 2) :: Nil

So much for theory, let's see how this helps us solve a real problem.

Six Degrees of Separation, Take 2

In part 2 of this series, we used Datalog to calculate degrees of separation between characters in Game of Thrones. The combination of logic programming and functional programming gave us a very simple solution. Now, we're going to use lattice semantics to make it even better.

As a refresher, here's the code from part 2 which creates our Related facts:

    let relationshipTypes = "parents" :: "parentOf" :: "killed" :: "killedBy" ::
        "serves" :: "servedBy" :: "guardianOf" :: "guardedBy" :: "siblings" ::
        "marriedEngaged" :: "allies" :: Nil;
    let relationships = Json.getRelationships(relationshipTypes);

    let related = inject relationships into Related;

And here is code which uses lattice semantics to calculate the degree of separation of every character from some root character (in this case Tyrion) in a single pass:

    let rules = #{
        Degree(x; Down(0)) :- Root(x).
        Degree(x; n + Down(1)) :- Degree(y; n), Related(y, x).
    };

    let root = inject "Tyrion Lannister" :: Nil into Root;

    println(query rules, related, root select (x, d) from Degree(x; d))

Before we go through this to see how it works, here's the (truncated) output:

(Aegon Targaryen, 3) :: (Aeron Greyjoy, 3) :: (Aerys II Targaryen, 2) ::
(Akho, 3) :: (Alliser Thorne, 3) :: (Alton Lannister, 2) :: ...

To see how this works, imagine what would happen if we wrote the above without lattice semantics:

        Degree(x, 0) :- Root(x).
        Degree(x, n + 1) :- Degree(y, n), Related(y, x).

This can be interpreted as saying: Character x is related to our root character by degree n + 1 if we can find any character y which is related by degree n, and where y is related to x.

So Cersei, for example, is related to Tyrion by degree 1 because Tyrion is related to Tyrion by degree 0, and Tyrion is related to Cersei.

But ... Cersei is also related to Tyrion by degree 2 because Tywin is related to Tyrion by degree 1, and Tywin is related to Cercei. And Cersei is related to Tyrion by degree 3 because Jaime is related to Tyrion by degree 2, and Jaime is related to Cersei. And so on.

By contrast, Arya is not related to Tyrion at degree 1 or 2, but is related to him by degree 3, because Eddard Stark is related to Tyrion at degree 2, and Eddard is related to Arya. And Arya is related to Tyrion by degree 4 because Walder Frey is related to Tyrion at degree 3, and Walder is related to Arya. And so on.

So, as you can probably already see, with non-lattice semantics this will never terminate; we'll just keep calculating higher and higher degrees of separation until we run out of either memory or patience.

But, with lattice semantics (and given that we're using Down to reverse the ordering) we can constrain all of our degrees (the numbers on the right of the semicolon) to a single minimum value (whichever is the lowest they can ever take), which means:

1. Our function terminates (always helpful!), and
2. The degree calculations always return the length of the "shortest path" from Tyrion to every other character.

Conclusion

So we're almost there: we now have a list of every character along with their minimum degree of separation from Tyrion. But if we're going to duplicate what we created in part 2, we need to go further and count how many characters there are at each degree of separation. We'll see how to do that in the next part of this series.

[Part 1 | Part 2 | Part 3 | Part 4]

The code to accompany this series is available here.

Injecting facts

In part 1 of this series, we used Datalog rules to infer new facts about characters in Game of Thrones.

But where do our initial facts come from? In part 1 we simply included them as part of our Datalog, but this isn't a scalable approach. We don't want to have to manually re-enter all the details of all of the characters in Game of Thrones!

What's more likely is that we'll read these facts from some external source; perhaps a database or a JSON file. This is where the power of Flix integration comes into play; we can use Flix, a powerful general purpose language, to read our data and get it into the right format. We then inject it into Datalog facts.

So instead of writing this (from part 1):

def main(): Unit \ IO =
    let got = #{
        ParentOf("Tywin Lannister", "Cersei Lannister").
        ParentOf("Tywin Lannister", "Jaime Lannister").
        ParentOf("Cersei Lannister", "Joffrey Baratheon").
        ParentOf("Cersei Lannister", "Myrcella Baratheon").

        SiblingOf(x, y) :- ParentOf(p, x), ParentOf(p, y), if x != y.
    };
    let siblings = query got select (x, y) from SiblingOf(x, y);
    println(siblings)

We can instead write:

def main(): Unit \ IO =
    let parentList = ("Tywin Lannister", "Cersei Lannister") ::
        ("Tywin Lannister", "Jaime Lannister") ::
        ("Cersei Lannister", "Joffrey Baratheon") ::
        ("Cersei Lannister", "Myrcella Baratheon") :: Nil;

    let parents = inject parentList into ParentOf;

    let got = #{
        SiblingOf(x, y) :- ParentOf(p, x), ParentOf(p, y), if x != y.
    };

    let siblings = query got, parents select (x, y) from SiblingOf(x, y);

    println(siblings)

Or, more realistically, we read our data from some external file (in this case, Jeffrey Lancaster's amazingly detailed Game of Thrones dataset):

def main(): Unit \ IO =

    let parents = inject Json.getParents() into ParentOf;

    let got = #{
        SiblingOf(x, y) :- ParentOf(p, x), ParentOf(p, y), if x != y.
    };

    let siblings = query got, parents select (x, y) from SiblingOf(x, y);

    println(siblings)

Which gives us the following (truncated!) output:

(Aegon Targaryen, Jon Snow) :: (Aegon Targaryen, Rhaenys Targaryen) ::
(Arya Stark, Bran Stark) :: (Arya Stark, Rickon Stark) ::
(Arya Stark, Robb Stark) :: (Arya Stark, Sansa Stark) ::
(Benjen Stark, Brandon Stark) :: (Benjen Stark, Eddard Stark) :: ...

Chaining Datalog

The ability to move data back and forth between Datalog and Flix makes it very easy to use Datalog as part of solving a larger problem. To illustrate this, let's consider working out whether Game of Thrones obeys the "six degrees of separation" rule (the idea that all people are six or fewer connections away from each other; you might have come across it from the "Six Degrees of Kevin Bacon" game, or a mathematician's Erdős number).

As well as parentage, the dataset we've been using includes all kinds of other relationships between characters such as alliances, marriages and engagements, who serves whom, who killed whom, etc. Taking all of these relationships together, how many degrees of separation does it take from a major character (say Tyrion) to cover the entire cast of characters?

(You can find the complete source here).

First, we extract our long list of relationships from our dataset:

    let relationshipTypes = "parents" :: "parentOf" :: "killed" :: "killedBy" ::
        "serves" :: "servedBy" :: "guardianOf" :: "guardedBy" :: "siblings" ::
        "marriedEngaged" :: "allies" :: Nil;
    let relationships = Json.getRelationships(relationshipTypes);

We then inject that list into Datalog as a series of Related facts, just like we've seen above:

    let related = inject relationships into Related;

So now we have a set of facts like Related("Tyrion Lannister", "Shae") because Tyrion killed Shae, and Related("Arya Stark", "Nymeria") because Arya is guarded by Nymeria, and so on.

Next, here is a Datalog rule which, given a set of characters we've already found, finds the characters with the next degree of separation:

    let rules = #{
        NextDegree(x) :- AlreadyFound(y), Related(y, x), not AlreadyFound(x).
    };

So a character x is in the next degree of separation if there is at least one character y who we have already and where y is related to x. Finally we exclude characters that we've already found.

Here's a function which uses the above to repeatedly calculate degrees of separation until we've exhausted all our characters:

    def degreesOfSeparation(deg: Int32, cs: List[String]): Unit \ IO = {
        let alreadyFound = inject cs into AlreadyFound;
        let nextDegree = query rules, alreadyFound, related select (x) from NextDegree(x);
        match List.length(nextDegree) {
            case count if count > 0 =>
                println("Separated by degree ${deg}: ${count}");
                degreesOfSeparation(deg + 1, cs ::: nextDegree)
            case _ =>
                println("Nobody separated by degree ${deg}")
        }
    };

This function takes deg (a number representing the current degree of separation we're looking at) and cs a list of characters we've already found.

We start by creating our AlreadyFound facts from this list of characters, and then use the Datalog rules we defined above, along with our AlreadyFound and Related facts to generate NextDegree characters.

List.length(nextDegree) returns the length of this list, and we use match to determine whether this number is greater than zero.

If it is, then we output the number of characters we just found, and recursively call degreesOfSeparation with deg increased by one, and a new set of characters which includes both our original root set and the characters we just found.

Finally, we kick the whole thing off by calling degreesOfSeparation with our initial root character, Tyrion:

    degreesOfSeparation(1, "Tyrion Lannister" :: Nil)

Here's what we get when we run it:

Separated by degree 1: 6
Separated by degree 2: 56
Separated by degree 3: 104
Separated by degree 4: 60
Separated by degree 5: 14
Separated by degree 6: 2
Nobody separated by degree 7

So yes, it does look like Game of Thrones does obey the six degrees of separation rule.

Here's the whole thing. I suggest trying to implement this in your favourite language to see just how easy the combination of Flix and Datalog makes this:

def main(): Unit \ IO =

    let relationshipTypes = "parents" :: "parentOf" :: "killed" :: "killedBy" ::
        "serves" :: "servedBy" :: "guardianOf" :: "guardedBy" :: "siblings" ::
        "marriedEngaged" :: "allies" :: Nil;
    let relationships = Json.getRelationships(relationshipTypes);

    let related = inject relationships into Related;
    let rules = #{
        NextDegree(x) :- AlreadyFound(y), Related(y, x), not AlreadyFound(x).
    };

    def degreesOfSeparation(deg: Int32, cs: List[String]): Unit \ IO = {
        let alreadyFound = inject cs into AlreadyFound;
        let nextDegree = query rules, alreadyFound, related select (x) from NextDegree(x);
        match List.length(nextDegree) {
            case count if count > 0 =>
                println("Separated by degree ${deg}: ${count}");
                degreesOfSeparation(deg + 1, cs ::: nextDegree)
            case _ =>
                println("Nobody separated by degree ${deg}")
        }
    };

    degreesOfSeparation(1, "Tyrion Lannister" :: Nil)

Conclusion

Flix allows us to seamlessly move data back and forth between Flix and Datalog by injecting facts from Flix to Datalog and querying data from Datalog to Flix. This allows us to use the strengths of a general purpose language like Flix for things it's good at (e.g. reading JSON files) and Datalog for things that it's good at (e.g. inferring new facts from existing ones).

In the next part of this series, we'll look at one of the most powerful aspects of Flix's implementation of Datalog: lattice semantics.

[Part 1 | Part 2 | Part 3 | Part 4]

The code to accompany this series is available here.

Flix

Flix is an exciting new programming language which provides the power of strongly typed functional programming with the accessibility of "programmer friendly" languages like Go, Rust, or Ruby.

This article isn't about Flix, though, it's about Datalog, and the reason why Flix plus Datalog creates something greater than the sum of its parts.

Logic Programming

Logic programming is one of the "big three" programming language paradigms:

Imperative programming: Languages like C, C++, Java, JavaScript, Go, Kotlin, …

Functional programming: Languages like Haskell, Clojure, Elixir, F#, …

Logic programming: Prolog, Datalog, Answer-set Programming, Mercury, …

But ... if it's one of the three major programming paradigms, how come nobody uses it? Perhaps you created a couple of toy Prolog programs at college, but you've almost certainly not used it since then.

There are good reasons why logic programming hasn't caught on outside of a very few specialised areas, but that may be about to change. By including logic programming as a first-class element of the language, Flix allows us to seamlessly move between functional and logic programming, leveraging the strengths of both.

This is analogous to the way that C# embeds database queries directly in the language through LINQ (and of course, you don't use LINQ to implement every single method; instead you use C# for most of your code and drop into LINQ when it makes sense).

In this first article, we'll explore the basics of Datalog. And then in the rest of the series we'll see how the combination of Datalog and Flix brings huge expressive power.

Facts

Datalog programs are built from two components: facts and rules. Here are some facts about Game of Thrones:

GreatHouse("Stark").
GreatHouse("Targaryen").

ParentOf("Rhaegar Targaryen", "Jon Snow").
ParentOf("Aerys II Targaryen", "Daenerys Targaryen").

Coordinates("Kings Landing", 6.697, 12.759).
Coordinates("Winterfell", 5.666, 6.218).

You can read these facts as:

• Stark is one of the Great Houses.
• Rhaegar Targaryen is a parent of Jon Snow.
• Kings Landing is located at coordinates 6.697, 12.759.

A fact can have any number of terms of (almost) any type.

Rules

Rules create new facts by making logical inferences from existing facts. Imagine, for example, that we know the following facts:

ParentOf("Tywin Lannister", "Cersei Lannister").
ParentOf("Tywin Lannister", "Jaime Lannister").
ParentOf("Cersei Lannister", "Joffrey Baratheon").
ParentOf("Cersei Lannister", "Myrcella Baratheon").

Then we can create additional facts about who is who's grandparent with the following rule:

GrandparentOf(x, y) :- ParentOf(x, c), ParentOf(c, y).

Which you can read as "x is a grandparent of y if we can find at least one person c where x is a parent of c, and c is a parent of y".

Here's a complete Flix program which does this:

def main(): Unit \ IO =
    let got = #{
        ParentOf("Tywin Lannister", "Cersei Lannister").
        ParentOf("Tywin Lannister", "Jaime Lannister").
        ParentOf("Cersei Lannister", "Joffrey Baratheon").
        ParentOf("Cersei Lannister", "Myrcella Baratheon").

        GrandparentOf(x, y) :- ParentOf(x, c), ParentOf(c, y).
    };
    let grandparents = query got select (x, y) from GrandparentOf(x, y);
    println(grandparents)

The code within the #{ ... } block is our datalog, the rest of the code is the Flix program that it's embedded within. Our Datalog program is executed when we call query to extract the new GrandparentOf facts.

Here's what it outputs:

(Tywin Lannister, Joffrey Baratheon) :: (Tywin Lannister, Myrcella Baratheon) :: Nil

This is a list of 2-element tuples, with :: indicating the list "cons" operator and Nil the end of the list. We can interpret these results as saying that Tywin Lannister is the grandparent of both Joffrey Baratheon and Myrcella Baratheon.

Here's a slightly different program which uses the same facts to generate additional facts about siblings:

def main(): Unit \ IO =
    let got = #{
        ParentOf("Tywin Lannister", "Cersei Lannister").
        ParentOf("Tywin Lannister", "Jaime Lannister").
        ParentOf("Cersei Lannister", "Joffrey Baratheon").
        ParentOf("Cersei Lannister", "Myrcella Baratheon").

        SiblingOf(x, y) :- ParentOf(p, x), ParentOf(p, y).
    };
    let siblings = query got select (x, y) from SiblingOf(x, y);
    println(siblings)

The key line is:

SiblingOf(x, y) :- ParentOf(p, x), ParentOf(p, y).

Which you can read as "x is a sibling of y if we can find at least one person p where p is a parent of x and p is a parent of y".

And here's what it outputs:

(Cersei Lannister, Cersei Lannister) :: (Cersei Lannister, Jaime Lannister) ::
(Jaime Lannister, Cersei Lannister) :: (Jaime Lannister, Jaime Lannister) ::
(Joffrey Baratheon, Joffrey Baratheon) :: (Joffrey Baratheon, Myrcella Baratheon) ::
(Myrcella Baratheon, Joffrey Baratheon) :: (Myrcella Baratheon, Myrcella Baratheon) ::
Nil

Were you surprised that there were so many answers?

The list is as long as it is because:

1. If SiblingOf(x, y) is true, then so is Sibling(y, x) (i.e. if you are my sibling, then I am also your sibling).
2. As we've written it, Sibling considers anyone with a parent to be their own sibling.

The first point above is probably exactly what we want (i.e. we really do want the sibling relationship to be reflexive). But perhaps we don't want people being their own siblings.

Here's a slightly modified version of our program which adds a guard to our rule, which eliminates cases where x and y represent the same person:

def main(): Unit \ IO =
    let got = #{
        ParentOf("Tywin Lannister", "Cersei Lannister").
        ParentOf("Tywin Lannister", "Jaime Lannister").
        ParentOf("Cersei Lannister", "Joffrey Baratheon").
        ParentOf("Cersei Lannister", "Myrcella Baratheon").

        SiblingOf(x, y) :- ParentOf(p, x), ParentOf(p, y), if x != y.
    };
    let siblings = query got select (x, y) from SiblingOf(x, y);
    println(siblings)

And here's what it outputs:

(Cersei Lannister, Jaime Lannister) :: (Jaime Lannister, Cersei Lannister) ::
(Joffrey Baratheon, Myrcella Baratheon) :: (Myrcella Baratheon, Joffrey Baratheon) ::
Nil

Conclusion

A Datalog program consists of facts and rules. Rules generate new facts from existing ones.

In the next part of this series, we'll dig deeper into the integration between Flix and Datalog.

[Part 1 | Part 2 | Part 3 | Part 4]